473,322 Members | 1,287 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

mmap disk performance

Hello all,

i am using the mmap module (python2.4) to access contents of a file.

My question regards the relative performance of mmap.seek() vs
mmap.tell(). I have a generator that returns stuff from the file,
piece by piece. Since other things may happen to the mmap object in
between consecutive next() calls (such as another iterator's next()),
i have to store the file position before yield and restore it
afterwards by means of tell() and seek(). Is this correct?

When restoring, is there a penalty for mmap.seek(pos) where the file
position is already at pos (i.e., nothing happened to the file
position in between, a common scenario)? If there is, is it worth
doing

if mmap.tell() != pos:
mmap.seek(pos)

or such?

Cheers!
Nov 20 '07 #1
1 2668
On Nov 20, 2007 10:31 AM, koara <ko***@atlas.czwrote:
Hello all,

i am using the mmap module (python2.4) to access contents of a file.

My question regards the relative performance of mmap.seek() vs
mmap.tell(). I have a generator that returns stuff from the file,
piece by piece. Since other things may happen to the mmap object in
between consecutive next() calls (such as another iterator's next()),
i have to store the file position before yield and restore it
afterwards by means of tell() and seek(). Is this correct?

When restoring, is there a penalty for mmap.seek(pos) where the file
position is already at pos (i.e., nothing happened to the file
position in between, a common scenario)? If there is, is it worth
doing

if mmap.tell() != pos:
mmap.seek(pos)

or such?
Measure it and see. I suspect that the cost of the check in Python
will outweigh any extra work the C code might do, but you should never
guess - just measure it.

This is also pretty unlikely to be any sort of hotspot in your
application - again, measure and see. Unless your profiler says you
spend a lot of time in mmap.seek calls, don't worry about it.
Nov 20 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: netbogus | last post by:
hi, I have a file stored in memory using mmap() and I'd like to parse to read line by line. Also, there are several threads that read this buffer so I think strtok(p, "\n") wouldnt be a good...
7
by: Michael | last post by:
I'm writing an application that decodes a file containing binary records. Each record is a particular event type. Each record is translated into ASCII and then written to a file. Each file contains...
26
by: myeates | last post by:
Hi Anyone ever done this? It looks like Python2.4 won't take a length arg Mathew
13
by: George Sakkis | last post by:
I've been trying to track down a memory leak (which I initially attributed erroneously to numpy) and it turns out to be caused by a memory mapped file. It seems that mmap caches without limit the...
2
by: beejisbrigit | last post by:
Hi there, I was wondering if anyone had experience with File I/O in Java vs. C++ using mmap(), and knew if the performance was better in one that the other, or more or less negligible. My...
1
by: James T. Dennis | last post by:
I've been thinking about the Python mmap module quite a bit during the last couple of days. Sadly most of it has just been thinking ... and reading pages from Google searches ... and very little...
9
by: Owen Zhang | last post by:
I have a file loaded into virtual memory space by mmap. I need to search some key word inside the memory opened by mmap. What is the best and efficient way to do?
6
by: osnot | last post by:
Looking at the php manual site, I don't see where php supports memory mapped file IO. Is that right? Or are mmap and munmap called something else?
5
by: castironpi | last post by:
Hi, I've got an "in-place" memory manager that uses a disk-backed memory- mapped buffer. Among its possibilities are: storing variable-length strings and structures for persistence and...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.