472,950 Members | 2,347 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,950 software developers and data experts.

mmap -- memory mapped file


hi, i'm new to this list and new to python as well.

i have a question on the memory mapped file ability python has. when i
use a mmap on a file, will it copy the whole thing to ram or just
whatever part of it i'm working on? basically, i'm wondering if it would
be ok for me to have multiple mmap's open on very large files as i read
or write from them.
Jun 29 '06 #1
1 2132
Carl Mackey <cj******@seas.upenn.edu> wrote:
hi, i'm new to this list and new to python as well.

i have a question on the memory mapped file ability python has. when i
use a mmap on a file, will it copy the whole thing to ram or just
whatever part of it i'm working on? basically, i'm wondering if it would
be ok for me to have multiple mmap's open on very large files as i read
or write from them.


Python relies on the operating system to do the "memory mapping"
sensibly; generally, that means the whole file is mapped in the
*virtual* memory of the process -- the process's address space -- but
only the necessary pages actually get into RAM. This should work fine
with any modern Linux, *BSD, MacOSX, or just about any other Unix that
is still sold, or, also, on Windows (at least in the NT/2000/XP line, I
wouldn't be so sure on some ancient Windows/98 box:-).

Still, assuming that my "very large files" you mean several gigabytes,
mmap will not work well on them with a 32-bit machine, or a 64-bit
machine hobbled by a 32-bit operating system -- the process's address
space being limited to no more 4GB, you may be unable to mmap even just
one "very large file" (this is totally unrelated to how much RAM you may
have: the limitation is with the *address space* of each process).
Alex
Jun 29 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

26
by: myeates | last post by:
Hi Anyone ever done this? It looks like Python2.4 won't take a length arg Mathew
13
by: George Sakkis | last post by:
I've been trying to track down a memory leak (which I initially attributed erroneously to numpy) and it turns out to be caused by a memory mapped file. It seems that mmap caches without limit the...
1
by: sam_cit | last post by:
Hi Everyone, I searched for mmap() and i found the following in wikipedia, 'Anonymous mappings are mappings of physical RAM to virtual memory. This is similar to malloc, and is used in some...
1
by: James T. Dennis | last post by:
I've been thinking about the Python mmap module quite a bit during the last couple of days. Sadly most of it has just been thinking ... and reading pages from Google searches ... and very little...
5
by: Matias Surdi | last post by:
Suppose I've a process P1, which generates itself a lot of data , for example 2Mb. Then, I've a process P2 which must access P1 shared memory and, probably, modify this data. To accomplish this,...
8
by: Unknown Soldier | last post by:
Hello, I have a couple queries about mmap() that ppl here might be able to help with. 1. What's the best way to resize an mmap()d area when you've enlarged the file? Can you call mmap() again...
1
by: Seisouhen | last post by:
Hi all, I am using mmap to obtain some space(mapped anonymously) and am giving the address of the assigned space to a struct pointer. Then I want to access a member of the struct that the pointer...
6
by: osnot | last post by:
Looking at the php manual site, I don't see where php supports memory mapped file IO. Is that right? Or are mmap and munmap called something else?
6
by: castironpi | last post by:
Any interest in pursuing/developing/working together on a mmaped-xml class? Faster, not readable in text editor.
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Sept 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
0
by: Aliciasmith | last post by:
In an age dominated by smartphones, having a mobile app for your business is no longer an option; it's a necessity. Whether you're a startup or an established enterprise, finding the right mobile app...
2
by: giovanniandrean | last post by:
The energy model is structured as follows and uses excel sheets to give input data: 1-Utility.py contains all the functions needed to calculate the variables and other minor things (mentions...
3
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be using a very simple database which has Form (clsForm) & Report (clsReport) classes that simply handle making the calling Form invisible until the Form, or all...
1
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 1 Nov 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM) Please note that the UK and Europe revert to winter time on...
3
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...
0
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be focusing on the Report (clsReport) class. This simply handles making the calling Form invisible until all of the Reports opened by it have been closed, when it...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.