473,714 Members | 2,531 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

mmap -- memory mapped file


hi, i'm new to this list and new to python as well.

i have a question on the memory mapped file ability python has. when i
use a mmap on a file, will it copy the whole thing to ram or just
whatever part of it i'm working on? basically, i'm wondering if it would
be ok for me to have multiple mmap's open on very large files as i read
or write from them.
Jun 29 '06 #1
1 2172
Carl Mackey <cj******@seas. upenn.edu> wrote:
hi, i'm new to this list and new to python as well.

i have a question on the memory mapped file ability python has. when i
use a mmap on a file, will it copy the whole thing to ram or just
whatever part of it i'm working on? basically, i'm wondering if it would
be ok for me to have multiple mmap's open on very large files as i read
or write from them.


Python relies on the operating system to do the "memory mapping"
sensibly; generally, that means the whole file is mapped in the
*virtual* memory of the process -- the process's address space -- but
only the necessary pages actually get into RAM. This should work fine
with any modern Linux, *BSD, MacOSX, or just about any other Unix that
is still sold, or, also, on Windows (at least in the NT/2000/XP line, I
wouldn't be so sure on some ancient Windows/98 box:-).

Still, assuming that my "very large files" you mean several gigabytes,
mmap will not work well on them with a 32-bit machine, or a 64-bit
machine hobbled by a 32-bit operating system -- the process's address
space being limited to no more 4GB, you may be unable to mmap even just
one "very large file" (this is totally unrelated to how much RAM you may
have: the limitation is with the *address space* of each process).
Alex
Jun 29 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

26
9315
by: myeates | last post by:
Hi Anyone ever done this? It looks like Python2.4 won't take a length arg Mathew
13
3514
by: George Sakkis | last post by:
I've been trying to track down a memory leak (which I initially attributed erroneously to numpy) and it turns out to be caused by a memory mapped file. It seems that mmap caches without limit the chunks it reads, as the memory usage grows to several hundreds MBs according to the Windows task manager before it dies with a MemoryError. I'm positive that these chunks are not referenced anywhere else; in fact if I change the mmap object to a...
1
1457
by: sam_cit | last post by:
Hi Everyone, I searched for mmap() and i found the following in wikipedia, 'Anonymous mappings are mappings of physical RAM to virtual memory. This is similar to malloc, and is used in some malloc implementations for certain allocations' I understood that the memory contents are mapped to files using mmap()
1
5465
by: James T. Dennis | last post by:
I've been thinking about the Python mmap module quite a bit during the last couple of days. Sadly most of it has just been thinking ... and reading pages from Google searches ... and very little of it as been coding. Mostly it's just academic curiosity (I might be teaching an "overview of programming" class in a few months, and I'd use Python for most of the practical examples to cover a broad range of programming topics, including the...
5
6070
by: Matias Surdi | last post by:
Suppose I've a process P1, which generates itself a lot of data , for example 2Mb. Then, I've a process P2 which must access P1 shared memory and, probably, modify this data. To accomplish this, I've been digging around python's mmap module, but I can't figure how to use it without files. Could anybody explain me how could this be accomplished? An example will be very appreciated. Thanks a lot for your help.
8
2418
by: Unknown Soldier | last post by:
Hello, I have a couple queries about mmap() that ppl here might be able to help with. 1. What's the best way to resize an mmap()d area when you've enlarged the file? Can you call mmap() again with the existing mapped location, or do you need to remap it from scratch? 2. If I have a file descriptor returned by a call to socket(), can I
1
7851
by: Seisouhen | last post by:
Hi all, I am using mmap to obtain some space(mapped anonymously) and am giving the address of the assigned space to a struct pointer. Then I want to access a member of the struct that the pointer points to. The code is: //----begin struct foo{ int test; }foo; struct foo one,*one_ptr; one_ptr=mmap(NULL,sizeof(one),PROT_READ|PROT_WRITE,MAP_ANON,-1,0);
6
10013
by: osnot | last post by:
Looking at the php manual site, I don't see where php supports memory mapped file IO. Is that right? Or are mmap and munmap called something else?
6
1654
by: castironpi | last post by:
Any interest in pursuing/developing/working together on a mmaped-xml class? Faster, not readable in text editor.
0
9307
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
9071
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9009
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7946
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4462
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4715
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3155
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2514
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2105
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.