Hi,
I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.
Has anybody encountered this problem before?
Thanks in advance for any ideas/suggestions.
Best Regards,
André M. Descombes 7 1487
amdescombes wrote:
Hi,
I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.
Has anybody encountered this problem before?
Thanks in advance for any ideas/suggestions.
Best Regards,
André M. Descombes
How are you reading the large files? IMO, large files are better read in
chunks:
target_file = open(f, 'rb')
while 1:
data = target_file.read(8192000)
if data:
DO SOMETHING
else:
break
The above reads 8MB at a time until the file has been completely read.
Change the 8MB to whatever you like.
amdescombes wrote:
Hi,
I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.
Has anybody encountered this problem before?
Thanks in advance for any ideas/suggestions.
Best Regards,
André M. Descombes
I forgot to mention that the OS itself or other processes may be using a
lot of memory. So, just because you have 2GB, that does not mean you can
access all of that at once. I would guess that 25% of memory is in
constant use by the OS. So, do your IO/reads in smaller chunks similar
to the example I gave earlier.
Brad
Hi Brad,
I do the reading one line at a time, the problem seems to be with the
dictionary I am creating.
Andre
amdescombes wrote:
>Hi,
I am using Python 2.5.1 I have an application that reads a file and generates a key in a dictionary for each line it reads. I have managed to read a 1GB file and generate more than 8 million keys on an Windows XP machine with only 1GB of memory and all works as expected. When I use the same program on a Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both Python.exe and Python25.dll and setting the /3GB flag on the boot.ini file to no avail. I still get the MemoryError exceptions.
Has anybody encountered this problem before?
Thanks in advance for any ideas/suggestions.
Best Regards,
André M. Descombes
I forgot to mention that the OS itself or other processes may be using a
lot of memory. So, just because you have 2GB, that does not mean you can
access all of that at once. I would guess that 25% of memory is in
constant use by the OS. So, do your IO/reads in smaller chunks similar
to the example I gave earlier.
Brad
AMD <am*********@gmail.comwrote:
> I do the reading one line at a time, the problem seems to be with the dictionary I am creating.
I don't know whether Python dictionaries must live in a contiguous piece of
memory, but if so, that could be the issue. The system DLLs in Server 2003
have been "rebased" in such a way that they chop up the virtual address
space more than XP. Even though there is more virtual memory available, it
is fragmented.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
Yes, I think that might be the issue, perhaps I could implement the
solution using several dictionaries instead of just one.
Are there any classes that implement disk based dictionaries?
Thanks,
Andre
>
I don't know whether Python dictionaries must live in a contiguous piece of
memory, but if so, that could be the issue. The system DLLs in Server 2003
have been "rebased" in such a way that they chop up the virtual address
space more than XP. Even though there is more virtual memory available, it
is fragmented.
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote:
Are there any classes that implement disk based dictionaries?
Take a look at the `shelve` module from the standard library.
Or object databases like ZODB or Durus.
Ciao,
Marc 'BlackJack' Rintsch
Thanks Marc,
I just tried shelve but it is very slow :(
I haven't tried the dbs yet.
Andre
Marc 'BlackJack' Rintsch a écrit :
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote:
>Are there any classes that implement disk based dictionaries?
Take a look at the `shelve` module from the standard library.
Or object databases like ZODB or Durus.
Ciao,
Marc 'BlackJack' Rintsch
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: scott_mcarthur2003 |
last post by:
We are running asp (not asp.net) and vb com dll web sites on a machine
which also has sql server 2000 sp3a running and uses Windows Server
2003. We recently migrated from Windows 2000 (same...
|
by: Justice |
last post by:
Currently I'm doing some experimenting with the XMLHTTP object in
Javascript. Now,
the XMLHttp object is asynchronous (at least in this case), and the
following code causes a significant memory...
|
by: ccuthbert |
last post by:
I have a MCMS 2002 site that we recently had several hotfixes
installed. After installing the hotfixes, both AOL users and Earthlink
users experience tha same problem, namely that intermittently...
|
by: Larry Bertolini |
last post by:
I was browsing Microsoft's SQL Server site, looking for
some details about SQL Server 2005. Didn't find what
I was looking for...
I'm thinking about moving an existing SQL Server 2000
workload...
|
by: Jon Trickey |
last post by:
We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the
weekend, but we have a light user load then (about 200 users.) Today
when we had close to 600 users connecting and running...
|
by: JCauble |
last post by:
We have a large Asp.net application that is currently crashing our production
servers. What we are seeing is the aspnet_wp eat up a bunch of memory and
then stop unexpectedly. Does not recycle.
...
|
by: Anders K. Jacobsen [DK] |
last post by:
Hi,
We are running a intranet application (.net 1.1 asp.net) on a windows 2003
server. We are having some issues that causes extreem slowdowns (like 1 min
to show one page)
This happens...
|
by: Florin |
last post by:
Hi all,
I have a problem related to memory grow on a server application
which is basically stateless (I have some static info loaded). The
client accesses this server using remoting and it has...
|
by: Pablo Bianco SE |
last post by:
Hi everyone, I am new in this group. I have a question and would like to have
some support.
A customer is experiencing memory problems with the asp.net worker process.
The have an application...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new...
| |