471,337 Members | 1,372 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,337 software developers and data experts.

Memory Problems in Windows 2003 Server

Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes
Oct 12 '07 #1
7 1427
amdescombes wrote:
Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes
How are you reading the large files? IMO, large files are better read in
chunks:

target_file = open(f, 'rb')
while 1:
data = target_file.read(8192000)
if data:
DO SOMETHING
else:
break

The above reads 8MB at a time until the file has been completely read.
Change the 8MB to whatever you like.
Oct 12 '07 #2
amdescombes wrote:
Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes
I forgot to mention that the OS itself or other processes may be using a
lot of memory. So, just because you have 2GB, that does not mean you can
access all of that at once. I would guess that 25% of memory is in
constant use by the OS. So, do your IO/reads in smaller chunks similar
to the example I gave earlier.

Brad
Oct 12 '07 #3
AMD
Hi Brad,

I do the reading one line at a time, the problem seems to be with the
dictionary I am creating.

Andre
amdescombes wrote:
>Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file
and generate more than 8 million keys on an Windows XP machine with
only 1GB of memory and all works as expected. When I use the same
program on a Windows 2003 Server with 2GB of RAM I start getting
MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes

I forgot to mention that the OS itself or other processes may be using a
lot of memory. So, just because you have 2GB, that does not mean you can
access all of that at once. I would guess that 25% of memory is in
constant use by the OS. So, do your IO/reads in smaller chunks similar
to the example I gave earlier.

Brad
Oct 13 '07 #4
AMD <am*********@gmail.comwrote:
>
I do the reading one line at a time, the problem seems to be with the
dictionary I am creating.
I don't know whether Python dictionaries must live in a contiguous piece of
memory, but if so, that could be the issue. The system DLLs in Server 2003
have been "rebased" in such a way that they chop up the virtual address
space more than XP. Even though there is more virtual memory available, it
is fragmented.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
Oct 14 '07 #5
Yes, I think that might be the issue, perhaps I could implement the
solution using several dictionaries instead of just one.
Are there any classes that implement disk based dictionaries?

Thanks,

Andre
>
I don't know whether Python dictionaries must live in a contiguous piece of
memory, but if so, that could be the issue. The system DLLs in Server 2003
have been "rebased" in such a way that they chop up the virtual address
space more than XP. Even though there is more virtual memory available, it
is fragmented.
Oct 15 '07 #6
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote:
Are there any classes that implement disk based dictionaries?
Take a look at the `shelve` module from the standard library.

Or object databases like ZODB or Durus.

Ciao,
Marc 'BlackJack' Rintsch
Oct 15 '07 #7
AMD
Thanks Marc,

I just tried shelve but it is very slow :(
I haven't tried the dbs yet.

Andre

Marc 'BlackJack' Rintsch a écrit :
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote:
>Are there any classes that implement disk based dictionaries?

Take a look at the `shelve` module from the standard library.

Or object databases like ZODB or Durus.

Ciao,
Marc 'BlackJack' Rintsch
Oct 19 '07 #8

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by scott_mcarthur2003 | last post: by
7 posts views Thread by Jon Trickey | last post: by
1 post views Thread by Anders K. Jacobsen [DK] | last post: by
3 posts views Thread by Florin | last post: by
1 post views Thread by Pablo Bianco SE | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.