473,239 Members | 1,404 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,239 software developers and data experts.

Memory Problems in Windows 2003 Server

Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes
Oct 12 '07 #1
7 1482
amdescombes wrote:
Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes
How are you reading the large files? IMO, large files are better read in
chunks:

target_file = open(f, 'rb')
while 1:
data = target_file.read(8192000)
if data:
DO SOMETHING
else:
break

The above reads 8MB at a time until the file has been completely read.
Change the 8MB to whatever you like.
Oct 12 '07 #2
amdescombes wrote:
Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes
I forgot to mention that the OS itself or other processes may be using a
lot of memory. So, just because you have 2GB, that does not mean you can
access all of that at once. I would guess that 25% of memory is in
constant use by the OS. So, do your IO/reads in smaller chunks similar
to the example I gave earlier.

Brad
Oct 12 '07 #3
AMD
Hi Brad,

I do the reading one line at a time, the problem seems to be with the
dictionary I am creating.

Andre
amdescombes wrote:
>Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file
and generate more than 8 million keys on an Windows XP machine with
only 1GB of memory and all works as expected. When I use the same
program on a Windows 2003 Server with 2GB of RAM I start getting
MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes

I forgot to mention that the OS itself or other processes may be using a
lot of memory. So, just because you have 2GB, that does not mean you can
access all of that at once. I would guess that 25% of memory is in
constant use by the OS. So, do your IO/reads in smaller chunks similar
to the example I gave earlier.

Brad
Oct 13 '07 #4
AMD <am*********@gmail.comwrote:
>
I do the reading one line at a time, the problem seems to be with the
dictionary I am creating.
I don't know whether Python dictionaries must live in a contiguous piece of
memory, but if so, that could be the issue. The system DLLs in Server 2003
have been "rebased" in such a way that they chop up the virtual address
space more than XP. Even though there is more virtual memory available, it
is fragmented.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
Oct 14 '07 #5
Yes, I think that might be the issue, perhaps I could implement the
solution using several dictionaries instead of just one.
Are there any classes that implement disk based dictionaries?

Thanks,

Andre
>
I don't know whether Python dictionaries must live in a contiguous piece of
memory, but if so, that could be the issue. The system DLLs in Server 2003
have been "rebased" in such a way that they chop up the virtual address
space more than XP. Even though there is more virtual memory available, it
is fragmented.
Oct 15 '07 #6
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote:
Are there any classes that implement disk based dictionaries?
Take a look at the `shelve` module from the standard library.

Or object databases like ZODB or Durus.

Ciao,
Marc 'BlackJack' Rintsch
Oct 15 '07 #7
AMD
Thanks Marc,

I just tried shelve but it is very slow :(
I haven't tried the dbs yet.

Andre

Marc 'BlackJack' Rintsch a écrit :
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote:
>Are there any classes that implement disk based dictionaries?

Take a look at the `shelve` module from the standard library.

Or object databases like ZODB or Durus.

Ciao,
Marc 'BlackJack' Rintsch
Oct 19 '07 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: scott_mcarthur2003 | last post by:
We are running asp (not asp.net) and vb com dll web sites on a machine which also has sql server 2000 sp3a running and uses Windows Server 2003. We recently migrated from Windows 2000 (same...
5
by: Justice | last post by:
Currently I'm doing some experimenting with the XMLHTTP object in Javascript. Now, the XMLHttp object is asynchronous (at least in this case), and the following code causes a significant memory...
0
by: ccuthbert | last post by:
I have a MCMS 2002 site that we recently had several hotfixes installed. After installing the hotfixes, both AOL users and Earthlink users experience tha same problem, namely that intermittently...
1
by: Larry Bertolini | last post by:
I was browsing Microsoft's SQL Server site, looking for some details about SQL Server 2005. Didn't find what I was looking for... I'm thinking about moving an existing SQL Server 2000 workload...
7
by: Jon Trickey | last post by:
We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the weekend, but we have a light user load then (about 200 users.) Today when we had close to 600 users connecting and running...
16
by: JCauble | last post by:
We have a large Asp.net application that is currently crashing our production servers. What we are seeing is the aspnet_wp eat up a bunch of memory and then stop unexpectedly. Does not recycle. ...
1
by: Anders K. Jacobsen [DK] | last post by:
Hi, We are running a intranet application (.net 1.1 asp.net) on a windows 2003 server. We are having some issues that causes extreem slowdowns (like 1 min to show one page) This happens...
3
by: Florin | last post by:
Hi all, I have a problem related to memory grow on a server application which is basically stateless (I have some static info loaded). The client accesses this server using remoting and it has...
1
by: Pablo Bianco SE | last post by:
Hi everyone, I am new in this group. I have a question and would like to have some support. A customer is experiencing memory problems with the asp.net worker process. The have an application...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.