Memory error in python

[sangeeth]

I have a very big python file which uses lists,tuples and dictionary.
The problem is that the program always crashes with Memory error.

Can anyone suggest how to solve this problem??

[Motoma]

I have a very big python file which uses lists,tuples and dictionary.
The problem is that the program always crashes with Memory error.

Can anyone suggest how to solve this problem??

What is the environment you are using?
How big is the Python script getting?
What specific error messages are you getting?
Where in the code do you think things are going astray?
Are you performing any threading?

Usually, Python memory leaks are caused (seemingly intentionally) by the programmer. Python's data structures are large, and it is quite easy to waltz across 500 megs without realizing it.

[Extremist]

I have a very big python file which uses lists,tuples and dictionary.
The problem is that the program always crashes with Memory error.

Can anyone suggest how to solve this problem??

A few questions, otherwise there is no hope:
1. What is the error you are getting
2. What is the code you are using
3. What is the Python version you are using

I think it won't be a tuple,

[sangeeth]

A few questions, otherwise there is no hope:
1. What is the error you are getting
2. What is the code you are using
3. What is the Python version you are using

I think it won't be a tuple,

The error is Memory Error
The code is very big spanning some 41,000 lines, which I don't think will be useful if I share, but it uses a lot of lists and dictionaries
The Python version is 2.4.3

Can you help me now??
Maybe how to do better memory management in Python?

[Motoma]

The error is Memory Error
The code is very big spanning some 41,000 lines, which I don't think will be useful if I share, but it uses a lot of lists and dictionaries
The Python version is 2.4.3

Can you help me now??
Maybe how to do better memory management in Python?

When you get this error, does Python give you a Trackback? Try going into the code and looking at what is happening there.

You didn't answer my question as to how much memory the Python program is taking up when it starts running.

If you've written a 41000 line Python script, I expect you would have more than a cursory understanding of the language; do all of your classes have proper destructors? Are you properly disposing of dictionary entries, or entire dictionaries when you are done with the data they contain? What is the flow of your program; is it a big load operation, followed by a big process information followed by a big store operation, or is it getting caught up on a task that shouldn't have a problem with memory?
You haven't given me much to go on, the more helpful you are, the more helpful we can be.

[Extremist]

The error is Memory Error
The code is very big spanning some 41,000 lines, which I don't think will be useful if I share, but it uses a lot of lists and dictionaries
The Python version is 2.4.3

Can you help me now??
Maybe how to do better memory management in Python?

Well not to be mean or anything, I mean this in a very helpfull way,
but you are not saying anything. It's like saying: I have a typeError in 40000 lines of code. It really doensn't say anything.
And to tell you the truth, I got a Seg Fault until recently, when I discovered I havent installed a module or two.

I understand about the amount of code, perhaps you could write a little example app which will do the same and then it will be easier.

Answer the guy above's question as well
Keep me posted

[bvdet]

The error is Memory Error
The code is very big spanning some 41,000 lines, which I don't think will be useful if I share, but it uses a lot of lists and dictionaries
The Python version is 2.4.3

Can you help me now??
Maybe how to do better memory management in Python?

Memory leak? According to Python documentation http://docs.python.org/lib/module-gc.html:

To debug a leaking program call gc.set_debug(gc.DEBUG_LEAK). Notice that this includes gc.DEBUG_SAVEALL, causing garbage-collected objects to be saved in gc.garbage for inspection.

[bartonc]

Here is a link to a discussion on range vs xrange. Good luck.

[sudiptachatterjee]

Okay, so I have run into the same problem, and the offending statement is this:

 tempHolder[followWord] = 0 

The tempHolder is a dictionary whose length can run into a really BIG number, because I am trying to go through an entire corpus of text and counting occurences of each word, so to say. Any ideas about how I can specify to Python that it needs to be prepared for such big lengths of the dictionary? Or is this a theoretical limit which I'll have to work around by creating smaller sub-dictionaries?

Thanks a lot in advance.

[bartonc]

Okay, so I have run into the same problem, and the offending statement is this:

 tempHolder[followWord] = 0 

The tempHolder is a dictionary whose length can run into a really BIG number, because I am trying to go through an entire corpus of text and counting occurences of each word, so to say. Any ideas about how I can specify to Python that it needs to be prepared for such big lengths of the dictionary? Or is this a theoretical limit which I'll have to work around by creating smaller sub-dictionaries?

Thanks a lot in advance.

You should still post the entire error message to give a clue how to help.