473,320 Members | 1,810 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Tuple size and memory allocation for embedded Python

Hi Folks,

Python seems unstable, when allocating big memory. For example, the
following C++ code creates a tuple of tuples:

PyObject* arCoord = PyTuple_New(n);
double d = 1.5;
for(int i=0; i<n; i++)
{
PyObject* coord = PyTuple_New(2);
PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
PyTuple_SetItem(arCoord,i, coord);
}

When the n is small, say 100, the code works fine. when n is big, say
10,000, Python has trouble allocating memory, saying:

"Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
collection' ignored
Fatal Python error: unexpected exception during garbage collection
Aborted"

Could anyone please give me some insight or a fix for this?

Thanks in advance for your answer.

Jinming Xu

__________________________________________________ _______________
Is your PC infected? Get a FREE online computer virus scan from McAfee®
Security. http://clinic.mcafee.com/clinic/ibuy...n.asp?cid=3963

Jul 18 '05 #1
2 2844
Jinming Xu wrote:
Hi Folks,

Python seems unstable, when allocating big memory. For example, the
following C++ code creates a tuple of tuples:

PyObject* arCoord = PyTuple_New(n);
double d = 1.5;
for(int i=0; i<n; i++)
{
PyObject* coord = PyTuple_New(2);
PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
PyTuple_SetItem(arCoord,i, coord);
}

When the n is small, say 100, the code works fine. when n is big, say
10,000, Python has trouble allocating memory, saying:

"Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
collection' ignored
Fatal Python error: unexpected exception during garbage collection
Aborted"

Could anyone please give me some insight or a fix for this?

Thanks in advance for your answer.

I'm going to guess that the problem is related to incorrect reference
counts. I don't see any IncRefs in there. It seems probable that the
program will work until you make n high enough to trigger a garbage
collection sweep, then memory your program still regards as allocated is
garbage collected by Python and reused. Ugly :-P

Python is pretty stable, so it's usually best to suspect our own code
unless you're heavily into using the C API (which I'm not, so feel free
to ignore me).

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #2
On Fri, 2005-01-21 at 17:20 -0500, Steve Holden wrote:
Jinming Xu wrote:
Hi Folks,

Python seems unstable, when allocating big memory. For example, the
following C++ code creates a tuple of tuples:

PyObject* arCoord = PyTuple_New(n);
double d = 1.5;
for(int i=0; i<n; i++)
{
PyObject* coord = PyTuple_New(2);
PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
PyTuple_SetItem(arCoord,i, coord);
}

When the n is small, say 100, the code works fine. when n is big, say
10,000, Python has trouble allocating memory, saying:

"Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
collection' ignored
Fatal Python error: unexpected exception during garbage collection
Aborted"

Could anyone please give me some insight or a fix for this?

Thanks in advance for your answer.
I'm going to guess that the problem is related to incorrect reference
counts.


It's usually a safe bet, after all. Another biggie is unchecked return
codes leaving the exception state set, though... that can cause _really_
_weird_ problems. ALWAYS check return values.
I don't see any IncRefs in there.
In this case it looks OK. PyFloat_FromDouble() reuturns a new reference,
as does PyTuple_New(), and PyTuple_SetItem() steals a reference to its
PyObject* argument.

Of course, there could be refcount errors outside the shown code
segment, but in this case I'd say the immediate error will be because of
an unhandled exception. As to why that exception is being thrown....

Also, forget my comment in my last post about not resizing - I'd failed
to notice the initial creation size of the tuple (the creation of which
isn't checked, but would segfault the app on failure).
Python is pretty stable, so it's usually best to suspect our own code
unless you're heavily into using the C API (which I'm not, so feel free
to ignore me).


That's been my experience - stability issues in my Python/C code have
almost always come down to refcounting bugs and/or failing to detect and
handle or propagate an exception.

--
Craig Ringer

Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

50
by: Will McGugan | last post by:
Hi, Why is that a tuple doesnt have the methods 'count' and 'index'? It seems they could be present on a immutable object. I realise its easy enough to convert the tuple to a list and do this,...
6
by: benevilent | last post by:
Hey, I'm trying to debug the memory allocation in an embedded use of the Python interpreter. The longer I leave my program running the more memory it consumes. The total number of objects in...
29
by: George Sakkis | last post by:
Why does slicing a tuple returns a new tuple instead of a view of the existing one, given that tuples are immutable ? I ended up writing a custom ImmutableSequence class that does this, but I...
53
by: john67 | last post by:
The company I work for is about to embark on developing a commercial application that will cost us tens-of-millions to develop. When all is said and done it will have thousands of business...
14
by: Otto Meijer | last post by:
Hi everyone, for one of my projects I need to figure out the size of the swap file(s) of a certain system. The problem is I need to do this on a host of platforms like: HP_UX, Solaris, Linux,...
7
by: mef526 | last post by:
I have had this problem for months now and it has been nagging me. I have a large project that has several C++ DLL's, one of them uses malloc / calloc / free for the other DLL's. I process very...
10
by: Alex Vinokur | last post by:
The memory allocation issue in embedded systems is usually critical.. How can one manage that? 1. Via 'new' char* p = new (nothrow) char ; if (p == 0) { // He we know that it is impossible...
11
by: Andy Watson | last post by:
I have an application that scans and processes a bunch of text files. The content I'm pulling out and holding in memory is at least 200MB. I'd love to be able to tell the CPython virtual machine...
5
by: vishnu | last post by:
Hi there, I am embedding python 2.5 on embedded system running on RTOS where I had strict memory constraints. As python is a huge malloc intensive application, I observed huge memory...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.