473,224 Members | 1,610 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,224 software developers and data experts.

Python for embedded systems with memory constraints

Hi there,

I am embedding python 2.5 on embedded system running on RTOS where I
had strict memory constraints.
As python is a huge malloc intensive application, I observed huge
memory fragmentation in my system which is leading to out of memory
after running few scripts.
So I decided to re-initialise the python with out restarting the whole python.
I tried to use Py_Finalise() after completion of each script , then
call Py_Initialise as is done in below link.
http://mail.python.org/pipermail/pyt...er/114253.html
Which in every loop it causes a leak of 10K and after some iterations
it leaks of 200K etc. After few more runs this crashes. I read some
where this leak was solved in 2.5, but with 2.5 also I am having
problems....
And also I found Py_Finalise does not completely cleanup the memory,
So how do I re-initialise my memory pool?

Does anybody faced this problem earlier and got any solution hack to
run the python for a embedded system within own managed memory pool of
say 10MB?

Any help/ideas are greatly appreciated!. Thanks in advance.
Jun 4 '07 #1
5 3312
Who else is using python (programmers, scientists, finance)?

Me! Graduated in fine arts. Python is what I do when I am fed up with
all those colors. Much easier to manufacture sense with.


Jun 5 '07 #2
In article <ma***************************************@python. org>,
vishnu <gk*******@gmail.comwrote:
>Hi there,

I am embedding python 2.5 on embedded system running on RTOS where I
had strict memory constraints.
As python is a huge malloc intensive application, I observed huge
memory fragmentation in my system which is leading to out of memory
after running few scripts.
So I decided to re-initialise the python with out restarting the whole python.
I tried to use Py_Finalise() after completion of each script , then
call Py_Initialise as is done in below link.
http://mail.python.org/pipermail/pyt...er/114253.html
Which in every loop it causes a leak of 10K and after some iterations
it leaks of 200K etc. After few more runs this crashes. I read some
where this leak was solved in 2.5, but with 2.5 also I am having
problems....
And also I found Py_Finalise does not completely cleanup the memory,
So how do I re-initialise my memory pool?

Does anybody faced this problem earlier and got any solution hack to
run the python for a embedded system within own managed memory pool of
say 10MB?

Any help/ideas are greatly appreciated!. Thanks in advance.
Your report is interesting and important--and surprising! I thought
Python memory allocation is "cleaner" than you seem to be observing.

I hope one of the core Python maintainers can address this. I haven't
worked at this level recently enough to speculate on why it's happen-
ing, nor will I soon be in a position to volunteer to research it on
my own (although I'd eagerly contract to do so on a modestly paid
basis).

Depending on your schedule and technology, there are lots of technical
fixes that might apply:
A. quick-starting Python variations that encourage you
to manage memory on a whole-process level;
B. use of one of the many Python variants (even PyPy?)
that might give you a more favorable memory profile;
C. switch to Lua or Tcl as more easily embeddable
alternative languages;
D. custom memory allocator;
....
Jun 7 '07 #3
Hi,
Thanks Cameron for your suggestions.
In fact I am using custom memory sub-allocator where I preallocate a
pool of memory during initialization of my application and ensure that
Python doesn't make any system mallocs later . With this arrangement,
python seems to run out of preallocated memory (of 10MB) after running
few simple scripts due to huge external fragmentation. My memory
sub-allocator got a good design which uses the best-fit algorithm and
coaelescing the adjacent blocks during each free call.
If anybody out there used their own memory manager and ran Python
without fragmentation , could provide some inputs on this.

Thanks in advance.

On 6/7/07, Cameron Laird <cl****@lairds.uswrote:
In article <ma***************************************@python. org>,
vishnu <gk*******@gmail.comwrote:
Hi there,

I am embedding python 2.5 on embedded system running on RTOS where I
had strict memory constraints.
As python is a huge malloc intensive application, I observed huge
memory fragmentation in my system which is leading to out of memory
after running few scripts.
So I decided to re-initialise the python with out restarting the whole python.
I tried to use Py_Finalise() after completion of each script , then
call Py_Initialise as is done in below link.
http://mail.python.org/pipermail/pyt...er/114253.html
Which in every loop it causes a leak of 10K and after some iterations
it leaks of 200K etc. After few more runs this crashes. I read some
where this leak was solved in 2.5, but with 2.5 also I am having
problems....
And also I found Py_Finalise does not completely cleanup the memory,
So how do I re-initialise my memory pool?

Does anybody faced this problem earlier and got any solution hack to
run the python for a embedded system within own managed memory pool of
say 10MB?

Any help/ideas are greatly appreciated!. Thanks in advance.

Your report is interesting and important--and surprising! I thought
Python memory allocation is "cleaner" than you seem to be observing.

I hope one of the core Python maintainers can address this. I haven't
worked at this level recently enough to speculate on why it's happen-
ing, nor will I soon be in a position to volunteer to research it on
my own (although I'd eagerly contract to do so on a modestly paid
basis).

Depending on your schedule and technology, there are lots of technical
fixes that might apply:
A. quick-starting Python variations that encourage you
to manage memory on a whole-process level;
B. use of one of the many Python variants (even PyPy?)
that might give you a more favorable memory profile;
C. switch to Lua or Tcl as more easily embeddable
alternative languages;
D. custom memory allocator;
...
--
http://mail.python.org/mailman/listinfo/python-list
Jun 9 '07 #4
On Jun 9, 1:33 pm, vishnu <gkkvis...@gmail.comwrote:
Hi,
Thanks Cameron for your suggestions.
In fact I am using custom memory sub-allocator where I preallocate a
pool of memory during initialization of my application and ensure that
Python doesn't make any system mallocs later . With this arrangement,
python seems to run out of preallocated memory (of 10MB) after running
few simple scripts due to huge external fragmentation. My memory
sub-allocator got a good design which uses the best-fit algorithm and
coaelescing the adjacent blocks during each free call.
If anybody out there used their own memory manager and ran Python
without fragmentation , could provide some inputs on this.

From what I remember, the best-fit algorithm isn't a good idea because
unless the free block was exactly the right size you'd tend to get
left with lots of small fragments. (Suppose that the best fit was a
free block only 4 bytes bigger than what you want; what can you do
with a free block of 4 bytes?)

A worst-fit algorithm would leave larger free blocks which are more
useful subsequently, but I think that the recommendation was next-fit
(ie use the first free block that's big enough, starting from where
you found the last one).

Jun 9 '07 #5
En Mon, 11 Jun 2007 15:59:19 -0300, vishnu <gk*******@gmail.comescribió:
So now I only see the solution to clear my memory pool and restart Python
without restarting the system (i.e. no power cycle to hardware). I tried
to
do this when my memory pool is 60% used in these steps:
1) Py_Finalize( )
2) Reset my Memory pool (i.e. free list links)
3) Then Restart Python by calling Py_Initialize().

But this resulted in Python crash during Py_Initialize(), where I found
that the static variables within embedded Python source code are still
holding some of the references to my memory pool. So now my question is
how
do I restart Python (i.e. reinitialize Python) without restarting whole
system. Is there a way to reset/re-initilaize those static variables such
that it will be possible to re-Initialize Python.
Ouch... I think this should not happen, but anyway I don't know if this
was a design principle. One should inspect all Python code to locate all
static variable references... Maybe you could instrument your allocator to
see which references are still held?
C extensions may be problematic too - there is no way to "uninitialize"
them, and nothing forbids an extension to hold a reference to any object.

--
Gabriel Genellina

Jun 12 '07 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Jens Thiede | last post by:
I want to use python heavily on my embeded linux. What would the best way be to slim down Python. Only keep those modules you would acctually need, and not imparing Python's capabilities? ...
13
by: Alexander May | last post by:
Hi, I love Python! I've been using it for a couple of years now and have found it to be a highly productive language. I evangelize it to my developer friends and am probably responsible for...
137
by: Philippe C. Martin | last post by:
I apologize in advance for launching this post but I might get enlightment somehow (PS: I am _very_ agnostic ;-). - 1) I do not consider my intelligence/education above average - 2) I am very...
4
by: Dennis Clark | last post by:
Hi all, I've looked through the threads about embedded Python that are a year and a half old, and I thought that I'd ask this question now to see if anything has changed. Has anyone, or is...
5
by: James | last post by:
I heard C is better for programming embedded systems than C++? Is this true, if so why?
1
by: Tom Tempelaere | last post by:
Hi people, Somewhere in the near future I will have to develop a kernel for an industrial printer. Some parts of the kernel have real-time constraints, other parts do not. The kernel should be...
49
by: Alex Vinokur | last post by:
Are there any restrictions/problems for use of C++ STL in development in embedded systems? In particular: * Does STL require too much space/memory? * Is 'implementation of STL...
29
by: 63q2o4i02 | last post by:
Hi, I'm interested in using python to start writing a CAD program for electrical design. I just got done reading Steven Rubin's book, I've used "real" EDA tools, and I have an MSEE, so I know what...
2
by: Carl J. Van Arsdall | last post by:
I'm aware of a couple python projects for embedded systems. I am currently considering using Python on an embedded platform to develop a simple application as a personal project, mostly to see if...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: mar23 | last post by:
Here's the situation. I have a form called frmDiceInventory with subform called subfrmDice. The subform's control source is linked to a query called qryDiceInventory. I've been trying to pick up the...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.