473,326 Members | 2,048 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Overhead of individual python apps

I'm setting up a system that consists of several small python
applications that all communicate amongst each other on the same pc.

When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.

My Question: Is there any way to reduce this per-process overhead? eg:
can you set it somehow so that one python.exe instance handles multiple
processes?

One possibility considered is to run them as threads of a single
process rather than multiple processes, but this has other drawbacks
for my application and I'd rather not,

Another possibility I considered is to strip out all but the most
essential imports in each app, but I tested this out and it has
marginal benefits. I demonstrated to myself that a simple one liner
app consisting of 'x = raw_input()' still eats up > 2.7MB .

I also tried -O but it, not surprisingly, did nothing for the
one-liner.

I'm simply running the .py files and I am still on v2.3

All help appreciated!

Thanks,
Russ

Sep 27 '05 #1
15 1635
"Qopit" <ru************@gmail.com> writes:
When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.
The first step is to clarify what's being reported. If WTM is
reporting the total memory usage for each process, then it's over
estimating the total usage by a considerable amount. In particular,
all the Python executable code should be shared by all the
processes. Unless you load compiled extensions, anyway - those will
only be shared by the ones that use them. You'll need something that
will give you the stack, heap and code segment sizes separately to
work out what the memory usage really is.
My Question: Is there any way to reduce this per-process overhead? eg:
can you set it somehow so that one python.exe instance handles multiple
processes?
The OS should do that for you by default.
One possibility considered is to run them as threads of a single
process rather than multiple processes, but this has other drawbacks
for my application and I'd rather not,


That shouldn't help memory usage - the data that isn't across
processes would need to be thread-private in any case.

The reason for the uncertainty is that I'm not positive that Windows
behaves sanely in this area. It may be that Windows doesn't have
shared executables. In this case, one solution is to move to a modern
OS - like v6 Unix :-).

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Sep 27 '05 #2
Several apps using 4Mb each shouldn't be very much
memory (maybe 20Mb at most). You didn't say how
much memory was in your machine, but 256Mb of memory
will cost you no more than $50. Not really worth a
lot of effort.

-Larry

Qopit wrote:
I'm setting up a system that consists of several small python
applications that all communicate amongst each other on the same pc.

When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.

My Question: Is there any way to reduce this per-process overhead? eg:
can you set it somehow so that one python.exe instance handles multiple
processes?

One possibility considered is to run them as threads of a single
process rather than multiple processes, but this has other drawbacks
for my application and I'd rather not,

Another possibility I considered is to strip out all but the most
essential imports in each app, but I tested this out and it has
marginal benefits. I demonstrated to myself that a simple one liner
app consisting of 'x = raw_input()' still eats up > 2.7MB .

I also tried -O but it, not surprisingly, did nothing for the
one-liner.

I'm simply running the .py files and I am still on v2.3

All help appreciated!

Thanks,
Russ

Sep 28 '05 #3
Larry Bates <la*********@websafe.com> writes:
Several apps using 4Mb each shouldn't be very much memory (maybe
20Mb at most). You didn't say how much memory was in your machine,
but 256Mb of memory will cost you no more than $50. Not really
worth a lot of effort.


That is bogus reasoning. I can't add 256mb to the machine I'm typing
on right now for any amount of money, let alone $50. It's an older
laptop, it's maxed out at 512 meg, and I can't add any more to it,
period. An even older laptop that I still use from time to time is
maxed out at 72 meg. And my Sharp Zaurus (Linux handheld) has 32 meg
and is not expandable. If Python claims to be a lightweight language,
it should not present any obstacles to running on machines of that
class. It's already slightly annoying that Python can't run well in
an 8 meg Palm Pilot, since there were many Common Lisp implementations
that ran tolerably in less memory than that.
Sep 28 '05 #4
ncf
First, please don't get so upset at people's replies (if you weren't
upset, that's how it was coming across, so my apologies). No matter
what newsgroup/forum you go to, there's always someone who's going to
suggest something like that.

Anyways, I'm fairly certain there are some minimalistic Python variants
designed for older computers and such. Maybe something to check into?

HTH,
-Wes

Sep 28 '05 #5
On 27 Sep 2005 15:11:57 -0700, "Qopit" <ru************@gmail.com>
declaimed the following in comp.lang.python:
When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.
Ah, but is that physical memory consumed, or virtual memory MAPPED
to the processes.

Most of Python is a DLL; as I recall, most DLLs are shared -- just
the process local data needs to be allocated.
-- ================================================== ============ <
wl*****@ix.netcom.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
================================================== ============ <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.netcom.com/> <

Sep 28 '05 #6
Paul Rubin wrote:
Several apps using 4Mb each shouldn't be very much memory (maybe
20Mb at most). You didn't say how much memory was in your machine,
but 256Mb of memory will cost you no more than $50. Not really
worth a lot of effort.


That is bogus reasoning.


not if you're a professional software developer and someone's paying you
to develop an application that is to be run on a platform that they control.

people are expensive, but hardware is extremely cheap these days.

</F>

Sep 28 '05 #7
"Fredrik Lundh" <fr*****@pythonware.com> writes:
That is bogus reasoning.


not if you're a professional software developer and someone's paying you
to develop an application that is to be run on a platform that they control.


An awful lot of Python targeted users are not in that situation, so if
Python's usability suffers for them when it doesn't have to, then
something is wrong with Python.
Sep 28 '05 #8
Paul Rubin wrote:
An awful lot of Python targeted users are not in that situation, so if
Python's usability suffers for them when it doesn't have to, then
something is wrong with Python.


(and so we go from the OP:s "I'm setting up a system" to the usual c.l.python "but
I can come up with another example" crap...)

But back to the "memory measurements" and your claim that "something is wrong
with Python [if the task manager shows large numbers]"...

Have you looked at a task manager on a lightly loaded W2K or WXP system lately?
On this PC, I currently have a couple of browser windows open, a newsreader, and
a few other programs. The only real app I have that uses less than 4 megabytes
*according to the task manager* is a 2.5 megabyte putty terminal window; most
apps are in the 15-50 megabyte range.

Even a trivial operation, like opening another Firefox tab, can make that process
grow by more than 4 megabytes, and if you browse around a little, you can easily
end up with 30-50 megabytes in "mem usage" -- until you minimize the browser,
and find that Firefox only uses 1.5 megabytes. Maximize it again, and it uses 12
megabytes.

Play the same games with a Python interpreter, and you'll find that the task manager
may report anything from just over 50 kilobytes to just under 5 megabytes.

Here's a random google link that discusses the task manager figures in a little more
detail:

http://www.itwriting.com/dotnetmem.php

"Newcomer to .NET says: I've just been looking at Task Manager. Why
does my simple VB.NET Windows application use 12MB RAM?"

"Don't worry. The app doesn't really use that much RAM."

(for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
process, and some of that will only occupy space in the paging file... for firefox with
a simple page loaded into a single tab, the private space is ~10 megabytes)

</F>

Sep 28 '05 #9
Paul Rubin wrote:
"Fredrik Lundh" <fr*****@pythonware.com> writes:
That is bogus reasoning.


not if you're a professional software developer and someone's paying you
to develop an application that is to be run on a platform that they control.

An awful lot of Python targeted users are not in that situation, so if
Python's usability suffers for them when it doesn't have to, then
something is wrong with Python.


If a useful subset of Python can be crammed into a Nokia cell phone then
I really don't think there's much to complain about (except that "it
hasn't been done for *my* machine").

Even embedded systems are much larger now than the minicomputers of
yesteryear. Everything's relative. Just wait three years!

i-remember-when-we-'ad-ter-code-seven-kilobytes-of-assembly-language-wi'-nowt-bu'-a-teletype-ter-edit-t'-paper-tape-wi'-ly
y'rs - steve

--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.pycon.org

Sep 28 '05 #10
Steve Holden wrote:
Even embedded systems are much larger now than the minicomputers of
yesteryear. Everything's relative. Just wait three years!


Had you placed such a bet in 2000, you'd have cleaned up at the
"Moore's Law Casino", but there are various factors at work now which
complicate the once-inevitable trends of hardware performance and the
corresponding advice to people wishing to speed up or slim down their
software: the flattening out of the CPU frequency curve and the
tendency of CPU manufacturers to choose multiple core strategies are
two things which prevent applications from speeding up all by
themselves; whilst storage density is still increasing, as far as I
know, I'd imagine other strategies being adopted in system architecture
which could make performance tradeoffs more pronounced in accessing all
that storage.

In other words, expect many more threads about global interpreter locks
in the coming three years than we've seen in the last three or even six
years. ;-)

Paul

Sep 28 '05 #11
Paul Boddie wrote:
Steve Holden wrote:
Even embedded systems are much larger now than the minicomputers of
yesteryear. Everything's relative. Just wait three years!

Had you placed such a bet in 2000, you'd have cleaned up at the
"Moore's Law Casino", but there are various factors at work now which
complicate the once-inevitable trends of hardware performance and the
corresponding advice to people wishing to speed up or slim down their
software: the flattening out of the CPU frequency curve and the
tendency of CPU manufacturers to choose multiple core strategies are
two things which prevent applications from speeding up all by
themselves; whilst storage density is still increasing, as far as I
know, I'd imagine other strategies being adopted in system architecture
which could make performance tradeoffs more pronounced in accessing all
that storage.

In other words, expect many more threads about global interpreter locks
in the coming three years than we've seen in the last three or even six
years. ;-)

I seem to remember blogging about this a while ago, or perhaps just
ranting at c.l.py ... anyway, bottom line is that most people have more
than adequate power for their current needs in a present-day
single-processor computer.

Expect also to see computational methods focus more on memory-intensive
techniques impractical in the past.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.pycon.org

Sep 28 '05 #12
Thanks - this is all very interesting...
Ah, but is that physical memory consumed, or virtual memory MAPPED
to the processes.
and
for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
process, and some of that will only occupy space in the paging file... for firefox with
a simple page loaded into a single tab, the private space is ~10 megabytes)
and most useful:
http://www.itwriting.com/dotnetmem.php


I had no idea that the memory usage reported by Windows Task Manager
(WTM) was so different than what I expected it would be. It doesn't
seem terribly useful to me right now.

After looking into that link (and discovering the perfmon app... can't
believe I never knew about such an amazingly useful tool!) below are
the results of some memory reporting checks that I ran on the simple
one-liner app (x = raw_input()). The "On Launch" is what was reported
immediately after launching the app, and the "Window Min'd" is
following a simple minimization for the window.

Memory report: On Launch Window Min'd
-------------- --------- ------------
WTM Mem Usage: 2,756K 88K
PerfMon Private bytes: 1,540,096 1,540,096
PerfMon Working set: 2,822,144 90,112 **

Basically it looks like the privately allocated memory for a 2.3 app
that cannot be shared by other processes (close to what I want,
neglecting paging possibilities) is the 1.5 MB that Fredrik reported.
Presumably using perfmon as well?

Maybe I'm a stickler, but this still seems pretty high for each and
every application to need to privately hold. Am I still misreading
this somehow? If not, I'd still love for this to be smaller... is
there any way to reduce this further?

Russ

** There was an interesting transient in the Working set during
minimization all the way up to 9 MB... presumably this is because
during minimization the "set of memory pages touched recently" jumps up
because of the transactions needed for the minimization.

Sep 28 '05 #13
Fredrik Lundh wrote:
(for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
process, and some of that will only occupy space in the paging file... for firefox with
a simple page loaded into a single tab, the private space is ~10 megabytes)


I believe a relatively simple, if crude, way of measuring this sort of
thing (at least for console programs) is to open a handful of command
prompts, check the Physical Memory Available measurement in the Task
Manager, and then load one instance of the program in each console,
checking the drop in physical memory available at each step. After
three or four instances are loaded, the approximate real memory usage
for each should be readily apparent.

Obviously there are a dozen issues that can screw with this, including
having too little memory free at all, and having other programs open
which are very actively allocating or freeing memory.

It does seem to back up the "~1.5MB" number above, and certainly shows
that the total is nowhere near 4MB per Python interpreter.

-Peter
Sep 28 '05 #14
On 28 Sep 2005 07:46:23 -0700, "Qopit" <ru************@gmail.com>
declaimed the following in comp.lang.python:

Maybe I'm a stickler, but this still seems pretty high for each and
every application to need to privately hold. Am I still misreading
this somehow? If not, I'd still love for this to be smaller... is
there any way to reduce this further?
I've not investigated -- but what is the default stack size given to
an application?
-- ================================================== ============ <
wl*****@ix.netcom.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
================================================== ============ <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.netcom.com/> <

Sep 28 '05 #15
Dennis Lee Bieber:
I've not investigated -- but what is the default stack size given to
an application?


Python has a stack reserve of 2000000 bytes but that is an
allocation of virtual space. Pages are realised in that allocation as
required with an initial stack commit of one page, 4096 bytes.

Neil

C:\Python24>dumpbin /headers python.exe
Microsoft (R) COFF/PE Dumper Version 7.10.3077
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file python.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
14C machine (x86)
4 number of sections
424A51B8 time date stamp Wed Mar 30 17:14:00 2005
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
10F characteristics
Relocations stripped
Executable
Line numbers stripped
Symbols stripped
32 bit word machine

OPTIONAL HEADER VALUES
10B magic # (PE32)
7.10 linker version
400 size of code
A00 size of initialized data
0 size of uninitialized data
1062 entry point (1D001062)
1000 base of code
2000 base of data
1D000000 image base (1D000000 to 1D004FFF)
1000 section alignment
200 file alignment
4.00 operating system version
0.00 image version
4.00 subsystem version
0 Win32 version
5000 size of image
400 size of headers
0 checksum
3 subsystem (Windows CUI)
0 DLL characteristics
1E8480 size of stack reserve
1000 size of stack commit
100000 size of heap reserve
1000 size of heap commit
....
Sep 29 '05 #16

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: SS | last post by:
I am trying to execute some python commands within a perl script and running into problems. The following works: python -c "import mypkg;do this; do that" however when I try to do something like...
9
by: limor | last post by:
Hi, I am considering using Python in a new testing tool application we intend to build for out product. I must get references before starting develope in this language , since although lots of...
47
by: Michael Scarlett | last post by:
There is an amazing article by paul graham about python, and an even better discussion about it on slashdot. The reason I point this out, is the more I read both articles, the more I realised how...
35
by: Michael Kearns | last post by:
I've been using python to write a simple 'launcher' for one of our Java applications for quite a while now. I recently updated it to use python 2.4, and all seemed well. Today, one of my...
9
by: Andy Leszczynski | last post by:
Hi, I run Mandrake 10.0 with python 2.3 installed by default. I want to keep it as it is but need another, very customized Python installation based of 2.3 as well. I would prefer to have it the...
16
by: PyDenis | last post by:
Today, I found strange error while using py2exe: 1. I wrote simple program and save as 1.py: import win32ui import win32con win32ui.MessageBox('Test messageBox.' , 'Test', win32con.MB_OK |...
48
by: meyer | last post by:
Hi everyone, which compiler will Python 2.5 on Windows (Intel) be built with? I notice that Python 2.4 apparently has been built with the VS2003 toolkit compiler, and I read a post from Scott...
56
by: Omar | last post by:
I'd love the perfect editor that would be: a) free b) enable me to drag and drop code snippets from a sort of browser into the code c) can run programs right from within d) can edit
7
by: Yansky | last post by:
I asked my hosting company if they would upgrade Python on my server to the latest version. They responded with: "Sorry no. We tend to stick with what comes packaged with the unix distribution...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.