By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
444,002 Members | 1,171 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 444,002 IT Pros & Developers. It's quick & easy.

Multiple instances of a python program

P: n/a
Hi.
I am part of a group in my univ where we organize a programming
contest. In this contest we have a UDP based server. The server
simulates a game and each contestant is to develop a team of virtual
players. Each team is composed of 75 similar bots...i.e. governed by
the same logic. Thus the contestant submits a single copy of the client
and we instantiate the same program 75 times at the same time.
The problem is that while executables from C source files are small and
we can make 75 processes but we dont know what to do with python.

If you have a python script and you want that 75 copies of the script
be run simultaneously how will you do it? Is there anyway to do so
without running 75 copies of the python interpreter simultaneously?

Jul 19 '05 #1
Share this Question
Share on Google+
8 Replies


P: n/a
Most of the Python interpreter is a shared library, at least on
Windows. The only duplication is in loaded Python code, which includes
only your bot and the libraries it uses. If you have memory problems,
try to do without some libraries or to edit unused parts out of them.

Lorenzo Gatti

Jul 19 '05 #2

P: n/a
Rahul wrote:
Hi.
I am part of a group in my univ where we organize a programming
contest. In this contest we have a UDP based server. The server
simulates a game and each contestant is to develop a team of virtual
players. Each team is composed of 75 similar bots...i.e. governed by
the same logic. Thus the contestant submits a single copy of the client
and we instantiate the same program 75 times at the same time.
The problem is that while executables from C source files are small and
we can make 75 processes but we dont know what to do with python.

If you have a python script and you want that 75 copies of the script
be run simultaneously how will you do it? Is there anyway to do so
without running 75 copies of the python interpreter simultaneously?


The technical way would be to use threads.

Of course it could be that the rules of the game explicitly forbid that
bots of a group communicate other than through the server. And it is
easier to cheat there when you have only one Python program running.

Daniel
Jul 19 '05 #3

P: n/a
Daniel Dittmar wrote:
Rahul wrote:
Hi.
I am part of a group in my univ where we organize a programming
contest. In this contest we have a UDP based server. The server
simulates a game and each contestant is to develop a team of virtual
players. Each team is composed of 75 similar bots...i.e. governed by
the same logic. Thus the contestant submits a single copy of the client
and we instantiate the same program 75 times at the same time.
The problem is that while executables from C source files are small and
we can make 75 processes but we dont know what to do with python.

If you have a python script and you want that 75 copies of the script
be run simultaneously how will you do it? Is there anyway to do so
without running 75 copies of the python interpreter simultaneously?


The technical way would be to use threads.

Of course it could be that the rules of the game explicitly forbid that
bots of a group communicate other than through the server. And it is
easier to cheat there when you have only one Python program running.


yup...its not allowed to communicate through other means.

well we do realise that its not easy to catch cheating but we require
the source code of the participants too. and we dont want to provide
simple loopholes.

rahul

Jul 19 '05 #4

P: n/a
On Thu, 16 Jun 2005 11:47:10 -0700, Rahul wrote:
Hi.
I am part of a group in my univ where we organize a programming
contest. In this contest we have a UDP based server. The server
simulates a game and each contestant is to develop a team of virtual
players. Each team is composed of 75 similar bots...i.e. governed by
the same logic. Thus the contestant submits a single copy of the client
and we instantiate the same program 75 times at the same time.
The problem is that while executables from C source files are small and
we can make 75 processes but we dont know what to do with python.

If you have a python script and you want that 75 copies of the script
be run simultaneously how will you do it? Is there anyway to do so
without running 75 copies of the python interpreter simultaneously?


Have you actually tested the performance of 75 instances of Python
running? Do you know that it will be too slow for your server, or are you
trying to optimize before testing?

I wrote a short Python script, then launched 115 instances of Python
executing the script. There was no detectable slowdown of my system, which
is far from a high-end PC.

The details of the script aren't important. It may even be that what I
tested is not even close to the load your server needs to deal with. But
you may be surprised at just how easily even a low-end PC copes 75
instances of Python. Or perhaps not -- but the only way to tell is to try.

--
Steven.
Jul 19 '05 #5

P: n/a
On Thu, 16 Jun 2005 11:47:10 -0700, Rahul wrote:
If you have a python script and you want that 75 copies of the script be
run simultaneously how will you do it? Is there anyway to do so without
running 75 copies of the python interpreter simultaneously?


If you're running on Linux (and other Unixes perhaps), you could use the
os.fork() function to create independent child processes from a single
python process. I believe Linux forked processes share memory until a
section of memory is written to (copy on write functionality).

If most of python is in a shared library, then this probably won't make
much difference.

Jeremy

Jul 19 '05 #6

P: n/a
Hi.
I will look into it..thanks
rahul

Jeremy Sanders wrote:
On Thu, 16 Jun 2005 11:47:10 -0700, Rahul wrote:
If you have a python script and you want that 75 copies of the script be
run simultaneously how will you do it? Is there anyway to do so without
running 75 copies of the python interpreter simultaneously?


If you're running on Linux (and other Unixes perhaps), you could use the
os.fork() function to create independent child processes from a single
python process. I believe Linux forked processes share memory until a
section of memory is written to (copy on write functionality).

If most of python is in a shared library, then this probably won't make
much difference.

Jeremy


Jul 19 '05 #7

P: n/a
Steven D'Aprano wrote:
On Thu, 16 Jun 2005 11:47:10 -0700, Rahul wrote:
Hi.
I am part of a group in my univ where we organize a programming
contest. In this contest we have a UDP based server. The server
simulates a game and each contestant is to develop a team of virtual
players. Each team is composed of 75 similar bots...i.e. governed by
the same logic. Thus the contestant submits a single copy of the client
and we instantiate the same program 75 times at the same time.
The problem is that while executables from C source files are small and
we can make 75 processes but we dont know what to do with python.

If you have a python script and you want that 75 copies of the script
be run simultaneously how will you do it? Is there anyway to do so
without running 75 copies of the python interpreter simultaneously?


Have you actually tested the performance of 75 instances of Python
running? Do you know that it will be too slow for your server, or are you
trying to optimize before testing?

I wrote a short Python script, then launched 115 instances of Python
executing the script. There was no detectable slowdown of my system, which
is far from a high-end PC.

The details of the script aren't important. It may even be that what I
tested is not even close to the load your server needs to deal with. But
you may be surprised at just how easily even a low-end PC copes 75
instances of Python. Or perhaps not -- but the only way to tell is to try.


Well...i havent tried (yes i hear "Premature optimization is evil evil
evil i say") but the point was that if we can find a solution consuming
less memory than we can even increase the number from 75 to lets say
200. as for hardware we have machines with 1.7 Ghz P4 and 128 mb ram.
and i cant run them right now...since i am currently not in my
univ...but asked now since we are planning right now and wanted to see
which languages we can support. Probably c,c++,python and lisp using
clisp...and java if we can find a way to avoid running 75 jvms...this
year we supported c,c++,java and actually ran 75 jvms using 3 machines
and it was horrible so we are looking for better ways for the 2006
contest. And we may port our server from java to python too but it
seems not many people had python installed but most had java installed.

rahul

Jul 19 '05 #8

P: n/a
"Rahul" <co********@gmail.com> writes:
Steven D'Aprano wrote:
On Thu, 16 Jun 2005 11:47:10 -0700, Rahul wrote:
Hi.
I am part of a group in my univ where we organize a programming
contest. In this contest we have a UDP based server. The server
simulates a game and each contestant is to develop a team of virtual
players. Each team is composed of 75 similar bots...i.e. governed by
the same logic. Thus the contestant submits a single copy of the client
and we instantiate the same program 75 times at the same time.
The problem is that while executables from C source files are small and
we can make 75 processes but we dont know what to do with python.

If you have a python script and you want that 75 copies of the script
be run simultaneously how will you do it? Is there anyway to do so
without running 75 copies of the python interpreter simultaneously?


Have you actually tested the performance of 75 instances of Python
running? Do you know that it will be too slow for your server, or are you
trying to optimize before testing?

I wrote a short Python script, then launched 115 instances of Python
executing the script. There was no detectable slowdown of my system, which
is far from a high-end PC.

The details of the script aren't important. It may even be that what I
tested is not even close to the load your server needs to deal with. But
you may be surprised at just how easily even a low-end PC copes 75
instances of Python. Or perhaps not -- but the only way to tell is to try.


Well...i havent tried (yes i hear "Premature optimization is evil evil
evil i say") but the point was that if we can find a solution consuming
less memory than we can even increase the number from 75 to lets say
200. as for hardware we have machines with 1.7 Ghz P4 and 128 mb ram.
and i cant run them right now...since i am currently not in my
univ...but asked now since we are planning right now and wanted to see
which languages we can support. Probably c,c++,python and lisp using
clisp...and java if we can find a way to avoid running 75 jvms...this
year we supported c,c++,java and actually ran 75 jvms using 3 machines
and it was horrible so we are looking for better ways for the 2006
contest. And we may port our server from java to python too but it
seems not many people had python installed but most had java installed.

rahul


As others have said, start by testing. Given 1.7 MHz and 128MB, I'm
guessing your performance problems are coming from swapping. Simply
getting more RAM in the box would help.

A quick check on process load just for the instances themselves can be
done by gradually increasing max_processes (see below) until you swap.
On a box with 256MB total, 150MB free, I was getting over 150 children
before swap was noticeable. By watching "top" I found each child was
using about 3MB, of which about 2MB is "shared" (basically, the python
library). In other words, each instance was costing 1MB.

Of course, your actual apps will require more RAM than this little
test program.

---main program---
max_processes=10
progname="child.py"
for i in xrange(max_processes):
os.spawnv(os.P_NOWAIT,progname,[progname,str(i)])

---child program---
def msg(txt):
sys.stdout.write(txt)
sys.stdout.flush()
....
#get childnum from comline parameters
....
for i in xrange(10):
msg('%i ' % childnum)
time.sleep(1)

--
ha************@boeing.com
6-6M21 BCA CompArch Design Engineering
Phone: (425) 294-4718
Jul 19 '05 #9

This discussion thread is closed

Replies have been disabled for this discussion.