os.fork() different in cgi-script?

[Andreas Kuntzagk]

Hi,

following code:

#!/usr/bin/python

import os
import sys

def main():
print "Content-Type: text/plain\n\n"
print os.getpid()

childPID = os.fork()
if childPID == 0:
sys.exit()
else:
os.wait()

if __name__ == "__main__":
main()

when run on a shell, gives the result:
---------------------------
Content-Type: text/plain
20953
----------------------------

but run as a cgi-script it gives:
---------------------------
21039
Content-Type: text/plain
21039
---------------------------
So it looks the main() is run 2 times.
Is it so? And if yes, why?

I googled for this but only found a similar question from 1997 and no
answer.

Ciao, Andreas

[Andreas Kuntzagk]

Sorry, forgot:

this is Python 2.2.1, apache 1.3.26 on SuSE 8.1

Andreas

[Michael Coleman]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

def main():
print "Content-Type: text/plain\n\n"
print os.getpid()

childPID = os.fork()
if childPID == 0:
sys.exit()
else:
os.wait()

if __name__ == "__main__":
main()

when run on a shell, gives the result:
---------------------------
Content-Type: text/plain
20953
----------------------------

but run as a cgi-script it gives:
---------------------------
21039
Content-Type: text/plain
21039
---------------------------
So it looks the main() is run 2 times.
Is it so? And if yes, why?

Probably because stdout is being buffered here. The pid got written in the buffer (but not yet actually printed), then the process was forked, then the buffer got flushed (written out) by each child.

One solution would be to make sure everything is flushed before you fork.

Mike

--
Mike Coleman, Scientific Programmer, [phone]
Stowers Institute for Biomedical Research
1000 E. 50th St., Kansas City, MO 64110

[Andreas Kuntzagk]

On Wed, 02 Jul 2003 09:50:03 +0000, Michael Coleman wrote:

Probably because stdout is being buffered here. The pid got written in
the buffer (but not yet actually printed), then the process was forked,
then the buffer got flushed (written out) by each child.

One solution would be to make sure everything is flushed before you
fork.

Thanx, that's it. Is it possible/advisable to make stdout unbuffered?

Andreas

[Erwin S. Andreasen]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

following code: print "Content-Type: text/plain\n\n"
print os.getpid()

childPID = os.fork()
if childPID == 0:
sys.exit()
else:
os.wait()
[...]
So it looks the main() is run 2 times.
Is it so? And if yes, why?

The problem is buffering. When you run your script from the command
line, the standard output is a terminal and becomes line-buffered.
Thus at the time you fork, the stdout buffer has been flushed.

When you run it as a CGI script, stdout is a socket and thus not a
terminal -- so it becomes block buffered. At the time you fork, each
copy of stdout holds all of our unflushed output. When the two
processes exit they will both flush the output, resulting in it being
printed twice.

You can use sys.stdout.flush() to flush the pending output before
forking.

--
================================================== =============
<er***@andreasen.org> Herlev, Denmark
<URL:http://www.andreasen.org/> <*>
================================================== =============

[Juha Autero]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

So it looks the main() is run 2 times.
Is it so? And if yes, why?
No, it isn't. The problem is that Python uses C STDIO library that has
buffered input and output. The text written in stdout is written in
buffer. Since child process created by fork() is almost identical to
parent process, it will also have a copy of stdout buffer. When
process exits, it closes stdout, which flushes the buffer. So, both
buffers are written to stdout and that causes everything to be printed
twice.

The reason why this doesn happen on command line is that buffers to
terminals are flushed after newline. You can see that by redirecting
script output to file:

$ python script.py >output
$ cat output
Content-Type: text/plain
4309
Content-Type: text/plain
4309
I googled for this but only found a similar question from 1997 and no
answer.

Which is strange since this common problem and has nothing to do with
Python.
--
Juha Autero
http://www.iki.fi/jautero/
Eschew obscurity!

[Michael Coleman]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

def main():
print "Content-Type: text/plain\n\n"
print os.getpid()

childPID = os.fork()
if childPID == 0:
sys.exit()
else:
os.wait()

if __name__ == "__main__":
main()

when run on a shell, gives the result:
---------------------------
Content-Type: text/plain
20953
----------------------------

but run as a cgi-script it gives:
---------------------------
21039
Content-Type: text/plain
21039
---------------------------
So it looks the main() is run 2 times.
Is it so? And if yes, why?

Probably because stdout is being buffered here. The pid got written in the buffer (but not yet actually printed), then the process was forked, then the buffer got flushed (written out) by each child.

One solution would be to make sure everything is flushed before you fork.

Mike

--
Mike Coleman, Scientific Programmer, [phone]
Stowers Institute for Biomedical Research
1000 E. 50th St., Kansas City, MO 64110

[Andreas Kuntzagk]

I googled for this but only found a similar question from 1997 and no
answer.

Which is strange since this common problem and has nothing to do with
Python.

Not so strange, because assuming a Python feature I also included "python"
in the googlequest(tm).

Andreas

[Andreas Kuntzagk]

On Wed, 02 Jul 2003 09:50:03 +0000, Michael Coleman wrote:

Probably because stdout is being buffered here. The pid got written in
the buffer (but not yet actually printed), then the process was forked,
then the buffer got flushed (written out) by each child.

One solution would be to make sure everything is flushed before you
fork.

Thanx, that's it. Is it possible/advisable to make stdout unbuffered?

Andreas

[Erwin S. Andreasen]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

following code: print "Content-Type: text/plain\n\n"
print os.getpid()

childPID = os.fork()
if childPID == 0:
sys.exit()
else:
os.wait()
[...]
So it looks the main() is run 2 times.
Is it so? And if yes, why?

The problem is buffering. When you run your script from the command
line, the standard output is a terminal and becomes line-buffered.
Thus at the time you fork, the stdout buffer has been flushed.

When you run it as a CGI script, stdout is a socket and thus not a
terminal -- so it becomes block buffered. At the time you fork, each
copy of stdout holds all of our unflushed output. When the two
processes exit they will both flush the output, resulting in it being
printed twice.

You can use sys.stdout.flush() to flush the pending output before
forking.

--
================================================== =============
<er***@andreasen.org> Herlev, Denmark
<URL:http://www.andreasen.org/> <*>
================================================== =============

[Juha Autero]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

So it looks the main() is run 2 times.
Is it so? And if yes, why?
No, it isn't. The problem is that Python uses C STDIO library that has
buffered input and output. The text written in stdout is written in
buffer. Since child process created by fork() is almost identical to
parent process, it will also have a copy of stdout buffer. When
process exits, it closes stdout, which flushes the buffer. So, both
buffers are written to stdout and that causes everything to be printed
twice.

The reason why this doesn happen on command line is that buffers to
terminals are flushed after newline. You can see that by redirecting
script output to file:

$ python script.py >output
$ cat output
Content-Type: text/plain
4309
Content-Type: text/plain
4309
I googled for this but only found a similar question from 1997 and no
answer.

Which is strange since this common problem and has nothing to do with
Python.
--
Juha Autero
http://www.iki.fi/jautero/
Eschew obscurity!

[Andreas Kuntzagk]

I googled for this but only found a similar question from 1997 and no
answer.

Which is strange since this common problem and has nothing to do with
Python.

Not so strange, because assuming a Python feature I also included "python"
in the googlequest(tm).

Andreas

[Michael Coleman]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

Thanx, that's it. Is it possible/advisable to make stdout unbuffered?

Probably. I'd be more inclined to just do an explicit flush in the
few places where they are specifically needed, though.

Mike

--
Mike Coleman, Scientific Programmer, [phone]
Stowers Institute for Biomedical Research
1000 E. 50th St., Kansas City, MO 64110

[Michael Coleman]

"Andreas Kuntzagk" <an**************@mdc-berlin.de> writes:

Thanx, that's it. Is it possible/advisable to make stdout unbuffered?

Probably. I'd be more inclined to just do an explicit flush in the
few places where they are specifically needed, though.

Mike

--
Mike Coleman, Scientific Programmer, [phone]
Stowers Institute for Biomedical Research
1000 E. 50th St., Kansas City, MO 64110