Debugging segmentation faults

[George Sakkis]

I have a pure python program (no C extensions) that occasionally core
dumps in a non-reproducible way. The program is started by a (non-
python) cgi script when a form is submitted. It involves running a
bunch of other programs through subprocess in multiple threads and
writing its output in several files. So the only suspicious parts I
can think of is subprocess and/or multithreading. For the
multithreading part I'm using a modified version of threadpool.py
(http://www.chrisarndt.de/en/software...n/threadpool/), which is
built on top of the threading and Queue stdlib modules. Whatever bugs
may linger there, I'd hope that they would show up as normal Python
exceptions instead of segfaults.

All I have now is a few not particularly insightful core files (actual
path names and args changed for privacy):

/home/gsakkis/foo>gdb --core core.20140
GNU gdb Red Hat Linux (5.2-2)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux".
Core was generated by `python2.5 /home/gsakkis/foo/foo.py --XXX --
max=30 --bar=/tmp/83840`
Program terminated with signal 11, Segmentation fault.
#0 0x080b222d in ?? ()
(gdb) backtrace
#0 0x080b222d in ?? ()
#1 0x080b28d1 in ?? ()
#2 0x080fa8ab in ?? ()
#3 0x0805c918 in ?? ()
(...)
#28 0x080b310f in ?? ()
#29 0x080dbfdd in ?? ()
#30 0x40021fef in ?? ()

Any hints ?

[SPE - Stani's Python Editor]

If the error is reproducable and you can run wxPython, you could use
the excellent WinPdb debugger, which ships with SPE (python editor):
http://www.digitalpeers.com/pythondebugger/

.... but if you can only run your script on a remote server this won't
help you.

Stani

--
SPE - http://pythonide.stani.be

On 7 Mrz., 20:15, "George Sakkis" <[e-mail]:

I have a pure python program (no C extensions) that occasionally core
dumps in a non-reproducible way. The program is started by a (non-
python) cgi script when a form is submitted. It involves running a
bunch of other programs through subprocess in multiple threads and
writing its output in several files. So the only suspicious parts I
can think of is subprocess and/or multithreading. For the
multithreading part I'm using a modified version of threadpool.py
(http://www.chrisarndt.de/en/software...n/threadpool/), which is
built on top of the threading and Queue stdlib modules. Whatever bugs
may linger there, I'd hope that they would show up as normal Python
exceptions instead of segfaults.

All I have now is a few not particularly insightful core files (actual
path names and args changed for privacy):

/home/gsakkis/foo>gdb --core core.20140
GNU gdb Red Hat Linux (5.2-2)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux".
Core was generated by `python2.5 /home/gsakkis/foo/foo.py --XXX --
max=30 --bar=/tmp/83840`
Program terminated with signal 11, Segmentation fault.
#0 0x080b222d in ?? ()
(gdb) backtrace
#0 0x080b222d in ?? ()
#1 0x080b28d1 in ?? ()
#2 0x080fa8ab in ?? ()
#3 0x0805c918 in ?? ()
(...)
#28 0x080b310f in ?? ()
#29 0x080dbfdd in ?? ()
#30 0x40021fef in ?? ()

Any hints ?

[John Nagle]

You're using Python on a web server to do something
complicated. You must suffer.

Are you trying to fork off a subprocess in a multithreaded
program? That's unlikely to work. The sematics differ
from OS to OS (Solaris forks all the threads, most other
operating systems don't; most UNIX-based OSs copy all the
open file descriptors, but some give you control of which
open files are passed), and Python may not be thread-safe
in that area.

John Nagle

SPE - Stani's Python Editor wrote:

If the error is reproducable and you can run wxPython, you could use
the excellent WinPdb debugger, which ships with SPE (python editor):
http://www.digitalpeers.com/pythondebugger/

... but if you can only run your script on a remote server this won't
help you.

Stani

--
SPE - http://pythonide.stani.be

On 7 Mrz., 20:15, "George Sakkis" <[e-mail]:

>>I have a pure python program (no C extensions) that occasionally core
dumps in a non-reproducible way. The program is started by a (non-
python) cgi script when a form is submitted. It involves running a
bunch of other programs through subprocess in multiple threads and
writing its output in several files. So the only suspicious parts I
can think of is subprocess and/or multithreading. For the
multithreading part I'm using a modified version of threadpool.py
(http://www.chrisarndt.de/en/software...n/threadpool/), which is
built on top of the threading and Queue stdlib modules. Whatever bugs
may linger there, I'd hope that they would show up as normal Python
exceptions instead of segfaults.

All I have now is a few not particularly insightful core files (actual
path names and args changed for privacy):

/home/gsakkis/foo>gdb --core core.20140
GNU gdb Red Hat Linux (5.2-2)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux".
Core was generated by `python2.5 /home/gsakkis/foo/foo.py --XXX --
max=30 --bar=/tmp/83840`
Program terminated with signal 11, Segmentation fault.
#0 0x080b222d in ?? ()
(gdb) backtrace
#0 0x080b222d in ?? ()
#1 0x080b28d1 in ?? ()
#2 0x080fa8ab in ?? ()
#3 0x0805c918 in ?? ()
(...)
#28 0x080b310f in ?? ()
#29 0x080dbfdd in ?? ()
#30 0x40021fef in ?? ()

Any hints ?

[George Sakkis]

On Mar 7, 4:15 pm, John Nagle <[e-mail]:

You're using Python on a web server to do something
complicated. You must suffer.

Are you trying to fork off a subprocess in a multithreaded
program? That's unlikely to work. The sematics differ
from OS to OS (Solaris forks all the threads, most other
operating systems don't; most UNIX-based OSs copy all the
open file descriptors, but some give you control of which
open files are passed), and Python may not be thread-safe
in that area.

I see the potential problem in general, but in my case every thread is
exclusively responsible for the subprocesses it forks; no subprocess
is inherited from the main thread or is shared in among the worker
threads.

George

[John Nagle]

George Sakkis wrote:

On Mar 7, 4:15 pm, John Nagle <[e-mail]:

> You're using Python on a web server to do something
complicated. You must suffer.

Are you trying to fork off a subprocess in a multithreaded
program? That's unlikely to work. The sematics differ
from OS to OS (Solaris forks all the threads, most other
operating systems don't; most UNIX-based OSs copy all the
open file descriptors, but some give you control of which
open files are passed), and Python may not be thread-safe
in that area.

I see the potential problem in general, but in my case every thread is
exclusively responsible for the subprocesses it forks; no subprocess
is inherited from the main thread or is shared in among the worker
threads.

Forking itself may not be thread safe. Forking is a process-level
operation, for historical reasons.

John Nagle