Python signal delivery under BSD 4.4

[Elf M. Sternberg]

I'm having a devil of a time with signal handling under BSD 4.4 with
Python and I was hoping that this might ring a bell with someone. Using
the Webware Application Server (a multi-threaded Python application), I
fork off a process to run independently and do some heavy-duty multi-
threaded I/O churning. The process is launched with os.spawnvp().

The user may, at any time, elect to terminate this process
before it reaches its end. I do that by having the Application Server
send a SIGUSR1 signal to the process ID returned by os.spawnvp(), which
so far has been reliably returned and stored in the appserver's session
mananger. So far, so good.

This worked well under Python 2.1, where I used os.fork() instead
of os.spawnvp().

We have upgraded to Python 2.2.3 and it has mysteriously stopped
working. Processes launched off the appserver, whether using os.fork()
or os.spawnvp(), are now completely deaf to signals. Nothing gets its
attention except, of course, SIGKILL. Not when sent from the appserver,
and not when sent from the shell, not even by root.

If the process is launched from the command line, however, it
is well-behaved and responds to signal reliably.

What gives?

[Donn Cave]

In article <m3************@drizzle.com>,
"Elf M. Sternberg" <el*@drizzle.com> wrote:

I'm having a devil of a time with signal handling under BSD 4.4 with
Python and I was hoping that this might ring a bell with someone. Using
the Webware Application Server (a multi-threaded Python application), I
fork off a process to run independently and do some heavy-duty multi-
threaded I/O churning. The process is launched with os.spawnvp().

The user may, at any time, elect to terminate this process
before it reaches its end. I do that by having the Application Server
send a SIGUSR1 signal to the process ID returned by os.spawnvp(), which
so far has been reliably returned and stored in the appserver's session
mananger. So far, so good.

This worked well under Python 2.1, where I used os.fork() instead
of os.spawnvp().

We have upgraded to Python 2.2.3 and it has mysteriously stopped
working. Processes launched off the appserver, whether using os.fork()
or os.spawnvp(), are now completely deaf to signals. Nothing gets its
attention except, of course, SIGKILL. Not when sent from the appserver,
and not when sent from the shell, not even by root.

If the process is launched from the command line, however, it
is well-behaved and responds to signal reliably.

What gives?

Don't know. Can you duplicate it in a simpler environment?
I'd like to run the following C program from spawnv, from a
Python program like

import os
import time
import signal

p = os.spawnv(os.P_NOWAIT, 'prsig', ['prsig'])
time.sleep(1.0)
os.kill(p, signal.SIGTERM)
print 'process exit', os.waitpid(p, 0)

if you think that represents what you're doing. This seems
to work as expected with Python 2.2 on a BSD platform (if
your "BSD 4.4" is actually NetBSD 1.6 or FreeBSD 5.1, I could
try that.)

The reasons I can think of in principle are
1. handler set to SIG_IGN (if the child process is supposed
to exit due to the default signal handler.)
2. blocked via signal mask
3. wrong process

Donn Cave, do**@u.washington.edu/do**@drizzle.com

include <signal.h>
#include <stdio.h>
#include <string.h>

static void
handler(int sig)
{
switch (sig) {
case SIGALRM:
fprintf(stderr, "caught SIGALRM\n");
break;
case SIGUSR1:
fprintf(stderr, "caught SIGUSR1\n");
break;
default:
fprintf(stderr, "caught %d\n", sig);
}
exit(0);
}

static void
prsig(int sig)
{
struct sigaction sa;
int status;
memset(&sa, 0, sizeof(sa));
status = sigaction(sig, 0, &sa);
if (status == 0) {
printf("sigaction {\n");
printf(" sa_handler 0x%08x,\n", sa.sa_handler);
printf(" sa_flags 0x%08x\n", sa.sa_flags);
/* printf(" sa_mask 0x%08x\n", sa.sa_mask); */
printf("}\n");
} else {
perror("sigaction query");
exit(2);
}
}

static void
prmask()
{
sigset_t mask;
int status;
status = sigprocmask(SIG_SETMASK, 0, &mask);
if (status == 0)
fprintf(stdout, "mask 0x%lx\n", mask);
else {
perror("sigprocmask");
exit(2);
}
}

static void
sethandler(int sig, void (*hf)(int))
{
struct sigaction sa;
int status;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = hf;
status = sigaction(sig, &sa, 0);
if (status == 0) {
} else {
perror("sigaction");
exit(2);
}
}

static void
pause()
{
char buffer[4];
read(0, buffer, 4);
}

int
main(int argc, char **argv)
{
prsig(SIGUSR1);
prmask();
sethandler(SIGUSR1, handler);
sethandler(SIGALRM, handler);
alarm(3);
pause();
return 0;
}

[Elf M. Sternberg]

Donn Cave <do**@u.washington.edu> writes:

In article <m3************@drizzle.com>,
"Elf M. Sternberg" <el*@drizzle.com> wrote:

This worked well under Python 2.1, where I used os.fork() instead
of os.spawnvp(). We have upgraded to Python 2.2.3 and it has mysteriously stopped
working. Processes launched off the appserver, whether using os.fork()
or os.spawnvp(), are now completely deaf to signals. Nothing gets its
attention except, of course, SIGKILL. Not when sent from the appserver,
and not when sent from the shell, not even by root.

I'm sorry about the confusion. FreeBSD 4.4 is what we're
running where I work.
Can you duplicate it in a simpler environment?
I'd like to run the following C program from spawnv, from a
Python program like import os
import time
import signal p = os.spawnv(os.P_NOWAIT, 'prsig', ['prsig'])
time.sleep(1.0)
os.kill(p, signal.SIGTERM)
print 'process exit', os.waitpid(p, 0)

I'm going to assume you meant 'SIGUSR1' here since that's what
you're catching in the C code you included.

In any event, your code worked as expected.

However, this variant worked as I've reported:

import os
import sys
import time
import signal
import thread
def do_it():
p = os.spawnv(os.P_NOWAIT, 'prsig', ['prsig'])
print >>sys.stderr, "Sleeping"
time.sleep(1.0)
os.kill(p, signal.SIGUSR1)
print 'process exit', os.waitpid(p, 0)
sys.exit()

thread.start_new_thread(do_it, ())
time.sleep(4.0)

Not only did it behave as I reported, but it slept the full four
seconds even though the sys.exit() call was made. This leads me to the
conclusion that something rather unusual is going on within Python's
threading module and the libc_r.a within FreeBSD4.4.

Right now, I just want a way to make signal delivery work.

Thanks for your help so far, though.

Elf

[Donn Cave]

In article <m3************@drizzle.com>,
"Elf M. Sternberg" <el*@drizzle.com> wrote:
....

I'm going to assume you meant 'SIGUSR1' here since that's what
you're catching in the C code you included.
Right, sorry.
However, this variant worked as I've reported:

[... forks from thread ]

I tried it from FreeBSD 5.1, using 2.2.2 built from ports,
and didn't see this problem. I believe 5.1 has a significantly
different thread implementation from 4.x, though.

Note that there is pthread_sigmask(3) with the same arguments as
sigprocmask(2), and Python's thread_pthread.h does appear to block
all signals that way when starting a new thread. That's a clue,
perhaps. If sigprocmask() didn't show any signals blocked, maybe
pthread_sigmask() would, or maybe at any rate you can set the
mask to 0 using one or the other of those functions.

Donn Cave, do**@u.washington.edu

[Andrew MacIntyre]

On Wed, 13 Aug 2003, Donn Cave wrote:

In article <m3************@drizzle.com>,
"Elf M. Sternberg" <el*@drizzle.com> wrote:

However, this variant worked as I've reported: [... forks from thread ]

I tried it from FreeBSD 5.1, using 2.2.2 built from ports,
and didn't see this problem. I believe 5.1 has a significantly
different thread implementation from 4.x, though.

And later releases of FreeBSD 4.x have a number of bugfixes to the pthread
support, including some signal related changes. The 4.x pthread support
still has some issues though.

Elf refers to this working with Python 2.1 - I vaguely recall some changes
to Python's pthreads support to try and "harmonise" the behaviour across a
number of platforms, and signal delivery may have been affected such that
signals are only delivered to the primary thread. As this is only a vague
recollection, treat with caution...

Does building Python with the linuxthreads port, instead of FreeBSD's
native pthreads, behave as expected?
Note that there is pthread_sigmask(3) with the same arguments as
sigprocmask(2), and Python's thread_pthread.h does appear to block
all signals that way when starting a new thread. That's a clue,
perhaps. If sigprocmask() didn't show any signals blocked, maybe
pthread_sigmask() would, or maybe at any rate you can set the
mask to 0 using one or the other of those functions.

I also recall Michael Hudson trying to get signal mask support working in
the presence of threads, but he could only get it to work on Linux. We
did eventually get a required fix to FreeBSD's libc_r committed (in the
4.7 timeframe I think), but Michael had given up by then and canned his
code.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: an*****@bullseye.apana.org.au (pref) | Snail: PO Box 370
an*****@pcug.org.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Elf M. Sternberg]

Andrew MacIntyre <an*****@bullseye.apana.org.au> writes:

I tried it from FreeBSD 5.1, using 2.2.2 built from ports,
and didn't see this problem. I believe 5.1 has a significantly
different thread implementation from 4.x, though.

And later releases of FreeBSD 4.x have a number of bugfixes to the pthread
support, including some signal related changes. The 4.x pthread support
still has some issues though.

I went and looked again at the evidence.

When calling the function with apply(), the child process spawned off
reports:

mask 0x0

When calling the function with thread.start_new_thread(), the child
process reports:

mask 0xfffefeff

So I think the two of you have more or less hit it squarely on the
head. Now, I just have to find a way around it... *Sigh*

Elf

[Alex Martelli]

Michael Hudson wrote:

"Elf M. Sternberg" <el*@drizzle.com> writes:

Right now, I just want a way to make signal delivery work.

At some point, the threads implementation got changed so that new
threads get their signal mask set to "block everything and it's cat"
(there was a reason for this, but I'm not really sure what it was).
As I see it, you have two options: 1) write a little C to unset the
procmask after fork() but before execve() 2) only launch apps from the
main thread.

Incidentally, these aren't far from the advice in "Pthreads Programming"
(by Nichols, Buttlar and Proulx Farrell -- O'Reilly) in Chapter 5, under
"Threads and Process Management". The authors are dealing with C issues,
not Python by any means, but still they conclude that "forking from a
multithreaded program is no picnic" and suggest 1) fork before you've
created any threads, and 2) consider the surrogate parent model (fork a
child process at init time, IPC with the child -- which remains single
threaded -- to delegate forking and execing on our behalf).
Alex

[Elf M. Sternberg]

Alex Martelli <al*****@yahoo.com> writes:

Incidentally, these aren't far from the advice in "Pthreads Programming"
(by Nichols, Buttlar and Proulx Farrell -- O'Reilly) in Chapter 5, under
"Threads and Process Management". The authors are dealing with C issues,
not Python by any means, but still they conclude that "forking from a
multithreaded program is no picnic" and suggest 1) fork before you've
created any threads, and 2) consider the surrogate parent model (fork a
child process at init time, IPC with the child -- which remains single
threaded -- to delegate forking and execing on our behalf).

(1) wasn't really viable since the parent process is ruled by an
application server (Webware, in this case) and I have no desire to go
trolling through the guts of that excellent program to make it "work"
the way I want. I almost went with (2). But, fortunately, we're
developing a single-OS appliance and don't really care about
cross-platform compatibility, so I worked with the FreeBSD 4.4 "model"
of forking from multithreaded applications. It's fortunate that the
next call is an exec(); I'd hate to have to manage a forked copy of the
process with that one thread...

Elf

[Donn Cave]

Quoth "Elf M. Sternberg" <el*@drizzle.com>:
| ... But, fortunately, we're
| developing a single-OS appliance and don't really care about
| cross-platform compatibility, so I worked with the FreeBSD 4.4 "model"
| of forking from multithreaded applications. It's fortunate that the
| next call is an exec(); I'd hate to have to manage a forked copy of the
| process with that one thread...

So, you resolved the problem and the spawnv child process can now
receive signals? Did you use sigprocmask() to clear the signal
mask? What is the FreeBSD 4.4 model of forking from multithreaded
applications? (And is it the same as the FreeBSD 4.7 model, 5.1, etc.?)

Donn Cave, do**@drizzle.com