"ulimit -s" has no effect?

[Maciej Kalisiak]

I use this simple test in Python:

def foo(i):
print i
foo(i+1)
import sys
sys.setrecursio nlimit(1000000)
foo(0)

Now, my understanding is that I get the segfault when Python overruns the C
stack. Naturally the last number printed should go up when I increase the
stack size in the shell before calling Python, by using "ulimit -s <number
of k>". But this seems to not be the case. Whatever value I use with
ulimit, Python always segfaults at the same recursion depth. Any advice on
how to get things to work proper with this?

% python -V
Python 2.3.3
% uname -a
Linux isildur 2.4.24-1-686 #1 ...

[Martin v. Löwis]

Maciej Kalisiak wrote:

Now, my understanding is that I get the segfault when Python overruns the C
stack. Naturally the last number printed should go up when I increase the
stack size in the shell before calling Python, by using "ulimit -s <number
of k>". But this seems to not be the case. Whatever value I use with
ulimit, Python always segfaults at the same recursion depth. Any advice on
how to get things to work proper with this?

As a starting point, verify that the limit you set actually does get
set. It may be that you don't have permission to increase your stack
limit.

Regards,
Martin

[Maciej Kalisiak]

"Martin v. Löwis" <ma****@v.loewi s.de> writes:

As a starting point, verify that the limit you set actually does get
set. It may be that you don't have permission to increase your stack
limit.

I check that it is set by running "ulimit -s" without a numerical argument.
The reported value is the one I have set, viz. much larger than the default of
8192 (kbytes). This is the soft limit. The hard limit is reported to be
"unlimited" .

If it matters, let me add that I'm running Python from within zsh.
% zsh --version
zsh 4.0.4 (i686-pc-linux-gnu)

--
Maciej Kalisiak | <mac at dgp.toronto.edu > | http://www.dgp.toronto.edu/~mac [McQ]

[Mathias Waack]

Maciej Kalisiak wrote:

If it matters, let me add that I'm running Python from within zsh.
% zsh --version
zsh 4.0.4 (i686-pc-linux-gnu)

Have you tried it with a different shell? Your example works for me
as expected. But I'm using a bash where ulimit is a builtin. Don't
know about zsh.

Mathias

[Maciej Kalisiak]

Mathias Waack <M.*****@gmx.de > writes:

Have you tried it with a different shell? Your example works for me
as expected. But I'm using a bash where ulimit is a builtin. Don't
know about zsh.

"ulimit" is a builtin in zsh as well. But I did try under "bash" from a Linux
console. Same behaviour. On two different machines. This must be something
embarassingly simple. Sanity check, just to be on the same page, here's my
procedure:

1. % python
2. # type in routine

def foo(i):

print i
foo(i+1)

3. >>> foo(0)
4. this properly raises exception at depth 998 or so
5. import sys
6. sys.setrecursio nlimit(1000000)
7. >>> foo(0)
8. segfaults at about depth 7000+
9. % ulimit -s 32000
10. repeat steps 1 through 7
11. same segfault point

--
Maciej Kalisiak | <ma*@dgp.toront o.edu> | http://www.dgp.toronto.edu/~mac

[Andrew MacIntyre]

On Fri, 5 Feb 2004, Maciej Kalisiak wrote:

I check that it is set by running "ulimit -s" without a numerical argument.
The reported value is the one I have set, viz. much larger than the default of
8192 (kbytes). This is the soft limit. The hard limit is reported to be
"unlimited" .

I have no idea whether this affects Linux, but some pthreads
implementations hard code the stack size for the "primary" or
"initial" thread.

Python is built with thread support if possible (incl on Linux), but
without explicit use of the thread support Python code runs in the context
of the "primary" thread.

ISTR that RedHat, amongst others, changed thread implementations
relatively recently.

I know that there is/was a bug in the Python bug tracker on SF related
to this issue on Linux (symptom is test_sre dumping core) with Python
2.3.3. This particular issue is sensitive to the version of gcc and
optimisation settings (& on Linux, whether the Python core is in a SO),
as newer releases/higher optimisation settings result in larger stack
frames.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: an*****@bullsey e.apana.org.au (pref) | Snail: PO Box 370
an*****@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Maciej Kalisiak]

Andrew MacIntyre <an*****@bullse ye.apana.org.au > writes:

I have no idea whether this affects Linux, but some pthreads
implementations hard code the stack size for the "primary" or
"initial" thread.

Python is built with thread support if possible (incl on Linux), but
without explicit use of the thread support Python code runs in the context
of the "primary" thread.

ISTR that RedHat, amongst others, changed thread implementations
relatively recently.

I know that there is/was a bug in the Python bug tracker on SF related
to this issue on Linux (symptom is test_sre dumping core) with Python
2.3.3. This particular issue is sensitive to the version of gcc and
optimisation settings (& on Linux, whether the Python core is in a SO),
as newer releases/higher optimisation settings result in larger stack
frames.

Interesting.

1. Is there a test or code snippet that I can run to confirm that this is what
ails my system?

2. If this *is* the problem, what would be a good workaround?
For the record I am using Debian's sid/unstable distribution.
debs used:
python 2.3.2.91-1
libc6 2.3.2.ds1-8

"ldd" shows python uses libpthreads, and it comes from the above libc6
package.

Upgraded Python to debian package 2.3.3-5; no change.

[Andrew MacIntyre]

On Fri, 6 Feb 2004, Maciej Kalisiak wrote:

1. Is there a test or code snippet that I can run to confirm that this is what
ails my system?
A trivial C program can be used to demonstrate:

---8<--- stest.c ---8<---
#include <stdio.h>

int recursion_func( int t1, int t2, int t3);

int main(void)
{
printf("counter = %d\n", recursion_func( 25, 15, 1));
return 0;
}
---8<------8<-------8<---
---8<--- rfunc.c ---8<---
int recursion_func( int t1, int t2, int t3)
{
int z1, z2, z3;
double x1, x2, x3;

z1 = t1 << 2;
z2 = t2 << 2;
z3 = t3 << 2;

x1 = 3.47 * z1;
x2 = 1.29 * z2;
x3 = -2.47 * z3;

return x1 * x2 - x3 > 1.5e30 ? t3 : recursion_func( ++t1, ++t2, ++t3);
}
---8<------8<-------8<---

Note that the recursion function was made more complex than might have
been needed to try to force larger stack frames. I was also trying to
avoid gcc's optimiser undoing the recursion at higher levels of
optimisation - this worked with gcc 2.95, but gcc 3.3.2 seems to be able
to make this test non-recursive even with -Os ... :-|

On FreeBSD, this is built by

$ gcc -g -O -pthread -c rfunc.c
$ gcc -g -O -pthread -c stest.c
$ gcc -g -O -pthread -o stest stest.o rfunc.o

You'll have to adapt for Linux. Running this:

$ ./stest
Bus error (core dumped)
$ gdb stest stest.core
GNU gdb 4.18 (FreeBSD)
{...copyright stuff elided...}
Program terminated with signal 10, Bus error.
Reading symbols from /usr/lib/libc_r.so.4...d one.
Reading symbols from /usr/libexec/ld-elf.so.1...done .
#0 0x80484ce in recursion_func (t1=21846, t2=21836, t3=21822)
at pthread_recursi on_test_aux.c:3
3 int recursion_func( int t1, int t2, int t3)
(gdb)
libc_r is the pthread-supporting libc.

Building without the "-pthread" switch uses the non-threaded libc, which
results in:

$ ./stest
Segmentation fault (core dumped)
$ gdb stest stest.core
{...}
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libc.so.4...don e.
Reading symbols from /usr/libexec/ld-elf.so.1...done .
#0 0x8048530 in recursion_func (t1=1398102, t2=1398092, t3=1398078)
at pthread_recursi on_test_aux.c:1 6
16 return x1 * x2 - x3 > 1.5e30 ? t3 : recursion_func( ++t1,
++t2, +
+t3);
(gdb)
In this case, I know the stack size is 64MB; the ratio t1 values between
the two matches 1MB to 64MB.
2. If this *is* the problem, what would be a good workaround?

If you can do without threads, build the Python interpreter that way.

I suggest that you enquire of glibc forums whether there is any way to
tweak the primary thread stack size by some runtime means, otherwise
the only option would probably require changing a library header file and
recompiling glibc :-( (which is what's required on FreeBSD 4.x - don't
know about 5.x).

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: an*****@bullsey e.apana.org.au (pref) | Snail: PO Box 370
an*****@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Josiah Carlson]

> def foo(i):

print i
foo(i+1)
import sys
sys.setrecursio nlimit(1000000)
foo(0)

This entire thread begs the question: Why are you recursing this deep?
It would be faster to iterate.

- Josiah

[Maciej Kalisiak]

Josiah Carlson <jc******@nospa m.uci.edu> writes:

def foo(i):
print i foo(i+1)
import sys
sys.setrecursio nlimit(1000000)
foo(0)

This entire thread begs the question: Why are you recursing this deep?
It would be faster to iterate.

The algorithm calls for it: Tarjan's algorithm for finding the
Strongly-Connected Component (SCC) of a graph. If there is an equally
efficient iterative approach, I'd like to hear of it.

I don't think I need to recurse to depth 1,000,000 , but definitely higher
than 7k.