Python 2.5 Core Dump on Solaris 8

[Melissa Evans]

Hi. I'm new to Python. :)

I've modified grappy.py,
http://www.stacken.kth.se/~mattiasa/projects/grappy/, a postfix policy
daemon for greylisting. to use LDAP as a backend instead of SQL (with
python-ldap.) The daemon runs fine when testing but when I put it under
load it core dumps quickly. What little analysis I know how to do shows
similar information every time. Any advice on where to go from here?

Thanks!
Melissa
Python 2.5 on Solaris 8:

truss:

write(1, " g r e y l i s t e d : ".., 82) = 82
Action:defer_if_permit Temporary failure
write(1, " A c t i o n : d e f e r".., 41) = 41
Incurred fault #5, FLTACCESS %pc = 0xFF142794
siginfo: SIGBUS BUS_ADRALN addr=0x0000000F
Received signal #10, SIGBUS [default]
siginfo: SIGBUS BUS_ADRALN addr=0x0000000F
*** process killed ***

pflags:

core './core' of 12227: python grappy-ldap.py
data model = _ILP32 flags = PR_RLC
flttrace = 0xfffffbff
sigtrace = 0xfffffeff 0xffffffff
entryset = 0x00000401 0x04000000 0x00000000 0x00000028
0x80000000 0x00000000 0x00000000 0x00000000
exitset = 0xfffffffe 0xffffffff 0xffffffff 0xffffffd7
0x7fffffff 0xffffffff 0xffffffff 0xffffffff
/16: flags = PR_PCINVAL
sigmask = 0xffffbefc,0x00001fff cursig = SIGBUS
/17: flags = PR_STOPPED
why = PR_SUSPENDED
/18: flags = PR_STOPPED
why = PR_SUSPENDED
/19: flags = PR_STOPPED
why = PR_SUSPENDED
/20: flags = PR_STOPPED
why = PR_SUSPENDED
/21: flags = PR_STOPPED
why = PR_SUSPENDED
/1: flags = PR_STOPPED
why = PR_SUSPENDED
/2: flags = PR_STOPPED|PR_ASLWP
why = PR_SUSPENDED
sigmask = 0xffbffeff,0x00001fff
/3: flags = PR_STOPPED
why = PR_SUSPENDED
/4: flags = PR_STOPPED
why = PR_SUSPENDED
/5: flags = PR_STOPPED
why = PR_SUSPENDED
/6: flags = PR_STOPPED
why = PR_SUSPENDED
/7: flags = PR_STOPPED
why = PR_SUSPENDED
/8: flags = PR_STOPPED
why = PR_SUSPENDED
/9: flags = PR_STOPPED
why = PR_SUSPENDED
/10: flags = PR_STOPPED
why = PR_SUSPENDED
/11: flags = PR_STOPPED
why = PR_SUSPENDED
/12: flags = PR_STOPPED
why = PR_SUSPENDED
/13: flags = PR_STOPPED
why = PR_SUSPENDED
/14: flags = PR_STOPPED
why = PR_SUSPENDED
/15: flags = PR_STOPPED
why = PR_SUSPENDED

pstack for the suspicious thread:

core './core' of 12227: python grappy-ldap.py
----------------- lwp# 16 / thread# 13 --------------------
ff142794 t_delete (ffffffff, 15df78, fffffffd, 1590f0, 3c4cf0, 4e80) +
c
ff14240c realfree (ffffffff, ff1c2858, ff1bc008, 1590f0, 4e83, 1590f8)
+ d0
ff142cb0 cleanfree (0, ff1bc008, ff1c27cc, ff1c284c, ff1c281c, 0) + 58
ff141de4 _malloc_unlocked (10, 0, ff1bc008, 10, 1, 0) + f0
ff141fec realloc (ff1c0608, 10, ff1bc008, 4cd80, 10, 0) + 5c
00040278 app1 (26c788, 264610, 1292ec, 0, 25ddd0, 25dddc) + a4
00044324 listappend (26c788, 264610, 1291c4, 8, ff1bfc78, 106418) + 8
0008f1f8 call_function (fe1086a0, 1, 8cc30, 4cd80, 6, 15c45c) + 594
0008cc38 PyEval_EvalFrameEx (2c9668, 1, 0, 2c9668, 1, 1b00f0) + 2be0
0008f73c fast_function (1daaf0, 3d7240, 2, 2c9668, 1b00f0, 2639c0) + c4
0008f5ec call_function (fe108868, 2, 8cc30, 4cd80, 5, 21191c) + 988
0008cc38 PyEval_EvalFrameEx (3d70e8, 1, 0, 3d70e8, 1, 1b00f0) + 2be0
0008f73c fast_function (268c30, 2c8dc4, 1, 3d70e8, 1b00f0, 1b00f0) + c4
0008f5ec call_function (fe108a30, 1, 8cc30, 0, 4, 156ac4) + 988
0008cc38 PyEval_EvalFrameEx (2c8c78, 0, 0, 2c8c78, 1, 1b00f0) + 2be0
0008e03c PyEval_EvalCodeEx (1d4a88, 1cc4b0, 0, 2639cc, 4, 0) + 838
000e2010 function_call (1e0130, 2639c0, 0, e1ed0, 146a98, 14c8e4) + 140
00025c28 PyObject_Call (1e0130, 2639c0, 0, 2639dc, 3, 2639c0) + 20
0002e2ac instancemethod_call (1e0130, 2639c0, 0, 2e09c, 1249f8, 26a9ad)
+ 210
00025c28 PyObject_Call (26c3a0, 269a58, 0, e2730, 2, 156a5c) + 20
0008ec2c PyEval_CallObjectWithKeywords (26c3a0, 269a58, 0, 332420, 1,
1b00f0) + f4
000290a4 PyInstance_New (269580, 269a58, 0, 28f88, 12462c, 14c8e4) +
11c
00025c28 PyObject_Call (2632d0, 269a58, 0, 0, 269a58, 269a64) + 20
00091ba0 do_call (2632d0, fe1091a0, ffffffff, 0, 26976c, 269760) + 94
0008f604 call_function (fe1091a0, 3, 8cc30, 4, 3, 265754) + 9a0
0008cc38 PyEval_EvalFrameEx (330f98, 3, 0, 330f98, 1, 1b00f0) + 2be0
0008f73c fast_function (1da1b0, 332570, 3, 330f98, 1b00f0, 332108) + c4
0008f5ec call_function (fe109368, 3, 8cc30, 0, 2, 156a5c) + 988
0008cc38 PyEval_EvalFrameEx (332420, 2, 0, 332420, 1, 1b00f0) + 2be0
0008e03c PyEval_EvalCodeEx (1d4728, 1cc4b0, 0, 26976c, 3, 2c0510) + 838
000e2010 function_call (1e00b0, 269760, 26ddb0, e1ed0, 146a98, 14c8e4)
+ 140
00025c28 PyObject_Call (1e00b0, 269760, 26ddb0, 0, 26976c, 269760) + 20
0008fe8c ext_do_call (1e00b0, fe109604, 3, ffffffff, 0, 23d4b4) + 3ec
0008cd30 PyEval_EvalFrameEx (3, 3323d4, 0, 332298, 1, 1b00f0) + 2cd8
0008f73c fast_function (242f30, 3cd2d4, 1, 332298, 1b00f0, 0) + c4
0008f5ec call_function (fe1097d0, 1, 8cc30, 0, 0, 2a79f4) + 988
0008cc38 PyEval_EvalFrameEx (3cd188, 0, 0, 3cd188, 1, 1b00f0) + 2be0
0008e03c PyEval_EvalCodeEx (23bba8, 2398a0, 0, 25dffc, 1, 0) + 838
000e2010 function_call (242f70, 25dff0, 0, e1ed0, 146a98, 14c8e4) + 140
00025c28 PyObject_Call (242f70, 25dff0, 0, ff1c284c, 0, 25dff0) + 20
0002e2ac instancemethod_call (242f70, 25dff0, 0, 2e09c, 1249f8, 0) +
210
00025c28 PyObject_Call (37da08, 150030, 0, 0, 0, 0) + 20
0008ec2c PyEval_CallObjectWithKeywords (37da08, 150030, 0, 0, 0, 0) +
f4
000bdcc0 t_bootstrap (2c08e0, ff09d658, 1, 1, ff09c000, 0) + 1c
ff08b01c _thread_start (2c08e0, 0, 0, 0, 0, 0) + 40

[Martin v. Löwis]

Melissa Evans schrieb:

I've modified grappy.py,
http://www.stacken.kth.se/~mattiasa/projects/grappy/, a postfix policy
daemon for greylisting. to use LDAP as a backend instead of SQL (with
python-ldap.) The daemon runs fine when testing but when I put it under
load it core dumps quickly. What little analysis I know how to do shows
similar information every time. Any advice on where to go from here?

It looks like a memory allocation bug, e.g. caused by a double free,
or possibly a buffer overrun. It's unlikely that the Python interpreter
itself has such a bug, so it's likely rather caused in an extension
module (e.g. the ldap module).

Analysing such a problem is tedious. You should create a debug build
of Python (which already has some memory checks) and see whether it
reports anything. Then, you could try a debug malloc next (such
as dmalloc.com). If you have a license, you can use Purify, if not,
try Valgrind (not sure whether it runs on Solaris, though). Another
such library is ElectricFence.

Good luck,
Martin

[Anthon]

Hi Melissa,

I run into similar problems after compiling python-ldap 2.2.0 for
Python 2.5 on SuSE Linux 9.3
and running a small commandline application
*** glibc detected *** double free or corruption (out): 0x40180788 ***
I realy suspect the _ldap.so file. I have not found a solution yet, but
will let you know if I do.

Did you compile python-ldap yourself for Python 2.5?
Which version of python-ldap? Which compiler?

Regards
Anthon

Melissa Evans wrote:

Hi. I'm new to Python. :)

I've modified grappy.py,
http://www.stacken.kth.se/~mattiasa/projects/grappy/, a postfix policy
daemon for greylisting. to use LDAP as a backend instead of SQL (with
python-ldap.) The daemon runs fine when testing but when I put it under
load it core dumps quickly. What little analysis I know how to do shows
similar information every time. Any advice on where to go from here?

Thanks!
Melissa

[Anthon]

You can set an environment variable MALLOC_CHECK_ to influence the
behaviour of glibc
0 -no error message program continues
1 -error message, program continues
2 -no error message, kills program
3 -error message, kills program

Since the message only occured with my two programs at exit time, I set
the environment var to 0

HTH
Anthon

[Michael Ströder]

Martin v. Löwis wrote:

Melissa Evans schrieb:

>>I've modified grappy.py,
http://www.stacken.kth.se/~mattiasa/projects/grappy/, a postfix policy
daemon for greylisting. to use LDAP as a backend instead of SQL (with
python-ldap.) The daemon runs fine when testing but when I put it under
load it core dumps quickly. What little analysis I know how to do shows
similar information every time. Any advice on where to go from here?

It looks like a memory allocation bug, e.g. caused by a double free,
or possibly a buffer overrun.

Unfortunately the C part of python-ldap is not in a good shape regarding
Python 2.5 since I'm not a C programmer. Release 2.2.0 works only with
Python 2.3.x. and 2.4.x. :-/

But this seems to help (tested on my local system):
http://sourceforge.net/tracker/index...72&atid=102072

Generally I also asked for help regarding "PEP 353 - preparation for
Python 2.5":
http://sourceforge.net/tracker/index...72&atid=102072

I posted a related message to python-ldap-dev list with some patches:
http://sourceforge.net/mailarchive/f...&forum_id=4346

During the next days I hope to commit some of the changes I've made
since then. Contributions welcome.

Ciao, Michael.

[Michael Ströder]

Michael Ströder wrote:

>
But this seems to help (tested on my local system):
http://sourceforge.net/tracker/index...72&atid=102072

Released python-ldap 2.2.1 yesterday which contains this fix.

Ciao, Michael.