Dear ng,
I use the thread module (not threading) for a client/server app where I
distribute large amounts of pickled data over ssh tunnels.
Now I get regular Segmentation Faults during high load episodes. I use a
semaphore to have pickle/unpickle run nonthreaded, but I still get
frequent nondeterministi c segmentation faults.
Since there is no traceback after a sf, I have no clue what exactly
happened, and debugging a multithreaded app is no fun anyway :(
Can someone recommend me how to get extra info during such a crash?
Or any other opinion on where the problem might lie?
Thanks a lot,
Mathias 8 3638
I use the thread module (not threading) for a client/server app where I
distribute large amounts of pickled data over ssh tunnels.
Now I get regular Segmentation Faults during high load episodes. I use a
semaphore to have pickle/unpickle run nonthreaded, but I still get
frequent nondeterministi c segmentation faults.
Since there is no traceback after a sf, I have no clue what exactly
happened, and debugging a multithreaded app is no fun anyway :(
Can someone recommend me how to get extra info during such a crash?
Or any other opinion on where the problem might lie?
Hi, it would be helpful if you posted a minimalistic code snippet
which showed the problem you describe.
Daniel
Hi, it would be helpful if you posted a minimalistic code snippet
which showed the problem you describe.
Daniel
I wish I could! If I knew exactly where the effect takes place I could
probably circumvent it. All I know know is that it happens under high
load and with a lot of waitstates I can reduce the propability of
crashing. So there must be some race condition somewhere I think.
Is there a way to analyze where the crash took place? I guess I can have
a core dumped and somehow analyze it, but that's probably very hard to do.
Would a profiler work which records the function call structure?
Does someone have experience with threading in python - are there
non-threadsafe functions I should know about?
Thanks,
Mathias
Daniel Nogradi wrote:
>I use the thread module (not threading) for a client/server app where I distribute large amounts of pickled data over ssh tunnels.
What module are you using for SSH?
What's in your program that isn't pure Python?
The problem is probably in some non-Python component; you shouldn't
be able to force a memory protection error from within Python code.
Also note that the "marshal" module may be unsafe.
John Nagle
John Nagle wrote:
Daniel Nogradi wrote:
>>I use the thread module (not threading) for a client/server app where I distribute large amounts of pickled data over ssh tunnels.
What module are you using for SSH?
What's in your program that isn't pure Python?
The problem is probably in some non-Python component; you shouldn't
be able to force a memory protection error from within Python code.
Also note that the "marshal" module may be unsafe.
John Nagle
I'm using os.popen2() to pipe into an ssh session via stdin/stdout.
That's probably not the elegant way...
Other modules: scipy 0.3.2 (with Numeric 24.2) and python 2.4
Does pickle/cPickle count as part of the marshal module?
Mathias
PS: setting sys.setcheckint erval(1) reduces the probablilty of a failure
as well, but definetely at a performance cost.
"Mathias" <ma************ ****@webDELETEM E.dewrote:
Does someone have experience with threading in python - are there
non-threadsafe functions I should know about?
how do your threads communicate with one another - are there any
globals that are accessed from different threads?
strange this - you should get an exception, not a segment fault if its
in the python bits..
- Hendrik
>
What module are you using for SSH?
What's in your program that isn't pure Python?
The problem is probably in some non-Python component; you shouldn't
be able to force a memory protection error from within Python code.
It looks like the error could be in scipy/Numeric, when a large array's
type is changed, like this:
>>from scipy import * a=zeros(10000 0000,'b') #100 MiB b=a.copy().as type('d') #800 MiB, ok a=zeros(10000 00000,'b') #1GiB b=a.copy().as type('d') #8GiB, fails with sf
Segmentation fault
if I use zeros directly for allocation of the doubles it works as expected:
>>from scipy import * a=zeros(10000 00000,'d') #8GiB, fails with python exception
Traceback (most recent call last):
File "<stdin>", line 1, in ?
MemoryError: can't allocate memory for array
>>>
I use python 2.4, but my scipy and Numeric aren't quite up-to-date:
scipy version 0.3.2, Numeric v 24.2
Mathias wrote:
>> What module are you using for SSH?
What's in your program that isn't pure Python? The problem is probably in some non-Python component; you shouldn't be able to force a memory protection error from within Python code.
It looks like the error could be in scipy/Numeric, when a large array's
type is changed, like this:
>>from scipy import *
>>a=zeros(10000 0000,'b') #100 MiB
>>b=a.copy().as type('d') #800 MiB, ok
>>a=zeros(10000 00000,'b') #1GiB
>>b=a.copy().as type('d') #8GiB, fails with sf
Segmentation fault
if I use zeros directly for allocation of the doubles it works as expected:
>>from scipy import *
>>a=zeros(10000 00000,'d') #8GiB, fails with python exception
Traceback (most recent call last):
File "<stdin>", line 1, in ?
MemoryError: can't allocate memory for array
>>>
I use python 2.4, but my scipy and Numeric aren't quite up-to-date:
scipy version 0.3.2, Numeric v 24.2
That sounds like the case where the array has to be reallocated from
4-byte floats to 8-byte doubles is being botched.
Take a look at
"http://www.mail-archive.com/nu************* *@lists.sourcef orge.net/msg02033.html"
and then at array_cast in arrraymethods.c of scipy. There may be a
reference count bug in that C code. I'm not familiar enough with
Python reference count internals to be sure, though.
John Nagle This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Dave Kirby |
last post by:
I am working on a network management program written in python that has
multiple threads (typically 20+) spawning subprocesses which are used
to communicate with other systems on the network. This runs fine for a
while, but eventually slows down to a crawl. Running sar shows that
when it is running slowly there is an exceptionally large number of
minor page faults - there are continuously 14000 faults/sec, with a
variation of about...
|
by: sysxperts |
last post by:
Hello,
Having an issue that is specific to PHP compiled with PGSQL support
with versions noted in subject line. I understand that there are many
variables to consider here but believe I have narrowed down the Apache
Client Cert failures to my PHP/PGSQL build.
1. Apache PHP without PGSQL works as expected using client
certificates
2. Apache PHP with PGSQL works fine with standard SSL connections
3 Apache PHP with PGSQL fails with...
|
by: Digital Puer |
last post by:
Hi, I'm coming over from Java to C++, so please bear with me.
In C++, is there a way for me to use exceptions to catch
segmentation faults (e.g. when I access a location off
the end of an array)?
Thanks.
|
by: lawrence k |
last post by:
We made some changes to our server yesterday, and ever since, every
single installation of WordPress that was on the server has stopped
running. Other PHP scripts still run fine, but WordPress is dead.
I logged into the server using ssh and looked the Apache error_log. The
only thing there was a whole bunch of lines like this:
child pid 27827 exit signal
Segmentation fault (11)
|
by: pycraze |
last post by:
I would like to ask a question. How do one handle the exception due to
Segmentation fault due to Python ? Our bit operations and arithmetic
manipulations are written in C and to some of our testcases we
experiance Segmentation fault from the python libraries.
If i know how to handle the exception for Segmentation fault , it will
help me complete the run on any testcase , even if i experiance Seg
Fault due to any one or many functions in...
| |
by: Michael B. Trausch |
last post by:
Is there a way to debug scripts that cause segmentation faults? I can
do a backtrace in gdb on Python, but that doesn't really help me all
that much since, well, it has nothing to do with my script... :-P
Any ideas?
Thanks,
Mike
|
by: George Sakkis |
last post by:
I have a pure python program (no C extensions) that occasionally core
dumps in a non-reproducible way. The program is started by a (non-
python) cgi script when a form is submitted. It involves running a
bunch of other programs through subprocess in multiple threads and
writing its output in several files. So the only suspicious parts I
can think of is subprocess and/or multithreading. For the
multithreading part I'm using a modified...
|
by: John Williams |
last post by:
I've written a simple program to do XOR encryption as my first foray
into understanding how encryption works. The code compiles fine, however
it segmentation faults on every run. using gdb to debug it let me
narrow the problem down to the Cipher function I think it faults at line
84 or 85. The program makes it's first read/cipher/write pass without
issue but the second pass kills it. Using gdb to print the variables
left showed me the...
|
by: IanWright |
last post by:
I've got a segmentation fault problem and am wondering if anyone can help me. I'll apologise now for the fact that I've about a week experience using C++, but I'll try my best to follow any replies :)
Firstly I'm developing in C++ on a Linux machine (using g++ compiler).
My application iterates through a process around about 120 times during testing. It appears that I randomly receive segmentation faults during one of these iterations...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |