472,811 Members | 1,448 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,811 software developers and data experts.

help! multi-threading problem on hyperthreading smp linux server

a colleague of mine has seen an odd problem in some code of ours.
we initially noticed it on webware, but in distilling a test case it seems
to be strictly a python issue. in the real system, it manifests as
webware just locking up, for no apparent reason, until we kill it.
we've also had the python interpreter running webware die on occasion.
it works fine on our desktop linux and windows machines, but fails
on the production hardware. the main difference being that the
production hardware is a dual Xenon machine with
hyperthreading enabled.

has anyone run into problems like this, or have any clues what
the problem might be? we pushed pretty hard on a skeptical
project manager to get python and webware accepted for this
project, and it's embarrassing to have it acting flaky.

i'd really appreciate any insight anyone's got on this. we need
to resolve it quickly. mike's description of the test case follows.

thanks
"Michael W. Balk" wrote:
You asked me yesterday to give you some info on the multi-threading problem
so that you could post a query on comp.lang.python.

Here is what I know so far.

The machine has two Xenon CPUs, both with hyper-threading enabled.
The linux kernel running is: 2.4.20-8smp

The pure python test I ran did the following:

Starts up 50 threads (using the threading module).
Each thread calculates 500 random numbers and writes them to a file,
repeating this 50 times to generate 50 output files. So there are 2500
files expected once all 50 threads complete.

The observation is that only a few threads actually produce output files,
and none of those threads produce all 50 of the files they are to generate.

The main thread does a join on each of the 50 threads sequentially, so that
after the last thread completes, then the program should exit. However, the
observation is that the main thread never exits, presumably since it has
joined to a child thread whose run() method never returns.

Now if I change the test so that a single thread runs the test 50 times
sequentially, all expected files are produced and the program terminates normally.

----
Garry Hodgson, Technology Consultant, AT&T Labs

Be happy for this moment.
This moment is your life.

Jul 18 '05 #1
5 2342
GIL = Global interpreter lock. It shold cause trouble on SMP. As someone
recently mentioned, Jython does not have it, so you have options.
Garry Hodgson wrote:
a colleague of mine has seen an odd problem in some code of ours.
we initially noticed it on webware, but in distilling a test case it seems
to be strictly a python issue. in the real system, it manifests as
webware just locking up, for no apparent reason, until we kill it.
we've also had the python interpreter running webware die on occasion.
it works fine on our desktop linux and windows machines, but fails
on the production hardware. the main difference being that the
production hardware is a dual Xenon machine with
hyperthreading enabled.

has anyone run into problems like this, or have any clues what
the problem might be? we pushed pretty hard on a skeptical
project manager to get python and webware accepted for this
project, and it's embarrassing to have it acting flaky.

i'd really appreciate any insight anyone's got on this. we need
to resolve it quickly. mike's description of the test case follows.

thanks
"Michael W. Balk" wrote:

You asked me yesterday to give you some info on the multi-threading problem
so that you could post a query on comp.lang.python.

Here is what I know so far.

The machine has two Xenon CPUs, both with hyper-threading enabled.
The linux kernel running is: 2.4.20-8smp

The pure python test I ran did the following:

Starts up 50 threads (using the threading module).
Each thread calculates 500 random numbers and writes them to a file,
repeating this 50 times to generate 50 output files. So there are 2500
files expected once all 50 threads complete.

The observation is that only a few threads actually produce output files,
and none of those threads produce all 50 of the files they are to generate.

The main thread does a join on each of the 50 threads sequentially, so that
after the last thread completes, then the program should exit. However, the
observation is that the main thread never exits, presumably since it has
joined to a child thread whose run() method never returns.

Now if I change the test so that a single thread runs the test 50 times
sequentially, all expected files are produced and the program terminates normally.


----
Garry Hodgson, Technology Consultant, AT&T Labs

Be happy for this moment.
This moment is your life.

Jul 18 '05 #2
> GIL = Global interpreter lock. It shold cause trouble on SMP. As someone
recently mentioned, Jython does not have it, so you have options.


Are you sure it is the GIL? I run quite a few threaded applications on
my SMP machine, and have had no isses with the GIL killing it...well,
except for that one wxPython thing. Maybe the GIL is the cause.

- Josiah
Jul 18 '05 #3
In article <20********************@k2.sage.att.com>,
Garry Hodgson <ga***@research.att.com> wrote:

a colleague of mine has seen an odd problem in some code of ours. we
initially noticed it on webware, but in distilling a test case it seems
to be strictly a python issue. in the real system, it manifests as
webware just locking up, for no apparent reason, until we kill it.
we've also had the python interpreter running webware die on occasion.
it works fine on our desktop linux and windows machines, but fails on
the production hardware. the main difference being that the production
hardware is a dual Xenon machine with hyperthreading enabled.


What version of Python? What happens if you enable both CPUs without
hyperthreading? What happens if you only start ten threads (fifty
threads isn't huge, but it's definitely large)? The last time I saw or
heard of significant problems like this was in the Python 1.5.1 days, and
Python 2.x fixed a few more bugs.

Have you applied all relevant OS patches and/or tried upgrading to a
newer kernel? Can you test the production system (or an equivalent) with
Windows? What happens if you run an equivalent test using C code?
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR
Jul 18 '05 #4
aa**@pythoncraft.com (Aahz) wrote:
What version of Python?
2.2.2, on all machines but the windows one, which runs 2.3.
What happens if you enable both CPUs without
hyperthreading?
same thing. it fails on the one machine, works on the rest,
including another (older) dual processor machine running
linux smp kernel.
What happens if you only start ten threads (fifty
threads isn't huge, but it's definitely large)?
same thing. works fine for 1 thread :-).
Have you applied all relevant OS patches and/or tried upgrading to a
newer kernel?
it's looking like this may be the problem, since the machine it fails on
is running the oldest linux kernel of all of them. all are 2.4.20's, but
with different minor release numbers.
Can you test the production system (or an equivalent) with
Windows?
nope. we do have another machine that's identical hardware,
though with slightly newer kernel patches. we intend to test
on that tomorrow, during a maintenance window (that's a
production machine).
What happens if you run an equivalent test using C code?


C? yuk!

----
Garry Hodgson, Technology Consultant, AT&T Labs

Be happy for this moment.
This moment is your life.

Jul 18 '05 #5
In article <20********************@k2.sage.att.com>,
Garry Hodgson <ga***@research.att.com> wrote:
aa**@pythoncraft.com (Aahz) wrote:

What happens if you only start ten threads (fifty
threads isn't huge, but it's definitely large)?


same thing. works fine for 1 thread :-).
Have you applied all relevant OS patches and/or tried upgrading to a
newer kernel?


it's looking like this may be the problem, since the machine it fails on
is running the oldest linux kernel of all of them. all are 2.4.20's, but
with different minor release numbers.


Those two combined make me agree.
Can you test the production system (or an equivalent) with
Windows?


nope. we do have another machine that's identical hardware,
though with slightly newer kernel patches. we intend to test
on that tomorrow, during a maintenance window (that's a
production machine).


One thing I learned during my threading problems, five years ago:
*always* have a QA machine with the same specs as the production machine.
Among other things, I found a bug with MS SQL Server that only showed up
on SMP machines. :-(
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"Argue for your limitations, and sure enough they're yours." --Richard Bach
Jul 18 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

21
by: Dave | last post by:
After following Microsofts admonition to reformat my system before doing a final compilation of my app I got many warnings/errors upon compiling an rtf file created in word. I used the Help...
9
by: Tom | last post by:
A question for gui application programmers. . . I 've got some GUI programs, written in Python/wxPython, and I've got a help button and a help menu item. Also, I've got a compiled file made with...
4
by: Sarir Khamsi | last post by:
Is there a way to get help the way you get it from the Python interpreter (eg, 'help(dir)' gives help on the 'dir' command) in the module cmd.Cmd? I know how to add commands and help text to...
6
by: wukexin | last post by:
Help me, good men. I find mang books that introduce bit "mang header files",they talk too bit,in fact it is my too fool, I don't learn it, I have do a test program, but I have no correct doing...
3
by: Colin J. Williams | last post by:
Python advertises some basic service: C:\Python24>python Python 2.4.1 (#65, Mar 30 2005, 09:13:57) on win32 Type "help", "copyright", "credits" or "license" for more information. >>> With...
7
by: Corepaul | last post by:
Missing Help Files When I enter "recordset" as the keyword and search the Visual Basic Help index, I get many topics of interest in the resulting list. But there isn't any information available...
5
by: Steve | last post by:
I have written a help file (chm) for a DLL and referenced it using Help.ShowHelp My expectation is that a developer using my DLL would be able to access this help file during his development time...
10
by: JonathanOrlev | last post by:
Hello everybody, I wrote this comment in another message of mine, but decided to post it again as a standalone message. I think that Microsoft's Office 2003 help system is horrible, probably...
1
by: trunxnirvana007 | last post by:
'UPGRADE_WARNING: Array has a new behavior. Click for more: 'ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?keyword="9B7D5ADD-D8FE-4819-A36C-6DEDAF088CC7"' 'UPGRADE_WARNING: Couldn't resolve...
0
by: hitencontractor | last post by:
I am working on .NET Version 2003 making an SDI application that calls MS Excel 2003. I added a menu item called "MyApp Help" in the end of the menu bar to show Help-> About. The application...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 2 August 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
by: erikbower65 | last post by:
Using CodiumAI's pr-agent is simple and powerful. Follow these steps: 1. Install CodiumAI CLI: Ensure Node.js is installed, then run 'npm install -g codiumai' in the terminal. 2. Connect to...
0
by: kcodez | last post by:
As a H5 game development enthusiast, I recently wrote a very interesting little game - Toy Claw ((http://claw.kjeek.com/))。Here I will summarize and share the development experience here, and hope it...
0
by: Taofi | last post by:
I try to insert a new record but the error message says the number of query names and destination fields are not the same This are my field names ID, Budgeted, Actual, Status and Differences ...
14
DJRhino1175
by: DJRhino1175 | last post by:
When I run this code I get an error, its Run-time error# 424 Object required...This is my first attempt at doing something like this. I test the entire code and it worked until I added this - If...
0
by: Rina0 | last post by:
I am looking for a Python code to find the longest common subsequence of two strings. I found this blog post that describes the length of longest common subsequence problem and provides a solution in...
5
by: DJRhino | last post by:
Private Sub CboDrawingID_BeforeUpdate(Cancel As Integer) If = 310029923 Or 310030138 Or 310030152 Or 310030346 Or 310030348 Or _ 310030356 Or 310030359 Or 310030362 Or...
0
by: lllomh | last post by:
How does React native implement an English player?
2
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.