The reliability of python threads

In article <ma************ *************** ************@py thon.org>,
"Carl J. Van Arsdall" <cv*********@mv ista.comwrites:
|Hey everyone, I have a question about python threads. Before anyone
|goes further, this is not a debate about threads vs. processes, just a
|question.
|>
|With that, are python threads reliable? Or rather, are they safe? I've
|had some strange errors in the past, I use threading.lock for my
|critical sections, but I wonder if that is really good enough.
|>
|Does anyone have any conclusive evidence that python threads/locks are
|safe or unsafe?

Unsafe. They are built on top of unsafe primitives (POSIX, Microsoft
etc.) Python will shield you from some problems, but not all.

There is precious little that you can do, because the root cause is
that the standards and specifications are hopelessly flawed.
Regards,
Nick Maclaren.

On 24 Jan 2007 17:12:19 GMT, Nick Maclaren <nm**@cus.cam.a c.ukwrote:

>
In article <ma************ *************** ************@py thon.org>,
"Carl J. Van Arsdall" <cv*********@mv ista.comwrites:
|Hey everyone, I have a question about python threads. Before anyone
|goes further, this is not a debate about threads vs. processes, just a
|question.
|>
|With that, are python threads reliable? Or rather, are they safe? I've
|had some strange errors in the past, I use threading.lock for my
|critical sections, but I wonder if that is really good enough.
|>
|Does anyone have any conclusive evidence that python threads/locks are
|safe or unsafe?

Unsafe. They are built on top of unsafe primitives (POSIX, Microsoft
etc.) Python will shield you from some problems, but not all.

There is precious little that you can do, because the root cause is
that the standards and specifications are hopelessly flawed.

This is sufficiently inaccurate that I would call it FUD. Using
threads from Python, as from any other language, requires knowledge of
the tradeoffs and limitations of threading, but claiming that their
usage is *inherently* unsafe isn't true. It is almost certain that
your code and locking are flawed, not that the threads underneath you
are buggy.

In article <ma************ *************** ************@py thon.org>,
"Chris Mellon" <ar*****@gmail. comwrites:
| |>
| |Does anyone have any conclusive evidence that python threads/locks are
| |safe or unsafe?
|
| Unsafe. They are built on top of unsafe primitives (POSIX, Microsoft
| etc.) Python will shield you from some problems, but not all.
|
| There is precious little that you can do, because the root cause is
| that the standards and specifications are hopelessly flawed.
|>
|This is sufficiently inaccurate that I would call it FUD. Using
|threads from Python, as from any other language, requires knowledge of
|the tradeoffs and limitations of threading, but claiming that their
|usage is *inherently* unsafe isn't true. It is almost certain that
|your code and locking are flawed, not that the threads underneath you
|are buggy.

I suggest that you find out rather more about the ill-definition of
POSIX threading memory model, to name one of the better documented
aspects. A Web search should provide you with more information on
the ghastly mess than any sane person wants to know.

And that is only one of many aspects :-(
Regards,
Nick Maclaren.

On 24 Jan 2007 18:21:38 GMT, Nick Maclaren <nm**@cus.cam.a c.ukwrote:

>
In article <ma************ *************** ************@py thon.org>,
"Chris Mellon" <ar*****@gmail. comwrites:
| |>
| |Does anyone have any conclusive evidence that python threads/locks are
| |safe or unsafe?
|
| Unsafe. They are built on top of unsafe primitives (POSIX, Microsoft
| etc.) Python will shield you from some problems, but not all.
|
| There is precious little that you can do, because the root cause is
| that the standards and specifications are hopelessly flawed.
|>
|This is sufficiently inaccurate that I would call it FUD. Using
|threads from Python, as from any other language, requires knowledge of
|the tradeoffs and limitations of threading, but claiming that their
|usage is *inherently* unsafe isn't true. It is almost certain that
|your code and locking are flawed, not that the threads underneath you
|are buggy.

I suggest that you find out rather more about the ill-definition of
POSIX threading memory model, to name one of the better documented
aspects. A Web search should provide you with more information on
the ghastly mess than any sane person wants to know.

And that is only one of many aspects :-(

I'm aware of the issues with the POSIX threading model. I still stand
by my statement - bringing up the problems with the provability of
correctness in the POSIX model amounts to FUD in a discussion of
actual problems with actual code.

Logic and programming errors in user code are far more likely to be
the cause of random errors in a threaded program than theoretical
(I've never come across a case in practice) issues with the POSIX
standard.

Emphasizing this means that people will tend to ignore bugs as being
"the fault of POSIX" rather than either auditing their code more
carefully, or avoiding threads entirely (the second being what I
suspect your goal is).

As a last case, I should point out that while the POSIX memory model
can't be proven safe, concrete implementations do not necessarily
suffer from this problem.

Chris Mellon wrote:

On 24 Jan 2007 18:21:38 GMT, Nick Maclaren <nm**@cus.cam.a c.ukwrote:

>[snip]

I'm aware of the issues with the POSIX threading model. I still stand
by my statement - bringing up the problems with the provability of
correctness in the POSIX model amounts to FUD in a discussion of
actual problems with actual code.

Logic and programming errors in user code are far more likely to be
the cause of random errors in a threaded program than theoretical
(I've never come across a case in practice) issues with the POSIX
standard.

Yea, typically I would think that. The problem I am seeing is
incredibly intermittent. Like a simple pyro server that gives me a
problem maybe every three or four months. Just something funky will
happen to the state of the whole thing, some bad data, i'm having an
issue tracking it down and some more experienced programmers mentioned
that its most likely a race condition. THe thing is, I'm really not
doing anything too crazy, so i'm having difficult tracking it down. I
had heard in the past that there may be issues with threads, so I
thought to investigate this side of things.

It still proves difficult, but reassurance of the threading model helps
me focus my efforts.

Emphasizing this means that people will tend to ignore bugs as being
"the fault of POSIX" rather than either auditing their code more
carefully, or avoiding threads entirely (the second being what I
suspect your goal is).

As a last case, I should point out that while the POSIX memory model
can't be proven safe, concrete implementations do not necessarily
suffer from this problem.

Would you consider the Linux implementation of threads to be concrete?

-carl

--

Carl J. Van Arsdall
cv*********@mvi sta.com
Build and Release
MontaVista Software

In article <ma************ *************** ************@py thon.org>,
"Carl J. Van Arsdall" <cv*********@mv ista.comwrites:
|Chris Mellon wrote:
|
| Logic and programming errors in user code are far more likely to be
| the cause of random errors in a threaded program than theoretical
| (I've never come across a case in practice) issues with the POSIX
| standard.
|
|Yea, typically I would think that. The problem I am seeing is
|incredibly intermittent. Like a simple pyro server that gives me a
|problem maybe every three or four months. Just something funky will
|happen to the state of the whole thing, some bad data, i'm having an
|issue tracking it down and some more experienced programmers mentioned
|that its most likely a race condition. THe thing is, I'm really not
|doing anything too crazy, so i'm having difficult tracking it down. I
|had heard in the past that there may be issues with threads, so I
|thought to investigate this side of things.

I have seen that many dozens of times on half a dozen Unices, but have
only tracked down the cause in a handful of cases. Of those,
implementation defects that are sanctioned by the standards have
accounted for about half.

Note that the term "race condition" is accurate but misleading! One
of the worst problems with POSIX is that it does not define how
non-memory global state is synchronised. For example, it is possible
for a memory update and an associated signal to occur on different
sides of a synchronisation boundary. Similarly, it is possible for
I/O to sidestep POSIX's synchronisation boundaries. I have seen both.

Perhaps the nastiest is that POSIX leaves it unclear whether the
action of synchronisation is transitive. So, if A synchronises with
B, and then B with C, A may not have synchronised with C. Again, I
have seen that. It can happen on Intel systems, according to the
experts I know.

|Would you consider the Linux implementation of threads to be concrete?

In this sort of area, Linux tends to be saner than most systems, but
remember that it has had MUCH less stress testing on threaded codes
than many other Unices. In fact, it was only a few years ago that
Linux threads became stable enough to be worth using.

Note that failures due to implementation defects and flaws in the
standards are likely to show up in very obscure ways; ones due to
programmer error tend to be much simpler.

If you want to contact me by Email, and can describe technically
what you are doing and (most importantly) what you are assuming, I
may be able to give some hints. But no promises.
Regards,
Nick Maclaren.

In article <ma************ *************** ************@py thon.org>,
Carl J. Van Arsdall <cv*********@mv ista.comwrote:

>
Hey everyone, I have a question about python threads. Before anyone
goes further, this is not a debate about threads vs. processes, just a
question.

With that, are python threads reliable? Or rather, are they safe? I've
had some strange errors in the past, I use threading.lock for my
critical sections, but I wonder if that is really good enough.

Does anyone have any conclusive evidence that python threads/locks are
safe or unsafe?

My response is that you're asking the wrong questions here. Our database
server locked up hard Sunday morning, and we still have no idea why (the
machine itself, not just the database app). I think it's more important
to focus on whether you have done all that is reasonable to make your
application reliable -- and then put your efforts into making your app
recoverable.

I'm particularly making this comment in the context of your later point
about the bug showing up only every three or four months.

Side note: without knowing what error messages you're getting, there's
not much anybody can say about your programs or the reliability of
threads for your application.
--
Aahz (aa**@pythoncra ft.com) <* http://www.pythoncraft.com/

Help a hearing-impaired person: http://rule6.info/hearing.html

In article <ep**********@p anix3.panix.com >,
aa**@pythoncraf t.com (Aahz) writes:
|>
|My response is that you're asking the wrong questions here. Our database
|server locked up hard Sunday morning, and we still have no idea why (the
|machine itself, not just the database app). I think it's more important
|to focus on whether you have done all that is reasonable to make your
|application reliable -- and then put your efforts into making your app
|recoverable.

Absolutely! Shit happens. In a well-designed world, that would not be
the case, but we don't live in one. Until you have identified the cause,
you can't tell if threading has anything to do with the failure - given
what we know, it seems likely, but what Aahz says is how to tackle the
problem WHATEVER the cause.
Regards,
Nick Maclaren.

On Jan 24, 6:43 pm, "Carl J. Van Arsdall" <cvanarsd...@mv ista.com>
wrote:

Chris Mellon wrote:

On 24 Jan 2007 18:21:38 GMT, Nick Maclaren <n...@cus.cam.a c.ukwrote:

[snip]

I'm aware of the issues with the POSIX threading model. I still stand
by my statement - bringing up the problems with the provability of
correctness in the POSIX model amounts to FUD in a discussion of
actual problems with actual code.

Logic and programming errors in user code are far more likely to be
the cause of random errors in a threaded program than theoretical
(I've never come across a case in practice) issues with the POSIX
standard.

Yea, typically I would think that. The problem I am seeing is
incredibly intermittent. Like a simple pyro server that gives me a
problem maybe every three or four months. Just something funky will
happen to the state of the whole thing, some bad data, i'm having an
issue tracking it down and some more experienced programmers mentioned
that its most likely a race condition. THe thing is, I'm really not
doing anything too crazy, so i'm having difficult tracking it down. I
had heard in the past that there may be issues with threads, so I
thought to investigate this side of things.

It still proves difficult, but reassurance of the threading model helps
me focus my efforts.

<SNIP>

-carl

Three to four months before `strange errors`? I'd spend some time
correlating logs; not just for your program, but for everything running

on the server. Then I'd expect to cut my losses and arrange to safely
re-start the program every TWO months.
(I'd arrange the re-start after collecting logs but before their
analysis.
Life is too short).

- Paddy.