Python reliability

[Ville Voipio]

I would need to make some high-reliability software
running on Linux in an embedded system. Performance
(or lack of it) is not an issue, reliability is.

The piece of software is rather simple, probably a
few hundred lines of code in Python. There is a need
to interact with network using the socket module,
and then probably a need to do something hardware-
related which will get its own driver written in
C.

Threading and other more error-prone techniques can
be left aside, everything can run in one thread with
a poll loop.

The software should be running continously for
practically forever (at least a year without a reboot).
Is the Python interpreter (on Linux) stable and
leak-free enough to achieve this?

- Ville

--
Ville Voipio, Dr.Tech., M.Sc. (EE)

[Paul Rubin]

Ville Voipio <vv*****@kosh.h ut.fi> writes:

The software should be running continously for
practically forever (at least a year without a reboot).
Is the Python interpreter (on Linux) stable and
leak-free enough to achieve this?

I would say give the app the heaviest stress testing that you can
before deploying it, checking carefully for leaks and crashes. I'd
say that regardless of the implementation language.

[Steven D'Aprano]

On Sun, 09 Oct 2005 23:00:04 +0300, Ville Voipio wrote:

I would need to make some high-reliability software
running on Linux in an embedded system. Performance
(or lack of it) is not an issue, reliability is.
[snip]
The software should be running continously for
practically forever (at least a year without a reboot).
Is the Python interpreter (on Linux) stable and
leak-free enough to achieve this?

If performance is really not such an issue, would it really matter if you
periodically restarted Python? Starting Python takes a tiny amount of time:

$ time python -c pass
real 0m0.164s
user 0m0.021s
sys 0m0.015s

If performance isn't an issue, your users may not even care about ten
times that delay even once an hour. In other words, built your software to
deal gracefully with restarts, and your users won't even notice or care if
it restarts.

I'm not saying that you will need to restart Python once an hour, or even
once a month. But if you did, would it matter? What's more important is
the state of the operating system. (I'm assuming that, with a year uptime
the requirements, you aren't even thinking of WinCE.)
--
Steven.

[Paul Rubin]

Steven D'Aprano <st***@REMOVETH IScyber.com.au> writes:

If performance is really not such an issue, would it really matter if you
periodically restarted Python? Starting Python takes a tiny amount of time:

If you have to restart an application, every network peer connected to
it loses its connection. Think of a phone switch. Do you really want
your calls dropped every few hours of conversation time, just because
some lame application decided to restart itself? Phone switches go to
great lengths to keep running through both hardware failures and
software upgrades, without dropping any calls. That's the kind of
application it sounds like the OP is trying to run.

To the OP: besides Python you might also consider Erlang.

[George Sakkis]

Steven D'Aprano wrote:

On Sun, 09 Oct 2005 23:00:04 +0300, Ville Voipio wrote:

I would need to make some high-reliability software
running on Linux in an embedded system. Performance
(or lack of it) is not an issue, reliability is.

[snip]

The software should be running continously for
practically forever (at least a year without a reboot).
Is the Python interpreter (on Linux) stable and
leak-free enough to achieve this?

If performance is really not such an issue, would it really matter if you
periodically restarted Python? Starting Python takes a tiny amount of time:

You must have missed or misinterpreted the "The software should be
running continously for practically forever" part. The problem of
restarting python is not the 200 msec lost but putting at stake
reliability (e.g. for health monitoring devices, avionics, nuclear
reactor controllers, etc.) and robustness (e.g. a computation that
takes weeks of cpu time to complete is interrupted without the
possibility to restart from the point it stopped).

George

[Neal Norwitz]

Ville Voipio wrote:

The software should be running continously for
practically forever (at least a year without a reboot).
Is the Python interpreter (on Linux) stable and
leak-free enough to achieve this?

Jp gave you the answer that he has done this.

I've spent quite a bit of time since 2.1 days trying to improve the
reliability. I think it has gotten much better. Valgrind is run on
(nearly) every release. We look for various kinds of problems. I try
to review C code for these sorts of problems etc.

There are very few known issues that can crash the interpreter. I
don't know of any memory leaks. socket code is pretty well tested and
heavily used, so you should be in fairly safe territory, particularly
on Unix.

n

[Steven D'Aprano]

George Sakkis wrote:

Steven D'Aprano wrote:

On Sun, 09 Oct 2005 23:00:04 +0300, Ville Voipio wrote:

I would need to make some high-reliability software
running on Linux in an embedded system. Performance
(or lack of it) is not an issue, reliability is.

[snip]

The software should be running continously for
practicall y forever (at least a year without a reboot).
Is the Python interpreter (on Linux) stable and
leak-free enough to achieve this?

If performance is really not such an issue, would it really matter if you
periodicall y restarted Python? Starting Python takes a tiny amount of time:

You must have missed or misinterpreted the "The software should be
running continously for practically forever" part. The problem of
restarting python is not the 200 msec lost but putting at stake
reliability (e.g. for health monitoring devices, avionics, nuclear
reactor controllers, etc.) and robustness (e.g. a computation that
takes weeks of cpu time to complete is interrupted without the
possibility to restart from the point it stopped).

Er, no, I didn't miss that at all. I did miss that it
needed continual network connections. I don't know if
there is a way around that issue, although mobile
phones move in and out of network areas, swapping
connections when and as needed.

But as for reliability, well, tell that to Buzz Aldrin
and Neil Armstrong. The Apollo 11 moon lander rebooted
multiple times on the way down to the surface. It was
designed to recover gracefully when rebooting unexpectedly:

http://www.hq.nasa.gov/office/pao/Hi...1.1201-pa.html

I don't have an authoritive source of how many times
the computer rebooted during the landing, but it was
measured in the dozens. Calculations were performed in
an iterative fashion, with an initial estimate that was
improved over time. If a calculation was interupted the
computer lost no more than one iteration.

I'm not saying that this strategy is practical or
useful for the original poster, but it *might* be. In a
noisy environment, it pays to design a system that can
recover transparently from a lost connection.

If your heart monitor can reboot in 200 ms, you might
miss one or two beats, but so long as you pick up the
next one, that's just noise. If your calculation takes
more than a day of CPU time to complete, you should
design it in such a way that you can save state and
pick it up again when you are ready. You never know
when the cleaner will accidently unplug the computer...
--
Steven.

[Ville Voipio]

In article <7x************ @ruckus.brouhah a.com>, Paul Rubin wrote:

I would say give the app the heaviest stress testing that you can
before deploying it, checking carefully for leaks and crashes. I'd
say that regardless of the implementation language.

Goes without saying. But I would like to be confident (or as
confident as possible) that all bugs are mine. If I use plain
C, I think this is the case. Of course, bad memory management
in the underlying platform will wreak havoc. I am planning to
use Linux 2.4.somethingne w as the OS kernel, and there I have
not experienced too many problems before.

Adding the Python interpreter adds one layer on uncertainty.
On the other hand, I am after the simplicity of programming
offered by Python.

- Ville

--
Ville Voipio, Dr.Tech., M.Sc. (EE)

[Ville Voipio]

In article <pa************ *************** *@REMOVETHIScyb er.com.au>,
Steven D'Aprano wrote:

If performance is really not such an issue, would it really matter if you
periodically restarted Python? Starting Python takes a tiny amount of time:
Uhhh. Sounds like playing with Microsoft :) I know of a mission-
critical system which was restarted every week due to some memory
leaks. If it wasn't, it crashed after two weeks. Guess which
platform...
$ time python -c pass
real 0m0.164s
user 0m0.021s
sys 0m0.015s
This is on the limit of being acceptable. I'd say that a one-second
time lag is the maximum. The system is a safety system after all,
and there will be a hardware watchdog to take care of odd crashes.
The software itself is stateless in the sense that its previous
state does not affect the next round. Basically, it is just checking
a few numbers over the network. Even the network connection is
stateless (single UDP packet pairs) to avoid TCP problems with
partial closings, etc.

There are a gazillion things which may go wrong. A stray cosmic
ray may change the state of one bit in the wrong place of memory,
and that's it, etc. So, the system has to be able to recover from
pretty much everything. I will in any case build an independent
process which probes the state of the main process. However,
I hope it is never really needed.
I'm not saying that you will need to restart Python once an hour, or even
once a month. But if you did, would it matter? What's more important is
the state of the operating system. (I'm assuming that, with a year uptime
the requirements, you aren't even thinking of WinCE.)

Not even in my worst nightmares! The platform will be an embedded
Linux computer running 2.4.somethingne w.

- Ville

--
Ville Voipio, Dr.Tech., M.Sc. (EE)

[Paul Rubin]

Ville Voipio <vv*****@kosh.h ut.fi> writes:

Goes without saying. But I would like to be confident (or as
confident as possible) that all bugs are mine. If I use plain
C, I think this is the case. Of course, bad memory management
in the underlying platform will wreak havoc. I am planning to
use Linux 2.4.somethingne w as the OS kernel, and there I have
not experienced too many problems before.

You might be better off with a 2.6 series kernel. If you use Python
conservatively (be careful with the most advanced features, and don't
stress anything too hard) you should be ok. Python works pretty well
if you use it the way the implementers expected you to. Its
shortcomings are when you try to press it to its limits.

You do want reliable hardware with ECC and all that, maybe with multiple
servers and automatic failover. This site might be of interest:

http://www.linux-ha.org/