Windows service and pyc files

[Laszlo Nagy]

Hello,

I have a win32 service written in Python that starts a plain
application, written in Python.

The win32 service tries to launch the application in a while loop and
logs the return value of the os.system call. That's all.

The application is a simple Python program that connects to an https
xml/rpc server, and works with the data retrieved from that server. It
is written as an application because it must be able to download updates
for itself. Here is how it works:

a.) connect to server
b.) compare current version with the latest
c.) if there is a difference, then download all sources from the server,
delete all pyc files and exit; otherwise start processing

I could not write a self-restarting server, and I did not want to
write/install two services for one problem. The win32 service is very
small and primitive so probably I will not need to update it. I think
the basic idea is good, but...

When there is a client update available, my application updates itself
cleanly and exists. Then the service tries to restart the application,
but it cannot. os.system returns with OS error code -1. The pyc files
are NOT generated for the application. However, if I start the
application from the command line, then pyc files are created, and then
the service will also start the application immediatelly. The win32
service is running as "Localsyste m" so it is sure that it has write
permission on all files.

I cannot log out the error from the application since it is not started.
The only error message I have is OSError -1, but it tells me nothing
about the nature of the error.

Thanks,

Laszlo

[kyosohma]

On Apr 4, 10:48 am, Laszlo Nagy <gand...@design aproduct.bizwro te:

Hello,

I have a win32 service written in Python that starts a plain
application, written in Python.

The win32 service tries to launch the application in a while loop and
logs the return value of the os.system call. That's all.

The application is a simple Python program that connects to an https
xml/rpc server, and works with the data retrieved from that server. It
is written as an application because it must be able to download updates
for itself. Here is how it works:

a.) connect to server
b.) compare current version with the latest
c.) if there is a difference, then download all sources from the server,
delete all pyc files and exit; otherwise start processing

I could not write a self-restarting server, and I did not want to
write/install two services for one problem. The win32 service is very
small and primitive so probably I will not need to update it. I think
the basic idea is good, but...

When there is a client update available, my application updates itself
cleanly and exists. Then the service tries to restart the application,
but it cannot. os.system returns with OS error code -1. The pyc files
are NOT generated for the application. However, if I start the
application from the command line, then pyc files are created, and then
the service will also start the application immediatelly. The win32
service is running as "Localsyste m" so it is sure that it has write
permission on all files.

I cannot log out the error from the application since it is not started.
The only error message I have is OSError -1, but it tells me nothing
about the nature of the error.

Thanks,

Laszlo

Have you tried the subprocess module for this rather than os.system?
You might be able to pipe errors to a file with it. You might be able
to use the traceback module for more verbose error catching too.

Mike

[Laszlo Nagy]

>
Have you tried the subprocess module for this rather than os.system?
You might be able to pipe errors to a file with it. You might be able
to use the traceback module for more verbose error catching too.

Okay, I'm tried this instead of os.system:

def dumpexc(e):
import sys,traceback,S tringIO
f = StringIO.String IO('')
ei = sys.exc_info()
traceback.print _exception(ei[0],ei[1],ei[2],file=f)
return f.getvalue()

def spawn():
# ... setup mydir here....
os.chdir(mydir)
prog = os.path.join(my dir,"Applicatio n.py")
params = [sys.executable, prog]
logger.info("Sp awing %s",str(params) )
fout = file(os.path.jo in(mydir,'error log.txt'),'wb+' )
try:
p = subprocess.Pope n(params, bufsize=1, stdout=fout.fil eno(),
stderr=fout.fil eno())
except Exception, e:
logger.error(du mpexc(e))
return -1
retcode = p.wait()
logger.info("Su bprocess exited, return code: %d",retcode)
fout.close()
return retcode

When I call spawn() from a service, this is written into the logfile:

2007-04-05 17:52:53,828 INFO .Spawner Spawing
['C:\\Python25\\ lib\\site-packages\\win32 \\PythonService .exe',
'T:\\Python\\Pr ojects\\Test\\A pplication.py']
2007-04-05 17:52:53,828 ERROR .Spawner Traceback (most recent call last):
File "T:\Python\Proj ects\Test\Proce ssor.py", line 40, in spawn_downloade r
p = subprocess.Pope n(params, bufsize=1, stdout=fout.fil eno(),
stderr=fout.fil eno())
File "C:\Python25\li b\subprocess.py ", line 586, in __init__
errread, errwrite) = self._get_handl es(stdin, stdout, stderr)
File "C:\Python25\li b\subprocess.py ", line 681, in _get_handles
p2cread = self._make_inhe ritable(p2cread )
File "C:\Python25\li b\subprocess.py ", line 722, in _make_inheritab le
DUPLICATE_SAME_ ACCESS)
TypeError: an integer is required
errorlog.txt is - of course - becomes an empty file.

When I call spawn() from an application, it works just fine. Any ideas?

Thanks,

Laszlo

[Laszlo Nagy]

When I call spawn() from a service, this is written into the logfile:

2007-04-05 17:52:53,828 INFO .Spawner Spawing
['C:\\Python25\\ lib\\site-packages\\win32 \\PythonService .exe',
'T:\\Python\\Pr ojects\\Test\\A pplication.py']
2007-04-05 17:52:53,828 ERROR .Spawner Traceback (most recent call last):
File "T:\Python\Proj ects\Test\Proce ssor.py", line 40, in spawn_downloade r
p = subprocess.Pope n(params, bufsize=1, stdout=fout.fil eno(),
stderr=fout.fil eno())
File "C:\Python25\li b\subprocess.py ", line 586, in __init__
errread, errwrite) = self._get_handl es(stdin, stdout, stderr)
File "C:\Python25\li b\subprocess.py ", line 681, in _get_handles
p2cread = self._make_inhe ritable(p2cread )
File "C:\Python25\li b\subprocess.py ", line 722, in _make_inheritab le
DUPLICATE_SAME_ ACCESS)
TypeError: an integer is required

Okay, here is what I learnt:

1. subprocess.Pope n cannot redirect stderr and stdout when called from a
win32 service. This is not documented, and makes debugging almost
impossible.
2. sys.executable becomes "pythonservice. exe" inside a win32 service.

If I specify r"C:\Python25\p ython.exe" instead of sys.executable, and
if I do not specify stdout and stderr parameters for subprocess.Pope n,
then my program starts to work. Here arises the question: how can I
find r"C:\Python25\p ython.exe" from inside a win32 service? Can I use this:

interpreter = os.path.join( os.path.split(s ys.executable),[0],
os.sep,os.sep,o s.sep,'Python.e xe' )

Is it safe? Please advise.
Thanks,

Laszlo

[Gabriel Genellina]

En Thu, 05 Apr 2007 13:00:52 -0300, Laszlo Nagy
<ga*****@design aproduct.bizesc ribió:

p = subprocess.Pope n(params, bufsize=1, stdout=fout.fil eno(),
stderr=fout.fil eno())

When I call spawn() from a service, this is written into the logfile:

2007-04-05 17:52:53,828 INFO .Spawner Spawing
['C:\\Python25\\ lib\\site-packages\\win32 \\PythonService .exe',
'T:\\Python\\Pr ojects\\Test\\A pplication.py']
2007-04-05 17:52:53,828 ERROR .Spawner Traceback (most recent call last):
File "T:\Python\Proj ects\Test\Proce ssor.py", line 40, in spawn_downloade r
p = subprocess.Pope n(params, bufsize=1, stdout=fout.fil eno(),
stderr=fout.fil eno())
File "C:\Python25\li b\subprocess.py ", line 586, in __init__
errread, errwrite) = self._get_handl es(stdin, stdout, stderr)
File "C:\Python25\li b\subprocess.py ", line 681, in _get_handles
p2cread = self._make_inhe ritable(p2cread )
File "C:\Python25\li b\subprocess.py ", line 722, in _make_inheritab le
DUPLICATE_SAME_ ACCESS)
TypeError: an integer is required

With a bit of guessing, I think I've found what's happening.
Since you don't provide a value for stdin, None is used. Inside
subprocess.py, method _get_handles, line 670, GetStdHandle *may* return
None; in that case _make_inheritab le fails.
If you print the value of p2cread in line 670 I bet you'll get None.
The fix is to test for None in _make_inheritab le (line 720):

if handle is not None:
return DuplicateHandle (...)

(else return None, implicit)

When I call spawn() from an application, it works just fine. Any ideas?

According to http://msdn2.microsoft.com/en-us/library/ms683231.aspx
GetStdHandle may return NULL (translated to None in Python) when invoked
from a service with no redirected standard handles. From an application,
there is no problem.

Please try the simple fix above to comfirm it works; I'll submit a patch
if that's the case.

--
Gabriel Genellina

[Gabriel Genellina]

En Thu, 05 Apr 2007 13:50:16 -0300, Laszlo Nagy
<ga*****@design aproduct.bizesc ribió:

1. subprocess.Pope n cannot redirect stderr and stdout when called from a
win32 service. This is not documented, and makes debugging almost
impossible.

Without the patch menctioned in my previous message, you must redirect all
stdin, stdout AND stderr (because the child cannot inherit the handles
from the parent service, as a service has no standard handles assigned
usually) or none of them.

2. sys.executable becomes "pythonservice. exe" inside a win32 service.

If I specify r"C:\Python25\p ython.exe" instead of sys.executable, and
if I do not specify stdout and stderr parameters for subprocess.Pope n,
then my program starts to work. Here arises the question: how can I
find r"C:\Python25\p ython.exe" from inside a win32 service? Can I use
this:

interpreter = os.path.join( os.path.split(s ys.executable),[0],
os.sep,os.sep,o s.sep,'Python.e xe' )

I think you meant to write: os.path.join(os .path.split(sys .executable)[0],
os.pardir, os.pardir, os.pardir, 'python.exe')

pythonservice.e xe is so Windows-specific that using os.pardir only makes
the code harder to read.
Anyway I'd use os.path.join(sy s.prefix, 'python.exe') (sys.prefix would be
C:\Python25 in your case)

--
Gabriel Genellina

[Laszlo Nagy]

>1. subprocess.Pope n cannot redirect stderr and stdout when called from a
win32 service. This is not documented, and makes debugging almost
impossible.

Without the patch menctioned in my previous message, you must redirect all
stdin, stdout AND stderr (because the child cannot inherit the handles
from the parent service, as a service has no standard handles assigned
usually) or none of them.

The truth is that subprocess.Pope n raises an exception when I try to
redirect stdin,stdout or stderr. If I do not redirect any of them, then
everything works fine for me. (Well, of course standard output is lost.)

>interpreter = os.path.join( os.path.split(s ys.executable),[0],
os.sep,os.sep, os.sep,'Python. exe' )

I think you meant to write: os.path.join(os .path.split(sys .executable)[0],
os.pardir, os.pardir, os.pardir, 'python.exe')

Yes, sorry.

pythonservice.e xe is so Windows-specific that using os.pardir only makes
the code harder to read.
Anyway I'd use os.path.join(sy s.prefix, 'python.exe') (sys.prefix would be
C:\Python25 in your case)

Oh, I did not know about that variable. Thanks! :-)

Best,

Laszlo

[Laszlo Nagy]

With a bit of guessing, I think I've found what's happening.
Since you don't provide a value for stdin, None is used. Inside
subprocess.py, method _get_handles, line 670, GetStdHandle *may* return
None; in that case _make_inheritab le fails.
If you print the value of p2cread in line 670 I bet you'll get None.
The fix is to test for None in _make_inheritab le (line 720):

if handle is not None:
return DuplicateHandle (...)

(else return None, implicit)

>When I call spawn() from an application, it works just fine. Any ideas?

According to http://msdn2.microsoft.com/en-us/library/ms683231.aspx
GetStdHandle may return NULL (translated to None in Python) when invoked
from a service with no redirected standard handles. From an application,
there is no problem.

Please try the simple fix above to comfirm it works; I'll submit a patch
if that's the case.

Yes, it fixed the problem. Please submit the patch. Thank you!

Laszlo