Duplex communication with pipes - is possible ?

[Dara Durum]

Hi !

I want to create a Process Pool Object.
I can hold started processes, and can communicate with them.

I tryed with many ipc methods, but every of them have bug or other problem.
Sockets are unavailabe (because Windows Firewall hold them).

I think I will use pipe.

The object's pseudocode:
while not Quit:
CheckProcessOut puts;
ProcessReceived Data;
SendDataToSubPr ocesses;
if NoMoreData: Quit=1

If I used pipes and subprocess module in Windows, I got big freezes,
and deadlocks.

Main proc:
subprocpipe.wri te('aaaa\n')
subprocpipe.rea dlines()

Sub proc:
input=sys.stdin .readlines().st rip()
print input+'!!!'

It is working. But when I move this client code to cycle, I got deadlock.
while not Quit:
input=sys.stdin .readlines().st rip()
print input+'!!!'
Quit=input=='q'

Why ? Why I cannot create cyclic communication with client ?
Subprocess must "staying alive" (don't die), and need to stay in
reuseable state ?

Simply: subprocess pool needed !!!

Thanks for help:
dd

[Daniel Dittmar]

Dara Durum wrote:

Hi !

I want to create a Process Pool Object.
I can hold started processes, and can communicate with them.

I tryed with many ipc methods, but every of them have bug or other problem.
Sockets are unavailabe (because Windows Firewall hold them).

I think I will use pipe.

The object's pseudocode:
while not Quit:
CheckProcessOut puts;
ProcessReceived Data;
SendDataToSubPr ocesses;
if NoMoreData: Quit=1

If I used pipes and subprocess module in Windows, I got big freezes,
and deadlocks.

Main proc:
subprocpipe.wri te('aaaa\n')
subprocpipe.rea dlines()

Sub proc:
input=sys.stdin .readlines().st rip()
print input+'!!!'

It is working. But when I move this client code to cycle, I got deadlock.
while not Quit:
input=sys.stdin .readlines().st rip()
print input+'!!!'
Quit=input=='q'

Why ? Why I cannot create cyclic communication with client ?
Subprocess must "staying alive" (don't die), and need to stay in
reuseable state ?

Simply: subprocess pool needed !!!

Thanks for help:
dd

readlines () will try to read until the stream/socket is closed. Try to
read only one line. This of course means that you cannot sent \n as part
of the data, you have to escape them somehow.

I'm not sure how the pipe code is searching for the \n. Trying to read
too much could lead to deadlocks as well. (Although I'm sure that the
code is written to return fewerbytes than requested if there isn't
enough data pending in the pipe). A safer variant might be to transfer a
fixed number of bytes containing the length n of the following data, and
then n bytes containing the actual data.

Daniel

[alisonken1]

<snip

readlines () will try to read until the stream/socket is closed. Try to
read only one line. This of course means that you cannot sent \n as part
of the data, you have to escape them somehow.

<snip>

If I remember correctly, if you want to pass '\n' so readline won't
stop, you should be able to escape the escape '\\n', then remove the
extra '\' when actually processing.

Of course, readline will continue to read until buffer filled or a real
'\n' is passed (g).

[Dara Durum]

Hi !

Ahhh !

It's working !

This is the simple client code:
if 'C' in sys.argv:
#sys.stdout=ope n(r'c:\tpp2clie nt.log','w')
print "clt start"
while 1:
head=sys.stdin. read(4)
print "clt head",[head]
if head.lower()==' quit':
break
dsize=int(head)
if dsize:
data=sys.stdin. read(dsize)
print "clt get data"#,[data]
print "clt end\n"
And the master:
else:
print "MS"
p=subprocess.Po pen([r'c:\python24\p ython.exe','tpp 2.py','C'], \
stdin=subproces s.PIPE,stdout=s ubprocess.PIPE)
print "MSS"
(child_stdout, child_stdin) = (p.stdout, p.stdin)
data=range(1000 )
import cPickle
bdata=cPickle.d umps(data,1)
print len(bdata)
import binascii
hdata=binascii. hexlify(bdata)
print len(hdata)
import base64
hdata=base64.en codestring(bdat a)
print len(hdata)
child_stdin.wri te('%04d'%len(h data))
child_stdin.wri te(hdata)
child_stdin.wri te('quit')
#child_stdin.wr ite('quit')
#child_stdin.wr ite('quit\n')
output=child_st dout.readlines( )
print [output]
print "MEE"

Now I trying with packet size decreasing. Are PIPE-s can handle the
binary data packets, or I need to convert them with base64 ?

Thanks for your help:
dd

[Daniel Dittmar]

Dara Durum wrote:

Now I trying with packet size decreasing. Are PIPE-s can handle the
binary data packets, or I need to convert them with base64 ?

In the client, you need to set the mode of sys.stdin to binary,
otherwise, you get the DOS translation of linefeeds. See
http://mail.python.org/pipermail/pyt...ry/020463.html
for some hints.

I assume that the stream returned by subprocess.pope n are also opened in
text mode and you have to change them tobinary, see the second hint in
the link mentioned above.

Daniel

[Dara Durum]

Hi !

See this shortened, simplified example. It is not working, but I don't
understand why...

# Client Process

import os, sys
from subprocess import Popen, PIPE
from time import sleep, time
from cPickle import loads, dumps
from binascii import hexlify, unhexlify
from base64 import encodestring, decodestring
from time import time

def WriteData(WStm, Data):
data=dumps(Data ,1)
bdata=hexlify(d ata)
msg='%s#'%bdata
WStm.write(msg)

def ReadData(RStm):
tmpl=[]
while 1:
c=RStm.read(1)
if c=='#':
break
tmpl.append(c)
bdata=''.join(t mpl)
data=unhexlify( bdata)
orgdata=loads(d ata)
return orgdata

def SubProcessFunct ions():
f=open('spp_clt .log','w')
print >>f,"m1"
while 1:
print >>f,"m2"
data=ReadData(s ys.stdin)
print >>f,"m3",[data]
if data=='quit':
print >>f,"m4"
WriteData(sys.s tdout,'')
break
print >>f,"m5"
WriteData(sys.s tdout,'>>%s<<'%[data])
print >>f,"m6"

if __name__=='__ma in__':
SubProcessFunct ions()

# The Master process
from spp_clt import ReadData, WriteData
from subprocess import Popen, PIPE
import time

def MasterProcess() :
print "m1"
p=Popen([r'c:\python24\p ython.exe','spp _clt.py'], \
stdin=PIPE,stdo ut=PIPE)
print "m2"
(child_stdout, child_stdin) = (p.stdout, p.stdin)
print "m3"
for s in range(2):
print "m4"
WriteData(child _stdin,s)
print "m5"
ReadData(child_ stdout)
print "m6"
print "m7"
WriteData(child _stdin,'quit')
print "m8"
ReadData(child_ stdout)
print "m9"

MasterProcess()

It is freezed, because I got deadlock. Every process got "Read" state,
and never get back.
What I do wrong ?
I was trying with this packet managing mode:
def WriteData(WStm, Data):
data=dumps(Data ,1)
bdata=encodestr ing(data)
dlen=len(bdata)
msg='%12d%s'%(d len,bdata)
if WStm==None:
print msg
else:
WStm.write(msg)

def ReadData(RStm):
dlen=int(RStm.r ead(12))
bdata=RStm.read (dlen)
data=decodestri ng(bdata)
orgdata=loads(d ata)
return orgdata

but it also freezed.
Why the master doesn't got the packets are sended by client ?

Thanks for your help:
dd


2006/6/21, Dara Durum <du*******@gmai l.com>:

Hi !
Sorry, but I need "multios" application...

This version for text mode is working:

import subprocess
import os,sys

if 'C' in sys.argv:
#sys.stdout=ope n(r'c:\tpp2clie nt.log','w')
print "clt start"
while 1:
head=sys.stdin. read(4)
print "clt head",[head]
if head.lower()==' quit':
break
dsize=int(head)
if dsize:
data=sys.stdin. read(dsize)
print "clt get data",len(data)
print "clt end\n"
else:
print "MS"
p=subprocess.Po pen([r'c:\python24\p ython.exe','tpp 2.py','C'], \
stdin=subproces s.PIPE,stdout=s ubprocess.PIPE)
print "MSS"
(child_stdout, child_stdin) = (p.stdout, p.stdin)
data=range(1000 )
import cPickle
bdata=cPickle.d umps(data,1)
print len(bdata)
import binascii
hdata=binascii. hexlify(bdata)
print len(hdata)
import base64
hdata=base64.en codestring(bdat a)
print len(hdata)
child_stdin.wri te('%04d'%len(h data))
child_stdin.wri te(hdata)
child_stdin.wri te('quit')
output=child_st dout.readlines( )
print "Client's answer:\n",'<'* 80
for s in output:
print s.strip()
print '>'*80
print "MEE"

I will see your idea: the bittorrent...

Thanx:
dd

2006/6/20, Daniel Dittmar <da************ @sap.corp>:

Dara Durum wrote:

Now I trying with packet size decreasing. Are PIPE-s can handle the
binary data packets, or I need to convert them with base64 ?

In the client, you need to set the mode of sys.stdin to binary,
otherwise, you get the DOS translation of linefeeds. See
http://mail.python.org/pipermail/pyt...ry/020463.html
for some hints.

I assume that the stream returned by subprocess.pope n are also opened in
text mode and you have to change them tobinary, see the second hint in
the link mentioned above.

Daniel
--
http://mail.python.org/mailman/listinfo/python-list