473,573 Members | 2,945 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Use subprocesses in simple way...

Hi !

Py2.4, Win32.

I need to optimize a program that have a speciality: hash (MD5/SHA1)
the file contents (many files).

Now I do this in a simple python program, because (what a pity) the
FSUM utility died in a file with unicode filename... (It is unknown
error: I used alternate file name, but it not found).
And FSUM have some problematics with it's parameterizing.

So now I want choose the best solution to get hash values:
1.) Creating pyshamd5.exe with py2exe and python, and call this
program with CreateProcess and Params ?
2.) Creating pyshamd5.exe with py2exe and python, and call this
program with popen - use pipes ?
3.) Using external utility ?
4.) Other things ?

So I want to avoid the passive times like "many process starting and
ending", "many file operation", etc.
Need to use unicode filenames, so the outer program can handle these
files (alternate file names ?).
I "send" the filename to the subprocess, and it is send the hash
results (or error codes) to my main process.

Pipe good for this, but PIPE-s are sync. (synchronous) things. For
sync. things I need threads.

If I use subprocess.Pope n object, it is better, because I not use
threads (I can use poll to get return code) - I can manage the
subprocesses from onethreaded code (from main thread).
(I need more process, because it speeding up my hashing. If I have one
ISO, the hashing of this file block all operations in one
thread/process. A multiprocessor code can hash more files in one
time.)

So: I need a solution that not need multithreading chaos, where I can
poll my processes.
Better, If I have persistent suprocesses, because I avoid
starting/ending "neutral gear" of subprocesses.

The working in pseudocode:

while HaveAnyFile or HaveOpenedProce sses:
if not Processes:
CreateProcess(M axProcesses)
if HaveUsableProce sses:
fn=GetNextFile
if fn:
SendToSubProces s(fn)
PollProcessesAn dStoreResults()

The questions
1.)
Do you know a "method" that can send/receive datas to/from presistent
subprocesses without blocking ?
2.)
Do you know command line to just like FSUM that can compute file
hashes (MD5/SHA1), and don't have any problems with unicode alt. file
names ?
3.)
PIPE-s better than CreateProcess(p arams) ?

Important: the sockets are unavailable in this project. I don't use
them, because users forbid to me.

Thanks for any help:
dd

_______________ _______________ _______________ _______________ ______
Python mailing list
Levelcim: Py****@lists.fs n.hu
Webfelulet: https://lists.fsn.hu/mailman/listinfo/python
Web oldalak: http://www.python.hu vagy http://www.python.org

Állományba menteni | Nem biztoságos képek elrejtése

Delete & Prev | Delete & Next
Move to:
May 10 '06 #1
3 2026

Dara Durum wrote:

[snip design of a multi-processor algorithm]

I thought md5 algorithm is pretty light, so you'll be I/O-bound, then
why bother with multi-processor algorithm?
2.)
Do you know command line to just like FSUM that can compute file
hashes (MD5/SHA1), and don't have any problems with unicode alt. file
names ?


I believe you can wrap the broken program with a simple python wrapper.
Use win32api.GetSho rtPathName to convert non-ascii file names to DOS
filenames.

May 10 '06 #2
10 May 2006 04:57:17 -0700, Serge Orlov <Se*********@gm ail.com>:
I thought md5 algorithm is pretty light, so you'll be I/O-bound, then
why bother with multi-processor algorithm?
This is an assessor utility.
The program's architecture must be flexible, because I don't know,
where it need to run (only I have a possibility to fix this: I write
to user's guide).

But I want to speedup my alg. with native code, and multiprocess code.
I not tested yed, but I think that 4 subprocess quickly as one large
process.
2.)
Do you know command line to just like FSUM that can compute file
hashes (MD5/SHA1), and don't have any problems with unicode alt. file
names ?


I believe you can wrap the broken program with a simple python wrapper.
Use win32api.GetSho rtPathName to convert non-ascii file names to DOS
filenames.


I use FindFilesW with 8. (?) parameter. This is the alternative name
of the file, but yet I found a file that not handled by FSUM
utility...

Thanx for help:
dd

Ps:
I wrote some code to test pipes and subprocesses. The name of the mod.
is testpipe.py.
The code is:

import sys,random,subp rocess,time,os, popen2,threadin g,thread

IsMaster=len(sy s.argv)==1

class ProcessThread(t hreading.Thread ):
def __init__(self,P aram):
threading.Threa d.__init__(self )
self.Param=Para m
self.RetVal=Non e
self.start()

def run(self):
param=self.Para m
print "New thread with param",param
po=os.popen2('c :\\python24\\py thon.exe testpipe.py 1')
child_stdin,chi ld_stdout=po
child_stdin.wri te(str(param)+' \n')
retval=child_st dout.readlines( )
child_stdin.clo se()
child_stdout.cl ose()
self.RetVal=ret val

if IsMaster:
print "M:",time.time( )
print "M: Start"
print "M: Open subprocess"
cnt=1
fcnt=0
pths=[]
ress=[None]*9
while True:
if cnt<10:
pt=ProcessThrea d(cnt)
pths.append(pt)
cnt+=1
pcnt=0
for pt in pths:
if pt: pcnt+=1
if pcnt:
for i in range(len(pths) ):
pt=pths[i]
if pt and pt.RetVal:
pths[i]=None
ress[i]=pt.RetVal
print [pt.RetVal]

else:
break
print "\M: The results are:"
for s in ress:
print s
print "M: End"
else:
print "S:",time.time( )
print "S: Start"
print "S: Data"
print "S: End"
s=sys.stdin.rea dline()
print "S: %s"%s
time.sleep(1)
print "Echo: %s"%s
print "Finished"

May 11 '06 #3
DurumDara wrote:
10 May 2006 04:57:17 -0700, Serge Orlov <Se*********@gm ail.com>:
I thought md5 algorithm is pretty light, so you'll be I/O-bound, then
why bother with multi-processor algorithm?


This is an assessor utility.
The program's architecture must be flexible, because I don't know,
where it need to run (only I have a possibility to fix this: I write
to user's guide).

But I want to speedup my alg. with native code, and multiprocess code.
I not tested yed, but I think that 4 subprocess quickly as one large
process.


I believe you need to look at Queue module. Using Queue will help you
avoid threading hell that you're afraid of (and rightly so!). Create
two queues: one for jobs, another one for results, the main thread
submits jobs and picks up results for results queue. As soon as number
of results == number of jobs, it's time to quit. Submit N special jobs
that indicate it's time to exit, where N is the number of worker
threads. Then "join" the main thread with worker threads and exit the
application.

May 11 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
6070
by: delisonews | last post by:
I'm looking for a simple, filesystem-based message board. (No MySQL!) Something that I could include easily in my code: include '../inc/messageboard.php'; .... so that the board shows up at the bottom of every PHP page. The board should have just the basic features, like: - posting capability
4
3359
by: Jane Austine | last post by:
Running Python 2.3 on Win XP It seems like socket is working interdependently with subprocesses of the process which created socket. ------------------------------------ #the server side >>> import socket >>> s=socket.socket(socket.AF_INET,socket.SOCK_STREAM) >>> s.bind(('localhost',9000))
2
2427
by: Marcos | last post by:
Hi guys, I realise this question has been answered in one form or another many times before but I can't quite find the solution I need. I am trying to run multiple subprocesses from a python script and then wait until all subprocesses have completed before continuing. The subprocesses run on other machines ie this is a coarse grained parallel...
8
6487
by: Dan | last post by:
Using XML::Simple in perl is extreemly slow to parse big XML files (can be up to 250M, taking ~1h). How can I increase my performance / reduce my memory usage? Is SAX the way forward?
2
2633
by: Dave Kirby | last post by:
I am working on a network management program written in python that has multiple threads (typically 20+) spawning subprocesses which are used to communicate with other systems on the network. This runs fine for a while, but eventually slows down to a crawl. Running sar shows that when it is running slowly there is an exceptionally large...
7
2273
by: abcd | last post by:
I am trying to set up client machine and investigatging which .net components are missing to run aspx page. I have a simple aspx page which just has "hello world" printed.... When I request that page like http://machinename/dir1/hellp.aspx instead of running that page it starts downloding ...whats missing here ....why the aspx engine...
1
1901
by: Gal Diskin | last post by:
Hi all, I'm writing a python program using threads to open several subprocesses concurrently (using module subprocess) and wait on them. I was wondering if there is a possibilty that a thread will return from wait even though the subprocess that finished was created by another thread thats also waiting. i.e. - 2 threads, each thread opens...
14
2966
by: Giancarlo Berenz | last post by:
Hi: Recently i write this code: class Simple { private: int value; public: int GiveMeARandom(void);
10
2119
by: Phillip Taylor | last post by:
Hi guys, I'm looking to develop a simple web service in VB.NET but I'm having some trivial issues. In Visual Studio I create a web services project and change the asmx.vb file to this: Imports System.Web.Services Imports System.Web.Services.Protocols Imports System.ComponentModel <System.Web.Services.WebService(Namespace:="http://...
0
7686
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8009
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8198
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7771
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8060
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5580
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5289
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
2194
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1296
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.