determine if os.system() is done

[Xah Lee]

suppose i'm calling two system processes, one to unzip, and one to
“tail� to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
xa*@xahlee.org
∑ http://xahlee.org/

[Alessandro Bottoni]

Xah Lee wrote:

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Well, if you use a function of the "popen" family, you get some kind of
return value from the subprocess on your "output" pipe. You should be able
to determine if your subprocess has terminated by examining (parsing) this
output pipe.

If you use os.system(), you should get a single return value (usually "None"
or a error code) that you can use for this task.

In both cases, you may have to use a loop (sleep(x)) to wait for the return
value (and check it) from within your code.

Have a look at the docu of the os.process module for details. (Maybe the
newer "subprocess" module is a better choice...)

HTH

-----------------------------------
Alessandro Bottoni

[Thomas Bellman]

"Xah Lee" <xa*@xahlee.org> writes:

suppose i'm calling two system processes, one to unzip, and one to
tail to get the last line. How can i determine when the first
process is done? Example: subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]); last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0] of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Have you tried reading the manual for the subprocess module? You
just *might* find the answer to your question if you look at what
you can do with Popen objects. Actually, just learning about the
exact semantics of the communicate() method might be enought to
solve your problem.
--
Thomas Bellman, Lysator Computer Club, Linköping University, Sweden
"I refuse to have a battle of wits with an ! bellman @ lysator.liu.se
unarmed person." ! Make Love -- Nicht Wahr!

[Fredrik Lundh]

Thomas Bellman wrote:

Have you tried reading the manual for the subprocess module?

han har försökt, men hans tourette tog överhanden:

http://mail.python.org/pipermail/pyt...er/297642.html

</F>

[Jeremy Jones]

Thomas Bellman wrote:

"Xah Lee" <xa*@xahlee.org> writes:

suppose i'm calling two system processes, one to unzip, and one to
tail to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Have you tried reading the manual for the subprocess module? You
just *might* find the answer to your question if you look at what
you can do with Popen objects.

Oh, come on. Don't you know that all Python documentation is rubbish
and not worth reading, written by IT idiots who throw around useless
jargon and indulge in extreme forms of self-gratification? Someone of
the caliber of Xah Lee would *never* stoop so low as to actually read
the documentation. It is beneath him. Instead, he posts messages to a
group of IT idiots who throw around useless jargon and indulge in
extreme forms of self-gratification in posting answers to questions.

<snip>

JMJ

[Martin Franklin]

Xah Lee wrote:

suppose i'm calling two system processes, one to unzip, and one to
“tail� to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
xa*@xahlee.org
∑ http://xahlee.org/

I think the idea is you wait for the first call to subprocess.call to
finish before executing the second...

http://docs.python.org/lib/node231.html
call( *args, **kwargs)
Run command with arguments. *Wait for command to complete*, then
return the returncode attribute.

The arguments are the same as for the Popen constructor. Example:

retcode = call(["ls", "-l"])

[Nainto]

Yeah, I agree. The Python documentation just merey describes what
arguements a function can take not as much how to use the actual
function.

[Fredrik Lundh]

"Nainto" <Na****@gmail.com> wrote:

Yeah, I agree. The Python documentation just merey describes what
arguements a function can take not as much how to use the actual
function.

yeah, that's a really relevant criticism when we're talking about a
module that contains one function and one class, and for which the
documentation contains *sixteen* examples.

</F>

[Lars Gustäbel]

[Fredrik Lundh]

han har försökt, men hans tourette tog överhanden:

IMHO it's more likely an Asperger's syndrome.

http://en.wikipedia.org/wiki/Asperger_Syndrome

--
Lars Gustäbel
la**@gustaebel.de

Any sufficiently advanced technology is indistinguishable
from magic.
(Arthur C. Clarke)

[Steve Horsley]

Xah Lee wrote:

suppose i'm calling two system processes, one to unzip, and one to
“tail� to get the last line. How can i determine when the first
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
xa*@xahlee.org
∑ http://xahlee.org/

As far as I can tell from the docs (worth reading),
system(command) doesn't return until the process has completed.
It would have to do some fancy footwork to return the exit code
BEFORE the process had completed!

Steve

[Martin P. Hellwig]

Lars Gustäbel wrote:

[Fredrik Lundh]

han har försökt, men hans tourette tog överhanden:

IMHO it's more likely an Asperger's syndrome.

http://en.wikipedia.org/wiki/Asperger_Syndrome

I disagree, in his writings I found no evidence of autisme.
Actually most of it can be classified as being stubborn against better
knowledge, what actually a common thing is.

But he does ask the question and by this does admits he is not
knowledged in that topic.

The only thing I am disappointed at his writing style, most likely he
has a disrupted view on social acceptable behavior and communication.
These skills might be still in development, so perhaps it is reasonable
to give him a chance and wait until he is out of his puberty.

--
mph

[Peter Hansen]

Martin P. Hellwig wrote:

The only thing I am disappointed at his writing style, most likely he
has a disrupted view on social acceptable behavior and communication.
These skills might be still in development, so perhaps it is reasonable
to give him a chance and wait until he is out of his puberty.

He's 37 years old! How long should one be given to mature?

-Peter

[Xah Lee]

Thanks all.

I found the answer, rather easily.
To make a system call and wait for it, do:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]).wait();

------
this post is archived at:
http://xahlee.org/perl-python/system_calls.html

Xah
xa*@xahlee.org
∑ http://xahlee.org/
Xah Lee wrote:

suppose i'm calling two system processes, one to unzip, and one to
“tail� to get the last line. How can i determine when thefirst
process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
xa*@xahlee.org
∑ http://xahlee.org/

[Martin P. Hellwig]

Peter Hansen wrote:

Martin P. Hellwig wrote:

The only thing I am disappointed at his writing style, most likely he
has a disrupted view on social acceptable behavior and communication.
These skills might be still in development, so perhaps it is
reasonable to give him a chance and wait until he is out of his puberty.

He's 37 years old! How long should one be given to mature?

Yeah well, eeh I guess that rules out the development part.
Although puberty can take a long time (according to my SO) :-)

--
mph

[Martin Franklin]

Xah Lee wrote:

suppose i'm calling two system processes, one to unzip, and one to
“tail� to get the las t line. How can i determine when the first process is done?

Example:

subprocess.Popen([r"/sw/bin/gzip","-d","access_log.4.gz"]);

last_line=subprocess.Popen([r"/usr/bin/tail","-n 1","access_log.4"],
stdout=subprocess.PIPE).communicate()[0]

of course, i can try workarounds something like os.system("gzip -d
thiss.gz && tail thiss"), but i wish to know if there's non-hack way to
determine when a system process is done.

Xah
xa*@xahlee.org
∑ http://xahlee.org/

I know you've found the answer to your question, however for the exact
example you gave a much better solution comes to mind...
import gzip

log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()
Regards
Martin

[Xah Lee]

Martin Franklin wrote:

import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()

does the
log_file.readlines()[-1]
actually read all the lines first?

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Xah
xa*@xahlee.org
∑ http://xahlee.org/

[Martin Franklin]

Xah Lee wrote:

Martin Franklin wrote:

import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()

does the
log_file.readlines()[-1]
actually read all the lines first?

Yes I'm afraid it does.


i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on :) (I have over a gig of memory here)

Xah
xa*@xahlee.org
∑ http://xahlee.org/

[Martin Franklin]

Martin Franklin wrote:

Xah Lee wrote:

Martin Franklin wrote:

import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()

does the
log_file.readlines()[-1]
actually read all the lines first?

Yes I'm afraid it does.

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on :) (I have over a gig of memory here)

And just a few minutes after I sent that... this...

import gzip

logfile = gzip.open("access_log.4.BIG.gz")

## seek relative to the end of the file
logfile.seek(-500)

last_line = logfile.readlines()[-1]

logfile.close()

print last_line
Works quite fast on my machine...

Regards
Martin

[Martin Franklin]

Martin Franklin wrote:

Martin Franklin wrote:

Xah Lee wrote:

Martin Franklin wrote:

import gzip
log_file = gzip.open("access_log.4.gz")
last_line = log_file.readlines()[-1]
log_file.close()
does the
log_file.readlines()[-1]
actually read all the lines first?

Yes I'm afraid it does.

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on :) (I have over a gig of memory here)

And just a few minutes after I sent that... this...

import gzip

logfile = gzip.open("access_log.4.BIG.gz")

## seek relative to the end of the file
logfile.seek(-500)

last_line = logfile.readlines()[-1]

logfile.close()

print last_line
Works quite fast on my machine...

whoops, no it doesn't looking at wrong window :( just ignore
me please :)

[Fredrik Lundh]

Martin Franklin wrote:

Ok, in that case stick to your shell based solution, although 100
megabytes does not sound that large to me I guess it is relative
to the system you are running on :) (I have over a gig of memory here)

since the file is gzipped, you need to read all of it to get the last line.
however, replacing

last_line = logfile.readlines()[-1]

with a plain

for last_line in logfile: pass

avoids storing the entire file in memory.

(note that seek() actually reads the file in 1024 blocks until it gets
to the right location; reading via a line iterator only seems to be
marginally slower. zcat|tail is a LOT faster. ymmv.)

</F>

[Christos Georgiou]

On Wed, 07 Sep 2005 23:28:13 -0400, rumours say that Peter Hansen
<pe***@engcorp.com> might have written:

Martin P. Hellwig wrote:

The only thing I am disappointed at his writing style, most likely he
has a disrupted view on social acceptable behavior and communication.
These skills might be still in development, so perhaps it is reasonable
to give him a chance and wait until he is out of his puberty.

He's 37 years old! How long should one be given to mature?

I (lots of female friends, actually :) believe many men remain in
puberty for longer than that.
--
TZOTZIOY, I speak England very best.
"Dear Paul,
please stop spamming us."
The Corinthians

[Leif K-Brooks]

Xah Lee wrote:

i switched to system call with tail because originally i was using a
pure Python solution

inF = gzip.GzipFile(ff, 'rb');
s=inF.readlines()
inF.close()
last_line=s[-1]

and since the log file is 100 megabytes it takes a long time and hogs
massive memory.

How about:

inF = gzip.GzipFile(ff, 'rb')
for line in inF:
pass
last_line = line

It's a bit slower than gzip and tail, but the memory usage is fine.

[Fredrik Lundh]

Fredrik Lundh wrote:

zcat|tail is a LOT faster.

and here's the "right way" to use that:

from subprocess import Popen, PIPE
p1 = Popen(["zcat", filename], stdout=PIPE)
p2 = Popen(["tail", "-1"], stdin=p1.stdout, stdout=PIPE)
last_line = p2.communicate()[0]

(on my small sample, this is roughly 15 times faster than the empty for loop)

</F>

[Xah Lee]

isn't there a way to implement tail in python with the same class of
performance?

how's tail implemented?

Xah
xa*@xahlee.org
∑ http://xahlee.org/

Fredrik Lundh wrote:

Fredrik Lundh wrote:

zcat|tail is a LOT faster.

and here's the "right way" to use that:

from subprocess import Popen, PIPE
p1 = Popen(["zcat", filename], stdout=PIPE)
p2 = Popen(["tail", "-1"], stdin=p1.stdout, stdout=PIPE)
last_line = p2.communicate()[0]

(on my small sample, this is roughly 15 times faster than the empty for loop)

</F>

[Michael Sparks]

Xah Lee wrote:

isn't there a way to implement tail in python with the same class of
performance?

how's tail implemented?:

Those crazy open source developers have an implementation here:
http://cvs.sourceforge.net/viewcvs.p....1&view=markup

It's not documented. It's pretty short though, I'm sure you won't have any
problems <strike>ripping it to pieces</strike> understanding it.

If it is a problem, a more documented version here (however I'm sorry, this
also by those open source/free software people you love to attack):

http://www.koders.com/c/fid8DEE98A42...4E25CF377.aspx

If you want something not by open source crazies (as you like to refer to
us), you may prefer this link:
http://minnie.tuhs.org/UnixTree/V7/u...md/tail.c.html

However that's even less documented (despite being not open source, just
"visible"), and more obfuscated.

As a compromise, you could always look at plan 9's implementation here:
* http://swtch.com/usr/local/plan9/src/cmd/tail.c

If you're after alternatives, you could always try here: (where all these
links came from)
http://www.google.com/search?q=tail.c

Fairly informative set of links provided by that highly non-obvious
search....

Hope that helps,

Best Regards,
Michael.