Help needed: file writing problem with subprocess

[Pekka Niiranen]

Hi,

I am running Python script in W2K or in WinXP.
The script is started from DOS prompt and writes text file to disk
with codecs.open() in UTF8.

The problem is: When script writes the file and tries to read it
with by calling itself thru subprocess() the created files are
NOT accessible because they have not been written into disk yet.
fh.flush() and os.fsync() do not work.

-- extract from "myscript.py" starts --

fh = codecs.open("myfile.txt", "w", "utf8")
fh.write("# Comment line\n")
fh.writelines("some list")
fh.flush()
fh.close()

then later:

mycmd = "%s -f %s" % ("myscript.py", "myfile.txt")
subprocess.call(mycmd, shell=True)

This fails: "myfile.txt" is not yet written into disk
and the "myscript.py" called thru subprocess fails.
Only after the main script exits will "myfile.txt" occur
into the directory.

Previously I had Perl script calling shell script
thru "system" -command. My plan is to rewrite everything with Python.

-pekka-

[Fredrik Lundh]

Pekka Niiranen wrote:

I am running Python script in W2K or in WinXP.
The script is started from DOS prompt and writes text file to disk
with codecs.open() in UTF8.

The problem is: When script writes the file and tries to read it
with by calling itself thru subprocess() the created files are
NOT accessible because they have not been written into disk yet.
fh.flush() and os.fsync() do not work.

-- extract from "myscript.py" starts --

fh = codecs.open("myfile.txt", "w", "utf8")
fh.write("# Comment line\n")
fh.writelines("some list")
fh.flush()
fh.close()

then later:

mycmd = "%s -f %s" % ("myscript.py", "myfile.txt")
subprocess.call(mycmd, shell=True)

This fails: "myfile.txt" is not yet written into disk
and the "myscript.py" called thru subprocess fails.
Only after the main script exits will "myfile.txt" occur
into the directory.

this should of course work, and it sure works for me.

does this involve any unconventional file systems? (file servers etc).

how much later is later ?

have you checked what the current directory is in all three cases? (before
you create the file, before you call the script, and before you try to open the
file in the script).

</F>

[pekka niiranen]

Fredrik Lundh wrote:

Pekka Niiranen wrote:

I am running Python script in W2K or in WinXP.
The script is started from DOS prompt and writes text file to disk
with codecs.open() in UTF8.

The problem is: When script writes the file and tries to read it
with by calling itself thru subprocess() the created files are
NOT accessible because they have not been written into disk yet.
fh.flush() and os.fsync() do not work.

-- extract from "myscript.py" starts --

fh = codecs.open("myfile.txt", "w", "utf8")
fh.write("# Comment line\n")
fh.writelines("some list")
fh.flush()
fh.close()

then later:

mycmd = "%s -f %s" % ("myscript.py", "myfile.txt")
subprocess.call(mycmd, shell=True)

This fails: "myfile.txt" is not yet written into disk
and the "myscript.py" called thru subprocess fails.
Only after the main script exits will "myfile.txt" occur
into the directory.

this should of course work, and it sure works for me.

does this involve any unconventional file systems? (file servers etc).

how much later is later ?

have you checked what the current directory is in all three cases? (before
you create the file, before you call the script, and before you try to open the
file in the script).

</F>

The problem was that full pathname to mytext.txt -file was wrong and my
handmade error message about it was misleading. Furthermore, Windows
Explorer showed created file on screen after considerable delay
(5sec) which made me think file was never created. Seems, that due to
operating system delays one cannot rely on his own eyes about
files true existence during script run.

Just one question more; how can I spawn/open another Dos -window
with subprocess()?

Thanks anyways,

-pekka-

[Pekka Niiranen]

Fredrik Lundh wrote:

Pekka Niiranen wrote:

I am running Python script in W2K or in WinXP.
The script is started from DOS prompt and writes text file to disk
with codecs.open() in UTF8.

The problem is: When script writes the file and tries to read it
with by calling itself thru subprocess() the created files are
NOT accessible because they have not been written into disk yet.
fh.flush() and os.fsync() do not work.

-- extract from "myscript.py" starts --

fh = codecs.open("myfile.txt", "w", "utf8")
fh.write("# Comment line\n")
fh.writelines("some list")
fh.flush()
fh.close()

then later:

mycmd = "%s -f %s" % ("myscript.py", "myfile.txt")
subprocess.call(mycmd, shell=True)

This fails: "myfile.txt" is not yet written into disk
and the "myscript.py" called thru subprocess fails.
Only after the main script exits will "myfile.txt" occur
into the directory.

this should of course work, and it sure works for me.

does this involve any unconventional file systems? (file servers etc).

how much later is later ?

have you checked what the current directory is in all three cases? (before
you create the file, before you call the script, and before you try to open the
file in the script).

</F>

The problem was full pathname to mytext.txt -file was wrong and my
handmade error message ("expect" part) about it was misleading.
Furthermore, Windows Explorer displayed created file on screen after
considerable delay (5sec) which made me think file was never created.

Due to operating system delays one cannot rely on his own
eyes about files true existence during script's run;).

Thanks anyways,

-pekka-

[Pekka Niiranen]

Fredrik Lundh wrote:

Pekka Niiranen wrote:

I am running Python script in W2K or in WinXP.
The script is started from DOS prompt and writes text file to disk
with codecs.open() in UTF8.

The problem is: When script writes the file and tries to read it
with by calling itself thru subprocess() the created files are
NOT accessible because they have not been written into disk yet.
fh.flush() and os.fsync() do not work.

-- extract from "myscript.py" starts --

fh = codecs.open("myfile.txt", "w", "utf8")
fh.write("# Comment line\n")
fh.writelines("some list")
fh.flush()
fh.close()

then later:

mycmd = "%s -f %s" % ("myscript.py", "myfile.txt")
subprocess.call(mycmd, shell=True)

This fails: "myfile.txt" is not yet written into disk
and the "myscript.py" called thru subprocess fails.
Only after the main script exits will "myfile.txt" occur
into the directory.

this should of course work, and it sure works for me.

does this involve any unconventional file systems? (file servers etc).

how much later is later ?

have you checked what the current directory is in all three cases? (before
you create the file, before you call the script, and before you try to open the
file in the script).

</F>

The problem was full pathname to mytext.txt -file was wrong and my
handmade error message ("expect" part) about it was misleading.
Furthermore, Windows Explorer displayed created file on screen after
considerable delay (5sec) which made me think file was never created.

Due to operating system delays one cannot rely on his own
eyes about files true existence during script's run;).

Thanks anyways,

-pekka-