tempfile.NamedTemporaryFile wont work

[Imbaud Pierre]

On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.
(I found a permanent workaround, so I dont ask for help)
I expected to write to a file, and access it thru a shell command.
This code, in a loop:
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
#tf.seek(0) # rewind the file
tf.write(chunk); tf.flush()
print >sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

[Bjoern Schliessmann]

Imbaud Pierre wrote:

tf = tempfile.NamedTemporaryFile()
tfName = tf.name
[...]
print >sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Just out of curiosity: Why did you assign tf.name to tfname?

Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Regards,
Björn

--
BOFH excuse #149:

Dew on the telephone lines.

[Steven D'Aprano]

On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:

On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.

[snip]

Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

I can verify this problem occurs on Fedora Core 5 too:

import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists(tfName):
print 'pre-check: %s not there' % tfName
subprocess.Popen(['strings', tfName])
if not os.path.exists(tfName):
print 'post-check: %s not there' % tfName
And here is a typical run, with the boring bits removed for ease of
reading:

>>test(30)

0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++
Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.

Is it possible this is a bug in strings?
--
Steven.

[Steven D'Aprano]

On Sun, 19 Nov 2006 13:18:39 +0100, Bjoern Schliessmann wrote:

Imbaud Pierre wrote:

> tf = tempfile.NamedTemporaryFile()
tfName = tf.name
[...]
print >sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Just out of curiosity: Why did you assign tf.name to tfname?

Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Well, yes, but if tf.name changed, that won't change the file name on disk
either:

>>tf = tempfile.NamedTemporaryFile()
tf.name

'/tmp/tmpYVV1Ij'

>>os.path.exists(tf.name)

True

>>oldname = tf.name
tf.name = "/tmp/something"
os.path.exists(tf.name)

False

>>os.path.exists(oldname)

True
I'm guessing that binding tf.name to tfName is a micro-optimization. In a
very tight loop, name lookups can take considerable time, and one
optimization can be to reduce the number of lookups:

method = something.method
while 1:
something.method # needs at least two lookups
method # needs a single lookup
--
Steve.

[Peter Otten]

Steven D'Aprano wrote:

On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:

>On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.

[snip]

>Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

I can verify this problem occurs on Fedora Core 5 too:

import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists(tfName):
print 'pre-check: %s not there' % tfName
subprocess.Popen(['strings', tfName])
if not os.path.exists(tfName):
print 'post-check: %s not there' % tfName
And here is a typical run, with the boring bits removed for ease of
reading:

>>>test(30)

0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++
Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.

Is it possible this is a bug in strings?

What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
therefore you may enter the second iteration -- which implicitly closes the
temporary file -- before strings actually tries to access it. Use call()
and everything should be fine.

Peter

[Imbaud Pierre]

Steven D'Aprano a écrit :

On Sun, 19 Nov 2006 13:18:39 +0100, Bjoern Schliessmann wrote:

>>Imbaud Pierre wrote:

>> tf = tempfile.NamedTemporaryFile()
tfName = tf.name
[...]
print >sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Just out of curiosity: Why did you assign tf.name to tfname?

Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Well, yes, but if tf.name changed, that won't change the file name on disk
either:

>>>>tf = tempfile.NamedTemporaryFile()
tf.name

'/tmp/tmpYVV1Ij'

>>>>os.path.exists(tf.name)

True

>>>>oldname = tf.name
tf.name = "/tmp/something"
os.path.exists(tf.name)

False

>>>>os.path.exists(oldname)

True
I'm guessing that binding tf.name to tfName is a micro-optimization.

indeed. And I dont see why tf.name would change.
In a

very tight loop, name lookups can take considerable time, and one
optimization can be to reduce the number of lookups:

method = something.method
while 1:
something.method # needs at least two lookups
method # needs a single lookup

[Imbaud Pierre]

Peter Otten a écrit :

Steven D'Aprano wrote:

>>On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:

>>>On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.

[snip]

>>>Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

I can verify this problem occurs on Fedora Core 5 too:

import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists(tfName):
print 'pre-check: %s not there' % tfName
subprocess.Popen(['strings', tfName])
if not os.path.exists(tfName):
print 'post-check: %s not there' % tfName
And here is a typical run, with the boring bits removed for ease of
reading:

>>>>>test(30)

0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++
Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.

Is it possible this is a bug in strings?

What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
therefore you may enter the second iteration -- which implicitly closesthe
temporary file -- before strings actually tries to access it. Use call()
and everything should be fine.

Thanks A LOT, works fine, I feel kind of silly; your diagnostic is
pretty obvious, afterward... I felt uneasy not closing the Popen, but
it worked, so why bother? Its so easy to make ugly code!