Help! File objects and scoping

[bens]

I'm trying to return an 'mmap' object from a function. The return
works, but some of the object's methods crash. Here are two examples

doesntwork.py
---------------------------
import mmap
import os

name="/any/real/file/on/my/hard/drive"

def getfile(name):
somefile=file(name, 'r+b')
mmapped = mmap.mmap(fileno=somefile.fileno(), \
length=os.path.getsize(name))
return mmapped

mmapped=getfile(name)
mmapped.seek(1) #this works, so mmapped is a working mmap object
print mmapped.size() #this dies with an error
mmapped.close()
--------------------------

doesntwork.py dies with the error:
EnvironmentError: [Errno 9] Bad file descriptor

On the other hand,
doeswork.py
---------------------------
import mmap
import os

name="/the/exact/same/file"

somefile=file(name, 'r+b')
mmapped = mmap.mmap(fileno=somefile.fileno(), \
length=os.path.getsize(name))

mmapped.seek(1) #works fine
print mmapped.size() #works fine, prints out the right number
mmapped.close()
--------------------------

The only difference between doeswork and doesntwork is whether 'mmapped'
is created in the global scope or created in a local scope and returned.
Why does that make a difference?

The strangest thing about this, I think, is that the error printed out
above, EnvironmentError, is listed in the docs under "only used as base
classes for other exceptions". In other words, it's never supposed to
be thrown.

I'll take any advice you can give me.
Thanks,
Ben Schwartz
www.mit.edu/~bens/

[Kent Johnson]

bens wrote:

I'm trying to return an 'mmap' object from a function. The return
works, but some of the object's methods crash. Here are two examples

doesntwork.py
---------------------------
import mmap
import os

name="/any/real/file/on/my/hard/drive"

def getfile(name):
somefile=file(name, 'r+b')
mmapped = mmap.mmap(fileno=somefile.fileno(), \
length=os.path.getsize(name))
return mmapped

mmapped=getfile(name)
mmapped.seek(1) #this works, so mmapped is a working mmap object
print mmapped.size() #this dies with an error
mmapped.close()
--------------------------

doesntwork.py dies with the error:
EnvironmentError: [Errno 9] Bad file descriptor
My guess is that when getfile() returns, somefile goes out of scope and the actual file is closed. Try returning both somefile and mmapped or some other way of keeping a reference to someefile.

Kent
On the other hand,
doeswork.py
---------------------------
import mmap
import os

name="/the/exact/same/file"

somefile=file(name, 'r+b')
mmapped = mmap.mmap(fileno=somefile.fileno(), \
length=os.path.getsize(name))

mmapped.seek(1) #works fine
print mmapped.size() #works fine, prints out the right number
mmapped.close()
--------------------------

The only difference between doeswork and doesntwork is whether 'mmapped'
is created in the global scope or created in a local scope and returned.
Why does that make a difference?

The strangest thing about this, I think, is that the error printed out
above, EnvironmentError, is listed in the docs under "only used as base
classes for other exceptions". In other words, it's never supposed to
be thrown.

I'll take any advice you can give me.
Thanks,
Ben Schwartz
www.mit.edu/~bens/

[Dave Benjamin]

bens wrote:

I'm trying to return an 'mmap' object from a function. The return
works, but some of the object's methods crash. Here are two examples

doesntwork.py
---------------------------
import mmap
import os

name="/any/real/file/on/my/hard/drive"

def getfile(name):
somefile=file(name, 'r+b')
mmapped = mmap.mmap(fileno=somefile.fileno(), \
length=os.path.getsize(name))
return mmapped

Here's the problem: the "mmap" object doesn't have a reference to
"somefile", only to its file descriptor number. So, when getfile() goes
out of scope, there are no references left, and the file gets closed
automatically.

One way to avoid this problem is to return the file object as well as
the "mmap" object, ie.:

def getfile(name):
somefile = file(name, 'r+b')
mmapped = mmap.mmap(fileno=somefile.fileno(),
length=os.path.getsize(name))
return (somefile, mmapped)

(somefile, mmapped) = getfile(name)

# do something with mmapped...

Now, as long as you keep "somefile" in scope, the file won't get closed.
You could also build an object and make both "somefile" and "mmapped"
attributes of this object.

Caveat: There may be a better way of doing this. I've never used "mmap".

Dave

[bens]

Dave Benjamin wrote:

Here's the problem: the "mmap" object doesn't have a reference to
"somefile", only to its file descriptor number. So, when getfile() goes
out of scope, there are no references left, and the file gets closed
automatically.

That makes a lot of sense. It's now how I would have built mmap, but I
can live with it.

Many thanks,
Ben