tarfile's tar.extractfile() file-like object incompatible with pickle.load()?

[Matt Doucleff]

Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r')
filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')

[Tom B.]

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...

Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r')
filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')

Maby you should,

import StringIO

hashcopy = pickle.load(StringIO.StringIO(filelike))

but im not sure,

Tom

[Tom B.]

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...

Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r')
filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')

It occurs to me that you need to do,

hashcopy = pickle.loads(filelike)

if filelike is a string.

Tom
P.S. have a look at pickle.dumps()

[Matt Doucleff]

"Tom B." <sb******@commspeed.net> wrote in message news:<10***************@news.commspeed.net>...

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...

Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r') filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')

It occurs to me that you need to do,

hashcopy = pickle.loads(filelike)

if filelike is a string.

Tom
P.S. have a look at pickle.dumps()

The tarfile.extractfile() method does not read the contents
of the encapsulated file into a string, but constructs a new
object that implements file operations (it is like a file)
and is intended to be used as if you had simply opened the
tar-encapsulated file directly.

Matt

[Tom B.]

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...

"Tom B." <sb******@commspeed.net> wrote in message

news:<10***************@news.commspeed.net>...

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...

Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r') filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')

It occurs to me that you need to do,

hashcopy = pickle.loads(filelike)

if filelike is a string.

Tom
P.S. have a look at pickle.dumps()

The tarfile.extractfile() method does not read the contents
of the encapsulated file into a string, but constructs a new
object that implements file operations (it is like a file)
and is intended to be used as if you had simply opened the
tar-encapsulated file directly.

Matt

How about,

filelike = tar.extractfile('pickledhash')
filetext = filelike.read()
hashcopy = pickle.load(StringIO.StringIO(filetext ))

Tom

[Tom B.]

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...

"Tom B." <sb******@commspeed.net> wrote in message

news:<10***************@news.commspeed.net>...

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...

Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os
Now that I take a closer look at your program it should read,

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = file(tmpfilename,'w+')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r') filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')

Your opening the tmpfilename in an odd way.

Tom