471,305 Members | 1,245 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,305 software developers and data experts.

tarfile's tar.extractfile() file-like object incompatible with pickle.load()?

Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r')
filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')
Jul 18 '05 #1
5 4343

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...
Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r')
filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')


Maby you should,

import StringIO

hashcopy = pickle.load(StringIO.StringIO(filelike))

but im not sure,

Tom
Jul 18 '05 #2

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...
Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r')
filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')


It occurs to me that you need to do,

hashcopy = pickle.loads(filelike)

if filelike is a string.

Tom
P.S. have a look at pickle.dumps()
Jul 18 '05 #3
"Tom B." <sb******@commspeed.net> wrote in message news:<10***************@news.commspeed.net>...
"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...
Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r') filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')


It occurs to me that you need to do,

hashcopy = pickle.loads(filelike)

if filelike is a string.

Tom
P.S. have a look at pickle.dumps()


The tarfile.extractfile() method does not read the contents
of the encapsulated file into a string, but constructs a new
object that implements file operations (it is like a file)
and is intended to be used as if you had simply opened the
tar-encapsulated file directly.

Matt
Jul 18 '05 #4

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...
"Tom B." <sb******@commspeed.net> wrote in message

news:<10***************@news.commspeed.net>...
"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...
Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = os.fdopen(fd, 'w')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r') filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')


It occurs to me that you need to do,

hashcopy = pickle.loads(filelike)

if filelike is a string.

Tom
P.S. have a look at pickle.dumps()


The tarfile.extractfile() method does not read the contents
of the encapsulated file into a string, but constructs a new
object that implements file operations (it is like a file)
and is intended to be used as if you had simply opened the
tar-encapsulated file directly.

Matt


How about,

filelike = tar.extractfile('pickledhash')
filetext = filelike.read()
hashcopy = pickle.load(StringIO.StringIO(filetext ))

Tom
Jul 18 '05 #5

"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...
"Tom B." <sb******@commspeed.net> wrote in message

news:<10***************@news.commspeed.net>...
"Matt Doucleff" <ma******@ugcs.caltech.edu> wrote in message
news:3e**************************@posting.google.c om...
Hi everyone! I must be doing something wrong here :) I have a
tarball that contains a single file whose contents are a pickled
object. I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile(). Attempts to do
this result in EOFError. However if I first extract to a temporary
file, then unpickle from there, it works. The below code reproduces
the problem (on my machine at least). I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed). Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os
Now that I take a closer look at your program it should read,

if __name__ == '__main__':
try:
hashtopickle = { 'a' : 1, 'b' : 2 }

# pickle to temp file
(fd, tmpfilename) = tempfile.mkstemp()
tmpfile = file(tmpfilename,'w+')
pickle.dump(hashtopickle, tmpfile)
tmpfile.close()

# create tar; add temp file
tar = tarfile.open('tarpickle.tar', 'w')
tar.add(tmpfilename, 'pickledhash')
tar.close()

# remove temp file
os.remove(tmpfilename)

# open tarfile for reading, get filelike
tar = tarfile.open('tarpickle.tar', 'r') filelike = tar.extractfile('pickledhash')

# fails
hashcopy = pickle.load(filelike)

finally:
# cleanup
os.remove('tarpickle.tar')

Your opening the tmpfilename in an odd way.

Tom
Jul 18 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by Lars Behrens | last post: by
8 posts views Thread by Jay Donnell | last post: by
5 posts views Thread by Uwe Mayer | last post: by
1 post views Thread by Beowulf | last post: by
3 posts views Thread by justin.vanwinkle | last post: by
3 posts views Thread by Anurag | last post: by
6 posts views Thread by Terry Carroll | last post: by
1 post views Thread by boblatest | last post: by
reply views Thread by rosydwin | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.