473,287 Members | 1,570 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

tarfile.open(mode='w:gz'|'w|gz'|..., fileobj=StringIO()) fails.

Hi,

is there a way to or at least a reason why I can not use tarfile to
create a gzip or bunzip2 compressed archive in the memory?

You might might wanna answer "use StringIO" but this isn't such easy
as it seems to be. ;) I am using Python 2.5.2, by the way. I think
this is a bug in at least in this version of python, but maybe
StringIO isn't just file-like enough for this "korky" tarfile module.
But this would conflict with its documentation.

"For special purposes, there is a second format for mode: 'filemode|
[compression]'. open() will return a TarFile object that processes its
data as a stream of blocks. No random seeking will be done on the
file. If given, fileobj may be any object that has a read() or write()
method (depending on the mode)."

Sounds good, but doesn't work. ;P StringIO provides a read() and
write() method amongst others. But tarfile has especially in this mode
problems with the StringIO object.

I extracted the code out of my project into a standalone python script
to proof this issue on the lowest level. You can run the script below
as following: ./StringIO-tarfile.py file1 [file2] [...]
#
# File: StringIO-tarfile.py
#
#!/usr/bin/env python

from StringIO import StringIO
import tarfile
import sys

def create_tar_file(filenames, fileobj, mode, result_cb=lambda f:
None):
tar_file = tarfile.open(mode=mode, fileobj=fileobj)
for f in filenames:
tar_file.add(f)
result = result_cb(fileobj)
tar_file.close()
return result

if __name__ == '__main__':
files = sys.argv[1:]
modes = ['w%s%s' % (x, y)for x in (':', '|') for y in ('', 'gz',
'bz2')]

string_io_cb = lambda f: f.getvalue()

for mode in modes:
ext = mode.replace('w|', '-pipe.tar.').replace('w:',
'.tar.').rstrip('.')
# StringIO test.
content = create_tar_file(files, StringIO(), mode, string_io_cb)
fd = open('StringIO%s' % ext, 'w')
fd.write(content)
fd.close()

# file object test.
fd = open('file%s' % ext, 'w')
create_tar_file(files, fd, mode)
As test input, I have used a directory with a single text file. As you
can see below, any tests using plain file objects were successful. But
when using StringIO, I can only create uncompressed tar files. Even
though I don't get any errors when creating them most of the files are
just empty or truncated.
$ for f in `ls *.tar{,.gz,.bz2}`; do echo -n $f; du -h $f | awk
'{print " ("$1"B)"}'; tar -tf $f; echo; done

file-pipe.tar (84KB)
foo/
foo/ksp-fosdem2008.txt

file-pipe.tar.bz2 (20KB)
foo/
foo/ksp-fosdem2008.txt

file-pipe.tar.gz (20KB)
foo/
foo/ksp-fosdem2008.txt

file.tar (84KB)
foo/
foo/ksp-fosdem2008.txt

file.tar.bz2 (20KB)
foo/
foo/ksp-fosdem2008.txt

file.tar.gz (20KB)
foo/
foo/ksp-fosdem2008.txt

StringIO-pipe.tar (76KB)
foo/
foo/ksp-fosdem2008.txt
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

StringIO-pipe.tar.bz2 (0B)
tar: This does not look like a tar archive
tar: Error exit delayed from previous errors

StringIO-pipe.tar.gz (0B)
tar: This does not look like a tar archive
tar: Error exit delayed from previous errors

StringIO.tar (76KB)
foo/
foo/ksp-fosdem2008.txt

StringIO.tar.bz2 (0B)
tar: This does not look like a tar archive
tar: Error exit delayed from previous errors

StringIO.tar.gz (4.0KB)

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error exit delayed from previous errors
Can somebody reproduce this problem? Did I misunderstood the API? What
would be the best work around, if I am right? I am thinking about
using the gzip and bz2 module directly.

Regards
Sebastian Noack
Jun 27 '08 #1
6 5930
En Mon, 26 May 2008 17:44:28 -0300, se*************@googlemail.com
<se*************@googlemail.comescribió:
is there a way to or at least a reason why I can not use tarfile to
create a gzip or bunzip2 compressed archive in the memory?

You might might wanna answer "use StringIO" but this isn't such easy
as it seems to be. ;) I am using Python 2.5.2, by the way. I think
this is a bug in at least in this version of python, but maybe
StringIO isn't just file-like enough for this "korky" tarfile module.
But this would conflict with its documentation.
def create_tar_file(filenames, fileobj, mode, result_cb=lambda f:
None):
tar_file = tarfile.open(mode=mode, fileobj=fileobj)
for f in filenames:
tar_file.add(f)
result = result_cb(fileobj)
tar_file.close()
return result
It's not a bug, you must extract the StringIO contents *after* closing
tar_file, else you won't get the last blocks pending to be written.

--
Gabriel Genellina

Jun 27 '08 #2
On May 27, 2:17 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
wrote:
It's not a bug, you must extract the StringIO contents *after* closing
tar_file, else you won't get the last blocks pending to be written.
I looked at tarfile's source code last night after I wrote this
message and figured it out. But the problem is that TarFile's close
method closes the underlying file object after the last block is
written and when you close StringIO you can not get its content
anymore. Wtf does it close the underlying file? There is absolute no
reason for doing this. Are you still sure this isn't a bug?

Regards
Sebastian Noack
Jun 27 '08 #3
I have written a FileWrapper class as workaround, which works for me
(see the code below). The FileWrapper object holds an internal file-
like object and maps its attributes, but prevents the user (in this
case tarfile) from closing the internal file, so I can still access
StringIO's content after closing the TarFile object.

But this should not be required to create in memory tar files. It is
definitely a bug, that TarFile closes external file objects passed to
tarfile.open, when closing the TarFile object. The code which opens a
file is also responsible for closing it.

Regards
Sebastian Noack
#
# File: StringIO-tarfile.py
#
#!/usr/bin/env python

from StringIO import StringIO
import tarfile
import sys

class FileWrapper(object):
def __init__(self, fileobj):
self.file = fileobj
self.closed = fileobj.closed

def __getattr__(self, name):
# Raise AttributeError, if it isn't a file attribute.
if name not in dir(file):
raise AttributeError(name)

# Get the attribute of the internal file object.
value = getattr(self.file, name)

# Raise a ValueError, if the attribute is callable (e.g. an instance
# method) and the FileWrapper is closed.
if callable(value) and self.closed:
raise ValueError('I/O operation on closed file')
return value

def close(self):
self.closed = True

def create_tar_file(filenames, fileobj, mode):
tar_file = tarfile.open(mode=mode, fileobj=fileobj)
for f in filenames:
tar_file.add(f)
tar_file.close()

if __name__ == '__main__':
files = sys.argv[1:]
modes = ['w%s%s' % (x, y) for x in (':', '|') for y in ('', 'gz',
'bz2')]

for mode in modes:
ext = mode.replace('w|', '-pipe.tar.').replace('w:',
'.tar.').rstrip('.')
# StringIO test.
stream = FileWrapper(StringIO())
create_tar_file(files, stream, mode)
fd = open('StringIO%s' % ext, 'w')
fd.write(stream.file.getvalue())
stream.file.close()
fd.close()

# file object test.
fd = open('file%s' % ext, 'w')
create_tar_file(files, fd, mode)
Jun 27 '08 #4
On Tue, May 27, 2008 at 01:51:47AM -0700, se*************@googlemail.com wrote:
I have written a FileWrapper class as workaround, which works for me
(see the code below). The FileWrapper object holds an internal file-
like object and maps its attributes, but prevents the user (in this
case tarfile) from closing the internal file, so I can still access
StringIO's content after closing the TarFile object.

But this should not be required to create in memory tar files. It is
definitely a bug, that TarFile closes external file objects passed to
tarfile.open, when closing the TarFile object. The code which opens a
file is also responsible for closing it.
You're right, _BZ2Proxy.close() calls the wrapped file object's close() method
and that is definitely not the desired behaviour. So, if you can do without 'bz2'
modes for now, you're problem is gone, all other modes work fine.

I fixed it (r63744), so the next beta release will work as expected. Your test
script helped a lot, thanks.

Regards,

--
Lars Gustäbel
la**@gustaebel.de

A casual stroll through a lunatic asylum shows that
faith does not prove anything.
(Friedrich Nietzsche)
Jun 27 '08 #5
En Tue, 27 May 2008 02:43:53 -0300, se*************@googlemail.com
<se*************@googlemail.comescribió:
On May 27, 2:17 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
wrote:
>It's not a bug, you must extract the StringIO contents *after* closing
tar_file, else you won't get the last blocks pending to be written.

I looked at tarfile's source code last night after I wrote this
message and figured it out. But the problem is that TarFile's close
method closes the underlying file object after the last block is
written and when you close StringIO you can not get its content
anymore. Wtf does it close the underlying file? There is absolute no
reason for doing this. Are you still sure this isn't a bug?
Ouch, sorry, I only tried with gzip (and worked fine), not bz2 (which is
buggy).

--
Gabriel Genellina

Jun 27 '08 #6
That is right, only bz2 is affected. I am happy that i could help. ;)

Regards
Sebastian Noack
Jun 27 '08 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Jay Donnell | last post by:
Is there a way to use the tarfile module to recursively compress the contents of a directory and maintain the directory structure in the tar archive? Simply doing os.system('tar -czvf ' +...
5
by: Matt Doucleff | last post by:
Hi everyone! I must be doing something wrong here :) I have a tarball that contains a single file whose contents are a pickled object. I would like to unpickle the object directly from the...
8
by: Dennis Hotson | last post by:
Hi, I'm trying to write a function that adds a file-like-object to a compressed tarfile... eg ".tar.gz" or ".tar.bz2" I've had a look at the tarfile module but the append mode doesn't support...
5
by: Uwe Mayer | last post by:
Hi, is it possible to delete a file from a tar-archive using the tarfile module? Thanks Uwe
1
by: Beowulf | last post by:
Hello, I'm using Python to automate admin tasks on my job. We use Windoze 2000 as desktop platform. When executing this daily backup scripts I get the following error: Traceback (most recent...
1
by: Matthew Thorley | last post by:
I've been using tarfile like this import tarfile tar = tarfile.open('path_to_tar_archive', 'r:gz') But I need to use it like this: archive = open('path_to_tar_archive', 'r') tar =...
7
by: aurora00 | last post by:
I have a program that generates a number of files that will be packaged into a tarball. Can I stream the content into TarFile without first writing them out to the file system? All add(), addfile()...
6
by: Terry Carroll | last post by:
I am trying to do something with a very large tarfile from within Python, and am running into memory constraints. The tarfile in question is a 4-gigabyte datafile from freedb.org,...
1
by: boblatest | last post by:
Hello, I'm trying to catch an "EOFError" exception that occurs when reading truncated tarfile. Here's my routine, and below that the callback trace. Note that although I'm trying to catch all...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.