471,872 Members | 1,413 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,872 software developers and data experts.

Create TarFile using string buffers

I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.

Thank you for your help

wy

Mar 19 '07 #1
7 7333
On Mon, Mar 19, 2007 at 12:06:39PM -0700, au******@gmail.com wrote:
I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.
addfile()'s fileobj argument can be anything that has a read()
method. The amount of bytes to read is taken from the
tarinfo.size attribute. You could let your program write its
data to a StringIO object and pass that object to addfile().

http://docs.python.org/lib/tarfile-objects.html

--
Lars Gustäbel
la**@gustaebel.de

Linux is like a wigwam - no Gates, no Windows, Apache inside.
Mar 19 '07 #2
En Mon, 19 Mar 2007 16:06:39 -0300, au******@gmail.com
<au******@gmail.comescribió:
I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.
You can create a TarInfo object directly, and use addfile with a StringIO
object:

import tarfile
from cStringIO import StringIO

data = "Some text, maybe containing\ntwo or more lines\n" + "The quick
brown fox jumps over the lazy dog\n" * 20
fobj = StringIO(data)

tar = tarfile.open("sample.tar", "w")
tarinfo = tarfile.TarInfo("foo.txt")
tarinfo.size = len(data)
tar.addfile(tarinfo, fobj)
tar.close()

tar = tarfile.open("sample.tar", "r")
tar.list()
foo = tar.extractfile("foo.txt")
data_read = foo.read()
print "foo.txt:\n%s" % data_read
tar.close()
assert data == data_read
--
Gabriel Genellina

Mar 19 '07 #3
I'm using the example of the site
http://docs.python.org/lib/module-struct.html :

import struct
pack('hhl', 1, 2, 3)

I should get:
>>'\x00\x01\x00\x02\x00\x00\x00\x03'
I get an empty line, a '\n':
>>>
_
I know it should work. The code is OK.
What could be wrong? Doesnt find the library module?
I repeat that the unpack works ok.
I use debian etch and sarge. Same situation in both of them.

Thanks.
Andrés M.
Mar 19 '07 #4

AndrésI'm using the example of the site
Andréshttp://docs.python.org/lib/module-struct.html :

Andrésimport struct
Andréspack('hhl', 1, 2, 3)

AndrésI should get:
>>>'\x00\x01\x00\x02\x00\x00\x00\x03'
AndrésI get an empty line, a '\n':
>>>
_
Try

import struct
struct.pack('hhl', 1, 2, 3)

or

from struct import pack
pack('hhl', 1, 2, 3)

instead.

Skip

Mar 19 '07 #5
En Mon, 19 Mar 2007 19:32:41 -0300, Andrés Martinelli
<an****@lacordilleraandina.com.arescribió:
I'm using the example of the site
http://docs.python.org/lib/module-struct.html :

import struct
pack('hhl', 1, 2, 3)
That code should raise a NameError. Either you are using *another* pack
function, or that is not what you actually typed.
This is what I get:

Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.
pyimport struct
pystruct.pack('hhl', 1, 2, 3)
'\x01\x00\x02\x00\x03\x00\x00\x00'
I should get:
>>>'\x00\x01\x00\x02\x00\x00\x00\x03'
Maybe on another platform - on i386 I get the byte ordering as above.

--
Gabriel Genellina

Mar 19 '07 #6
Thanks. It almost works. The problem is I don't know the size of the
file until it has finished streaming. It looks like the tar file
format need the file size written at the beginning :(

Mar 20 '07 #7
En Mon, 19 Mar 2007 21:55:30 -0300, au******@gmail.com
<au******@gmail.comescribió:
Thanks. It almost works. The problem is I don't know the size of the
file until it has finished streaming. It looks like the tar file
format need the file size written at the beginning :(
Yes, maybe because it's originally a tape format. Anyway it could be done;
addfile() could seek back to the header and patch it after the file size
is known...

--
Gabriel Genellina

Mar 20 '07 #8

This discussion thread is closed

Replies have been disabled for this discussion.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.