469,358 Members | 1,565 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,358 developers. It's quick & easy.

Create TarFile using string buffers

I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.

Thank you for your help

wy

Mar 19 '07 #1
7 6545
On Mon, Mar 19, 2007 at 12:06:39PM -0700, au******@gmail.com wrote:
I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.
addfile()'s fileobj argument can be anything that has a read()
method. The amount of bytes to read is taken from the
tarinfo.size attribute. You could let your program write its
data to a StringIO object and pass that object to addfile().

http://docs.python.org/lib/tarfile-objects.html

--
Lars Gustäbel
la**@gustaebel.de

Linux is like a wigwam - no Gates, no Windows, Apache inside.
Mar 19 '07 #2
En Mon, 19 Mar 2007 16:06:39 -0300, au******@gmail.com
<au******@gmail.comescribió:
I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.
You can create a TarInfo object directly, and use addfile with a StringIO
object:

import tarfile
from cStringIO import StringIO

data = "Some text, maybe containing\ntwo or more lines\n" + "The quick
brown fox jumps over the lazy dog\n" * 20
fobj = StringIO(data)

tar = tarfile.open("sample.tar", "w")
tarinfo = tarfile.TarInfo("foo.txt")
tarinfo.size = len(data)
tar.addfile(tarinfo, fobj)
tar.close()

tar = tarfile.open("sample.tar", "r")
tar.list()
foo = tar.extractfile("foo.txt")
data_read = foo.read()
print "foo.txt:\n%s" % data_read
tar.close()
assert data == data_read
--
Gabriel Genellina

Mar 19 '07 #3
I'm using the example of the site
http://docs.python.org/lib/module-struct.html :

import struct
pack('hhl', 1, 2, 3)

I should get:
>>'\x00\x01\x00\x02\x00\x00\x00\x03'
I get an empty line, a '\n':
>>>
_
I know it should work. The code is OK.
What could be wrong? Doesnt find the library module?
I repeat that the unpack works ok.
I use debian etch and sarge. Same situation in both of them.

Thanks.
Andrés M.
Mar 19 '07 #4

AndrésI'm using the example of the site
Andréshttp://docs.python.org/lib/module-struct.html :

Andrésimport struct
Andréspack('hhl', 1, 2, 3)

AndrésI should get:
>>>'\x00\x01\x00\x02\x00\x00\x00\x03'
AndrésI get an empty line, a '\n':
>>>
_
Try

import struct
struct.pack('hhl', 1, 2, 3)

or

from struct import pack
pack('hhl', 1, 2, 3)

instead.

Skip

Mar 19 '07 #5
En Mon, 19 Mar 2007 19:32:41 -0300, Andrés Martinelli
<an****@lacordilleraandina.com.arescribió:
I'm using the example of the site
http://docs.python.org/lib/module-struct.html :

import struct
pack('hhl', 1, 2, 3)
That code should raise a NameError. Either you are using *another* pack
function, or that is not what you actually typed.
This is what I get:

Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.
pyimport struct
pystruct.pack('hhl', 1, 2, 3)
'\x01\x00\x02\x00\x03\x00\x00\x00'
I should get:
>>>'\x00\x01\x00\x02\x00\x00\x00\x03'
Maybe on another platform - on i386 I get the byte ordering as above.

--
Gabriel Genellina

Mar 19 '07 #6
Thanks. It almost works. The problem is I don't know the size of the
file until it has finished streaming. It looks like the tar file
format need the file size written at the beginning :(

Mar 20 '07 #7
En Mon, 19 Mar 2007 21:55:30 -0300, au******@gmail.com
<au******@gmail.comescribió:
Thanks. It almost works. The problem is I don't know the size of the
file until it has finished streaming. It looks like the tar file
format need the file size written at the beginning :(
Yes, maybe because it's originally a tape format. Anyway it could be done;
addfile() could seek back to the header and patch it after the file size
is known...

--
Gabriel Genellina

Mar 20 '07 #8

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Beowulf | last post: by
1 post views Thread by Matthew Thorley | last post: by
3 posts views Thread by itzel | last post: by
6 posts views Thread by Terry Carroll | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.