473,390 Members | 1,456 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,390 software developers and data experts.

Create TarFile using string buffers

I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.

Thank you for your help

wy

Mar 19 '07 #1
7 7674
On Mon, Mar 19, 2007 at 12:06:39PM -0700, au******@gmail.com wrote:
I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.
addfile()'s fileobj argument can be anything that has a read()
method. The amount of bytes to read is taken from the
tarinfo.size attribute. You could let your program write its
data to a StringIO object and pass that object to addfile().

http://docs.python.org/lib/tarfile-objects.html

--
Lars Gustäbel
la**@gustaebel.de

Linux is like a wigwam - no Gates, no Windows, Apache inside.
Mar 19 '07 #2
En Mon, 19 Mar 2007 16:06:39 -0300, au******@gmail.com
<au******@gmail.comescribió:
I have a program that generates a number of files that will be
packaged into a tarball. Can I stream the content into TarFile without
first writing them out to the file system? All add(), addfile() and
gettarinfo() seems to assume there is a file in the disk. But for me I
seems inefficient to write all the content to the disk and then have
it read back by the TarFile module.
You can create a TarInfo object directly, and use addfile with a StringIO
object:

import tarfile
from cStringIO import StringIO

data = "Some text, maybe containing\ntwo or more lines\n" + "The quick
brown fox jumps over the lazy dog\n" * 20
fobj = StringIO(data)

tar = tarfile.open("sample.tar", "w")
tarinfo = tarfile.TarInfo("foo.txt")
tarinfo.size = len(data)
tar.addfile(tarinfo, fobj)
tar.close()

tar = tarfile.open("sample.tar", "r")
tar.list()
foo = tar.extractfile("foo.txt")
data_read = foo.read()
print "foo.txt:\n%s" % data_read
tar.close()
assert data == data_read
--
Gabriel Genellina

Mar 19 '07 #3
I'm using the example of the site
http://docs.python.org/lib/module-struct.html :

import struct
pack('hhl', 1, 2, 3)

I should get:
>>'\x00\x01\x00\x02\x00\x00\x00\x03'
I get an empty line, a '\n':
>>>
_
I know it should work. The code is OK.
What could be wrong? Doesnt find the library module?
I repeat that the unpack works ok.
I use debian etch and sarge. Same situation in both of them.

Thanks.
Andrés M.
Mar 19 '07 #4

AndrésI'm using the example of the site
Andréshttp://docs.python.org/lib/module-struct.html :

Andrésimport struct
Andréspack('hhl', 1, 2, 3)

AndrésI should get:
>>>'\x00\x01\x00\x02\x00\x00\x00\x03'
AndrésI get an empty line, a '\n':
>>>
_
Try

import struct
struct.pack('hhl', 1, 2, 3)

or

from struct import pack
pack('hhl', 1, 2, 3)

instead.

Skip

Mar 19 '07 #5
En Mon, 19 Mar 2007 19:32:41 -0300, Andrés Martinelli
<an****@lacordilleraandina.com.arescribió:
I'm using the example of the site
http://docs.python.org/lib/module-struct.html :

import struct
pack('hhl', 1, 2, 3)
That code should raise a NameError. Either you are using *another* pack
function, or that is not what you actually typed.
This is what I get:

Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.
pyimport struct
pystruct.pack('hhl', 1, 2, 3)
'\x01\x00\x02\x00\x03\x00\x00\x00'
I should get:
>>>'\x00\x01\x00\x02\x00\x00\x00\x03'
Maybe on another platform - on i386 I get the byte ordering as above.

--
Gabriel Genellina

Mar 19 '07 #6
Thanks. It almost works. The problem is I don't know the size of the
file until it has finished streaming. It looks like the tar file
format need the file size written at the beginning :(

Mar 20 '07 #7
En Mon, 19 Mar 2007 21:55:30 -0300, au******@gmail.com
<au******@gmail.comescribió:
Thanks. It almost works. The problem is I don't know the size of the
file until it has finished streaming. It looks like the tar file
format need the file size written at the beginning :(
Yes, maybe because it's originally a tape format. Anyway it could be done;
addfile() could seek back to the header and patch it after the file size
is known...

--
Gabriel Genellina

Mar 20 '07 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Beowulf | last post by:
Hello, I'm using Python to automate admin tasks on my job. We use Windoze 2000 as desktop platform. When executing this daily backup scripts I get the following error: Traceback (most recent...
1
by: Matthew Thorley | last post by:
I've been using tarfile like this import tarfile tar = tarfile.open('path_to_tar_archive', 'r:gz') But I need to use it like this: archive = open('path_to_tar_archive', 'r') tar =...
3
by: itzel | last post by:
I have a problem. I'm new in python and I need a script that group a list of files using Tar file utility and then, compress that block using a compress utility (gzip i think). I already found some...
6
by: Terry Carroll | last post by:
I am trying to do something with a very large tarfile from within Python, and am running into memory constraints. The tarfile in question is a 4-gigabyte datafile from freedb.org,...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.