473,699 Members | 2,433 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Using Python/CGI to stream large tar files on the fly??

4 New Member

I'm running a CGI script written in python that tars up the photos in user selected directories (via a form). It's running on Apache on Solaris 5.8.

It works well for a small amount of files, but when I try to use it for large archives (typically over 10MB, but this seems to vary) my download stops short. Occasionally I get an error message, this is usually a Broken Pipe (IO Error 32), but sometimes I don't see anything at all (but this is perhaps a flushing issue with stderr on my webspace, perhaps it's because the Broken Pipe is a red herring).

My understanding is that to get a broken pipe the client could close its download request prematurely (definately not the case), or Apache/Python is closing stdout on its side? I get no timeout error in the browser - it just stops dead, thinking the download is complete - the tar is always corrupted and well short of the size I'd expect.

I would have thought that the stdout pipe to the user would have stayed open as long data was being streamed to it? Is it possible that my webspace provider has set some time limit? Could some sort of buffer under/over-run be occurring on the stdout stream?

I've tried various other methods of solving the problem - I've ruled out using zips as I cannot create the objects in memory (using CStringIO or similar) before streaming them as a one complete string due to runtime memory limitations on my webspace (only 30MB as far as I can see!). The only other thing I can think of is to write a temporary file to disk and allow the user to download this - again, not practical as diskspace is limited, and is a bit ugly too.

The function causing the problem is below, it's pretty self explantory. If anyone has any ideas what might be causing the problem I'd be very greatful - it's driving me round the bend!



def download( folders ):

print "Content-type: application/x-tar"
print "Content-Disposition: attachment; filename=\"Down load.tar\"\n"

parentZipFile = tarfile.open( '', "w", sys.stdout )

#signal.signal( signal.SIGPIPE, signal.SIG_DFL )

for folder in folders:

photoDir = os.path.join( folder, "photos" )
if os.path.isdir( photoDir ):

# We have photos!
photos = glob.glob( photoDir + "/*.jpg" )
photos += glob.glob( photoDir + "/*.JPG" )
for photo in photos:

parentZipFile.a dd( photo, string.join( photo.split( "/" )[-3:], "/" ) )
parentZipFile.c lose()
May 29 '07 #1
2 3163
4 New Member
To clarify this is definately not a *byte* limit on streaming.

From my nice fast connection at work, I have no problem downloading 54MB for example, but if I crank it up to a larger 150-odd MB tar it falls over in exactly the same way.

With larger archives I can see as it tries to extract that the initial decompression goes fine - and from what I can see the end of the file is hit unexpectadly.

This does look like a timeout on the Apache server.

I was wondering if anyone could clarify, and perhaps suggest a workaround?

Thanks again,

May 30 '07 #2
6,596 Recognized Expert Expert
Can't help with CGI scripts. Can help with posting code:
We use [code] tags that will maintain the indentation of your code.
Great work-around on your part, using nested[indent] tags, though.
It's all right there, on the right hand side of the page when posting or replying: 4 little things to keep in mind in * GUIDELINES...
May 30 '07 #3

Sign in to post your reply or Sign up for a free account.

Similar topics

by: Richard Taylor | last post by:
User-Agent: OSXnews 2.07 Xref: number1.nntp.dca.giganews.com comp.lang.python:437315 Hi I am trying to use py2app (http://undefined.org/python/) to package a gnome-python application called gramps (http://www.gramps-project.org) for MAC OS X.
by: TokiDoki | last post by:
Hello there, I have been programming python for a little while, now. But as I am beginning to do more complex stuff, I am running into small organization problems. It is possible that what I want to obtain is not possible, but I would like the advice of more experienced python programmers. I am writing a relatively complex program in python that has now around 40 files.
by: syed javid | last post by:
hi all, I am getting an error when i process 1 MB txt file using StreamReader and i am reading char by char using streamReader.Read() method at some point it is throwing an exception System.ComponentModel.win32Exception:The handle is invalid. Any help for this problem or any kind of informaiton to process large files the file size can be 10 MB. thanks in advance
by: nuttydevil | last post by:
Hey everyone! I'm hoping someone will be able to help me, cause I haven't had success searching on the web so far... I have large chunks of text ( all in a long string) that are currently all in separate notebook files. I want to use python to read these strings of text, THREE CHARACTERS AT A TIME. (I'm studying the genetic code you see, so I need to read and analyse each sequence one codon at a time effectively.) Does anyone have any idea...
by: Julien | last post by:
Hello, I have some files located in a file server and managed by a SQL database from a web based interface using ASP + VBSCRIPT technology. I need to automatically copy those files to a web folder. I that possible? Thanks.
by: Michael Sperlle | last post by:
Is it possible? Bestcrypt can supposedly be set up on linux, but it seems to need changes to the kernel before it can be installed, and I have no intention of going through whatever hell that would cause. If I could create a large file that could be encrypted, and maybe add files to it by appending them and putting in some kind of delimiter between files, maybe a homemade version of truecrypt could be constructed. Any idea what it...
by: C.L. | last post by:
I was looking for a function or method that would return the index to the first matching element in a list. Coming from a C++ STL background, I thought it might be called "find". My first stop was the Sequence Types page of the Library Reference (http://docs.python.org/lib/typesseq.html); it wasn't there. A search of the Library Reference's index seemed to confirm that the function did not exist. A little later I realized it might be called...
by: Chi Yin Cheung | last post by:
Hi, Is there a way in python to output binary files? I need to python to write out a stream of 5 million floating point numbers, separated by some separator, but it seems that all python supports natively is string information output, which is extremely space inefficient. I'd tried using the pickle module, but it crashed whenever I tried using it due to the large amount of data involved. Thanks for your help!
by: falloutphil | last post by:
Hi, First of all sorry for the double post - this is on the Python page too, but as far as I can see this is an Apache issue now. Mods - feel free to delete the similar titled posts from me on Python, if this is the case (can't seem to do it myself!). Anyway, I'm running a CGI script written in python that tars up the photos in user selected directories (via a form). It's running on Apache on Solaris 5.8. It works well for a...
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.