473,322 Members | 1,352 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Creating Large zip files in php

Gday Guys

I'm attempting to create zip files on the fly for some highly
compressible, yet very large files stored on my Web server. At present
I'm using a class from the Zend library by Eric Mueller which I've
modified to suit my purposes. The problem occurs when I have files of
an arbitrarily large size.. say 100mb, whereby I end up with high
memory usage. At the moment I'm using a workaround whereby the scripts
may use a very large amount of memory and work on no time limit but
obviously this is something that must not be done in a production
environment.

What I'm hoping is that someone has written a class, or solved this by
'streaming' a zip file as it's being created to the user to avoid high
memory usage during the process. My understanding is that using the
deflate algorithm, you can create sequential 'blocks' of compressed data
which I imagine would play directly into the hands of what I'd want to
achieve

Time is not an issue for my clients, as the 2 minutes they may have to
wait for the script to execute are far better than the extra 200mb they
would have to download for an uncompressed file. (Example, reduction
from 420mb to 78mb), however I would like to be able to handle several
of these operations occurring at once.

Failing that.. looks like I'll have to write something to suit my needs.
--
--------------------
Arkady Renko
Network Admin
To email me directly
Remove all capitals
and underscores from
my posting address
Oct 3 '06 #1
1 7653
Hi,

You can use command-line zip to do this. For example, to zip a file to
stdout in unix:

zip - file.txt

Here's a function to zip and stream a file on the fly in PHP using the
zip command:

<?

// This function will zip and transmit a file on the
// fly using the zip command.
//
// Usage example:
//
// zipfly('/tmp/data.txt');

function zipfly($path)
{
// Try to disable the browser cache

header('Pragma: public');
header('Expires: 0');
header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
header('Cache-Control: private', false);

// Run the zip command

$zipErrorFile = tempnam('/tmp', 'zipfly-');

$cmd = 'zip - ' . escapeshellarg($path) .
' 2' . escapeshellarg($zipErrorFile);

$proc = popen($cmd, 'r') or die();

// Stream the zip data

if (is_resource($proc) && !feof($proc)) {
if (($data = fread($proc, 8192)) !== false && $data !== '') {
header('Content-Transfer-Encoding: binary');
header('Content-Type: application/zip');
$fileName = preg_replace('/\..*?$/', '', basename($path)) .
'.zip';
header('Content-Disposition: attachment; filename="' .
"$fileName\";");
echo $data;
if (!feof($proc)) {
fpassthru($proc);
}
}
}

// Get the exit code of the command

$exitCode = pclose($proc);

if ($exitCode != 0) {
$msg = "Command \"$cmd\" failed with exit code $exitCode";
$errors = is_file($zipErrorFile) ? file_get_contents($zipErrorFile)
: false;
$msg .= $errors !== false ? ": $errors" : '.';
trigger_error($msg);
}
}

?>

Best Regards,

John Peters

Arkady Renko wrote:
Gday Guys

I'm attempting to create zip files on the fly for some highly
compressible, yet very large files stored on my Web server. At present
I'm using a class from the Zend library by Eric Mueller which I've
modified to suit my purposes. The problem occurs when I have files of
an arbitrarily large size.. say 100mb, whereby I end up with high
memory usage. At the moment I'm using a workaround whereby the scripts
may use a very large amount of memory and work on no time limit but
obviously this is something that must not be done in a production
environment.

What I'm hoping is that someone has written a class, or solved this by
'streaming' a zip file as it's being created to the user to avoid high
memory usage during the process. My understanding is that using the
deflate algorithm, you can create sequential 'blocks' of compressed data
which I imagine would play directly into the hands of what I'd want to
achieve

Time is not an issue for my clients, as the 2 minutes they may have to
wait for the script to execute are far better than the extra 200mb they
would have to download for an uncompressed file. (Example, reduction
from 420mb to 78mb), however I would like to be able to handle several
of these operations occurring at once.

Failing that.. looks like I'll have to write something to suit my needs.
--
--------------------
Arkady Renko
Network Admin
To email me directly
Remove all capitals
and underscores from
my posting address
Oct 4 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Edvard Majakari | last post by:
Greetings, fellow Pythonistas! I'm about to create three modules. As an avid TDD fan I'd like to create typical 'use-cases' for each of these modules. One of them is rather large, and I wondered...
0
by: Oci-One Kanubi | last post by:
Everything works fine in Access, but when I double-click on the resultant Excel files the first one opens correctly, but subsequent ones, and any other Excel files I try to open, fail to display at...
3
by: D.M | last post by:
In the Access 2002 mdb database I want to disable creating MDE files. Thanks in advance.
2
by: Todd_M | last post by:
I was wondering what anyone might suggest as "best practice" patterns for streaming out fixed formatted text files with C#? Let's say we get our data in a dataset table and we need to iterate over...
2
by: jdev8080 | last post by:
We are looking at creating large XML files containing binary data (encoded as base64) and passing them to transformers that will parse and transform the data into different formats. Basically,...
0
by: kirknew2SQL | last post by:
I have AdventureWorks installed. I am using SQL Server 2005. I need to import some large tab delimited text files into SQL. From the research i did I thought that using BCO would be the best...
5
by: Aryan | last post by:
Hi, I am using C# with framework 2.0 and creating PDF files on-fly, along with this I am using Windows 2003 Server. I am using Byte to take the data input and then save into pdf format on...
7
by: DanielJohnson | last post by:
I have a small project which has around 10 .py files and I run this project using command line arguments. I have to distribute this project to somebody. I was wondering how can I make an...
1
by: =?Utf-8?B?UVNJRGV2ZWxvcGVy?= | last post by:
Using .NET 2.0 is it more efficient to copy files to a single folder versus spreading them across multiple folders. For instance if we have 100,000 files to be copied, Do we copy all of them to...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.