473,224 Members | 1,588 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,224 software developers and data experts.

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped file) then loading 50GB unzipped data? im using System.IO.Compression.GZipStream and its not maxing out the cpu while loading the gzip data! Im using the default buffer of the

DR
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
Sep 4 '08 #1
5 2901
On Sep 4, 3:54*pm, "DR" <softwareengineer98...@yahoo.comwrote:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
I have no idea. Perhaps your resident virus checker is checking the
zipped file? Try turning it off.

BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! *THIS* is evil! Do no evil My ARSE.
Ridiculous.

RL
Sep 4 '08 #2
"DR" <so*******************@yahoo.comwrote in message
news:eZ**************@TK2MSFTNGP02.phx.gbl...
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB
file of data takes a few minutes!

Define "loading." Do you have 50+ GB of ram? But seriously, you're adding
a lot of extra processing when opening compressed data. With uncompressed
data, you're just retrieving as much as you can hold in memory (virtual or
otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a week
straight without hearing the same song twice), I wouldn't use a zip format.

-Roger Frost

Sep 5 '08 #3
On Sep 5, 12:02*am, raylopez99 <raylope...@yahoo.comwrote:

<snip>
BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! **THIS* is evil! Do no evil My ARSE.
Ridiculous.
GMail works with zip attachments. However, it doesn't allow certain
file types to be sent/received, even in zip files.
See http://mail.google.com/support/bin/a...90&topic=12842

Jon
Sep 5 '08 #4
On Sep 4, 11:54*pm, "DR" <softwareengineer98...@yahoo.comwrote:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
[Please learn to use a short subject line. The subject isn't meant to
hold your whole post.]

Can you provide a short but complete program to demonstrate your
problem? See http://pobox.com/~skeet/csharp/complete.html for what I
mean by that.

Jon
Sep 5 '08 #5
"Roger Frost" <fr*****@hotmail.comwrote in message
news:A2**********************************@microsof t.com...
"DR" <so*******************@yahoo.comwrote in message
news:eZ**************@TK2MSFTNGP02.phx.gbl...
>Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB
file of data takes a few minutes!


Define "loading." Do you have 50+ GB of ram? But seriously, you're
adding a lot of extra processing when opening compressed data. With
uncompressed data, you're just retrieving as much as you can hold in
memory (virtual or otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a
week straight without hearing the same song twice), I wouldn't use a zip
format.

-Roger Frost

A 50Gb zip file is just a really, really bad idea. Try breaking it up into
smaller pieces.

Eric B.

Oct 6 '08 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Marek Möhling | last post by:
My server (Apache/1.3.28 - PHP/4.3.3) is configured to receive gzipped data via: Header append Accept-Encoding "gzip, deflate" PHP is configured to send gzipped data via: php_value...
8
by: Dennis Hotson | last post by:
Hi, I'm trying to write a function that adds a file-like-object to a compressed tarfile... eg ".tar.gz" or ".tar.bz2" I've had a look at the tarfile module but the append mode doesn't support...
3
by: rohisingh | last post by:
I have a tar file. The content of the file are as following. rohits@sandman 12-08-04 $ tar tvf 20041208.tar drwxr-xr-x root/root 0 2004-12-08 21:39:19 20041208/ -rw-r--r-- root/root ...
12
by: Sharon | last post by:
I’m wrote a small DLL that used the FreeImage.DLL (that can be found at http://www.codeproject.com/bitmap/graphicsuite.asp). I also wrote a small console application in C++ (unmanaged) that uses...
9
by: ReidarT | last post by:
How can I zip (compress) a file in vs2005 with windows own zip-program? reidarT
6
by: Aaron Gray | last post by:
Hi, Is there any tool that will concatinate JavaScript files compacting them spacewise and removing comment fields ? I have some sed script that sort of does the job but was wondering if there...
1
by: Paul Smith | last post by:
Hi, I'd like to read a series of sqlite database files that have already been gzipped and was wondering if this can be done on the fly. In other words, can I avoid explicitly unzipping the file...
13
by: fniles | last post by:
In VB.NET 2003 how can you programmatically unzip a ZIP file ? Thank you.
1
by: John Nagle | last post by:
I have a large (gigabytes) file which is encoded in UTF-8 and then compressed with gzip. I'd like to read it with the "gzip" module and "utf8" decoding. The obvious approach is fd =...
2
by: =?Utf-8?B?QVZM?= | last post by:
Hi I've a requirement wherein I need to unzip a zipped folder programatically in .net. I've tried out System.IO.Compression namespace..but it writes the content into bytes which I dont need..I...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: veera ravala | last post by:
ServiceNow is a powerful cloud-based platform that offers a wide range of services to help organizations manage their workflows, operations, and IT services more efficiently. At its core, ServiceNow...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.