471,337 Members | 988 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,337 software developers and data experts.

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped file) then loading 50GB unzipped data? im using System.IO.Compression.GZipStream and its not maxing out the cpu while loading the gzip data! Im using the default buffer of the

DR
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
Sep 4 '08 #1
5 2743
On Sep 4, 3:54*pm, "DR" <softwareengineer98...@yahoo.comwrote:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
I have no idea. Perhaps your resident virus checker is checking the
zipped file? Try turning it off.

BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! *THIS* is evil! Do no evil My ARSE.
Ridiculous.

RL
Sep 4 '08 #2
"DR" <so*******************@yahoo.comwrote in message
news:eZ**************@TK2MSFTNGP02.phx.gbl...
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB
file of data takes a few minutes!

Define "loading." Do you have 50+ GB of ram? But seriously, you're adding
a lot of extra processing when opening compressed data. With uncompressed
data, you're just retrieving as much as you can hold in memory (virtual or
otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a week
straight without hearing the same song twice), I wouldn't use a zip format.

-Roger Frost

Sep 5 '08 #3
On Sep 5, 12:02*am, raylopez99 <raylope...@yahoo.comwrote:

<snip>
BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! **THIS* is evil! Do no evil My ARSE.
Ridiculous.
GMail works with zip attachments. However, it doesn't allow certain
file types to be sent/received, even in zip files.
See http://mail.google.com/support/bin/a...90&topic=12842

Jon
Sep 5 '08 #4
On Sep 4, 11:54*pm, "DR" <softwareengineer98...@yahoo.comwrote:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
[Please learn to use a short subject line. The subject isn't meant to
hold your whole post.]

Can you provide a short but complete program to demonstrate your
problem? See http://pobox.com/~skeet/csharp/complete.html for what I
mean by that.

Jon
Sep 5 '08 #5
"Roger Frost" <fr*****@hotmail.comwrote in message
news:A2**********************************@microsof t.com...
"DR" <so*******************@yahoo.comwrote in message
news:eZ**************@TK2MSFTNGP02.phx.gbl...
>Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB
file of data takes a few minutes!


Define "loading." Do you have 50+ GB of ram? But seriously, you're
adding a lot of extra processing when opening compressed data. With
uncompressed data, you're just retrieving as much as you can hold in
memory (virtual or otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a
week straight without hearing the same song twice), I wouldn't use a zip
format.

-Roger Frost

A 50Gb zip file is just a really, really bad idea. Try breaking it up into
smaller pieces.

Eric B.

Oct 6 '08 #6

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by Marek Möhling | last post: by
8 posts views Thread by Dennis Hotson | last post: by
3 posts views Thread by rohisingh | last post: by
6 posts views Thread by Aaron Gray | last post: by
1 post views Thread by Paul Smith | last post: by
13 posts views Thread by fniles | last post: by
1 post views Thread by John Nagle | last post: by
2 posts views Thread by =?Utf-8?B?QVZM?= | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.