Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped file) then loading 50GB unzipped data? im using System.IO.Compression.GZipStream and its not maxing out the cpu while loading the gzip data! Im using the default buffer of the

[DR]

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!

[raylopez99]

On Sep 4, 3:54*pm, "DR" <[e-mail]:

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!

I have no idea. Perhaps your resident virus checker is checking the
zipped file? Try turning it off.

BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! *THIS* is evil! Do no evil My ARSE.
Ridiculous.

RL

[Roger Frost]

"DR" <so*******************@yahoo.comwrote in message
news:eZ**************@TK2MSFTNGP02.phx.gbl...

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB
file of data takes a few minutes!

Define "loading." Do you have 50+ GB of ram? But seriously, you're adding
a lot of extra processing when opening compressed data. With uncompressed
data, you're just retrieving as much as you can hold in memory (virtual or
otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a week
straight without hearing the same song twice), I wouldn't use a zip format.

-Roger Frost

[Jon Skeet [C# MVP]]

On Sep 5, 12:02*am, raylopez99 <[e-mail]:

<snip>

BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! **THIS* is evil! Do no evil My ARSE.
Ridiculous.

GMail works with zip attachments. However, it doesn't allow certain
file types to be sent/received, even in zip files.
See http://mail.google.com/support/bin/a...90&topic=12842

Jon

[Jon Skeet [C# MVP]]

On Sep 4, 11:54*pm, "DR" <[e-mail]:

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!

[Please learn to use a short subject line. The subject isn't meant to
hold your whole post.]

Can you provide a short but complete program to demonstrate your
problem? See http://pobox.com/~skeet/csharp/complete.html for what I
mean by that.

Jon

[Eric B.]

"Roger Frost" <fr*****@hotmail.comwrote in message
news:A2**********************************@microsof t.com...

"DR" <so*******************@yahoo.comwrote in message
news:eZ**************@TK2MSFTNGP02.phx.gbl...

>Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB
file of data takes a few minutes!

Define "loading." Do you have 50+ GB of ram? But seriously, you're
adding a lot of extra processing when opening compressed data. With
uncompressed data, you're just retrieving as much as you can hold in
memory (virtual or otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a
week straight without hearing the same song twice), I wouldn't use a zip
format.

-Roger Frost

A 50Gb zip file is just a really, really bad idea. Try breaking it up into
smaller pieces.

Eric B.