473,320 Members | 1,969 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

GZipStream

DR
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
Sep 4 '08 #1
2 2925
"DR" <so*******************@yahoo.comwrote in message
news:ek**************@TK2MSFTNGP02.phx.gbl...
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB
file of data takes a few minutes!
GZipStream is broken in many ways. To list a few:

https://connect.microsoft.com/Visual...edbackID=93636
https://connect.microsoft.com/Visual...edbackID=94784

My suggestion is to vote on those on Microsoft Connect in hope they will get
fixed eventually, and use SharpZipLib meanwhile.
Sep 5 '08 #2

"DR" wrote:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compression.GZipStream and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compression.GZipStream takes an hour! when just loading 50GB file
of data takes a few minutes!
Because loading compressed data is more complex than loading uncompressed
data. I don't know the internals of the zip algorithm, but I'm guessing the
amount of IO reading prevents choking the CPU.
--
Happy Coding!
Morten Wennevik [C# MVP]
Sep 5 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Craig Buchanan | last post by:
when i try to open a file that has been compressed with GZipStream (source from microsoft's 101 Samples for Visual Studio 2005), i get an "invalid archive" message from winzip, zipgenius and xp's...
1
by: Robinson | last post by:
I'm trying to work out how to decompress a GZipStream. I'm not sure how to allocate the destination buffer, given that I cannot query the stream to find out the total decompressed size of the...
1
by: prince.matt | last post by:
Hi, I am using GZipStream to compress a simple text file approx 24MB in size. I am finding that the destination "compressed" file is several MB larger than the source file (31MB). The code is...
1
by: hakan.thornqvist | last post by:
I am using GZipStream for compression, and the size and contents of the compressed byte array differs from time to time, using the same source. byte uncompressed =...
1
by: sedwick | last post by:
I'm using .NET 2.0.50727 in VS 2005 Pro, ENU Service Pack 1 (KB926601). I've been experimenting with the System.IO.Compression classes GZipStream and DeflateStream, and I found it interesting to...
0
by: pierreuk | last post by:
Hi All, I have been looking at the best way of doing this but can't find a solution that doesn't involve using bytes array. Basically I want to wrap a XmlTextWriter in a GZipStream so I produce...
4
by: Arnie | last post by:
Folks, The GZipStream class in .NET is throwing an exception during an OS unseal/power up (first boot experience). The .NET system is fully up as other apps seem to run fine.
0
by: DR | last post by:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped file) then loading 50GB unzipped data? im using System.IO.Compression.GZipStream and its not maxing out the cpu while...
2
by: =?Utf-8?B?TWFydGluIE1hZHJlemE=?= | last post by:
Hi, i've got problems with CryptoStream and GZipStream. Someone tried that? Or got any idea why this wont work? i tried every combination, first i used CryptoStream, after that GZipStream. Or...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.