473,795 Members | 2,999 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

GZipStream

DR
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compr ession.GZipStre am and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compr ession.GZipStre am takes an hour! when just loading 50GB file
of data takes a few minutes!
Sep 4 '08 #1
2 2949
"DR" <so************ *******@yahoo.c omwrote in message
news:ek******** ******@TK2MSFTN GP02.phx.gbl...
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compr ession.GZipStre am and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compr ession.GZipStre am takes an hour! when just loading 50GB
file of data takes a few minutes!
GZipStream is broken in many ways. To list a few:

https://connect.microsoft.com/Visual...edbackID=93636
https://connect.microsoft.com/Visual...edbackID=94784

My suggestion is to vote on those on Microsoft Connect in hope they will get
fixed eventually, and use SharpZipLib meanwhile.
Sep 5 '08 #2

"DR" wrote:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compr ession.GZipStre am and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compr ession.GZipStre am takes an hour! when just loading 50GB file
of data takes a few minutes!
Because loading compressed data is more complex than loading uncompressed
data. I don't know the internals of the zip algorithm, but I'm guessing the
amount of IO reading prevents choking the CPU.
--
Happy Coding!
Morten Wennevik [C# MVP]
Sep 5 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2196
by: Craig Buchanan | last post by:
when i try to open a file that has been compressed with GZipStream (source from microsoft's 101 Samples for Visual Studio 2005), i get an "invalid archive" message from winzip, zipgenius and xp's compressed folder app. using DeflateStream resulted in the same error. am i missing something simple? thanks, Craig Buchanan
1
3091
by: Robinson | last post by:
I'm trying to work out how to decompress a GZipStream. I'm not sure how to allocate the destination buffer, given that I cannot query the stream to find out the total decompressed size of the item. A hack here http://www.geekpedia.com/tutorial191_Unzipping-compressed-files-using-GZipStream.html shows that you can get the last 4 bytes of the stream and convert it into an integer "size". But this doesn't seem right to me. I could also...
1
2150
by: prince.matt | last post by:
Hi, I am using GZipStream to compress a simple text file approx 24MB in size. I am finding that the destination "compressed" file is several MB larger than the source file (31MB). The code is as follows: Dim sourcefile As FileStream = File.OpenRead(inFileName) Dim destFile As FileStream = File.Open(outFilename, FileMode.Create, FileAccess.Write)
1
3638
by: hakan.thornqvist | last post by:
I am using GZipStream for compression, and the size and contents of the compressed byte array differs from time to time, using the same source. byte uncompressed = Enc.GetBytes(xmlNode.OuterXml); byte compressed = compress(uncompressed); private byte compress(byte uncompressed) { MemoryStream ms = new MemoryStream();
1
10208
by: sedwick | last post by:
I'm using .NET 2.0.50727 in VS 2005 Pro, ENU Service Pack 1 (KB926601). I've been experimenting with the System.IO.Compression classes GZipStream and DeflateStream, and I found it interesting to note that they'll create different-sized compressed files depending on the procedure you use to take in the source file's data. With one, almost every file I feed them for compression ends up significantly bigger. That code is below, almost verbatim...
0
1251
by: pierreuk | last post by:
Hi All, I have been looking at the best way of doing this but can't find a solution that doesn't involve using bytes array. Basically I want to wrap a XmlTextWriter in a GZipStream so I produce a GZipped Xml file. I have been trying this: using (GZipStream gzipout = new GZipStream(File.Create(Path), CompressionMode.Compress)) {
4
3459
by: Arnie | last post by:
Folks, The GZipStream class in .NET is throwing an exception during an OS unseal/power up (first boot experience). The .NET system is fully up as other apps seem to run fine.
0
1337
by: DR | last post by:
Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped file) then loading 50GB unzipped data? im using System.IO.Compression.GZipStream and its not maxing out the cpu while loading the gzip data! Im using the default buffer of the stream that i open on the 20GB gzipped file and pass it into the GZipStream ctor. then System.IO.Compression.GZipStream takes an hour! when just loading 50GB file of data takes a few minutes!
2
4985
by: =?Utf-8?B?TWFydGluIE1hZHJlemE=?= | last post by:
Hi, i've got problems with CryptoStream and GZipStream. Someone tried that? Or got any idea why this wont work? i tried every combination, first i used CryptoStream, after that GZipStream. Or first i create a file with GZipStream and crypt that file. But become back the orginal is impossible... I hope anyone got an idea how this could work.
0
9673
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9522
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10448
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10217
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10167
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10003
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
5566
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
3730
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2922
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.