Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped file) then loading 50GB unzipped data? im using System.IO.Compr ession.GZipStre am and its not maxing out the cpu while loading the gzip data! Im using the default buffer of the

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compr ession.GZipStre am and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compr ession.GZipStre am takes an hour! when just loading 50GB file
of data takes a few minutes!

Sep 4 '08 #1

Subscribe Reply

2955

raylopez99

On Sep 4, 3:54*pm, "DR" <softwareengine er98...@yahoo.c omwrote:

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compr ession.GZipStre am and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compr ession.GZipStre am takes an hour! when just loading 50GB file
of data takes a few minutes!

I have no idea. Perhaps your resident virus checker is checking the
zipped file? Try turning it off.

BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! *THIS* is evil! Do no evil My ARSE.
Ridiculous.

RL

Sep 4 '08 #2

Roger Frost

"DR" <so************ *******@yahoo.c omwrote in message
news:eZ******** ******@TK2MSFTN GP02.phx.gbl...

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compr ession.GZipStre am and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compr ession.GZipStre am takes an hour! when just loading 50GB
file of data takes a few minutes!

Define "loading." Do you have 50+ GB of ram? But seriously, you're adding
a lot of extra processing when opening compressed data. With uncompressed
data, you're just retrieving as much as you can hold in memory (virtual or
otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a week
straight without hearing the same song twice), I wouldn't use a zip format.

-Roger Frost

Sep 5 '08 #3

Jon Skeet [C# MVP]

On Sep 5, 12:02*am, raylopez99 <raylope...@yah oo.comwrote:

<snip>

BTW who besides me has a beef with Google Gmail not accepting emails
that contain Zip attachments!?! **THIS* is evil! Do no evil My ARSE.
Ridiculous.

GMail works with zip attachments. However, it doesn't allow certain
file types to be sent/received, even in zip files.
See http://mail.google.com/support/bin/a...90&topic=12842

Jon

Sep 5 '08 #4

Jon Skeet [C# MVP]

On Sep 4, 11:54*pm, "DR" <softwareengine er98...@yahoo.c omwrote:

Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Compr ession.GZipStre am and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i open
on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Compr ession.GZipStre am takes an hour! when just loading 50GB file
of data takes a few minutes!

[Please learn to use a short subject line. The subject isn't meant to
hold your whole post.]

Can you provide a short but complete program to demonstrate your
problem? See http://pobox.com/~skeet/csharp/complete.html for what I
mean by that.

Jon

Sep 5 '08 #5

Eric B.

"Roger Frost" <fr*****@hotmai l.comwrote in message
news:A2******** *************** ***********@mic rosoft.com...

"DR" <so************ *******@yahoo.c omwrote in message
news:eZ******** ******@TK2MSFTN GP02.phx.gbl...
>Why is its substantialy slower to load 50GB of gzipped file (20GB gzipped
file) then loading 50GB unzipped data? im using
System.IO.Comp ression.GZipStr eam and its not maxing out the cpu while
loading the gzip data! Im using the default buffer of the stream that i
open on the 20GB gzipped file and pass it into the GZipStream ctor. then
System.IO.Comp ression.GZipStr eam takes an hour! when just loading 50GB
file of data takes a few minutes!

Define "loading." Do you have 50+ GB of ram? But seriously, you're
adding a lot of extra processing when opening compressed data. With
uncompressed data, you're just retrieving as much as you can hold in
memory (virtual or otherwise)... or simply copying data into memory.

Try opening a 5MB file, it should "load" about as fast as the 50GB
uncompressed file. That's because you're not actually "loading" the whole
50GB at once.

With the compressed file on the other hand, you are processing the whole
thing before any real progress can be made.

Personally if I had 50GB of data that I needed to compress (I don't have
that much on 2 hard drives by the way, and I can listen to music for a
week straight without hearing the same song twice), I wouldn't use a zip
format.

-Roger Frost

A 50Gb zip file is just a really, really bad idea. Try breaking it up into
smaller pieces.

Eric B.

Oct 6 '08 #6

Similar topics

2640

trouble measuring filesize of gzipped data

by: Marek Möhling | last post by:

My server (Apache/1.3.28 - PHP/4.3.3) is configured to receive gzipped data via: Header append Accept-Encoding "gzip, deflate" PHP is configured to send gzipped data via: php_value output_handler ob_gzhandler When downloading files from servers that send gzipped data as well, I'd like to get the filesize before and after my server decompresses them.

PHP

8850

Add a file to a compressed tarfile

by: Dennis Hotson | last post by:

Hi, I'm trying to write a function that adds a file-like-object to a compressed tarfile... eg ".tar.gz" or ".tar.bz2" I've had a look at the tarfile module but the append mode doesn't support compressed tarfiles... :( Any thoughts on what I can do to get around this?

Python

4591

Read a gzip file from inside a tar file

by: rohisingh | last post by:

I have a tar file. The content of the file are as following. rohits@sandman 12-08-04 $ tar tvf 20041208.tar drwxr-xr-x root/root 0 2004-12-08 21:39:19 20041208/ -rw-r--r-- root/root 1576 2004-12-08 21:39:19 20041208/README drwxr-xr-x root/root 0 2004-12-08 21:27:31 20041208/snapshot_01/ -rw-r--r-- was/was 103010606 2004-12-08 16:37:38 20041208/snapshot_01/tpv-2004 1208-1350.xml.gz

Python

6201

Large image load error that do work in C++

by: Sharon | last post by:

Iâ€™m wrote a small DLL that used the FreeImage.DLL (that can be found at http://www.codeproject.com/bitmap/graphicsuite.asp). I also wrote a small console application in C++ (unmanaged) that uses the DLL above. Now the application, together with the above DLLâ€™s is successfully loading a TIF image file (62992 x 113386 Pixels, Huffman RLE compression, 3200 x 3200 DPI resolution, binary colored (1 Bit Per Pixel), file on disk size 43.08...

C# / C Sharp

5519

Zipping a file programmatically in vs2005

by: ReidarT | last post by:

How can I zip (compress) a file in vs2005 with windows own zip-program? reidarT

Visual Basic .NET

2609

Javascript file compactor wanted

by: Aaron Gray | last post by:

Hi, Is there any tool that will concatinate JavaScript files compacting them spacewise and removing comment fields ? I have some sed script that sort of does the job but was wondering if there were any hopefully open source tools that will do the job. I don't want encryption or obverscation though.

Javascript

6969

Can sqlite read gzipped databases?

by: Paul Smith | last post by:

Hi, I'd like to read a series of sqlite database files that have already been gzipped and was wondering if this can be done on the fly. In other words, can I avoid explicitly unzipping the file into another file, but instead get an SQL connection to the zip file either directly (can't see an option to do this) or to an object in memory resulting from unzipping, eg. (hypothetically); import gzip from sqlite3 import dbapi2 as sqlite

Python

11513

How to unzip a file ?

by: fniles | last post by:

In VB.NET 2003 how can you programmatically unzip a ZIP file ? Thank you.

Visual Basic .NET

11141

How to read gzipped utf8 file in Python?

by: John Nagle | last post by:

I have a large (gigabytes) file which is encoded in UTF-8 and then compressed with gzip. I'd like to read it with the "gzip" module and "utf8" decoding. The obvious approach is fd = gzip.open(fname, 'rb',encoding='utf8') But "gzip.open" doesn't support an "encoding" parameter. (It probably should, for consistency.) Is there some way to do this? Is it possible to express "unzip, then decode utf8" via "codecs.open"?

Python

1476

reg zip file

by: =?Utf-8?B?QVZM?= | last post by:

Hi I've a requirement wherein I need to unzip a zipped folder programatically in .net. I've tried out System.IO.Compression namespace..but it writes the content into bytes which I dont need..I need the files to unzipped to a new folder.. HOw can I acheive it..are there any third party dlls for it..any samples

ASP.NET

9978

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

9818

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...

Windows Server

10913

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

10456

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...

General

9621

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

7165

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

6038

Windows Forms - .Net 8.0

by: adsilva | last post by:

A Windows Forms form does not have the event Unload, like VB6. What one acts like?

Visual Basic .NET

4670

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp

4267

How to add payments to a PHP MySQL app.

by: muto222 | last post by:

How can i add a mobile payment intergratation into php mysql website.

PHP