GZipStream Decompression Failure

[=?Utf-8?B?QkRSaWNoYXJkc29u?=]

Hi,

I have been using the new GZipStream classes, and have been experiencing
problems when attemping to decompress files, which from experience, seem to
be failing when the original file size exceeds something like 64 MB.

For example, when I attempt to decompress a text file of size 1.18 MB to its
original size of 106 MB, I receive the following error message:

System.IO.IOExc eption: Insufficient system resources exist to complete the
requested service.
Source: mscorlib
StackTrace: " at System.IO.__Err or.WinIOError(I nt32 errorCode, String
maybeFullPath)\ r\n at System.IO.FileS tream.WriteCore (Byte[] buffer, Int32
offset, Int32 count)\r\n at System.IO.FileS tream.Write(Byt e[] array, Int32
offset, Int32 count)\r\n at %MethodName% in %SourceFilePath %:line 96"
TargetSite: {Void WinIOError(Int3 2, System.String)}

However, it works perfectly fine for files of an original size which is,
approximately, less than 64 MB in size.

Can anyone kindly provide any assistance as to whether there is a solution,
or whether there is just a flaw in the class?

Thanks

[Jon Skeet [C# MVP]]

On Aug 30, 11:44 am, BDRichardson
<BDRichard...@d iscussions.micr osoft.comwrote:

I have been using the new GZipStream classes, and have been experiencing
problems when attemping to decompress files, which from experience, seem to
be failing when the original file size exceeds something like 64 MB.

<snip>

Can anyone kindly provide any assistance as to whether there is a solution,
or whether there is just a flaw in the class?

Well, you haven't shown any of the code you're using to decompress.

Could you produce a short but complete program which demonstrates the
problem?
See http://pobox.com/~skeet/csharp/complete.html for what I mean by
that.

Jon

[=?Utf-8?B?QkRSaWNoYXJkc29u?=]

The Method:

public static void DecompressFile( FileInfo FileToDecompres s, String
DestinationDire ctory, Boolean DeleteOriginalF ile)
{
FileStream fsSource = null;
FileStream fsDestination = null;
GZipStream compressedStrea m = null;

try
{
Byte[] buffer;

fsSource = new FileStream( FileToDecompres s.FullName, FileMode.Open,
FileAccess.Read , FileShare.Read) ;

// The original file size may be obtained from the footer of the
compressed file
buffer = new Byte[ 4];
fsSource.Positi on = Convert.ToInt32 ( fsSource.Length ) - 4;
fsSource.Read( buffer, 0, 4);
Int32 _OriginalFileSi ze = BitConverter.To Int32( buffer, 0);

// Read the decompressed file contents
buffer = new Byte[ _OriginalFileSi ze];
fsSource.Positi on = 0;

compressedStrea m = new GZipStream( fsSource, CompressionMode .Decompress,
true);
compressedStrea m.Read( buffer, 0, _OriginalFileSi ze);
compressedStrea m.Flush();
compressedStrea m.Close();

// Write the decompressed data to a new file
String _OriginalFileNa me = FileToDecompres s.Name.Substrin g( 0,
FileToDecompres s.Name.Length - 5);
fsDestination = new FileStream( DestinationDire ctory +
Path.DirectoryS eparatorChar + _OriginalFileNa me, FileMode.Create ,
FileAccess.Writ e, FileShare.Write );
fsDestination.W rite( buffer, 0, _OriginalFileSi ze);
fsDestination.C lose();
fsSource.Close( );

System.Diagnost ics.Debug.Write Line( String.Format( "The file {0} was
decompressed as {1}.", FileToDecompres s.Name, _OriginalFileNa me));

if( DeleteOriginalF ile)
{
FileToDecompres s.Delete();
System.Diagnost ics.Debug.Write Line( String.Format( "The following file
was deleted: {0}", FileToDecompres s.Name));
}
}
catch( System.Exceptio n ex)
{
System.Diagnost ics.Debug.Write Line( String.Format( "The following
exception occurred within the application:\n{ 0}", ex.Message));
}
finally
{
if( fsSource != null) fsSource.Close( );
if( compressedStrea m != null)
{
compressedStrea m.Flush();
compressedStrea m.Close();
}

if( fsDestination != null) fsDestination.C lose();
}
}

[Jon Skeet [C# MVP]]

On Aug 30, 12:08 pm, BDRichardson
<BDRichard...@d iscussions.micr osoft.comwrote:

The Method:

Please see http://pobox.com/~skeet/csharp/incomplete.html

However, it looks to me like the problem is that you're trying to read
the whole thing in one go. Aside from anything else, this is
horrendously inefficient in terms of memory. It's much better to use a
buffer, read into that, write from it, then repeat until you're done.

I've got a method in my MiscUtil library for copying a whole stream.
See
http://pobox.com/~skeet/csharp/miscutil

It won't *quite* work out of the box in this case because you've
appended the original size to the end of the compressed stream. Is
this absolutely required? If you really need the information, could
you not put it at the *start* of the stream rather than the end? (That
way it can be skipped over very easily, rather than giving a
compressed stream which has invalid data at the end.)

Jon

[Marc Gravell]

Yowser! Buffer bomb!

You shouldn't be allocating a buffer for the entire file (which could
be huge), but rather just create a small buffer and loop over the
data - something like (untested):

static void Main() {
using(Stream inFile = File.OpenRead(" in.gzip"))
using(GZipStrea m zip = new GZipStream(inFi le,
CompressionMode .Decompress))
using (Stream outFile = File.OpenWrite( "out.whatever") ) {
CopyStream(zip, outFile);
outFile.Close() ;
}
}
static long CopyStream(Stre am source, Stream destination) {
if (source == null) throw new ArgumentNullExc eption("source" );
if (!source.CanRea d) throw new ArgumentExcepti on("Cannot read
from source");
if (destination == null) throw new
ArgumentNullExc eption("destina tion");
if (!destination.C anWrite) throw new ArgumentExcepti on("Cannot
write to destination");
const int BUFFER_SIZE = 4096;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
long totalBytesRead = 0;
while ((bytesRead = source.Read(buf fer, 0, BUFFER_SIZE)) 0)
{
destination.Wri te(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
}
destination.Flu sh();
return totalBytesRead;
}

Marc

[=?Utf-8?B?QkRSaWNoYXJkc29u?=]

KABOOOOM!

OK, shoot me now!

I'm sure its not that obvious that I'm new to Streams... yeah right!
Although I didn't & don't totally understand streams, I do recall when I
first started carving the code, I suspected that it might be dumping the
whole buffer at once. Obvious really when you take a careful look at the
code!

I've altered it slightly so that it now only reads and writes a small buffer
at a time, and it works a treat.

Many thanks guys, you've given me the slap I needed!

[Marc Gravell]

[being new to streams]
Please rest assured that I also learnt this the hard way. No doubt Jon
or one of the other stalwarts set me on the right path - sorry if it
sounded condascending, it wasn't my intent - just to stress that a
radical change of direction was needed ;-p

Many thanks guys, you've given me the slap I needed!

No problem

Marc