GZip Compression :(

[Carlo Razzeto]

Hello there,

I'm having an odd issue with GZIP compression (having followed example code
found on MSDN). Basically, after running through the compression routine I
end up with a byte array several times larger than the source text file,
full of zero data. Below is the code used to do the compression, it's a part
of a web service to retreive a file, there's a compress option prior to
base64 encoding the data. In the following code all undeclared variables you
see are properties, compress repersents a compress attribute specified in
the xml request, FileName is a relitive path to the file on the server
inside the webroot.

Response.ContentType = "text/xml"
If Not File.Exists(Server.MapPath(FileName)) Then
Throw New GetBinaryFileException(FileName,
GetBinaryFileException.GetBinaryFileError.FileNotF ound)
End If

Dim FileData() As Byte = Nothing
Dim FStream As New FileStream(Server.MapPath(FileName),
FileMode.Open, FileAccess.Read, FileShare.ReadWrite)
If Compress Then
Dim TempData(FStream.Length - 1) As Byte
FStream.Read(TempData, 0, FStream.Length)
Dim MStream As New MemoryStream
Dim Compressor As New GZipStream(MStream,
CompressionMode.Compress, True)
Compressor.Write(TempData, 0, TempData.Length)

ReDim FileData(MStream.Length - 1)
Dim BytesRead As Integer = MStream.Read(FileData, 0,
MStream.Length)
MStream.Close()
MStream.Dispose()
Compressor.Close()
Compressor.Dispose()
Else
ReDim FileData(FStream.Length - 1)
FStream.Read(FileData, 0, FStream.Length)
End If
FStream.Close()
FStream.Dispose()

Dim Base64 As String = Convert.ToBase64String(FileData)

Dim FileDataNode As XmlNode =
XmlExchangeLib.GetOrSetXmlNode("FileData", Root)
XmlExchangeLib.AddAttributeWithValue(FileDataNode, "Compressed",
Compress.ToString().ToLower())
FileDataNode.InnerText = Base64
XmlResponse.Save(Response.OutputStream)

[Marc Gravell]

First, you need to make sure that you close the zip-stream (compressor)
before looking at the memory-stream - it won't have finished writing yet;
second, you then either need to rewind the memory stream, or just use
ToArray() to get the full contents.
Third - Read (on the file stream) is not strictly guaranteed to get
everything - and even if it did it isn't very efficient. But
File.ReadAllBytes would be a more reliable way of reading the entire file at
once.

You might also be allocating the FileData array one too short - I'm not sure
(VB...)

Marc

[Carlo Razzeto]

The last point is the one I know for a fact is fine, in VB you need to
declare it to length -1. But I'll take a look at the rest of the points.
Thanks very much for your thoughts, most helpful...
"Marc Gravell" <ma**********@gmail.comwrote in message
news:eE**************@TK2MSFTNGP02.phx.gbl...

First, you need to make sure that you close the zip-stream (compressor)
before looking at the memory-stream - it won't have finished writing yet;
second, you then either need to rewind the memory stream, or just use
ToArray() to get the full contents.
Third - Read (on the file stream) is not strictly guaranteed to get
everything - and even if it did it isn't very efficient. But
File.ReadAllBytes would be a more reliable way of reading the entire file
at once.

You might also be allocating the FileData array one too short - I'm not
sure (VB...)

Marc

[Carlo Razzeto]

Thanks for the advice, I swithed to autoclosing the zip stream and using
ToArray on the memory stream and it seems to be pulling bytes. Now my only
consern is I'm getting back a byte array much larger than my original 26
byte text file :(

"Marc Gravell" <ma**********@gmail.comwrote in message
news:eE**************@TK2MSFTNGP02.phx.gbl...

First, you need to make sure that you close the zip-stream (compressor)
before looking at the memory-stream - it won't have finished writing yet;
second, you then either need to rewind the memory stream, or just use
ToArray() to get the full contents.
Third - Read (on the file stream) is not strictly guaranteed to get
everything - and even if it did it isn't very efficient. But
File.ReadAllBytes would be a more reliable way of reading the entire file
at once.

You might also be allocating the FileData array one too short - I'm not
sure (VB...)

Marc

[=?Utf-8?B?RmFtaWx5IFRyZWUgTWlrZQ==?=]

Just to make sure...

You are talking about "before" the step of going to base64, correct? The
base 64 step will bloat the string by a factor of 1.37 plus header data, if I
recall correctly.

"Carlo Razzeto" wrote:

Thanks for the advice, I swithed to autoclosing the zip stream and using
ToArray on the memory stream and it seems to be pulling bytes. Now my only
consern is I'm getting back a byte array much larger than my original 26
byte text file :(

"Marc Gravell" <ma**********@gmail.comwrote in message
news:eE**************@TK2MSFTNGP02.phx.gbl...

First, you need to make sure that you close the zip-stream (compressor)
before looking at the memory-stream - it won't have finished writing yet;
second, you then either need to rewind the memory stream, or just use
ToArray() to get the full contents.
Third - Read (on the file stream) is not strictly guaranteed to get
everything - and even if it did it isn't very efficient. But
File.ReadAllBytes would be a more reliable way of reading the entire file
at once.

You might also be allocating the FileData array one too short - I'm not
sure (VB...)

Marc

[Carlo Razzeto]

Raw byte array size (prior to conversion to base64 string). I read in 26
bytes and typically get back 132 bytes worth of "compressed" data.

"Family Tree Mike" <Fa************@discussions.microsoft.comwrote in
message news:C6**********************************@microsof t.com...

Just to make sure...

You are talking about "before" the step of going to base64, correct? The
base 64 step will bloat the string by a factor of 1.37 plus header data,
if I
recall correctly.

"Carlo Razzeto" wrote:

>Thanks for the advice, I swithed to autoclosing the zip stream and using
ToArray on the memory stream and it seems to be pulling bytes. Now my
only
consern is I'm getting back a byte array much larger than my original 26
byte text file :(

"Marc Gravell" <ma**********@gmail.comwrote in message
news:eE**************@TK2MSFTNGP02.phx.gbl...

First, you need to make sure that you close the zip-stream (compressor)
before looking at the memory-stream - it won't have finished writing
yet;
second, you then either need to rewind the memory stream, or just use
ToArray() to get the full contents.
Third - Read (on the file stream) is not strictly guaranteed to get
everything - and even if it did it isn't very efficient. But
File.ReadAllBytes would be a more reliable way of reading the entire
file
at once.

You might also be allocating the FileData array one too short - I'm not
sure (VB...)

Marc

[Marc Gravell]

I wouldn't bother compressing 26 bytes... gzip itself has header overhead
etc. This also isn't enough space to actually get many useful compression
opportunities. Finally, it depends on what the data is: if it is fairly
random (a complex image, a security token, etc) then it simply won't
compress.

Marc

[Marc Gravell]

Demo; outputs "125"; compression just isn't going to help you with very
short inputs:

using(MemoryStream dest = new MemoryStream()) {
using(GZipStream zip = new GZipStream(dest,
CompressionMode.Compress, true))
using(StreamWriter writer = new StreamWriter(zip)) {
writer.Write("Hi hi hi");
writer.Close();
zip.Close();
}
Console.WriteLine(dest.Length);
}

Marc

[Carlo Razzeto]

Ah, yeah hadn't been considering the compression headers. Thanks for
reminding me of that, so that makes sense. IRL this code isn't going to be
used to compress 25 byte files, more like several K to an M or two pdf files
so it should be fine. Thanks,

Carlo

"Marc Gravell" <ma**********@gmail.comwrote in message
news:e2*************@TK2MSFTNGP04.phx.gbl...

Demo; outputs "125"; compression just isn't going to help you with very
short inputs:

using(MemoryStream dest = new MemoryStream()) {
using(GZipStream zip = new GZipStream(dest,
CompressionMode.Compress, true))
using(StreamWriter writer = new StreamWriter(zip)) {
writer.Write("Hi hi hi");
writer.Close();
zip.Close();
}
Console.WriteLine(dest.Length);
}

Marc

[Marc Gravell]

One approach would be to use the first byte to indicate whether compression
is on (and what) - i.e. 0x00 = none, 0x01 = gzip, etc. I use this trick
quite happily; pick a cutoff under which you won't even bother trying to
compress... otherwise try compressing it and see if it got shorter (even
some non-trivial data gets longer when "compressed"). Worth consideration
perhaps... And in reverse check the first byte - if 0 return the rest of the
stream vanilla, if 1 the gzip, etc...

Marc