473,700 Members | 3,099 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How do I inflate compressed data from an asynchronous socket?

Hi, I'm writing my own implementation of the Gnutella P2P protocol
using C#. I have implemented it using BeginReceive and EndReceive
calls so as not to block when waiting for data from the supernode.
Everything I have written works fine sending and receiving
uncompressed data. But now I want to implement compression using the
deflate algorithm as the Gnutella protocol accepts:
Accept-Encoding: deflate
Content-Encoding: deflate
in the HTTP connection headers.

At this stage I can't get the receive to work properly. I need this
first before I can tell if the send is working or not.

First I attempted to use the existing DeflateStream(n ew
NetworkStream(s ocket)) and call BeginRead and EndRead, but I kept
getting errors. (Can't remember what they were, this was a few days
ago now).

So I then tried using the SharpZipLib with a little bit of success. I
initially used the InflaterInputSt ream(new NetworkStream(s ocket)) but
it was causing too many exceptions when the supernode cut the
connection during the receive. The problem with this is, because I am
using asynchronous callbacks, the exceptions are not caught in my
code.

So I implemented it using the existing socket calls and passed the
received data to an Inflater and called SetInput and Inflate. With a
large receive buffer size I can sometimes uncompress some data from
the supernode. However when I shrink the buffer to a more manageable
size, I start getting "broken uncompressed block" exceptions.

I believe the problem is that I am trying to uncompress an incomplete
packet of data. But I am not sure how else to do it, because I have to
maintain a permanent connection to the supernode, so the data never
stops coming, and there is no way to tell how big individual messages
will be until I have them and have processed them.

I don't think a MemoryStream is the right way to go either, because if
I buffer all data from the session, due to the permanent connection,
eventually I will eat up all my memory.

Does anyone know how to inflate segmented chunks of data without
knowledge of the packet size?

Thanks,
Pat.

Jun 11 '07 #1
6 6996
Keeping in mind that I don't have first-hand knowledge of your specific
problem...

On Sun, 10 Jun 2007 21:34:48 -0700, Pat B <to****@hotmail .comwrote:
[...]
First I attempted to use the existing DeflateStream(n ew
NetworkStream(s ocket)) and call BeginRead and EndRead, but I kept
getting errors. (Can't remember what they were, this was a few days
ago now).
I'm not clear on why you would call BeginRead, or any kind of read, using
this mechanism. Once you've successfully wrapped the socket in a
NetworkStream, and passed that NetworkStream to DeflateStream, I would
expect that DeflateStream would at that point take over the network i/o by
accessing the NetworkStream directly. Of course, you would want this to
happen in its own thread so as to not block your GUI.

What is it that you expect to be able to do when your receive callback
passed to BeginRead is called?
So I then tried using the SharpZipLib with a little bit of success. I
initially used the InflaterInputSt ream(new NetworkStream(s ocket)) but
it was causing too many exceptions when the supernode cut the
connection during the receive. The problem with this is, because I am
using asynchronous callbacks, the exceptions are not caught in my
code.
You can broaden the chances of getting useful information by defining
terms not specific to C# or .NET Framework (the only real common ground
you are assured of here). For example, what's a "supernode" and how is it
relevant to your question? I assume it has something to do with the
Gnutella protocol, but that's not something I've spent time studying.

As far as the exceptions go, I have the same question as above: if you are
passing the NetworkStream to another component, why would you expect to
handle the receive yourself at all?
So I implemented it using the existing socket calls and passed the
received data to an Inflater and called SetInput and Inflate. With a
large receive buffer size I can sometimes uncompress some data from
the supernode. However when I shrink the buffer to a more manageable
size, I start getting "broken uncompressed block" exceptions.
I can't speak specifically about the library you're using for
decompression, but it may assume that you are passing a complete
compressed block of data to it in each operation. So if you don't
accumulate all of your data first before passing it to the decompressor, I
certainly could see why you'd see it complain about incomplete (or
"broken") blocks.
I believe the problem is that I am trying to uncompress an incomplete
packet of data. But I am not sure how else to do it, because I have to
maintain a permanent connection to the supernode, so the data never
stops coming, and there is no way to tell how big individual messages
will be until I have them and have processed them.
I don't really know. If the Gnutella protocol doesn't provide a way to
tell you in advance how long a single compressed block of data will be,
nor does it have a mechanism for delimiting a single compressed block of
data, _and_ the decompressor you're using does not accept partial blocks
of data, nor does it have a mechanism for reading a stream of data and
periodically emitting decompressed blocks as they are completed, I don't
really see a solution to your problem.

If you find a way to eliminate just one of the above factors, then I
believe that would be your solution.

It seems to me that, based on a couple of the methods you mentioned
(specifically, the two tht involve passing a NetworkStream to something),
I would think that you should not have any problem as long as the library
can deal with somehow telling you when it's finished a block. It would do
this either by continuing to run, but somehow informing you of a completed
block (for example, raising a .NET event), or it could only read the
stream up to the end of the block and then complete, at which point you
would create a new decompressor instance using the same NetworkStream
object so that it can start reading from where the last one left off.

But if your decompressors get confused when you provide a stream that's
too large, I don't have any ideas.

Finally, one thing to check on: you talking about "message" and "packet
size", but assuming you're dealing with TCP there's no such thing at the
network protocol level. If you are trying to receive the data manually
and don't take into account the fact that when you read from a TCP socket,
you can wind up getting back anywhere between 1 byte and the total number
of bytes sent since the previous byte most recently received, then it is
possible and quite likely to wind up with corrupted data.

Make sure you have this last point addressed before you move on to the
more specific stuff. :)

Pete
Jun 11 '07 #2
Thanks Pete.

When I was receiving the raw data I was not using NetworkStream, I was
just using the socket itself. I was calling BeginReceive and
EndReceive because I don't want the read to block. It is much simpler
to make use of the asynchronous calls than to implement threads or
thread pools.

Example in pseudo code with fictional functions:

OnConnect
{
socket.BeginRec eive(tempbuffer[256], OnReceive)
}

OnReceive
{
read = socket.EndRecei ve()
readData += CopyLeftBytes(t empbuffer, read)
ProcessData(rea dData)
socket.BeginRec eive(tempbuffer[256], OnReceive)
}

ProcessData
{
while (true)
{
msgLength = ReadMessageLeng thFrom(readData )
if (msgLength == 0 || msgLength readData.Length )
{
// wait for more data
return
}

message = CopyLeftBytes(r eadData, msgLength)
RemoveLeftBytes (readData, msgLength)
// process single message
}
}

This works without compression. The size of the messages coming from
the supernode (server) is specified within the data itself. The
problem with adding compression is that you first have to decompress
the data before you can read out the message length.

First I tried:

OnConnect
{
readStream = new SharpZipLib.Inf laterInputStrea m(new
NetworkStream(s ocket))
readStream.Begi nRead(tempbuffe r[256], OnReceive)
}

OnReceive
{
read = readStream.EndR ead()
readData += CopyLeftBytes(t empbuffer, read)
ProcessData(rea dData)
readStream.Begi nRead(tempbuffe r[256], OnReceive)
}

ProcessData
{
...
}

This works... when everything works. The problem is when the supernode
(server) terminates the connection while you are waiting for data, or
if some invalid data is sent by the supernode, the exception is thrown
on a different thread (initiated by windows?) and so I cannot catch it
an clean up. This is because the Begin and End are asynchronous and
the exception is being thrown in the middle.

Using the dotNET DeflateStream in the place of
SharpZipLib.Inf laterInputStrea m blew up with the error: "Block length
does not match with its complement."

So then I tried this:

OnConnect
{
inflater = new SharpZipLib.Inf later()
socket.BeginRec eive(tempbuffer[256], OnReceive)
}

OnReceive
{
read = readStream.EndR ead()
inflater.SetInp ut(CopyLeftByte s(tempbuffer, read))
read = inflater.Inflat e(inflateBuffer[1024])
readData += CopyLeftBytes(i nflateBuffer, read)
ProcessData(rea dData)
readStream.Begi nRead(tempbuffe r[256], OnReceive)
}

ProcessData
{
...
}

This works only when tempBuffer is large enough to receive the entire
packet before passing it into the inflater. If not it blows up on the
inflater.Inflat e(...) line with the "broken uncompressed block" error.

- So in each case I am catering for different size blocks of data
coming from the socket.
- I have no way of knowing how much data to expect until after I have
decompressed it.
- I don't want to have to implement using thread pools and blocking
calls.
- Using streams may be the solution, only if there is a good way of
catching the exceptions from the asynchronous reads.

Maybe I am supposed to expect the error from the inflater and take
that as an indication to wait for more data before trying to
decompress again?

Thanks,
Pat.

Jun 11 '07 #3
Maybe I am supposed to expect the error from the inflater and take
that as an indication to wait for more data before trying to
decompress again?
Nope. Doesn't work. Same error.

Jun 11 '07 #4
Again, keeping mind my lack of first-hand knowledge with the exact
situation...

On Mon, 11 Jun 2007 02:25:29 -0700, Pat B <to****@hotmail .comwrote:
When I was receiving the raw data I was not using NetworkStream, I was
just using the socket itself. I was calling BeginReceive and
EndReceive because I don't want the read to block. It is much simpler
to make use of the asynchronous calls than to implement threads or
thread pools.
Yes, I agree that the asynchronous methods of the Socket class are simple
and convenient. Do keep in mind that you are still using the thread pool
though; you just do it implicitly in this case, rather than explicitly.
[...]
First I tried:

OnConnect
{
readStream = new SharpZipLib.Inf laterInputStrea m(new
NetworkStream(s ocket))
readStream.Begi nRead(tempbuffe r[256], OnReceive)
}

OnReceive
{
read = readStream.EndR ead()
readData += CopyLeftBytes(t empbuffer, read)
ProcessData(rea dData)
readStream.Begi nRead(tempbuffe r[256], OnReceive)
}

ProcessData
{
...
}

This works... when everything works. The problem is when the supernode
(server) terminates the connection while you are waiting for data, or
if some invalid data is sent by the supernode, the exception is thrown
on a different thread (initiated by windows?) and so I cannot catch it
an clean up. This is because the Begin and End are asynchronous and
the exception is being thrown in the middle.
I see. So the decompressor (SharpZipLib) exposes a stream from which you
can read. IMHO, it seems to me that SharpZipLib ought to provide a
mechanism for cleanly dealing with errors on the NetworkStream you pass to
it. It could do this in a variety of ways, but I can't comment on whether
or how it does, never having used it.

If it does in fact provide no mechanism, even though it should, then one
thing you might consider is writing your own intermediate Stream class to
handle the errors. You would have this class simply forward all the basic
calls to the NetworkStream it contains, so that the SharpZipLib is calling
your class rather than NetworkStream directly. That way if and when an
exception or other error occurs, you have a chance to deal with it before
propogating it back to the SharpZipLib class.
Using the dotNET DeflateStream in the place of
SharpZipLib.Inf laterInputStrea m blew up with the error: "Block length
does not match with its complement."

So then I tried this:

OnConnect
{
inflater = new SharpZipLib.Inf later()
socket.BeginRec eive(tempbuffer[256], OnReceive)
}

OnReceive
{
read = readStream.EndR ead()
inflater.SetInp ut(CopyLeftByte s(tempbuffer, read))
read = inflater.Inflat e(inflateBuffer[1024])
readData += CopyLeftBytes(i nflateBuffer, read)
ProcessData(rea dData)
readStream.Begi nRead(tempbuffe r[256], OnReceive)
}

ProcessData
{
...
}

This works only when tempBuffer is large enough to receive the entire
packet before passing it into the inflater. If not it blows up on the
inflater.Inflat e(...) line with the "broken uncompressed block" error.
This sounds to me like the same error you get using the "dotNET
DeflateStream" mechanism: essentially, you've tried to decompress data in
one shot without having the complete data available to decompress. No
surprise then that doesn't work.
- So in each case I am catering for different size blocks of data
coming from the socket.
Yes, it seems to me that you're correctly handing the basic network
receiving correctly.
- I have no way of knowing how much data to expect until after I have
decompressed it.
This seems to me to be a fundamental mistake in the Gnutella protocol,
which requires you to make at least some extra effort beyond that which
should be necessary. I suspect you can get SharpZipLib to do what you
want, but in the worst case if you couldn't, the Gnutella protocol would
put you in a position where you'd have to do all the decompressing
yourself in your own code so that you can work around not knowing how much
data you're expecting.

Whether this limitation really exists in Gnutella, I can't say. It may be
that it does, or it may be that you're just overlooking something in the
protocol that would address this.
- I don't want to have to implement using thread pools and blocking
calls.
Well, as I said, you are already using the thread pool implicitly. Still,
it doesn't seem to me that you would wind up in a position where you have
to implicitly do some threading, given the apparent design of the
libraries you're trying touse.
- Using streams may be the solution, only if there is a good way of
catching the exceptions from the asynchronous reads.
See above: I suspect that if you simply write your own Stream-derived
wrapper for the NetworkStream, you can intercept the exceptions before
they get to the SharpZipLib class, allowing you to handle the exceptions
however you see fit.
Maybe I am supposed to expect the error from the inflater and take
that as an indication to wait for more data before trying to
decompress again?
You could do that as well. I don't know why it wouldn't, even though you
say it doesn't. When you tried this, are you sure that you retained the
data at the beginning of your array for future attempts? It seems to me
that if you keep trying to decompress the same data, except that new data
is appended between each attempt, eventually you would have enough data to
decompress and it would succeed. It doesn't seem like a very efficient
approach to the problem though, so I think you should explore the
Stream-derived wrapper solution first.

Pete
Jun 11 '07 #5
Ok, thanks for all your suggestions Pete. I'll give some of them a go
and see what I can come up with.

Thanks,
Pat.

Jun 11 '07 #6
On Jun 11, 2:25 pm, Pat B <to_...@hotmail .comwrote:
Thanks Pete.

When I was receiving the raw data I was not using NetworkStream, I was
just using the socket itself. I was calling BeginReceive and
EndReceive because I don't want the read to block. It is much simpler
to make use of the asynchronous calls than to implement threads or
thread pools.

Example in pseudo code with fictional functions:

OnConnect
{
socket.BeginRec eive(tempbuffer[256], OnReceive)

}

OnReceive
{
read = socket.EndRecei ve()
readData += CopyLeftBytes(t empbuffer, read)
ProcessData(rea dData)
socket.BeginRec eive(tempbuffer[256], OnReceive)

}

ProcessData
{
while (true)
{
msgLength = ReadMessageLeng thFrom(readData )
if (msgLength == 0 || msgLength readData.Length )
{
// wait for more data
return
}

message = CopyLeftBytes(r eadData, msgLength)
RemoveLeftBytes (readData, msgLength)
// process single message
}

}

This works without compression. The size of the messages coming from
the supernode (server) is specified within the data itself. The
problem with adding compression is that you first have to decompress
the data before you can read out the message length.

First I tried:

OnConnect
{
readStream = new SharpZipLib.Inf laterInputStrea m(new
NetworkStream(s ocket))
readStream.Begi nRead(tempbuffe r[256], OnReceive)

}

OnReceive
{
read = readStream.EndR ead()
readData += CopyLeftBytes(t empbuffer, read)
ProcessData(rea dData)
readStream.Begi nRead(tempbuffe r[256], OnReceive)

}

ProcessData
{
...

}

This works... when everything works. The problem is when the supernode
(server) terminates the connection while you are waiting for data, or
if some invalid data is sent by the supernode, the exception is thrown
on a different thread (initiated by windows?) and so I cannot catch it
an clean up. This is because the Begin and End are asynchronous and
the exception is being thrown in the middle.

Using the dotNET DeflateStream in the place of
SharpZipLib.Inf laterInputStrea m blew up with the error: "Block length
does not match with its complement."

So then I tried this:

OnConnect
{
inflater = new SharpZipLib.Inf later()
socket.BeginRec eive(tempbuffer[256], OnReceive)

}

OnReceive
{
read = readStream.EndR ead()
inflater.SetInp ut(CopyLeftByte s(tempbuffer, read))
read = inflater.Inflat e(inflateBuffer[1024])
readData += CopyLeftBytes(i nflateBuffer, read)
ProcessData(rea dData)
readStream.Begi nRead(tempbuffe r[256], OnReceive)

}

ProcessData
{
...

}

This works only when tempBuffer is large enough to receive the entire
packet before passing it into the inflater. If not it blows up on the
inflater.Inflat e(...) line with the "broken uncompressed block" error.

- So in each case I am catering for different size blocks of data
coming from the socket.
- I have no way of knowing how much data to expect until after I have
decompressed it.
- I don't want to have to implement using thread pools and blocking
calls.
- Using streams may be the solution, only if there is a good way of
catching the exceptions from the asynchronous reads.

Maybe I am supposed to expect the error from the inflater and take
that as an indication to wait for more data before trying to
decompress again?

Thanks,
Pat.
=============== =============== ===========

Hi ,

I also want to inflate compressed data from an asynchronous socket but
I would like to inflate ones data is received by BeginReceive and
EndReceive. Is that feasible ? or I have to replace this socket
calls with InflaterInputSt ream - BeginRead and EndRead ?

Thanks
Jul 17 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
6212
by: Niels Johansen | last post by:
Hello, When using the asynchronous read method in the BufferedStream class, , it seems to me that it blocks like the normal synchronous read method. Why is it so? Why does the BufferedStream.BeginRead() not behave similar to the NetworkStream.BeginRead() ?? The following short program illustrates my problem.: using System;
5
11703
by: mscirri | last post by:
The code below is what I am using to asynchronously get data from a PocketPC device. The data comes in fine in blocks of 1024 bytes but even when I send no data from the PocketPC constant blocks of 1024 with all values set to Null arrive. Other than examine a block of 1024 to see if the entire block is null, is there any other way to determine if , say a chat message "Hi Charlie" has been received completely?
9
3599
by: Macca | last post by:
Hi, I have a synchronous socket server which my app uses to read data from clients. To test this I have a simulated client that sends 100 byte packets. I have set up the socket server so that its buffer is bigger than this. I did this expecting the data to be read in one pass.
2
6874
by: Macca | last post by:
My app has an asynchronous socket server. It will have 20 clients connected to the server. Each client sends data every 500 millisecondsThe Connections once established will not be closed unless there is a problem with the connection. I need to know which client has sent the incoming data as each client has its own buffer on my "server" app. I am using the standard asynch socket code from MSDN to listen for connections and they...
0
4679
by: Macca | last post by:
Hi, I am writing an asychronous socket server to handle 20+ simulataneous connections. I have used the example in MSDN as a base. The code is shown at end of question. Each connection has a number of different types of data coming in. I have a databuffer for each type of data coming in.
2
7565
by: Ben | last post by:
I need to send data from a client to a server. In the server code I have: s = ServerSocket.Accept() If (s.Connected = False) Then MsgBox("Unable to connect", , "Server Error") Exit Sub End If While (s.Connected = True)
4
3602
by: Engineerik | last post by:
I am trying to create a socket server which will listen for connections from multiple clients and call subroutines in a Fortran DLL and pass the results back to the client. The asynchronous socket client and asynchronous socket server example code provided in the .NET framework developers guide is a great start but I have not dealt with sockets before and I am struggling with something. From what I can tell the sample server code ...
2
3428
by: Nicolas Le Gland | last post by:
Hello everyone here. This is my first post in this newsgroup, I hope I won't be to much off-topic. Feel free to redirect me to any better group. I am getting strange timing issues when failing to asynchronously connect sockets on closed or filtered ports, but I'm quite unsure if this is a PHP issue or my misunderstanding, as it seems that socket streams only wrap around <sys/socket.h>.
0
8712
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8639
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9203
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
8952
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
7794
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6555
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5895
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4395
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3082
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.