By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
446,259 Members | 1,346 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 446,259 IT Pros & Developers. It's quick & easy.

Determining the size of data being sent from Socket Server

Frinavale
Expert Mod 5K+
P: 9,731
I'm implementing a Silverlight application that uses Sockets to receive data that is pushed to it from a Socket Server. (Silverlight only supports the TCP protocol)

The Socket Server pushes a series of images to the Silverlight application which displays them. Every 2 seconds the next image in the series is sent until all of the images have been sent, at which time the Socket Server just starts over again.

Everything's working fine right now but that's only because I've made the receive buffer size for the SocketAsyncEventArgs large enough to accept the whole image file.

When the receive buffer size is smaller than the file more than one packet is sent from the server to the client until the whole file's been sent.

My question is: how do I know how large the file is that's being sent?


Thanks for your time,

-Frinny
Jul 13 '09 #1
Share this Question
Share on Google+
38 Replies


Plater
Expert 5K+
P: 7,872
Do you control the file sending? Send a datasize first?
Jul 13 '09 #2

Frinavale
Expert Mod 5K+
P: 9,731
Thanks Plater,

Yes I control both sides.

I have thought of that solution too but it seems that there should be another way to do this....

The BeginSend method even takes a parameter that indicates the size of the data being sent.

Why can't I see this value client side?

I was actually thinking about serializing each packet so that the size would be included at the beginning....I was reading an article on this last week but when I tried access it this morning it wasn't there any more.....

Maybe this serialization thing isn't the best idea?

-Frinny
Jul 13 '09 #3

Plater
Expert 5K+
P: 7,872
Well msdn.com and my local msdn library both seem to be broken when it comes to looking up that SocketAsyncEventArgs object.
If you have access to the socket, you can see how much data is currently on the stream though, that might help a bit?
I generally avoid using ASync stuff for these very "grouping" reasons.
Jul 13 '09 #4

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
I have no choice but to use the async stuff because Silverlight requires it.

I have access to the socket, and I can see how much data is currently on the stream but I have no idea how much data's supposed to be sent....how big the end result is supposed to be...so I have no idea how large to grow my "data collection" buffer or when the file actually ends and new data begins. I'm quite a newbie when it comes to Sockets so I might even be over looking a property/method that tells me this...but I did go through every single property/method intellisense displays looking for anything that could help me out.

-Frinny
Jul 13 '09 #5

Plater
Expert 5K+
P: 7,872
Well sending a header with the datasize is still my recomended action. This backwards client/server business boggles my mind (I prefer the polling process)
If you send the datasize first, your async thing can go "Ok I'm getting this many bytes" and you can keep appending the bytes to some sort of object until you have them all and create you image object from it.

How do you currently know what seperates image A from image B?
Jul 13 '09 #6

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
The whole image fits into the receive buffer right now because I've made the buffer large enough to accommodate the whole file... so it's pretty simple, when I receive the data (in the OnReceive event) I know that it's the whole file.

Right now, for practice, the Socket Server sending static image files (images that exist in a folder) to the client. This is how I was able to determine how big to make the buffer (just used the largest image size); however, in my real world application the socket server is going to be receiving raw byte data from a physical device and pushing that to the Silverlight application. So you can understand why it's important that I some how be able to tell how big the data is.

@Plater
I have no idea how to do this...what should I be keywords would you recommend that I research with to find the answer?

@Plater
Would you do this in a header or in a separate "init" push to the client?
Jul 13 '09 #7

Plater
Expert 5K+
P: 7,872
However you want really. You could steal the idea from how http works with a robust header system (you can include things like the name of the image, the size, the type, date modified, any meta-data, etc) or just something simple like the first 8 bytes represent a size and anything after it are bytes for the image.

What happens when a 2nd image is sent before the first one is done being received? Wouldn't the OnReceive events inter-mix with each other? There mose be some way to tell them apart? Perhaps the remote port from the Socket object could tell you which "image" the receive event belongs to?
Jul 13 '09 #8

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
I think I'll try sending the file information first and then send the file because at least I understand how that works...I don't know how to get at the header information using the SocketAsyncEventArgs right now and google's not helping me today.

@Plater
I've thought about this problem too but I'm not ready to solve it yet. That's why I have the timer for 2 seconds...and sometimes I extend this

When I'm debugging and have the client paused for more than 2 seconds the server throws an exception saying something or other...I'm not sure what happens when more than one file is sent within milliseconds of each other yet...I'm sure that the packets are going to be all mixed up. Asynchronous Socekts are pretty hard to work with.

I'll look at that later. First I just want to figure out how big the expected data is. Then the bigger can of worms will be opened.

One step at a time :)

-Frinny
Jul 13 '09 #9

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
On second thought I think I know how I'll be able to tell them apart......I Think.

When the application calls BeginSend, it will use a separate thread to execute the specified callback method, and will block on EndSend until the Socket sends the number of bytes requested.

I would assume that if a second file were to be sent before the first file it would be queued. I don't know for sure though.

This would kind of defeat the purpose of sending via asyc though.
I'll have to look into this later.

Err, I'm off track now...back to the original problem: determining the size of the file.
Jul 13 '09 #10

Plater
Expert 5K+
P: 7,872
Oh! When I said header i didn't specifically mean something relating to the Socket and TCP transfering protocol, I meant like create your own "header"
Jul 13 '09 #11

Frinavale
Expert Mod 5K+
P: 9,731
Thanks for your help Plater :)

I sent an "init package" to the client which contained the size of the data before I sent the actual data. From there I was able to allocate the correct amount of memory to the array I'm using to appended the received data to.
Jul 14 '09 #12

Frinavale
Expert Mod 5K+
P: 9,731
Hmm this solution is not working well at all.
Jul 14 '09 #13

Plater
Expert 5K+
P: 7,872
Are they getting mixed up now?
Jul 14 '09 #14

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
How can you tell?

All I know is it's easy to crash this.

I change the implementation to append 2 bytes to the data being sent to the client. The 2 bytes contains an Int16 used to indicate the length of data that's being sent to the client....this way I don't have to send an "init package" before the real send begins.

Now that I've been playing with this more I'm thinking of trying the other way again...where I send an "init package" to the client with the length, and then the data.

The problem that I was having with the original implementation is that if anything takes a bit longer to process than usual (lag) in the client everything sent from the server gets thrown out of sync.

I can't figure out when the begin send/end send is happening to get it back into sync.

I think the something similar is happening with this new implementation when I increase the the timer to a value of 100th of a second...some how the data sent from the server is getting out of sync with the client after some time passes and I open more socket connections to the server.

If I knew when the begin send and end send was happening in the client I would be happy...I think.

And I thought sockets were easy when I first started playing with them :P
It only took me 5 minutes to figure out how to get a synchronized solution to work using a normal windows application (client) and console application (server). Between using asynchronous sockets and Silverlight I'm starting to feel like a newbie.
Jul 14 '09 #15

Frinavale
Expert Mod 5K+
P: 9,731
I went back to the first solution again: sending a separate intializing package to the client before sending the data.

When I receive data from the server I check the size of the data and whether or not the I'm expecting an initializing package. If both requirements are met then I reset the Object I'm using to store the data I'm receiving from the server and begin the collection process.

If, while I'm gathering the data, something gets messed up I reset and wait for the initializing package again.

The client code:
Expand|Select|Wrap|Line Numbers
  1. Private Sub OnReceive(ByVal sender As Object, ByVal e As SocketAsyncEventArgs)
  2.         Try
  3.             If _isInitPackage And e.BytesTransferred = 4 Then
  4.                 Dim amountExpected As Integer = BitConverter.ToInt32(e.Buffer, 0)
  5.                 If amountExpected > 0 Then
  6.                     ReDim _receivedData(amountExpected - 1)
  7.                     _isInitPackage = False
  8.                 Else
  9.                     _receivedData = Nothing
  10.                 End If
  11.             Else
  12.                 If _receivedData IsNot Nothing Then
  13.                     If _amountOfDataReceivedSoFar + e.BytesTransferred <= _receivedData.Length Then
  14.                         Array.Copy(e.Buffer, 0, _receivedData, _amountOfDataReceivedSoFar, e.BytesTransferred)
  15.                         _amountOfDataReceivedSoFar += e.BytesTransferred
  16.                     Else
  17.                         ErrorOccurred = "File doesn't fit." + Environment.NewLine + "Amount received so far: " + _amountOfDataReceivedSoFar.ToString + Environment.NewLine + "Amount Expected: " + _receivedData.Length.ToString + Environment.NewLine + "Amount recieved this time: " + e.BytesTransferred.ToString
  18.                         ResetRecieve()
  19.                     End If
  20.  
  21.                     If _amountOfDataReceivedSoFar = _receivedData.Length Then
  22.                         CompletedData = _receivedData.Clone
  23.                         ResetRecieve()
  24.                     End If
  25.                 End If
  26.             End IF
  27.         Catch ex As Exception
  28.             ErrorOccurred = ex.Message + Environment.NewLine + ex.StackTrace
  29.             ResetRecieve()
  30.         End Try
  31.         _socket.ReceiveAsync(e)
  32.     End Sub
  33.  
  34.    Private Sub ResetRecieve()
  35.         _amountOfDataReceivedSoFar = 0
  36.         _isInitPackage = True
  37.         _receivedData = Nothing
  38.     End Sub
The Server Code:
Expand|Select|Wrap|Line Numbers
  1.  Public Sub Send(ByVal data() As Byte)
  2.         If Not _disposedOf Then
  3.             If IsConnected Then
  4.  
  5.                 SendInitPackage(data.Length)
  6.  
  7.                 Console.WriteLine("Sending data to client...")
  8.                 _client.BeginSend(data, 0, data.Length, SocketFlags.None, New AsyncCallback(AddressOf OnSendCompleted), Nothing)
  9.             Else
  10.                 Console.WriteLine("Client is closed: cannot send data to client")
  11.             End If
  12.         End If
  13.     End Sub
  14.  
  15.     Private Sub SendInitPackage(ByVal size As Integer)
  16.         If Not _disposedOf Then
  17.             Dim sizeInBytes() As Byte = BitConverter.GetBytes(size)
  18.             Console.WriteLine("Sending init package to client...")
  19.             _client.BeginSend(sizeInBytes, 0, sizeInBytes.Length, SocketFlags.None, New AsyncCallback(AddressOf OnInitSendComplete), Nothing)
  20.         Else
  21.             Console.WriteLine("Client is closed: cannot init package to client")
  22.         End If
  23.     End Sub
  24.  
  25.     Private Sub OnSendCompleted(ByVal res As IAsyncResult)
  26.         If Not _disposedOf Then
  27.             Try
  28.                 Console.WriteLine("Data Sent to client.")
  29.                 _client.EndSend(res)
  30.             Catch ex As ObjectDisposedException
  31.                 Me.Dispose()
  32.             Catch ex As Exception
  33.                 Console.WriteLine("Could not send data to client:")
  34.                 Console.WriteLine(ex.Message)
  35.                 Console.WriteLine()
  36.                 _client.Close()
  37.             End Try
  38.         End If
  39.     End Sub
  40.  
  41.     Private Sub OnInitSendComplete(ByVal res As IAsyncResult)
  42.         If Not _disposedOf Then
  43.             Try
  44.                 Console.WriteLine("Init package sent to client.")
  45.                 _client.EndSend(res)
  46.             Catch ex As ObjectDisposedException
  47.                 Me.Dispose()
  48.             Catch ex As Exception
  49.                 Console.WriteLine("Could not send init package to client:")
  50.                 Console.WriteLine(ex.Message)
  51.                 Console.WriteLine()
  52.                 _client.Close()
  53.             End Try
  54.         End If
  55.     End Sub
I haven't seen anything that indicates the data's out of sync since I've made this change. I can't reproduce the problems I was having yesterday with this new modification.

I am having problems with my socket connection collection being changed while in the middle of sending data to the clients. I think I'm going to have to look into using a proper, thread safe, object for this collection now...but this is a whole different issue.

I think I've solved the problem...for now at least.

Thanks again for your help :)
Jul 15 '09 #16

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
Ok, the data's all mixed up now!
Now, I'm experiencing what you were getting at earlier.

Before the socket server was sending the file when a timer ticked. The timer was slow and this allowed the sending process to work properly. I've removed the timer and the socket server now sends data whenever a file has finished being read into a byte array.

What I noticed was that the initializing package was being sent several times before the actual data was sent....and sometimes when the data was being sent a new initializing package would be sent.

The client corrects itself when the data doesn't fit properly so the images are being displayed (very rapidly) but it's constantly losing data too.

I'm looking into using the IAsyncResult to solve the problem....except that I don't have a reference to this when the data's being sent so I'm not able to make the thread wait until both the init package has been sent before the actual data has been sent....and wait until data has been sent before the next init package.

The problem is that there's synchronized processing that has to take place in an asynchronous setting.


I'm not sure if I'm approaching the problem correctly (trying to fix the problem on the Server side of things) so any guidance at this point would be greatly appreciated.

This mini project has been a really interesting adventure!
Jul 23 '09 #17

Plater
Expert 5K+
P: 7,872
I would say you might need a custom object, and then a colelction of them.
Whenever you get an intializing package, create an object and stick it on the collection. Use the socket's port# as a unique id (should work, i'm hoping) and maybe track the DateTime that it came in as.
Any data that is not an init package, gets appeneded to the data stored for the correct object, based on the socket port.
Once all the data (based on your init package's datasize) is in place, you can use and remove the object from your collection.
Same thing with if it hasn't received any data packets for awhile, or if you get another init package with the same port number (assuming they were coming up as different). Just wipe out the old object completely.
Jul 24 '09 #18

Frinavale
Expert Mod 5K+
P: 9,731
I changed the implementation back to pre-pending the size to the byte array (the file) being sent to the client.

This avoided the situation where the init packages didn't match with the data being sent.

I started experiencing other exceptions with this method but I think that they were mainly being caused by the speed at which I was sending the files. Every time the server read a file it sent it to the client (asynchronously), then read the next file.

I think that the garbage collector in the client could not clean up the released arrays fast enough and I would quickly get OutOfMemoryExceptions.

I returned to using the timer...setting the interval value to 10.

I ran the app for a good minute or two before I decided to update this thread. It didn't throw any exceptions until I opened the browser ....hmmmm out of memory again.

There's got to be a better way to do this. I don't know if I can actually use this timer method to stream data from the physical device when I start interfacing with it. It would be best to figure out how to avoid this problem now before I start the real project.

-Frinny
Jul 29 '09 #19

Banfa
Expert Mod 5K+
P: 8,916
I haven't read the details of your attempts but I thought some information on TCP/Sockets might be useful.

A TCP connection is a stream, that is the data is not packetised. While it is true that the transport layer packetises the data for transmission across the network at the level where you send and receive data to/from the socket it is, and needs to be treated as a simple byte stream.

One of the effects of this could be, for instance, if you sent 2 small items (or images) in rapid succession from the source then the network protocol layer might well place these into a single transport layer packet. The result would be that your receiver would only get 1 notification of data to read. In general you may not get as many notifications of data available as the number of times you sent data because of the transport layers ability to concatenate data for efficient transmission.

Another is that the transport layer has no way of knowing the size of the data objects you have encoded into its byte stream.

TCP provides the following guarantee, if data arrives then it is correct and it arrives in the order it was transmitted. If 2 images arrive mixed up in the stream it can only be because the data was transmitted in a mixed up fashion (or some wired .NET effect that I take no responsibility for).

I thought some additional knowledge of the underlying transport mechanism might help.
Jul 29 '09 #20

Frinavale
Expert Mod 5K+
P: 9,731
@Banfa
So this would explain why my idea of sending an "init" package before the data was not a good idea. It's a simple byte stream, not actual packets I'm dealing with.

Ok this is starting to make more sense.

@Banfa
I'm going to look into a way to determine if 2 or more items are in the stream. I never considered using a Stream object instead of a byte array....

@Banfa
This much I knew from my research (although I didn't know it when I originally asked the question). I've encoded it into the byte stream to get around this. But I'm wondering if I can use an actual Stream object instead of a byte array.

@Banfa
What is the order when everything's asynchronously sent.....hmmm.

@Banfa
It wasn't that the images were mixed up it was the problems I was facing by sending a separate "init package" before the actual data stream.

@Banfa
Thank you so much!
You've given me a lot to consider.
Jul 29 '09 #21

Frinavale
Expert Mod 5K+
P: 9,731
I cannot use streams because of the limitations of the controls I'm using.

I think that the control is taking the TCP stream and breaking it up into several pieces... it fills a buffer with the data. The buffer is a byte array.

Things are messed up when I try to put it back together again in the client.

I'm taking what's in the buffer and appending it to a byte array.

In order to determine how big this array is I've pre-pended the file size to the file sent down. When receiving data I fist grab the size of the file and start filling the array with what's in the buffer.

When I increase the frequency that the files are being sent down to the client something gets messed up.

When I finish reading in the the amount of bytes I'm expecting I try to grab the size of the next file....but sometimes the file sizes I grab are Huge. This is why I'm running out of memory. I think that it's grabbing the size from the middle of a file instead of the beginning:

@Banfa
I think this is how everything's being thrown out of sync.
Jul 29 '09 #22

Plater
Expert 5K+
P: 7,872
I wish I had a good way to test out these things for you finny. I have not done anything with silverlight really (Save for a brief example from TR )
I suppose I could just make a project that uses the send/receive async calls and it should be the same yes?
Jul 29 '09 #23

Frinavale
Expert Mod 5K+
P: 9,731
I could post what I have but it's a Really big mess right now.

PS re-read my previous post...I've edited it.
Jul 29 '09 #24

Plater
Expert 5K+
P: 7,872
Could you tell the server to not push the next file to the client until the client says its ok?
The only thing I can think of is that another even is coming in(and starting being processed) before you have finished with the previous event's handler code? (Known as re-entry)
Jul 29 '09 #25

Frinavale
Expert Mod 5K+
P: 9,731
I can't really tell the server to wait before sending the next image because I'm not entirely sure what my physical device is going to be sending in the future. Depending on the device manufacturer, it could be images or it could be streaming video that has no end. The first device that I'm going to be integrating with will be sending raw byte data of images.

Right now I just want to get the images to work. I'm not even connecting to the device...I'm simply reading files off of the hard drive and sending them down to the client.

I think that Banfa was on to something when he mentioned that more than one image is being placed into a single transport layer packet.

Instead of just displaying an error stating that the file doesn't fit I'm reading the amount left of what I was expecting....

Then I restart the process right there....reading the next 4 bytes to determine the size, resizing the _retrievedData array for the new file, and fill it with the remaining bytes in the buffer.

It works.......................kind of.
Jul 29 '09 #26

Plater
Expert 5K+
P: 7,872
You were not reading the exact amount you needed for each file, but were instead reading everything that was available and filling in your byte array from there?

Think of the TCP connection as a lazy front desk clerk. The clerk only looks up every once and awhile to see if someone is waiting and buzzes an office to notify them(think windows events). Sometimes there is no one, sometimes only one person (think one byte) and sometimes multiple people are waiting by the time they look up.
So X amount of people(bytes) could make it to the desk by the time the clerk goes "hey someone is here". Maybe they are all with each other as a group (or part of your image) maybe there is more then one group there.
The clerk just buzzes in however many "people" are seen and then looks back down. The next time they look up and see people, they will buzz them in too.
Each of those buzzes will be your receivedData event.

With a big image, not all the bytes will arrive before the clerk buzzes them in. And the next time the clerk buzzes, there could be part of the next image in with it.

So you need to only read exactly the amount of bytes you need (as the value you send before the image) and if there is MORE data available to be read in your receivedData event, you need to start over and read the byte size and start reading/appending data again.

As a side note: I should probably be banned from making annalogies.
Jul 29 '09 #27

Banfa
Expert Mod 5K+
P: 8,916
So I have some experience of this sort of scenario. I worked on a system where a microprocessor had to receive an image from a IP Camera, it then retransmitted the image over a GPRS link to a server. Because of the nature of the GPRS packet switch radio link the maximum size of a packet is about 1500 bytes. We had some overhead so I split the images into 1024 byte blocks to send. We sent a 2 byte start of block indicator, a 2 byte block size (because the last block is not 1024 bytes) some other data that isn't relevant here, the image data block and finally a checksum. On TCP IP the checksum is strictly speaking not required but our protocol was design to also work using modems on a PSTN.

The receive routine looked something like this (in pseudo code)

Expand|Select|Wrap|Line Numbers
  1. PERMANENTSTORE   ExpectedSize
  2. PERMANENTSTORE   DataBuffer
  3. PERMANENTSTORE   CheckSum
  4. PERMANENTSTORE   State = FindFirstStartByte
  5.  
  6. FUNCTION SocketDataReady  // Called when the socket has data to read
  7.  
  8.   LOCALSTORE Byte
  9.   LOCALSTORE Status
  10.  
  11.   Status = ReadByteFromSocket INTO Byte
  12.  
  13.   WHILE STATUS == OK
  14.  
  15.     SWITCH State
  16.     CASE FindFirstStartByte
  17.       IF Byte == FirstStartByte
  18.         State = FindSecondStartByte
  19.       ENDIF
  20.       BREAK
  21.  
  22.     CASE FindSecondStartByte
  23.       IF Byte == SecondStartByte
  24.         State = GetFirstSizeByte
  25.         ExpectedSize = 0
  26.         CheckSum = 0
  27.         CLEAR DataBuffer
  28.       ELSE
  29.         State = FindFirstStartByte
  30.       ENDIF
  31.       BREAK
  32.  
  33.     CASE GetFirstSizeByte
  34.       ExpectedSize = Byte << 8   // Assuming Big Endian Protocol
  35.       State = GetSecondSizeByte
  36.       BREAK
  37.  
  38.     CASE GetSecondSizeByte
  39.       ExpectedSize = ExpectedSize| Byte
  40.       State = FindFirstStartByte
  41.       BREAK
  42.  
  43.      CASE GetData
  44.       PUT Byte INTO DataBuffer
  45.       UPDATE CheckSum WITH BYTE
  46.       ExpectedSize = ExpectedSize - 1
  47.       IF ExpectedSize == 0
  48.         State = GetCheckSum
  49.       ENDIF
  50.       BREAK
  51.  
  52.     CASE GetCheckSum
  53.       IF TransmitedCheckSum == Byte
  54.         PASS DataBuffer TO ImageHandler
  55.       ENDIF
  56.       State = FindFirstStartByte
  57.       BREAK
  58.     ENDSWITCH
  59.  
  60.     Status = ReadByteFromSocket INTO Byte
  61.  
  62.   ENDWHILE
  63.  
  64. ENDFUNCTION
  65.  
Obviously for use the image handler had to stitch together the 1k blocks back into images which it did successfully and then put them into an SQL Server DB.

Here are the important points about this code though that make it able to handle a byte stream rather than a packet stream
  • The receive function does not assume that it will receive only one block every time it is called.
  • The receive function does not assume it is receiving a whole block every time it is called.
  • The receive function makes no assumption of where it is in the data stream the when it is called for the first time, it looks for synchronisation markers in the stream.
  • The receive function checks that when it thinks it is at the end of a block it actually is at the end of a block. In theory on TCP using a checksum for this is very over the top and end of block marker should work just as well.
  • Having received a block the receive function does not assume that it is automatically at the start of the next block but looks for the synchronisation marker again.

Im some ways this should not be necessary over TCP/IP, however every protocol I have ever seem does something like this.

If you are worried about the underlying correct functionality of you libraries then write a little program that transmits like this

Expand|Select|Wrap|Line Numbers
  1. UNSIGNED STORE Byte = 0
  2.  
  3. OPEN SOCKET
  4.  
  5. FOREVER
  6.   SEND Byte TO SOCKET
  7.   Byte = Byte + 1 // Assuming that 255 it will wrap to 0
  8. ENDFOREVER
  9.  
It should be simple to receive and verify that you are receiving every byte transmitted.
Jul 29 '09 #28

Frinavale
Expert Mod 5K+
P: 9,731
@Banfa
I'm not using a checksum right now but it might be a good idea to explore later.

I've used a very similar solution to yours but instead of infinite loops, I'm using something similar to recursion. I had to because of the nature of the asynchronous sockets.

Every time the socket receives data the OnReceive method is called. At the end of this method I tell the socket to begin receiving again.

The important thing was that the OnReceive method was implemented so that it:
@Banfa
My problem was that the function originally assumed that it was receiving a single block at a time.

This problem is solved now. The client is processing everything properly now :) Thank you so much for your advice.

@Banfa
This Very similar to what I'm going to be working with but not at a microprocessor level, and it's not just one camera...there's several channels.

When I remove the timer from my server "PictureGetter" component and simply push the images as soon as I read them in I'm running into a problem.

Sending the pictures using socket couldn't be preformed because "the system lacked sufficient buffer space or because the queue was full".

Is this happening because it's placing too much into a single transport layer packet?

How did you get around this problem?
Jul 30 '09 #29

Plater
Expert 5K+
P: 7,872
Somewhere you had to tell a listener socket to Listen(), which means you had to supply a backlog (how many connections can be "waiting to be accepted" in this case) You are probably filling up that as the queue being full?
As for an overall statement, the camera can probably produce more pictures then the bandwidth between applications will allow. Just something you need to be able to trap and recover rfom I would guess
Jul 30 '09 #30

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
The server's listener allows 10 connections to wait.

I don't know if that's the case....I only opened 1 socket connection to the server.

The server pushed many images but the client stopped receiving them very quickly. The computer's fans turned on because it was working hard...and I was really tempted to stop the server process but I waited it out see what the exception was.

@Plater
Slow it down on purpose to reduce the bandwidth?
It's a possibility I'll keep in mind.
Jul 30 '09 #31

Plater
Expert 5K+
P: 7,872
No I mean your bandwidth might be a bottle-neck, you will have to be prepared to "skip" images if you cannot send them. (Get your FPS figured out that way)
Just what kinda of computer are you using as a server? Something from the mid 90s? It should not be baulking at this little of a task?
Jul 30 '09 #32

Banfa
Expert Mod 5K+
P: 8,916
@Frinavale
Well you have to remember that I was using a low power microprocessor connected to a self contained GPRS modem module. Given the data sheet of the GPRS module it was safe to assume that the power in the GPRS module in terms of processor power and available memory far exceed what I had available on the microprocessor so anything I could do it could cope with.

I suspect you problem may be at the interface to the TCP stack coupled with your speed of transmission. The TCP stack is what handles the data transmission for you. When you send a byte through a TCP socket what you actually do is queue it on the TCP stack for transmission. The TCP stack sends the byte but can't discard it until it has received the confirmation that it has arrived.

TCP automatically acks received data and retransmits data not acked (roughly), that is how it guarantees correct reception in the correct order.

If you throw data at the TCP stack faster than it can transmit it then in the end it will run out of buffer space.


You can make an estimate of an upper (unreachable) limit of the number of pictures you can transmit.

For instance if you are sending 50kbyte (on average) pictures down a 50Mbit/s link then you can send roughly

50 * 1,000,000 / 50 * 10(convert to bits) * 1000 ~= 100 pictures / s

I know I've used 1000 instead of 1024 but the calculation is very rough because it does not even include transport protocol overhead for instance. This gives you an idea of the point at which the link will fail. If you are having trouble transmitting 10 pics/s then there is a problem to resolve somewhere. On the other hand in you are trying to transmit 90 pics/s and it is failing then may be you are sending too much.
Jul 30 '09 #33

Frinavale
Expert Mod 5K+
P: 9,731
It can handle 100 pictures/second very easily without crashing.

I've decided, for now, just to leave it at 30 pics/sec until I actually start looking at the device capabilities.

This little adventure would have been a lot more frustrating without the both of you here for help. Thanks a lot!

:)

-Frinny
Jul 30 '09 #34

Frinavale
Expert Mod 5K+
P: 9,731
@Plater
:) will have to check this out later.

@Plater
Haha :)
I think it actually might be. It doesn't have a lot of RAM and it's a processor that I used to use myself years ago (Pentium D)

I hate my DEV machine :) <--- fake plastered smile

@Plater
Precisely why I hate it so much.
It tortures me with making me wait when I'm in the middle of some thought process...sometimes I even forget what I was doing/thinking because it takes sooo long.
Jul 30 '09 #35

Banfa
Expert Mod 5K+
P: 8,916
No under-power DEV machines are good because if you can get it working on that then there certainly should be a problem when you install on the brand spanking new production machine.

The problem is if you do it the other way round, that is a sure route for disaster.
Jul 30 '09 #36

Frinavale
Expert Mod 5K+
P: 9,731
@Banfa
That's my boss's' exact same reasoning for not purchasing more RAM for my poor machine.

*eyes Banfa* conspiracy?
Jul 30 '09 #37

Plater
Expert 5K+
P: 7,872
@Banfa
That only works if the difference is not too significant.
Jul 30 '09 #38

Banfa
Expert Mod 5K+
P: 8,916
No if the software is correctly written in a robust manner it should work all the time. One of the main issues is using time for your timings and not processor clocks.
Jul 30 '09 #39

Post your reply

Sign in to post your reply or Sign up for a free account.