Select hangs after some reads

[alsmeirelles]

Hi,

I'm building a multithreaded application and I encountered a tiny and
annoying problem. I use a select to wait for data to be read from a
socket, after some reads, the select simply blocks and stays that way
until I close the connection on the other side of the socket. When the
socket is closed on the writer end the select releases and then I get
only empty strings from the socket.
My question is this: Why did it block? The reading has never ended,
every test I make I write 50 requests (wich are strings) to the socket
and I have read the maximum of 34 requests. I'm using winPdb to take a
closer look on what's happening and I see the threads blocked on this
same select. If I send anything more through the socket, the select
releases for a thread, despite the other data that is still unread.

This is the select:
rd,w,e = select.select([self.rfd],[],[])

self.rfd is the fileno of the file object returned by the makefile
method from the socket object.

I know that's some buffer behavior that I'm missing but I don't know
what it is.

anything is helpfull, If I'm beeing stupid you can say it.
thanks

[Steve Holden]

al**********@gm ail.com wrote:

Hi,

I'm building a multithreaded application and I encountered a tiny and
annoying problem. I use a select to wait for data to be read from a
socket, after some reads, the select simply blocks and stays that way
until I close the connection on the other side of the socket. When the
socket is closed on the writer end the select releases and then I get
only empty strings from the socket.
That's to be expected: the first return of a zero-length string from
socket.read() indicates end of file.
My question is this: Why did it block? The reading has never ended,
every test I make I write 50 requests (wich are strings) to the socket
and I have read the maximum of 34 requests. I'm using winPdb to take a
closer look on what's happening and I see the threads blocked on this
same select. If I send anything more through the socket, the select
releases for a thread, despite the other data that is still unread.
Are you sure that the received data is presenting in block the same size
as are being sent? There's no guarantee this will be so on a TCP socket,
and it may be that multiple sends are being coalesced into a single
read. The important thing to focus on is the total number of bytes sent
and received.
This is the select:
rd,w,e = select.select([self.rfd],[],[])

self.rfd is the fileno of the file object returned by the makefile
method from the socket object.
It would be simpler to use the result of the socket's .fileno() method
directly.
I know that's some buffer behavior that I'm missing but I don't know
what it is.

anything is helpfull, If I'm beeing stupid you can say it.
thanks

Without being able to see all your code it's hard to say whether or not
you are doing something daft, but you give a general impression of
competence.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[alsmeirelles]

Steve Holden wrote:

al**********@gm ail.com wrote:

Hi,

I'm building a multithreaded application and I encountered a tiny and
annoying problem. I use a select to wait for data to be read from a
socket, after some reads, the select simply blocks and stays that way
until I close the connection on the other side of the socket. When the
socket is closed on the writer end the select releases and then I get
only empty strings from the socket.
That's to be expected: the first return of a zero-length string from
socket.read() indicates end of file.

My question is this: Why did it block? The reading has never ended,
every test I make I write 50 requests (wich are strings) to the socket
and I have read the maximum of 34 requests. I'm using winPdb to take a
closer look on what's happening and I see the threads blocked on this
same select. If I send anything more through the socket, the select
releases for a thread, despite the other data that is still unread.

Are you sure that the received data is presenting in block the same size
as are being sent? There's no guarantee this will be so on a TCP socket,
and it may be that multiple sends are being coalesced into a single
read. The important thing to focus on is the total number of bytes sent
and received.

Well, actually I´m using a very simple protocol wich sends only
strings ended by newline. I need to send 3 chunks of information and a
newline after them. On the reader side I make 3 readline(), this way I
wouldn´t have to care about this problem, but maybe that´s where I´m
falling. If that´s the case, I´ll have to use a more complex
protocol.

This is the select:
rd,w,e = select.select([self.rfd],[],[])

self.rfd is the fileno of the file object returned by the makefile
method from the socket object.
It would be simpler to use the result of the socket's .fileno() method
directly.

Yes, you are right, this is something remanescent from old ideas.

I know that's some buffer behavior that I'm missing but I don't know
what it is.

anything is helpfull, If I'm beeing stupid you can say it.
thanks

Without being able to see all your code it's hard to say whether or not
you are doing something daft, but you give a general impression of
competence.

Thanks, the code has a significant size now, if I don´t solve it
quickly I´ll post the most important parts.

thanks again,

Andre LS Meirelles regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Grant Edwards]

On 2006-06-08, al**********@gm ail.com <al**********@g mail.com> wrote:

Well, actually I´m using a very simple protocol wich sends only
strings ended by newline. I need to send 3 chunks of information and a
newline after them. On the reader side I make 3 readline(), this way I
wouldn´t have to care about this problem, but maybe that´s where I´m
falling. If that´s the case, I´ll have to use a more complex
protocol.

You can't use readline() with select(). Select tells you
whether recv() called on the underlying socket will block or
not. What's probably happening is that all of the data has
been read from the underlying socket and is being held in a
buffer waiting to be read by readline().

The Select call has no way of knowing about that buffered data.
As far as it's concerned there's no more data left to read, so
it block until the socket is closed.

--
Grant Edwards grante Yow! My mind is a potato
at field...
visi.com

[alsmeirelles]

Grant Edwards escreveu:

On 2006-06-08, al**********@gm ail.com <al**********@g mail.com> wrote:

Well, actually I´m using a very simple protocol wich sends only
strings ended by newline. I need to send 3 chunks of information and a
newline after them. On the reader side I make 3 readline(), this way I
wouldn´t have to care about this problem, but maybe that´s where I´m
falling. If that´s the case, I´ll have to use a more complex
protocol.
You can't use readline() with select(). Select tells you
whether recv() called on the underlying socket will block or
not. What's probably happening is that all of the data has
been read from the underlying socket and is being held in a
buffer waiting to be read by readline().

Yes, as I expected, its the buffers. In my opinion the problem is that
the socket module
doesn't provide a way of reading all its internal buffer.

readlines() just make subsequent calls to readline and readline may
call recv, so we have a locked scene. I want to know if I will block
anyway. Of course I can clean the buffer myself, but I think the socket
module should provide a way of doing this. It's not a big problem
though.

The Select call has no way of knowing about that buffered data.
As far as it's concerned there's no more data left to read, so
it block until the socket is closed.
You're very right.
thanks,

Andre LS Meirelles
--
Grant Edwards grante Yow! My mind is a potato
at field...
visi.com

[Steve Holden]

al**********@gm ail.com wrote:

Grant Edwards escreveu:

On 2006-06-08, al**********@gm ail.com <al**********@g mail.com> wrote:

Well, actually I´m using a very simple protocol wich sends only
strings ended by newline. I need to send 3 chunks of information and a
newline after them. On the reader side I make 3 readline(), this way I
wouldn´t have to care about this problem, but maybe that´s where I´m
falling. If that´s the case, I´ll have to use a more complex
protocol.

You can't use readline() with select(). Select tells you
whether recv() called on the underlying socket will block or
not. What's probably happening is that all of the data has
been read from the underlying socket and is being held in a
buffer waiting to be read by readline().

Yes, as I expected, its the buffers. In my opinion the problem is that
the socket module
doesn't provide a way of reading all its internal buffer.

readlines() just make subsequent calls to readline and readline may
call recv, so we have a locked scene. I want to know if I will block
anyway. Of course I can clean the buffer myself, but I think the socket
module should provide a way of doing this. It's not a big problem
though.

The Select call has no way of knowing about that buffered data.
As far as it's concerned there's no more data left to read, so
it block until the socket is closed.

You're very right.
thanks,

Of course, if the client forces the TCP PSH flag true then the receiver
is guaranteed to debuffer the stream up to that point - this is how FTP
clients work, for example.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Fredrik Lundh]

al**********@gm ail.com wrote:

Yes, as I expected, its the buffers. In my opinion the problem is that
the socket module doesn't provide a way of reading all its internal buffer.

umm. that's what recv() does, of course, of you pass in a large enough
buffersize.

for your use case, I suggest looking at asyncore/asynchat instead of
trying to write your own asynchronous socket layer...

</F>

[Grant Edwards]

On 2006-06-08, al**********@gm ail.com <al**********@g mail.com> wrote:

Well, actually I´m using a very simple protocol wich sends
only strings ended by newline. I need to send 3 chunks of
information and a newline after them. On the reader side I
make 3 readline(), this way I wouldn´t have to care about this
problem, but maybe that´s where I´m falling. If that´s the
case, I´ll have to use a more complex protocol.
You can't use readline() with select(). Select tells you
whether recv() called on the underlying socket will block or
not. What's probably happening is that all of the data has
been read from the underlying socket and is being held in a
buffer waiting to be read by readline().

Yes, as I expected, its the buffers. In my opinion the problem
is that the socket module doesn't provide a way of reading all
its internal buffer.

What internal buffer?
readlines() just make subsequent calls to readline and
readline may call recv, so we have a locked scene. I want to
know if I will block anyway.
I'm lost.
Of course I can clean the buffer myself, but I think the
socket module should provide a way of doing this.

I'm afraid I don't understand what you mean.

Since you talked about calling readline(), I assumed that you
had called the socket object's makefile() method to create a
file-object. The select system call can only tell you whether
a recv on the socket will block or not. It knows nothing about
the state of the file object on which you're calling readline().

--
Grant Edwards grante Yow! Is this an out-take
at from the "BRADY BUNCH"?
visi.com

[Jon Ribbens]

In article <ma************ *************** ************@py thon.org>, Steve Holden wrote:

Of course, if the client forces the TCP PSH flag true then the receiver
is guaranteed to debuffer the stream up to that point - this is how FTP
clients work, for example.

I don't think that's right. You are confusing the PSH flag (which is
basically unused in Unix networking I think) and the URG flag (which
is extremely rarely used, but is indeed used by FTP to get abort
requests to 'jump the queue' as it were).

[Steve Holden]

Jon Ribbens wrote:

In article <ma************ *************** ************@py thon.org>, Steve Holden wrote:

Of course, if the client forces the TCP PSH flag true then the receiver
is guaranteed to debuffer the stream up to that point - this is how FTP
clients work, for example.

I don't think that's right. You are confusing the PSH flag (which is
basically unused in Unix networking I think) and the URG flag (which
is extremely rarely used, but is indeed used by FTP to get abort
requests to 'jump the queue' as it were).

Nope. The URG flag indicates that a packet contains out-of-band data,
whihc is what you describe above.

The PSH flag indicates that the data stream must be flushed right
through to the other end. This is essential for interactive protocols
such as FTP: without it the server has no way to know that the client
has sent a complete command, and vice versa.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden