Delays getting data on sys.stdin.readline() ?

[Christian Convey]

Hello,

I've got a program that (ideally) perpetually monitors sys.stdin for
lines of text. As soon as a line comes in, my program takes some
action.

The problem is, it seems like a very large amount of data must
accumulate on sys.stdin before even my first invocation of readline()
returns. This delay prevents my program from being responsive in the
way it must be.

Has anyone else seen this effect? If so, is there a reasonable workaround?

Thanks very much,
Christian

[Mike Meyer]

Christian Convey <ch**************@gmail.com> writes:

I've got a program that (ideally) perpetually monitors sys.stdin for
lines of text. As soon as a line comes in, my program takes some
action.
The problem is, it seems like a very large amount of data must
accumulate on sys.stdin before even my first invocation of readline()
returns. This delay prevents my program from being responsive in the
way it must be.
readline normally returns as soon as it sees a newline. External
conditions may cause this to change, or make it impossible. Without
knowing those external conditions, the best we can do is guess as to
what might be the problem.
Has anyone else seen this effect? If so, is there a reasonable workaround?

Yes, and maybe. Depends on what's causing the problem. Tell us more
about the program, and what sys.stdin is connected to, and the
platform you're running on, and someone should be able to provide
explicit information.
<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Mike Meyer]

Christian Convey <ch**************@gmail.com> writes:

I've got a program that (ideally) perpetually monitors sys.stdin for
lines of text. As soon as a line comes in, my program takes some
action.
The problem is, it seems like a very large amount of data must
accumulate on sys.stdin before even my first invocation of readline()
returns. This delay prevents my program from being responsive in the
way it must be.
readline normally returns as soon as it sees a newline. External
conditions may cause this to change, or make it impossible. Without
knowing those external conditions, the best we can do is guess as to
what might be the problem.
Has anyone else seen this effect? If so, is there a reasonable workaround?

Yes, and maybe. Depends on what's causing the problem. Tell us more
about the program, and what sys.stdin is connected to, and the
platform you're running on, and someone should be able to provide
explicit information.
<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Christian Convey]

On 11/19/05, Mike Meyer <mw*@mired.org> wrote:

Christian Convey <ch**************@gmail.com> writes:

I've got a program that (ideally) perpetually monitors sys.stdin for
lines of text. As soon as a line comes in, my program takes some
action.
The problem is, it seems like a very large amount of data must
accumulate on sys.stdin before even my first invocation of readline()
returns. This delay prevents my program from being responsive in the
way it must be.

readline normally returns as soon as it sees a newline. External
conditions may cause this to change, or make it impossible. Without
knowing those external conditions, the best we can do is guess as to
what might be the problem.

Has anyone else seen this effect? If so, is there a reasonable workaround?

Yes, and maybe. Depends on what's causing the problem. Tell us more
about the program, and what sys.stdin is connected to, and the
platform you're running on, and someone should be able to provide
explicit information.

OK, I've fixed it, but I don't understand why the fix works.

Let's say I've got two Python programs, I'll call "producer" and
"consumer". "producer" runs for a long time and occasionally while
running sends lines of text to stdout. "consumer" is typically blocked
in a call to sys.stdin.readline().

When I run "producer" on its own, I see its output appear on the
console pretty much immediately after calling the "print" command.

But when I pipe "producer"s output to "consumer"s stdin on the Linux
command line, "consumer" stays blocked on its first call to
sys.stdin.readline() until the "producer" program terminates. At that
point, "consumer" seems to immediately get access to all of the stdout
produced by "producer".

I've found I can fix this problem by modifying "producer" so that
immediately after each "print" command, I call sys.stdout.flush().
When I make this modification, I find that "consumer" has access to
the output of "producer" immediately after "producer" issues a "print"
statement.

So here's what I don't get: If "producer" was retaining its output for
a while for the sake of efficiency, I would expect to see that effect
when I just run "producer" on the command line. That is, I would
expect the console to not show any output from "producer" until
"producer" terminates. But instead, I see the output immediately. So
why, when I pipe the output to "consumer", doesn't "consumer" get
access to that data as its produced unless "consumer" is explicitely
calling sys.stdout.flush().

Any thoughts?

Thanks,
Christian

[Christian Convey]

On 11/19/05, Mike Meyer <mw*@mired.org> wrote:

Christian Convey <ch**************@gmail.com> writes:

I've got a program that (ideally) perpetually monitors sys.stdin for
lines of text. As soon as a line comes in, my program takes some
action.
The problem is, it seems like a very large amount of data must
accumulate on sys.stdin before even my first invocation of readline()
returns. This delay prevents my program from being responsive in the
way it must be.

readline normally returns as soon as it sees a newline. External
conditions may cause this to change, or make it impossible. Without
knowing those external conditions, the best we can do is guess as to
what might be the problem.

Has anyone else seen this effect? If so, is there a reasonable workaround?

Yes, and maybe. Depends on what's causing the problem. Tell us more
about the program, and what sys.stdin is connected to, and the
platform you're running on, and someone should be able to provide
explicit information.

OK, I've fixed it, but I don't understand why the fix works.

Let's say I've got two Python programs, I'll call "producer" and
"consumer". "producer" runs for a long time and occasionally while
running sends lines of text to stdout. "consumer" is typically blocked
in a call to sys.stdin.readline().

When I run "producer" on its own, I see its output appear on the
console pretty much immediately after calling the "print" command.

But when I pipe "producer"s output to "consumer"s stdin on the Linux
command line, "consumer" stays blocked on its first call to
sys.stdin.readline() until the "producer" program terminates. At that
point, "consumer" seems to immediately get access to all of the stdout
produced by "producer".

I've found I can fix this problem by modifying "producer" so that
immediately after each "print" command, I call sys.stdout.flush().
When I make this modification, I find that "consumer" has access to
the output of "producer" immediately after "producer" issues a "print"
statement.

So here's what I don't get: If "producer" was retaining its output for
a while for the sake of efficiency, I would expect to see that effect
when I just run "producer" on the command line. That is, I would
expect the console to not show any output from "producer" until
"producer" terminates. But instead, I see the output immediately. So
why, when I pipe the output to "consumer", doesn't "consumer" get
access to that data as its produced unless "consumer" is explicitely
calling sys.stdout.flush().

Any thoughts?

Thanks,
Christian

[Fredrik Lundh]

Christian Convey wrote:

So here's what I don't get: If "producer" was retaining its output for
a while for the sake of efficiency, I would expect to see that effect
when I just run "producer" on the command line. That is, I would
expect the console to not show any output from "producer" until
"producer" terminates. But instead, I see the output immediately. So
why, when I pipe the output to "consumer", doesn't "consumer" get
access to that data as its produced unless "consumer" is explicitely
calling sys.stdout.flush().

Any thoughts?

your terminal is not a pipe.

$ man stdout

....

CONSIDERATIONS
The stream stderr is unbuffered. The stream stdout is line-buffered when
it points to a terminal. Partial lines will not appear until fflush(3) or
exit(3) is called, or a newline is printed. This can produce unexpected
results, especially with debugging output. The buffering mode of the
standard streams (or any other stream) can be changed using the setbuf(3)
or setvbuf(3) call. Note that in case stdin is associated with a termi*
nal, there may also be input buffering in the terminal driver, entirely
unrelated to stdio buffering. (Indeed, normally terminal input is line
buffered in the kernel.) This kernel input handling can be modified
using calls like tcsetattr(3); see also stty(1), and termios(3).

....

</F>

[Fredrik Lundh]

Christian Convey wrote:

So here's what I don't get: If "producer" was retaining its output for
a while for the sake of efficiency, I would expect to see that effect
when I just run "producer" on the command line. That is, I would
expect the console to not show any output from "producer" until
"producer" terminates. But instead, I see the output immediately. So
why, when I pipe the output to "consumer", doesn't "consumer" get
access to that data as its produced unless "consumer" is explicitely
calling sys.stdout.flush().

Any thoughts?

your terminal is not a pipe.

$ man stdout

....

CONSIDERATIONS
The stream stderr is unbuffered. The stream stdout is line-buffered when
it points to a terminal. Partial lines will not appear until fflush(3) or
exit(3) is called, or a newline is printed. This can produce unexpected
results, especially with debugging output. The buffering mode of the
standard streams (or any other stream) can be changed using the setbuf(3)
or setvbuf(3) call. Note that in case stdin is associated with a termi*
nal, there may also be input buffering in the terminal driver, entirely
unrelated to stdio buffering. (Indeed, normally terminal input is line
buffered in the kernel.) This kernel input handling can be modified
using calls like tcsetattr(3); see also stty(1), and termios(3).

....

</F>