stdin read delay when using pipes [solved]

[ShawnD]

I'm having some issues when trying to read input off of a pipe using a python script. I'm trying to process packet data from tcpdump in real-time, so it's a filter that needs to read data while the process being piped from is still reading.

On both my Linux and OSX system, it seems that there's a large delay reading from stdin if the process being read from is still running, even if it's producing output to be read. It will wait until it has a really large amount of data, and then suddenly it will read it all, process it all, and then idle again waiting for a huge chunk of data. I think that it's something that either Python or I am doing because when I pipe the output through awk it processes it in real-time.

Anyhow, here's the source for my *very simple* test to try to get the pipe working:

while 1:
    print "input: " + raw_input()
 

The command I'm running is:

sudo tcpdump -i en1 -xx -l |awk '{print $2$3$4$5$6$7$8$ 9}' | python test.py

That command will hang until there's a good amount of data. Removing the python script from the end produces real-time awk formatted data without delay.

I've also tried this for the script so that I'm only reading a single character at a time, same problem:

import sys
 
buffer = []
while 1:
    char = sys.stdin.read(1)
    if not char == "\n":
        buffer.append(char)
    else:
        print "input: " + "".join(buffer)
        buffer = []
 

Weirdness, this has gotta be a simple problem that I'm overlooking.

[ShawnD]

Found my answer, it would seem that awk is buffering the output and won't flush/write it until it has a bunch when piping to another process. Removing awk and just using tcpdump in line-buffered mode and processing the output with Python seems to do the trick, but it seems to be a hack.

Someone with more Unix-fu than I have may be able to figure out how to get awk to line buffer or flush.

[tasoth]

The 'mawk' implementation of awk has an extra switch for interactive mode: -Wi. This will do what you want. This switch is not in the POSIX spec, but other implementations may have similar options.