469,271 Members | 1,785 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,271 developers. It's quick & easy.

stdin read delay when using pipes [solved]

2
I'm having some issues when trying to read input off of a pipe using a python script. I'm trying to process packet data from tcpdump in real-time, so it's a filter that needs to read data while the process being piped from is still reading.

On both my Linux and OSX system, it seems that there's a large delay reading from stdin if the process being read from is still running, even if it's producing output to be read. It will wait until it has a really large amount of data, and then suddenly it will read it all, process it all, and then idle again waiting for a huge chunk of data. I think that it's something that either Python or I am doing because when I pipe the output through awk it processes it in real-time.

Anyhow, here's the source for my *very simple* test to try to get the pipe working:

Expand|Select|Wrap|Line Numbers
  1. while 1:
  2.     print "input: " + raw_input()
  3.  
The command I'm running is:

sudo tcpdump -i en1 -xx -l |awk '{print $2$3$4$5$6$7$8$9}' | python test.py

That command will hang until there's a good amount of data. Removing the python script from the end produces real-time awk formatted data without delay.

I've also tried this for the script so that I'm only reading a single character at a time, same problem:

Expand|Select|Wrap|Line Numbers
  1. import sys
  2.  
  3. buffer = []
  4. while 1:
  5.     char = sys.stdin.read(1)
  6.     if not char == "\n":
  7.         buffer.append(char)
  8.     else:
  9.         print "input: " + "".join(buffer)
  10.         buffer = []
  11.  
Weirdness, this has gotta be a simple problem that I'm overlooking.
Apr 12 '06 #1
2 5617
ShawnD
2
Found my answer, it would seem that awk is buffering the output and won't flush/write it until it has a bunch when piping to another process. Removing awk and just using tcpdump in line-buffered mode and processing the output with Python seems to do the trick, but it seems to be a hack.

Someone with more Unix-fu than I have may be able to figure out how to get awk to line buffer or flush.
Apr 13 '06 #2
tasoth
1
The 'mawk' implementation of awk has an extra switch for interactive mode: -Wi. This will do what you want. This switch is not in the POSIX spec, but other implementations may have similar options.
Oct 25 '06 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by David Bear | last post: by
7 posts views Thread by Yandos | last post: by
reply views Thread by Diogo Bastos | last post: by
16 posts views Thread by fbertasso | last post: by
5 posts views Thread by Luis Zarrabeitia | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.