Pipe problem, Is it a bug in Perl

[stbrian]

Hello everyone,

I have a C program that constantly returned the % of the job completed to my Perl script so to execute this program I have these lines:

use IO::Handle;
$| = 1;
 
$data1 = "./myC.exe 10 10 1 abc";
 
open PS, "$data1 |";
PS->autoflush(1);
while ($line = readline(*PS)) {
    print $line;
    print "\n";
}
 

I have noticed that the results are only returned to my Perl script after my C program is completed and not while it is running. The C program usually takes about 10 hours to complete and every half an hour or so it prints to STDOUT how many percent of the job has been completed so that the Perl script can take it and push it to the web page. I have tried

use IO::Handle;
$| = 1;
 
$data1 = "./myC.exe 10 10 1 abc";
 
open PS, "$data1 |";
PS->autoflush(1);
while (<PS>) {
    print;
    print "\n";
}
 

but no luck. It appeared that the result are buffered or stucked somewhere while the C program is running and only get returned to my Perl script after the C program exited. Any help would be appreciated.
Thanks in advance.

[KevinADC]

I've never used IO::Handle like you are doing. I have always used it to create an IO::Handle object and use methods on the object. But since you are using a pipe, look into IO::Pipe, which is also a core module.

http://perldoc.perl.org/IO/Pipe.html

You could also just use open() and not use a IO module.

use strict;
use warnings;
my $data1 = './myC.exe 10 10 1 abc';
 
$|=1;
open PIPE, "$data1 |" or die "$!";
while (<PIPE>) {
   print $_,"\n"
}
close PIPE;

[stbrian]

Same problem. The output did not get displayed on the screen until the C program finished running. Any other suggestions?

[KevinADC]

are you running this as a cgi from the web browser or from the command line?

[stbrian]

From the command line. I have 2 terminals displayed. On one terminal I see the C program sending requests to a database and sending back how many percent of the requests have been processed as thing progress. But on the Perl terminal, nothing is printed to the screen until the C program has finished.

The responses from the C program are:
+::010

+::020

+::030

+::040

+::050

+::060

+::070

+::080

+::090

+::100


where +::010 means 10% of the requests have been processed. When I run the c program directly from the command line, I can see the % completed being printed on the screen as the C program progressed. However, when calling it from Perl, all the requests are done before all of the above lines get printed at the same time. I just don't get it. Thanks for your help.

[KevinADC]

Sorry mate, I don't know what the problem is, seems like it should work. Hopefully Miller of Jeff or someone else will have an idea.

[numberwhun]

Wow, that is really odd. To me, it should work as well. My next suggestion, unless Miller knows, would be to post this up on to Perlmonks.org and see if they know why?

Regards,

Jeff

[miller]

open used in this context is a blocking operation. Meaning, open will not actually finish until the secondary process has exitted.

Based off the script that you provided, what you probably want is this:

use strict;
 
my $data1 = "./myC.exe 10 10 1 abc";
 
system("$data1 &");
 

This executes the command as a background process that share the same STDOUT and STDERR as the perl script. The system command is also blocking, so you can still list commands afterwords that you want executed sequentially. But all the output of $data1 will be sent to STDOUT immediately instead of waiting until the very end.

If you want to do some type of translation of the output of the command, then you'll have to do use IPC instead.

- Miller

[KevinADC]

Good suggestion. IPC::Open2 might be appropriate for this task. Maybe this link will help:

http://perldoc.perl.org/perlipc.html

[stbrian]

I posted the question on Perl Monk and a few folks helped me solved this problem. The problem was the C program did not flushed its buffer after each printf statement. I can see the output on STDOUT b/c it is not buffered but the pipe is actually buffered so printf needs to be flushed with fflush.
Thanks for all who contributed.

[miller]

Yes, I observed your question there, and was happy to let the other monks throw in their vast wisdom. Glad you were able to figure out an answer.

- Miller