producer-consumer? no, a race to stop memory filling up!

[dh]

Runtime.getRuntime().exec() ... mad buffering of stdout ...

It buffers stdout without limit ...
so if your Java program doesn't keep up with the Process output then
memory fills up!
I wish it just blocked instead because my Java program is loosing
the race against the C program that it exec()s.

The pattern I want is "producer-consumer"
and i'm suprised that it seems imposible.

what can I do?

Here is a small example that illustrates the issue ...

5523:fosters:~: cat Test.java

class Test {
public static void main(String[] args) {
try {
Process p =
Runtime.getRuntime().exec("/usr/bin/yes");
for (;;) {
Thread.sleep(1000);
System.out.println("tick");
Thread.sleep(1000);
System.out.println("tock");
}
}
catch (Exception x) {
x.printStackTrace();
}
}
}
5524:fosters:~: rm *.class
5525:fosters:~: /usr/local/IBMJava2-13/bin/javac Test.java
5526:fosters:~: /usr/local/IBMJava2-13/bin/java Test
tick
tock
tick
tock
java.lang.OutOfMemoryError
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
[etc....]

[Silvio Bierman]

What do you mean by "loosing the race" if you perform sleeps and do not even
attemt to read the proces output?

Silvio Bierman

"dh" <[e-mail]> wrote in message
news:c7*************************@posting.google.co m...

Runtime.getRuntime().exec() ... mad buffering of stdout ...

It buffers stdout without limit ...
so if your Java program doesn't keep up with the Process output then
memory fills up!
I wish it just blocked instead because my Java program is loosing
the race against the C program that it exec()s.

The pattern I want is "producer-consumer"
and i'm suprised that it seems imposible.

what can I do?

Here is a small example that illustrates the issue ...

5523:fosters:~: cat Test.java

class Test {
public static void main(String[] args) {
try {
Process p =
Runtime.getRuntime().exec("/usr/bin/yes");
for (;;) {
Thread.sleep(1000);
System.out.println("tick");
Thread.sleep(1000);
System.out.println("tock");
}
}
catch (Exception x) {
x.printStackTrace();
}
}
}
5524:fosters:~: rm *.class
5525:fosters:~: /usr/local/IBMJava2-13/bin/javac Test.java
5526:fosters:~: /usr/local/IBMJava2-13/bin/java Test
tick
tock
tick
tock
java.lang.OutOfMemoryError
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
tock
tick
[etc....]

[Kai Grossjohann]

"Silvio Bierman" <sb******@idfix.nl> writes:

What do you mean by "loosing the race" if you perform sleeps and do not even
attemt to read the proces output?

He means that "yes" is printing output like mad, and Java is reading
it into a buffer, leading to an OOM.

He expected "yes" to stop producing output when Java didn't feel like
reading it anymore.

If you do "yes | more" then yes will print 24 lines of output, and
more will show them. Then more waits for the user to hit space,
causing the yes process to stall. When the user hits space, then
more reads another 24 lines and stops. This causes yes again to
stall. Even if the user never hits space, then memory never gets
full, because the yes process is prevented from running.

But in the Java case, the yes process keeps running, and Java keeps
buffering its output. Why?
I'm not an expert, however, and I might be misinterpreting the
situation. I wonder what the situation really is ;-)

Kai

[chris]

I ran your test on Wonka and it works fine (even with -Xmx=4M), so it looks
like this is a bug in IBM's implementation (and maybe in Sun's too).

Normally a process writing to a pipe will block when there is a certain
amount of unread data in the pipe, so they must indeed be deliberately
buffering inside the VM heap. Report a bug.

--
Chris Gray ch***@kiffer.eunet.be
/k/ Embedded Java Solutions