473,288 Members | 1,743 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,288 software developers and data experts.

Reading Java byte[] data stream over standard input

Hello,
I am using HadoopStreaming using a BinaryInputStream. What this
basically does is send a stream of bytes ( the java type is : private
byte[] bytes) to my python program.

I have done a test like this,
while 1:
x=sys.stdin.read(100)
if x:
print x
else:
break

Now, the incoming data is binary(though mine is actually merely ascii
text) but the output is not what is expected. I expect for e.g

all/86000/114.310.151.209.60370-121.110.5.176.113\n62485.9718
118.010.241.12 60370 128.210.5.176

However i get a 1 before all and a 4 just after \n and before the 6.

My question is : how do i read binary data(Java's byte stream) from
stdin?
Or is this actually what i'm getting?

Thanks
Sapsi
Jun 27 '08 #1
6 3127
I should also mention that for some reason there are several binay
values popping in between for some reason. This behavior (for the
inputr stream) is not expected

Now, the incoming data is binary(though mine is actually merely ascii
text) but the output is not what is expected. I expect for e.g

all/86000/114.310.151.209.60370-121.110.5.176.113\n62485.9718
118.010.241.12 60370 128.210.5.176

However i get a 1 before all and a 4 just after \n and before the 6.

My question is : how do i read binary data(Java's byte stream) from
stdin?
Or is this actually what i'm getting?

Thanks
Sapsi
Jun 27 '08 #2
On Sun, 18 May 2008 22:11:33 -0700, sapsi wrote:
I am using HadoopStreaming using a BinaryInputStream. What this
basically does is send a stream of bytes ( the java type is : private
byte[] bytes) to my python program.

I have done a test like this,
while 1:
x=sys.stdin.read(100)
if x:
print x
else:
break

Now, the incoming data is binary(though mine is actually merely ascii
text) but the output is not what is expected. I expect for e.g

all/86000/114.310.151.209.60370-121.110.5.176.113\n62485.9718
118.010.241.12 60370 128.210.5.176

However i get a 1 before all and a 4 just after \n and before the 6.

My question is : how do i read binary data(Java's byte stream) from
stdin?
Or is this actually what i'm getting?
If there's extra data in `x` then it was sent to stdin. Maybe there's
some extra information like string length, Java type information, or
checksums encoded in that data!?

Ciao,
Marc 'BlackJack' Rintsch
Jun 27 '08 #3
Yes, that could be the case. Browsing through hadoop's source, i see
stdin in the above code is reading from piped Java DataOutputStream.
I read of a libray on the net Javadata.py that reads this but it has
disappeared.
What is involved in reading from a Dataoutputstream?

Thank you
Sapsi
Jun 27 '08 #4
On Mon, 19 May 2008 00:14:25 -0700, sapsi wrote:
Yes, that could be the case. Browsing through hadoop's source, i see
stdin in the above code is reading from piped Java DataOutputStream.
I read of a libray on the net Javadata.py that reads this but it has
disappeared.
What is involved in reading from a Dataoutputstream?
According to the Java docs of `DataInput` and `DataOutput` it is quite
simple. Most methods just seem to write the necessary bytes for the
primitive types except `writeUTF()` which prefixes the string data with
length information.

So if it is not Strings you are writing then "hadoop" seems to throw in
some information into the stream.

Ciao,
Marc 'BlackJack' Rintsch
Jun 27 '08 #5
On 19 May, 06:11, sapsi <saptarshi.g...@gmail.comwrote:
Hello,
I am using HadoopStreaming using a BinaryInputStream. What this
basically does is send a stream of bytes ( the java type is : private
byte[] bytes) to my python program.

I have done a test like this,
while 1:
x=sys.stdin.read(100)
if x:
print x
else:
break

Now, the incoming data is binary(though mine is actually merely ascii
text) but the output is not what is expected. I expect for e.g

all/86000/114.310.151.209.60370-121.110.5.176.113\n62485.9718
118.010.241.12 60370 128.210.5.176

However i get a 1 before all and a 4 just after \n and before the 6.

My question is : how do i read binary data(Java's byte stream) from
stdin?
Or is this actually what i'm getting?

Thanks
Sapsi
In the past I've sent binary data to a java applet reading
DataInputStream using xdrlib from the standard library. I'd expect
that it would work in the reverse direction so I suggest you have a
look at that.

Giles
Jun 27 '08 #6
sapsi wrote:
I should also mention that for some reason there are several binay
values popping in between for some reason. This behavior (for the
inputr stream) is not expected

>Now, the incoming data is binary(though mine is actually merely ascii
text) but the output is not what is expected. I expect for e.g

all/86000/114.310.151.209.60370-121.110.5.176.113\n62485.9718
118.010.241.12 60370 128.210.5.176

However i get a 1 before all and a 4 just after \n and before the 6.

My question is : how do i read binary data(Java's byte stream) from
stdin?
Or is this actually what i'm getting?
Consider changing "print x" to "print repr(x)" ... this would mean that
you have a better chance of understanding what the extra or unexpected
popping-in bytes are.
Jun 27 '08 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

73
by: RobertMaas | last post by:
After many years of using LISP, I'm taking a class in Java and finding the two roughly comparable in some ways and very different in other ways. Each has a decent size library of useful utilities...
30
by: Richard | last post by:
Level: Java newbie, C experienced Platform: Linux and Win32, Intel Another programmer and I are working on a small project together. He's writing a server process in Java that accepts input...
3
by: Fernando Arbeiza | last post by:
Hi: I need some clarification about a code like this: printf("%s", "a string with NO trailing newline"); scanf("%d", &i); Regarding if a fflush() of the standard output is needed or not. I...
3
by: Nick | last post by:
I have found a class that compresses and uncompresses data but need some help with how to use part of it below is the deflate method which compresses the string that I pass in, this works OK. At...
1
by: Vitaly | last post by:
// Open input file and create the BinaryReader. br = new BinaryReader(new FileStream("Test.dat", FileMode.Open, FileAccess.Read)); // Read binary data. d = br.ReadDouble(); A question is...
8
by: junk5 | last post by:
Hi I need to read raw 16 bit data from a file, where the first byte is the most significant byte of the first data value and the second byte is the least significant byte of the first data value...
3
by: Marc Gravell | last post by:
It might just be my tired eyes, but I can't see what is wrong in the following: I have a byte array filled with random data (with a fixed seed to make reproducable); I then compress and...
3
by: Sir Psycho | last post by:
Hi, For some reason, when i step over this code, it returns the full byte stream im expecting from the server, however when I let it run with no intervention, it only seems to grab a small chunk...
4
by: radhikams | last post by:
hi I have written a code for writing the data of a column from database into a file and again reading that file and displaying. Im writing into an .html file....Now the problem is im...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.