Parsing data from pyserial

[Lone Wolf]

I'm trying to get data through my serial port from a CMUcam.
This gizmo tracks a color and returns a packet of data. The
packet has nine data points (well, really eight since the first
point is just a packet header) separated by spaces as follows: M
xxx xxx xxx xxx xxx xxx xxx xxx

Here is the code I am using (python v24):

import serial

ser=serial.Seri al('com1',baudr ate=115200, bytesize=8,
parity='N', stopbits=1,xonx off=0, timeout=1)

ser.write("PM 1") #This sets the CMUcam to poll mode

for i in range(0,100,1):
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)
print reading
components = reading.split()
print components
ser.close

Here is an example output:

M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']

My problem is that I am trying to get each data point of the
packet into a separate variable. Ordinarily, this would be easy,
as I would just parse the packet, read the array and assign each
element to a variable eg. mx = components[1]. However, that
doesn't work here because the original packet and the array that
I got from using the split() method are different. If I were to
try read the array created in the first example output, mx would
be 123 instead of 37 like it is in the packet. In the second
example, the array is 85 while the packet is 38.

As near as I can figure out, pyserial is reading a stream of
data and helpfully rearranging it so that it fits the original
packet format M xxx xxx xxx xxx xxx xxx xxx xxx. I would have
thought the split() method that I used on original packet (ie
the "reading" variable) would have just returned an array with
nine elements like the packet has. This is not the case, and I
am at a loss about how to fix this.

I've searched the archive here and elsewhere with no luck. Any
help REALLY appreciated!

Wolf :)

_______________ _______________ _______________ ___
Get your own "800" number
Voicemail, fax, email, and a lot more
http://www.ureach.com/reg/tag

[John Machin]

Lone Wolf wrote:

I'm trying to get data through my serial port from a CMUcam.
This gizmo tracks a color and returns a packet of data. The
packet has nine data points (well, really eight since the first
point is just a packet header) separated by spaces as follows: M
xxx xxx xxx xxx xxx xxx xxx xxx

Here is the code I am using (python v24):

import serial

ser=serial.Seri al('com1',baudr ate=115200, bytesize=8,
parity='N', stopbits=1,xonx off=0, timeout=1)

ser.write("PM 1") #This sets the CMUcam to poll mode

for i in range(0,100,1):
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)

You are asking for 40 bytes of data. You will get 40 bytes of data.

However your packets are (presumably) variable length, (presumably)
terminated by CR and/or LF. What does the documentation for the device
tell you?

print reading

What you see from the print statement is not necessarily what you've
got.
Change that to print repr(reading) and show us what you then see.

components = reading.split()
print components
ser.close

Here is an example output:

M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']

Let's try to reconstruct "reading":

| >>a = ['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
| ... '124', '86', '25', 'M']
| >>astrg = ' '.join(a)
| >>astrg
| '59 123 87 25 M 37 79 3 4 59 124 86 25 M'
| >>len(astrg)
| 39 <<<<<==== ooh! almost 40!!
| >>b = ['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124',
'86',
| ... '25', 'M', '38'
| ... , '7']
| >>bstrg = ' '.join(b)
| >>bstrg
| '39 85 26 M 38 77 3 2 59 124 86 25 M 38 7'
| >>len(bstrg)
| 40 <<<<<==== ooh! exactly 40!!!

My guess: the device is pumping out packets faster than you can handle
them. So you are getting 40-byte snatches of bytes. A snatch is long
enough to cover a whole packet with possible fragments of packets at
each end. You will need to discard the fragments. If you need all the
data, you'd better get some help on how to implement flow control --
I've never used pyserial and I'm not going to read _all_ the docs for
you :-)

I'm very interested to see what print repr(reading) actually shows. I'm
strongly suspecting there is a CR (no LF) at the end of each packet; in
the two cases shown, this would cause the "print reading" to appear as
only one packet ... think about it: carriage return, with no linefeed,
would cause overwriting. It is a coincidence with those two samples
that the first part of the line doesn't appear strange, with a 4, 5, or
6-digit number showing up where the trailing fragment ends

My problem is that I am trying to get each data point of the
packet into a separate variable. Ordinarily, this would be easy,
as I would just parse the packet, read the array and assign each
element to a variable eg. mx = components[1].

better would be:

mx, foo, bar, ......, eighth_vbl = components[start:start + 8]
once you have worked out what start should be, e.g. start =
components.inde x('M') + 1

However, that
doesn't work here because the original packet and the array that
I got from using the split() method are different. If I were to
try read the array created in the first example output, mx would
be 123 instead of 37 like it is in the packet. In the second
example, the array is 85 while the packet is 38.

As near as I can figure out, pyserial is reading a stream of
data and helpfully rearranging it so that it fits the original
packet format M xxx xxx xxx xxx xxx xxx xxx xxx.

How, if you've read the docstring for the Serial.read() method, did you
come to that conclusion?

pyserial knows nothing about your packet format.

I would have
thought the split() method that I used on original packet (ie
the "reading" variable) would have just returned an array with
nine elements like the packet has. This is not the case, and I
am at a loss about how to fix this.

I've searched the archive here and elsewhere with no luck. Any
help REALLY appreciated!

With a bit of repr() and a bit of RTFM, one can often manage without
help :-)

Cheers,
John

[Giovanni Bajo]

Lone Wolf wrote:

reading = ser.read(40)

Simply try ser.readline() here, or maybe ser.readline(eo l="\r").

--
Giovanni Bajo

[Grant Edwards]

On 2006-12-03, Lone Wolf <lo*******@urea ch.comwrote:

import serial

ser=serial.Seri al('com1',baudr ate=115200, bytesize=8,
parity='N', stopbits=1,xonx off=0, timeout=1)

ser.write("PM 1") #This sets the CMUcam to poll mode

for i in range(0,100,1):
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)
print reading
components = reading.split()
print components
ser.close

Here is an example output:

M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']

My problem is that I am trying to get each data point of the
packet into a separate variable. Ordinarily, this would be
easy, as I would just parse the packet, read the array and
assign each element to a variable eg. mx = components[1].
However, that doesn't work here because the original packet
and the array that I got from using the split() method are
different.

I doubt it. Try printing `reading` instead of reading. I
suspect that the string you're getting from ser.read() has a
carraige-return in it that you aren't seeing when you do print
reading.

If I were to try read the array created in the first example
output, mx would be 123 instead of 37 like it is in the
packet. In the second example, the array is 85 while the
packet is 38.

As near as I can figure out, pyserial is reading a stream of
data and helpfully rearranging it so that it fits the original
packet format M xxx xxx xxx xxx xxx xxx xxx xxx.

No, it isn't. I wrote the Posix low-level code that's in
pyserial. I've used pyserial extensively on both Windows and
Linux. It doesn't rearrange anything.

I would have thought the split() method that I used on
original packet (ie the "reading" variable) would have just
returned an array with nine elements like the packet has. This
is not the case, and I am at a loss about how to fix this.

When something odd seems to be happening with strings, always
print `whatever` rather than whatever

I've searched the archive here and elsewhere with no luck. Any
help REALLY appreciated!

--
Grant Edwards grante Yow! There's a SALE on
at STRETCH SOCKS down at the
visi.com "7-11"!!

[Si Ballenger]

On Sat, 2 Dec 2006 23:02:06 -0500, Lone Wolf
<lo*******@urea ch.comwrote:

>I'm trying to get data through my serial port from a CMUcam.
This gizmo tracks a color and returns a packet of data. The
packet has nine data points (well, really eight since the first
point is just a packet header) separated by spaces as follows: M
xxx xxx xxx xxx xxx xxx xxx xxx

Here is the code I am using (python v24):

import serial

ser=serial.Ser ial('com1',baud rate=115200, bytesize=8,
parity='N', stopbits=1,xonx off=0, timeout=1)

ser.write("P M 1") #This sets the CMUcam to poll mode

for i in range(0,100,1):
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)
print reading
components = reading.split()
print components
ser.close

In my dealing with serial gizmos I have to put a delay between
the request sent to the gizmo and the reading of the serial input
buffer for returned data. Serial ports and gizmos need some time
to do their thing.

[Grant Edwards]

On 2006-12-03, Si Ballenger <sh**********@c omporium.netwro te:

In my dealing with serial gizmos I have to put a delay between
the request sent to the gizmo and the reading of the serial input
buffer for returned data. Serial ports and gizmos need some time
to do their thing.

I doubt that's the issue. He's reading with a 1-second timeout
value.

--
Grant Edwards grante Yow! The Korean War must
at have been fun.
visi.com

[Si Ballenger]

On Sun, 03 Dec 2006 16:52:33 -0000, Grant Edwards
<gr****@visi.co mwrote:

>On 2006-12-03, Si Ballenger <sh**********@c omporium.netwro te:

>In my dealing with serial gizmos I have to put a delay between
the request sent to the gizmo and the reading of the serial input
buffer for returned data. Serial ports and gizmos need some time
to do their thing.

I doubt that's the issue. He's reading with a 1-second timeout
value.

I would think a time delay would be needed between the below two
lines in the code if he expects to get a useable data string back
from the gizmo for the command sent to it.

ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)

[John Machin]

Grant Edwards wrote:

When something odd seems to be happening with strings, always
print `whatever` rather than whatever

:-)

Unholy perlism, Batman!

For the benefit of gentle readers who are newish and might not have
seen the ` character in Python code outside a string literal, or for
those who'd forgotten, there is a cure:

| >>re.sub(r"`(.* ?)`", r"repr(\1)", "print `whatever`, `foo`, `bar`")
| 'print repr(whatever), repr(foo), repr(bar)'
:-)

[Grant Edwards]

On 2006-12-03, Si Ballenger <sh**********@c omporium.netwro te:

>>In my dealing with serial gizmos I have to put a delay between
the request sent to the gizmo and the reading of the serial input
buffer for returned data. Serial ports and gizmos need some time
to do their thing.

I doubt that's the issue. He's reading with a 1-second timeout
value.

I would think a time delay would be needed between the below two
lines in the code if he expects to get a useable data string back
from the gizmo for the command sent to it.

ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)

No. A delay isn't needed as long as the device responds within
1 second. The read() call will wait up to 1 second for the
first byte of the response.

--
Grant Edwards grante Yow! NEWARK has been
at REZONED!! DES MOINES has
visi.com been REZONED!!

[Grant Edwards]

On 2006-12-03, John Machin <sj******@lexic on.netwrote:

Grant Edwards wrote:

>When something odd seems to be happening with strings, always
print `whatever` rather than whatever

:-)

Unholy perlism, Batman!

OK, make that "print repr(whatever)" . :)

--
Grant Edwards grante Yow! I selected E5... but
at I didn't hear "Sam the Sham
visi.com and the Pharoahs"!