Parsing data from pyserial, an (inefficient) solution

[Lone Wolf]

I want to thank everybody who tried to help me, and also to post
my solution, even though I don’t think it is a very good one.

Many of you correctly guessed that there was an “\r” included
with the packet from the CUMcam, and you were correct. The
actual format of the packet is: M xxx xxx xxx xxx xxx xxx
xxx xxx\r. Unfortunately, splitting the packet using \r wouldn’t
help because the format of the data stream that I get with the
components variable (after I split the reading file according to
“M”) generally doesn’t include a complete packet at first. For
example, I get: [xxx xxx xxx\r, yyy yyy yyy yyy yyy yyy yyy
yyy/r, zzz zzz zzz] Therefore, data from before the
first \r (which I have shown as xxx) generally is incomplete
and I need to go on to the second packet.

Also, for those of you who suggested some kind of delay before
reading the serial port, you were right. The first packet from
the CMUcam is always a null.

Anyway, here is my code:

################################################## ##############

# This program reads a serial port hookup up to a CMUcam version
1.
# It tracks the middle of a green object and provides a
confidence estimate

import serial

ser=serial.Serial('com1',baudrate=115200, bytesize=8,
parity='N', stopbits=1,xonxoff=0, timeout=1)

ser.write("TC 016 240 100 240 016 240\r\n") #This line orders
the CMUcam to track green
reading = ser.read(40) # CMUcam's first data packet is null, so
this line gets it out of the way

for i in range(0,100,1):
reading = ser.read(40)
components = reading.split("M")
components = components[1]
if len(components) 23: # If shorter than 24 it won't have
enough data for a full packet
subcomponents = components.split()
mx = int(subcomponents[0])
my = int(subcomponents[1])
confidence = int(subcomponents[7])
print mx, my, confidence
ser.close
The really sad thing is that I get a perfectly constructed
packet from the reading variable, and that gets butchered when I
try to slice it up to pick out individual elements. Since
pyserial doesn’t do anything to rearrange the data, then the
CMUcam must do the heavy lifting of extracting a perfect packet
from the data stream. It’s a real shame I couldn’t use it
because the program would be more efficient. FWIW, this code
will analyze 2-3 frames per second on my computer, which is
enough for my purposes.

In case you couldn’t tell from the questions/code, I am a total
beginner, and I really appreciate this list. All I needed was a
hand, not a handout. Wolves are willing to hunt for their
supper.
________________________________________________
Get your own "800" number
Voicemail, fax, email, and a lot more
http://www.ureach.com/reg/tag

[John Machin]

Lone Wolf wrote:

Your code has a problem when the first character of reading is 'M': you
will miss the full packet and pick up a fragment. The length test that
you are doing to reject the fragment is a kludge. If the average length
of a packet is say 25, then you are throwing away 4% of all packets on
average. Try this [untested]:

reading = ser.read(40)
# components = reading.split("M")
# components = components[1]
# if len(components) 23: # If shorter than 24 it won't have
enough data for a full packet
# subcomponents = components.split()
pos_past_m = reading.index('M') + 1
subcomponents = reading[pos_past_m:].split()
mx = int(subcomponents[0])
my = int(subcomponents[1])
confidence = int(subcomponents[7])
print mx, my, confidence

>
The really sad thing is that I get a perfectly constructed
packet from the reading variable,

What we tell you three times is true, you are *NOT* getting a perfectly
formed packet from the reading variable; you are getting 40 bytes of
guff which will be long enough to contain a packet with possible stray
fragments at either end.

Do this:
print len(reading)
print repr(reading)
and see for yourself.

and that gets butchered when I
try to slice it up to pick out individual elements. Since
pyserial doesn't do anything to rearrange the data, then the
CMUcam must do the heavy lifting of extracting a perfect packet
from the data stream.

Huh? I thought the CMUcam was *creating* the data stream -- how could
it be *extracting* a "perfect packet" from it?

It's a real shame I couldn't use it
because the program would be more efficient.

Somebody else gave you a clue: use the readline method instead of the
read method; have you tried that? It's more likely to stop on a packet
boundary that what you are doing.

As far as starting on a packet boundary is concerned, have you explored
your options with buffering and flow control?

FWIW, this code
will analyze 2-3 frames per second on my computer, which is
enough for my purposes.

In case you couldn't tell from the questions/code, I am a total
beginner, and I really appreciate this list. All I needed was a
hand, not a handout. Wolves are willing to hunt for their
supper.

Independence is one thing. Ignoring plausible truth told to you by
multiple independent people with no axe to grind is another :-)

HTH,
John

[Giovanni Bajo]

Dennis Lee Bieber wrote:

>

>The really sad thing is that I get a perfectly constructed
packet from the reading variable, and that gets butchered when I
try to slice it up to pick out individual elements. Since
pyserial doesn’t do anything to rearrange the data, then the
CMUcam must do the heavy lifting of extracting a perfect packet
from the data stream. It’s a real shame I couldn’t use it

More likely it is running fast enough to keep up with the serial
port, and synchronizes on the "\rM". Your code seems to be reading in
bursts and somehow losing parts during your processing (which is
strange, I'd have expected pyserial to buffer some data between reads).

Lacking that, I'd probably end up creating a thread to handle the
serial port reading, which basically looks like:

while ser.read(1) != "\r": pass #synchronize
while not Done:
bfr = []
while True:
c = ser.read(1)
bfr.append(c)
if c == "\r": break
queue.put("".join(bfr))

This should result in complete packets (from the "M" to a "\r")

Oh well. readline(eol="\r") will do that much better.

while 1:
data = ser.readline(eol="\r")
args = data.split()
if args[0] == "M":
# command M; do something with args[1:]
elif args[0] == "KK":
# command KK; do something with args[1:]
[.. process other commands ..]
else:
# Invalid command: might be an incomplete
# packet. Just ignore it.
pass

[Grant Edwards]

On 2006-12-04, Giovanni Bajo <no***@ask.mewrote:

[...]

>This should result in complete packets (from the "M" to a "\r")

Oh well. readline(eol="\r") will do that much better.

Yup. Using readline() has been suggested several times. It
sure seems like the obvious solution to me as well.

while 1:
data = ser.readline(eol="\r")
args = data.split()
if args[0] == "M":
# command M; do something with args[1:]
elif args[0] == "KK":
# command KK; do something with args[1:]
[.. process other commands ..]
else:
# Invalid command: might be an incomplete
# packet. Just ignore it.
pass

--
Grant Edwards grante Yow! Hmmm... A hash-singer
at and a cross-eyed guy were
visi.com SLEEPING on a deserted
island, when...