Problem using Optional pyparsing

[Nathan Harmston]

Hi,

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work. My file generally looks like this:

ALIGNMENT 1020 YS2-10a02.q1k chr09 1295 42 141045
142297 C 1254 95.06 1295 reject_bad_brea k 0

or this:

ALIGNMENT 36 YS2-10a08.q1k chrm 208 165 10745
10788 C 44 95.45 593 reject_low 10,14

and my grammar work well for these lines, however somethings the row looks like:
ALIGNMENT 53 YS2-10b03.p1k chr12 180 125 1067465
1067520 C 56 98.21 532|5,2 reject_low 25

So I try to parse the 532 using

from pyparsing import *

integer = Word( nums )
float = Word( nums+".")
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNM ENT ").suppress ()
row_1 = integer.setResu ltsName("row_1" )#.setParseActi on(make_int)
src_id = identifier.setR esultsName("src _id")
dest_id = identifier.setR esultsName("des t_id")
src_start = integer.setResu ltsName("src_st art")#.setParse Action(make_int )
src_stop = integer.setResu ltsName("src_st op")#.setParseA ction(make_int)
dest_start = integer.setResu ltsName("dest_s tart")#.setPars eAction(make_in t)
dest_stop = integer.setResu ltsName("dest_s top")#.setParse Action(make_int )
row_8 = oneOf("F C").setResultsN ame("row_8")
length = integer.setResu ltsName("length ")#.setParseAct ion(make_int)
percent_id = float.setResult sName("percent_ id")#.setParseA ction(make_floa t)
row_11 = integer + Optional(Litera l("|") + commaSeparatedL ist )
)#.setResultsNa me("row_11")#.s etParseAction(m ake_int)
result = Word(alphas+"_" ).setResultsNam e("result")
row_13 = commaSeparatedL ist.setResultsN ame("row_13")

def make_alilines_s tatus_parser():
return alignment + row_1 + src_id + dest_id + src_start + src_stop
+ dest_start + dest_stop + row_8 + length + percent_id + row_11 +
result + row_13

def parse_alilines_ status(ifile):
alilines = make_alilines_s tatus_parser()
for l in ifile:
yield alilines.parseS tring( l )

However my parser always fails on lines of type 3. Does anyone know
why the Optional part is not working.

Many Thanks in advance

Nathan

[Peter Otten]

Nathan Harmston wrote:

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work. My file generally looks like this:

ALIGNMENT *1020 *YS2-10a02.q1k chr09 * * 1295 * * * 42 * *141045
142297 * C * *1254 95.06 1295 reject_bad_brea k 0

or this:

ALIGNMENT *36 * *YS2-10a08.q1k chrm * * *208 * * *165 * * 10745
10788 * C * * *44 95.45 593 reject_low 10,14

and my grammar work well for these lines, however somethings the row looks

like:

ALIGNMENT *53 * *YS2-10b03.p1k chr12 * * *180 * * *125 * 1067465
1067520 * C * * *56 98.21 532|5,2 reject_low 25

So I try to parse the 532 using

from pyparsing import *

integer = Word( nums )
float = Word( nums+".")
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNM ENT ").suppress ()
row_1 = integer.setResu ltsName("row_1" )#.setParseActi on(make_int)
src_id = identifier.setR esultsName("src _id")
dest_id = identifier.setR esultsName("des t_id")
src_start = integer.setResu ltsName("src_st art")#.setParse Action(make_int )
src_stop = integer.setResu ltsName("src_st op")#.setParseA ction(make_int)
dest_start =

integer.setResu ltsName("dest_s tart")#.setPars eAction(make_in t)

dest_stop = integer.setResu ltsName("dest_s top")#.setParse Action(make_int )
row_8 = oneOf("F C").setResultsN ame("row_8")
length = integer.setResu ltsName("length ")#.setParseAct ion(make_int)
percent_id =

float.setResult sName("percent_ id")#.setParseA ction(make_floa t)

row_11 = integer + Optional(Litera l("|") + commaSeparatedL ist )
)#.setResultsNa me("row_11")#.s etParseAction(m ake_int)
result = Word(alphas+"_" ).setResultsNam e("result")
row_13 = commaSeparatedL ist.setResultsN ame("row_13")

def make_alilines_s tatus_parser():
* * return alignment + row_1 + src_id + dest_id + src_start + src_stop
+ dest_start + dest_stop + row_8 + length + percent_id + row_11 +
result + row_13

def parse_alilines_ status(ifile):
* * alilines = make_alilines_s tatus_parser()
* * for l in ifile:
* * * * yield alilines.parseS tring( l )

However my parser always fails on lines of type 3. Does anyone know
why the Optional part is not working.

The commaSeparatedL ist includes the rest of the line into its last item:

>>commaSeparate dList.parseStri ng("a,b c")

(['a', 'b c'], {})

You can fix this by defining your own delimitedList that doesnt accept
whitespace, e. g.:

>>delimitedList (Word(alphanums )).parseString( "a,b c")

(['a', 'b'], {})

Peter

[Paul McGuire]

On Aug 16, 2:09 am, "Nathan Harmston" <ratchetg...@go oglemail.com>
wrote:

Hi,

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work.

<snip>

Thanks, Peter, your comments are dead-on.

Pyparsing-related posts crop up here every so often, usually with me
as the culprit. But to address your question, I've added a "Getting
Help" page to the pyparsing wiki, with links to the various mailing
lists and support pages on SourceForge. Fortunately, I don't think
this group minds an occasional trip down Pyparsing Lane (there've been
no serious complaints so far).

Some other suggestions/comments on your parser:
- Good use of results names. For debugging, you can print out the
parse results by using the dump() method - this gives you this kind of
output:
[1020, 'YS2-10a02.q1k', 'chr09', 1295, 42, 141045, 142297, 'C',
1254, 95.060000000000 002, 1295, 'reject_bad_bre ak', '0']
- dest_id: chr09
- dest_start: 141045
- dest_stop: 142297
- length: 1254
- percent_id: 95.06
- result: reject_bad_brea k
- row_1: 1020
- row_11: [1295]
- row_13: ['0']
- row_8: C
- src_id: YS2-10a02.q1k
- src_start: 1295
- src_stop: 42
This may give you some ideas on some more meaningful names ('row_8'
looks like an obvious candidate for replacement, for instance).

- If you are using pyparsing 1.4.7, you can abbreviate your calls to
setResultsName to just ("name"), that is, this:
src_id = identifier.setR esultsName("src _id")
can be written as just:
src_id = identifier("src _id")

- Don't use the name 'float' for the expression for a real number,
since this masks the builtin Python type float, and more importantly,
the builtin Python conversion function float (which you will want to
use in the parse action for this expression). Perhaps a name like
'real' or 'realNumber' would do.

- There is no need for the trailing space in defining
Literal("ALIGNM ENT").

- It is simpler to attach the parse actions to integer and real
themselves, rather than on each line where they are used. If you end
up with some integer field that you don't want converted, you can
define it in situ, like this:
zipCode = Word(nums) # don't use integer for this, don't want it
converted
or define a intString expression that does not do the conversion.

- With these changes, your grammar section becomes a little more
readable:
make_int = lambda t: int(t[0])
make_float = lambda t: float(t[0])
integer = Word( nums ).setParseActio n(make_int)
real = Word( nums+".").setPa rseAction(make_ float)
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNM ENT").suppress( )
row_1 = integer("row_1" )
src_id = identifier("src _id")
dest_id = identifier("des t_id")
src_start = integer("src_st art")
src_stop = integer("src_st op")
dest_start = integer("dest_s tart")
dest_stop = integer("dest_s top")
row_8 = oneOf("F C")("row_8")
length = integer("length ")
percent_id = real("percent_i d")
row_11 = (integer +
Optional(Litera l("|") +
delimitedList(W ord(nums))("sub Items"))
)("row_11")
result = Word(alphas+"_" )("result")
row_13 = commaSeparatedL ist("row_13")

(although all the ()'s make row_11 a little tough to follow - this
might deserve getting broken up into several lines.) And yes, I know
that PEP8 says not to align '=' signs on successive assignment
statments, but I truly believe it does help readability at times.

-- Paul