473,549 Members | 2,862 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Problem using Optional pyparsing

Hi,

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work. My file generally looks like this:

ALIGNMENT 1020 YS2-10a02.q1k chr09 1295 42 141045
142297 C 1254 95.06 1295 reject_bad_brea k 0

or this:

ALIGNMENT 36 YS2-10a08.q1k chrm 208 165 10745
10788 C 44 95.45 593 reject_low 10,14

and my grammar work well for these lines, however somethings the row looks like:
ALIGNMENT 53 YS2-10b03.p1k chr12 180 125 1067465
1067520 C 56 98.21 532|5,2 reject_low 25

So I try to parse the 532 using

from pyparsing import *

integer = Word( nums )
float = Word( nums+".")
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNM ENT ").suppress ()
row_1 = integer.setResu ltsName("row_1" )#.setParseActi on(make_int)
src_id = identifier.setR esultsName("src _id")
dest_id = identifier.setR esultsName("des t_id")
src_start = integer.setResu ltsName("src_st art")#.setParse Action(make_int )
src_stop = integer.setResu ltsName("src_st op")#.setParseA ction(make_int)
dest_start = integer.setResu ltsName("dest_s tart")#.setPars eAction(make_in t)
dest_stop = integer.setResu ltsName("dest_s top")#.setParse Action(make_int )
row_8 = oneOf("F C").setResultsN ame("row_8")
length = integer.setResu ltsName("length ")#.setParseAct ion(make_int)
percent_id = float.setResult sName("percent_ id")#.setParseA ction(make_floa t)
row_11 = integer + Optional(Litera l("|") + commaSeparatedL ist )
)#.setResultsNa me("row_11")#.s etParseAction(m ake_int)
result = Word(alphas+"_" ).setResultsNam e("result")
row_13 = commaSeparatedL ist.setResultsN ame("row_13")

def make_alilines_s tatus_parser():
return alignment + row_1 + src_id + dest_id + src_start + src_stop
+ dest_start + dest_stop + row_8 + length + percent_id + row_11 +
result + row_13

def parse_alilines_ status(ifile):
alilines = make_alilines_s tatus_parser()
for l in ifile:
yield alilines.parseS tring( l )

However my parser always fails on lines of type 3. Does anyone know
why the Optional part is not working.

Many Thanks in advance

Nathan
Aug 16 '07 #1
2 1978
Nathan Harmston wrote:
I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work. My file generally looks like this:

ALIGNMENT *1020 *YS2-10a02.q1k chr09 * * 1295 * * * 42 * *141045
142297 * C * *1254 95.06 1295 reject_bad_brea k 0

or this:

ALIGNMENT *36 * *YS2-10a08.q1k chrm * * *208 * * *165 * * 10745
10788 * C * * *44 95.45 593 reject_low 10,14

and my grammar work well for these lines, however somethings the row looks
like:
ALIGNMENT *53 * *YS2-10b03.p1k chr12 * * *180 * * *125 * 1067465
1067520 * C * * *56 98.21 532|5,2 reject_low 25

So I try to parse the 532 using

from pyparsing import *

integer = Word( nums )
float = Word( nums+".")
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNM ENT ").suppress ()
row_1 = integer.setResu ltsName("row_1" )#.setParseActi on(make_int)
src_id = identifier.setR esultsName("src _id")
dest_id = identifier.setR esultsName("des t_id")
src_start = integer.setResu ltsName("src_st art")#.setParse Action(make_int )
src_stop = integer.setResu ltsName("src_st op")#.setParseA ction(make_int)
dest_start =
integer.setResu ltsName("dest_s tart")#.setPars eAction(make_in t)
dest_stop = integer.setResu ltsName("dest_s top")#.setParse Action(make_int )
row_8 = oneOf("F C").setResultsN ame("row_8")
length = integer.setResu ltsName("length ")#.setParseAct ion(make_int)
percent_id =
float.setResult sName("percent_ id")#.setParseA ction(make_floa t)
row_11 = integer + Optional(Litera l("|") + commaSeparatedL ist )
)#.setResultsNa me("row_11")#.s etParseAction(m ake_int)
result = Word(alphas+"_" ).setResultsNam e("result")
row_13 = commaSeparatedL ist.setResultsN ame("row_13")

def make_alilines_s tatus_parser():
* * return alignment + row_1 + src_id + dest_id + src_start + src_stop
+ dest_start + dest_stop + row_8 + length + percent_id + row_11 +
result + row_13

def parse_alilines_ status(ifile):
* * alilines = make_alilines_s tatus_parser()
* * for l in ifile:
* * * * yield alilines.parseS tring( l )

However my parser always fails on lines of type 3. Does anyone know
why the Optional part is not working.
The commaSeparatedL ist includes the rest of the line into its last item:
>>commaSeparate dList.parseStri ng("a,b c")
(['a', 'b c'], {})

You can fix this by defining your own delimitedList that doesnt accept
whitespace, e. g.:
>>delimitedList (Word(alphanums )).parseString( "a,b c")
(['a', 'b'], {})

Peter

Aug 16 '07 #2
On Aug 16, 2:09 am, "Nathan Harmston" <ratchetg...@go oglemail.com>
wrote:
Hi,

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work.
<snip>

Thanks, Peter, your comments are dead-on.

Pyparsing-related posts crop up here every so often, usually with me
as the culprit. But to address your question, I've added a "Getting
Help" page to the pyparsing wiki, with links to the various mailing
lists and support pages on SourceForge. Fortunately, I don't think
this group minds an occasional trip down Pyparsing Lane (there've been
no serious complaints so far).

Some other suggestions/comments on your parser:
- Good use of results names. For debugging, you can print out the
parse results by using the dump() method - this gives you this kind of
output:
[1020, 'YS2-10a02.q1k', 'chr09', 1295, 42, 141045, 142297, 'C',
1254, 95.060000000000 002, 1295, 'reject_bad_bre ak', '0']
- dest_id: chr09
- dest_start: 141045
- dest_stop: 142297
- length: 1254
- percent_id: 95.06
- result: reject_bad_brea k
- row_1: 1020
- row_11: [1295]
- row_13: ['0']
- row_8: C
- src_id: YS2-10a02.q1k
- src_start: 1295
- src_stop: 42
This may give you some ideas on some more meaningful names ('row_8'
looks like an obvious candidate for replacement, for instance).

- If you are using pyparsing 1.4.7, you can abbreviate your calls to
setResultsName to just ("name"), that is, this:
src_id = identifier.setR esultsName("src _id")
can be written as just:
src_id = identifier("src _id")

- Don't use the name 'float' for the expression for a real number,
since this masks the builtin Python type float, and more importantly,
the builtin Python conversion function float (which you will want to
use in the parse action for this expression). Perhaps a name like
'real' or 'realNumber' would do.

- There is no need for the trailing space in defining
Literal("ALIGNM ENT").

- It is simpler to attach the parse actions to integer and real
themselves, rather than on each line where they are used. If you end
up with some integer field that you don't want converted, you can
define it in situ, like this:
zipCode = Word(nums) # don't use integer for this, don't want it
converted
or define a intString expression that does not do the conversion.

- With these changes, your grammar section becomes a little more
readable:
make_int = lambda t: int(t[0])
make_float = lambda t: float(t[0])
integer = Word( nums ).setParseActio n(make_int)
real = Word( nums+".").setPa rseAction(make_ float)
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNM ENT").suppress( )
row_1 = integer("row_1" )
src_id = identifier("src _id")
dest_id = identifier("des t_id")
src_start = integer("src_st art")
src_stop = integer("src_st op")
dest_start = integer("dest_s tart")
dest_stop = integer("dest_s top")
row_8 = oneOf("F C")("row_8")
length = integer("length ")
percent_id = real("percent_i d")
row_11 = (integer +
Optional(Litera l("|") +
delimitedList(W ord(nums))("sub Items"))
)("row_11")
result = Word(alphas+"_" )("result")
row_13 = commaSeparatedL ist("row_13")

(although all the ()'s make row_11 a little tough to follow - this
might deserve getting broken up into several lines.) And yes, I know
that PEP8 says not to align '=' signs on successive assignment
statments, but I truly believe it does help readability at times.

-- Paul

Aug 16 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
1877
by: Guy Robinson | last post by:
I have the code below which parses an expression string and creates tokens. Can anyone suggest the best of error checking for things like: Valid variable only obj.attribute -whitespace allowed test( "ff*2/dd.r..ss r") #additional ..ss -invalid variable. test( "ff*$24..55/ddr") #double .. and $ -invalid number test( "ff*2/dd.r.ss...
10
1910
by: Angelo Secchi | last post by:
Hi, I have string of numbers and words like ',,,,,,23,,,asd,,,,,"name,surname",,,,,,,\n' and I would like to split (I'm using string.split()) it using comma as separator but I do not want to split in two also the "name,surname" field. In other word I would like python in separating fields to skip that particular comma.
15
17857
by: could ildg | last post by:
In re, the punctuation "^" can exclude a single character, but I want to exclude a whole word now. for example I have a string "hi, how are you. hello", I want to extract all the part before the world "hello", I can't use ".*" because "^" only exclude single char "h" or "e" or "l" or "o". Will somebody tell me how to do it? Thanks.
3
1986
by: rh0dium | last post by:
Hi all, I have a file which I need to parse and I need to be able to break it down by sections. I know it's possible but I can't seem to figure this out. The sections are broken by <> with one or more keywords in the <>. What I want to do is to be able to pars a particular section of the file. So for example I need to be able to look...
1
1856
by: David Hirschfield | last post by:
Anyone out there use simpleparse? If so, I have a problem that I can't seem to solve...I need to be able to parse this line: """Cen2 = Cen(OUT, "Cep", "ies", wh, 544, (wh/ht));""" with this grammar: grammar = r''' declaration := ws, line, (ws, line)*, ws line := (statement / assignment), ';', ws
13
2046
by: 7stud | last post by:
To the developer: 1) I went to the pyparsing wiki to download the pyparsing module and try it 2) At the wiki, there was no index entry in the table of contents for Downloads. After searching around a bit, I finally discovered a tiny link buried in some text at the top of the home page. 3) Link goes to sourceforge. At sourceforge, there...
1
1555
by: napolpie | last post by:
----Messaggio originale---- Da: napolpie@tin.it Data: 3-mag-2007 10.02 A: <python-list@python.org> Ogg: problem with meteo datas Hello, I'm Peter and I'm new in python codying and I'm using parsying to
1
2633
by: Steve | last post by:
Hi All (especially Paul McGuire!) Could you lend a hand in the grammar and paring of the output from the function win32pdhutil.ShowAllProcesses()? This is the code that I have so far (it is very clumsy at the moment) : import string
3
2639
by: John Carlyle-Clarke | last post by:
Hi. I'm new to Python and trying to use it to solve a specific problem. I have an XML file in which I need to locate a specific text node and replace the contents with some other text. The text in question is actually about 70k of base64 encoded data. I wrote some code that works on my Linux box using xml.dom.minidom, but it will not...
0
7518
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7715
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7956
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
7808
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5368
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3498
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1935
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1057
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
757
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.