473,288 Members | 2,350 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,288 software developers and data experts.

parser recommendation

I have a project that uses a proprietary format and I've been using
regex to extract information from it. I haven't hit any roadblocks
yet, but I'd like to use a parsing library rather than maintain my own
code base of complicated regex's. I've been intrigued by the parsers
available in python, which may add some much needed flexibility.

I've briefly looked at PLY and pyparsing. There are several others,
but too many to enumerate. My understanding is that PLY (although
more difficult to use) has much more flexibility than pyparsing. I'm
basically looking to make an informed choice. Not just for this
project, but for the long haul. I'm not afraid of using a difficult
(to use or learn) parser either if it buys me something like
portability (with other languages) or flexibility).

I've been to a few websites that enumerate the parsers, but not all
that very helpful when it came to comparisons...

http://nedbatchelder.com/text/python-parsers.html
http://www.python.org/community/sigs...ards-standard/

I'm not looking to start a flame war... I'd just like some honest opinions.. ;)

thanks,
filipe
Jun 27 '08 #1
8 2878
On Jun 3, 8:43*am, "Filipe Fernandes" <fernandes...@gmail.comwrote:
>
I've briefly looked at PLY and pyparsing. *There are several others,
but too many to enumerate. *My understanding is that PLY (although
more difficult to use) has much more flexibility than pyparsing. *I'm
basically looking to make an informed choice. *Not just for this
project, but for the long haul. *I'm not afraid of using a difficult
(to use or learn) parser either if it buys me something like
portability (with other languages) or flexibility).
Short answer: try them both. Learning curve on pyparsing is about a
day, maybe two. And if you are already familiar with regex, PLY
should not seem too much of a stretch. PLY parsers will probably be
faster running than pyparsing parsers, but I think pyparsing parsers
will be quicker to work up and get running.

Longer answer: PLY is of the lex/yacc school of parsing libraries
(PLY=Python Lex/Yacc). Use regular expressions to define terminal
token specifications (a la lex). Then use "t_XXX" and "p_XXX" methods
to build up the parsing logic - docstrings in these methods capture
regex or BNF grammar definitions. In contrast, pyparsing is of the
combinator school of parsers. Within your Python code, you compose
your parser using '+' and '|' operations, building up the parser using
pyparsing classes such as Literal, Word, OneOrMore, Group, etc. Also,
pyparsing is 100% Python, so you wont have any portability issues
(don't know about PLY).

Here is a link to a page with a PLY and pyparsing example (although
not strictly a side-by-side comparison): http://www.rexx.com/~dkuhlman/python_201/.
For comparison, here is a pyparsing version of the PLY parser on that
page (this is a recursive grammar, not necessarily a good beginner's
example for pyparsing):
===============
term = Word(alphas,alphanums)

func_call = Forward()
func_call_list = Forward()
comma = Literal(",").suppress()
func_call_list << Group( func_call + Optional(comma +
func_call_list) )

lpar = Literal("(").suppress()
rpar = Literal(")").suppress()
func_call << Group( term + lpar +
Optional(func_call_list,default=[""]) + rpar )
command = func_call

prog = OneOrMore(command)

comment = "#" + restOfLine
prog.ignore( comment )
================
With the data set given at Dave Kuhlman's web page, here is the
output:
[['aaa', ['']],
['bbb', [['ccc', ['']]]],
['ddd',
[['eee', ['']],
[['fff', [['ggg', ['']], [['hhh', ['']], [['iii', ['']]]]]]]]]]

Pyparsing makes some judicious assumptions about how you will want to
parse, most significant being that whitespace can be ignored during
parsing (this *can* be overridden in the parser definition).
Pyparsing also supports token grouping (for building parse trees),
parse-time callbacks (called 'parse actions'), and assigning names
within subexpressions (called 'results names'), which really helps in
working with the tokens returned from the parsing process.

If you learn both, you may find that pyparsing is a good way to
quickly prototype a particular parsing problem, which you can then
convert to PLY for performance if necessary. The pyparsing prototype
will be an efficient way to work out what the grammar "kinks" are, so
that when you get around to PLY-ifying it, you already have a clear
picture of what the parser needs to do.

But, really, "more flexible"? I wouldn't really say that was the big
difference between the two.

Cheers,
-- Paul

(More pyparsing info at http://pyparsing.wikispaces.com.)
Jun 27 '08 #2
On Tue, Jun 3, 2008 at 10:41 AM, Paul McGuire <pt***@austin.rr.comwrote:
If you learn both, you may find that pyparsing is a good way to
quickly prototype a particular parsing problem, which you can then
convert to PLY for performance if necessary. The pyparsing prototype
will be an efficient way to work out what the grammar "kinks" are, so
that when you get around to PLY-ifying it, you already have a clear
picture of what the parser needs to do.
Thanks (both Paul and Kay) for responding. I'm still looking at Trail
in EasyExtend and pyparsing is very nicely objected oriented but PLY
does seems to have the speed advantage, so I'm leaning towards PLY

But I do have more questions... when reading the ply.py header (in
2.5) I found the following paragraph...

# The current implementation is only somewhat object-oriented. The
# LR parser itself is defined in terms of an object (which allows multiple
# parsers to co-exist). However, most of the variables used during table
# construction are defined in terms of global variables. Users shouldn't
# notice unless they are trying to define multiple parsers at the same
# time using threads (in which case they should have their head examined).

Now, I'm invariably going to have to use threads... I'm not exactly
sure what the author is alluding to, but my guess is that to overcome
this limitation I need to acquire a thread lock first before
"defining/creating" a parser object before I can use it?

Has anyone ran into this issue....? This would definitely be a
showstopper (for PLY anyway), if I couldn't create multiple parsers
because of threads. I'm not saying I need more than one, I'm just not
comfortable with that limitation.

I have a feeling I'm just misunderstanding since it doesn't seem to
hold you back from creating multiple parsers under a single process.

filipe
Jun 27 '08 #3
On 3 Jun., 19:34, "Filipe Fernandes" <fernandes...@gmail.comwrote:
# The current implementation is only somewhat object-oriented. The
# LR parser itself is defined in terms of an object (which allows multiple
# parsers to co-exist). However, most of the variables used during table
# construction are defined in terms of global variables. Users shouldn't
# notice unless they are trying to define multiple parsers at the same
# time using threads (in which case they should have their head examined).

Now, I'm invariably going to have to use threads... I'm not exactly
sure what the author is alluding to, but my guess is that to overcome
this limitation I need to acquire a thread lock first before
"defining/creating" a parser object before I can use it?
Nope. It just says that the parser-table construction itself relies on
global state. But you will most likely build your parser offline in a
separate run.

Jun 27 '08 #4
On Jun 3, 12:34*pm, "Filipe Fernandes" <fernandes...@gmail.comwrote:
On Tue, Jun 3, 2008 at 10:41 AM, Paul McGuire <pt...@austin.rr.comwrote:
But I do have more questions... when reading the ply.py header (in
2.5) I found the following paragraph...

# The current implementation is only somewhat object-oriented. The
# LR parser itself is defined in terms of an object (which allows multiple
# parsers to co-exist). *However, most of the variables used during table
# construction are defined in terms of global variables. *Users shouldn't
# notice unless they are trying to define multiple parsers at the same
# time using threads (in which case they should have their head examined).

Now, I'm invariably going to have to use threads... *I'm not exactly
sure what the author is alluding to, but my guess is that to overcome
this limitation I need to acquire a thread lock first before
"defining/creating" a parser object before I can use it?

Has anyone ran into this issue....? *This would definitely be a
showstopper (for PLY anyway), if I couldn't create multiple parsers
because of threads. *I'm not saying I need more than one, I'm just not
comfortable with that limitation.

I have a feeling I'm just misunderstanding since it doesn't seem to
hold you back from creating multiple parsers under a single process.

filipe
You can use pyparsing from any thread, and you can create multiple
parsers each running in a separate thread, but you cannot concurrently
use one parser from two different threads. Some users work around
this by instantiating a separate parser per thread using pickle to
quickly construct the parser at thread start time.

-- Paul
Jun 27 '08 #5
On Jun 3, 12:34 pm, "Filipe Fernandes" <fernandes...@gmail.comwrote:
>On Tue, Jun 3, 2008 at 10:41 AM, Paul McGuire <pt...@austin.rr.comwrote:
But I do have more questions... when reading the ply.py header (in
2.5) I found the following paragraph...

# The current implementation is only somewhat object-oriented. The
# LR parser itself is defined in terms of an object (which allows multiple
# parsers to co-exist). However, most of the variables used during table
# construction are defined in terms of global variables. Users shouldn't
# notice unless they are trying to define multiple parsers at the same
# time using threads (in which case they should have their head examined).

Now, I'm invariably going to have to use threads... I'm not exactly
sure what the author is alluding to, but my guess is that to overcome
this limitation I need to acquire a thread lock first before
"defining/creating" a parser object before I can use it?

Has anyone ran into this issue....? This would definitely be a
showstopper (for PLY anyway), if I couldn't create multiple parsers
because of threads. I'm not saying I need more than one, I'm just not
comfortable with that limitation.
On Tue, Jun 3, 2008 at 1:53 PM, Kay Schluehr <ka**********@gmx.netwrote:
Nope. It just says that the parser-table construction itself relies on
global state. But you will most likely build your parser offline in a
separate run.
Thanks Kay for the context.., I misunderstood completely, but your
last sentence coupled with a few running examples, cleared things
right up...

On Tue, Jun 3, 2008 at 4:36 PM, Paul McGuire <pt***@austin.rr.comwrote:
You can use pyparsing from any thread, and you can create multiple
parsers each running in a separate thread, but you cannot concurrently
use one parser from two different threads. Some users work around
this by instantiating a separate parser per thread using pickle to
quickly construct the parser at thread start time.
I didn't know that pyparsing wasn't thread safe. I kind of just
assumed because of it's OO approach. Thanks for the work around. I
haven't given up on pyparsing, although I'm now heavily leaning
towards PLY as an end solution since lex and yacc parsing is available
on other platforms as well.

Thanks Kay and Paul for the advice... I'm still using the first two I
started looking at, but I'm much for confident in the choices made.

filipe
Jun 27 '08 #6
On Jun 3, 2:55 pm, "Filipe Fernandes" <fernandes...@gmail.comwrote:
I haven't given up on pyparsing, although I'm now heavily leaning
towards PLY as an end solution since lex and yacc parsing is available
on other platforms as well.
Keep in mind that PLY's "compatibility" with YACC is functional,
not syntactical. That is, you can not take a YACC file, replace
the actions with Python actions and feed it to PLY.

It's a shame that the Python world has no truly YACC compatible
parser like YAPP in the Perl world.
Jun 27 '08 #7
One other possibility:
SimpleParse (for speed).
<URL:http://simpleparse.sourceforge.net/>
It is very nice.
Alan Isaac
Jun 27 '08 #8
On 6 Jun., 01:58, Alan Isaac <ais...@american.eduwrote:
One other possibility:
SimpleParse (for speed).
<URL:http://simpleparse.sourceforge.net/>
It is very nice.
Alan Isaac
How does SimpleParse manage left-factorings, left-recursion and other
ambiguities?

For example according to [1] there are two non-terminals

UNICODEESCAPEDCHAR_16

and

UNICODEESCAPEDCHAR_32

with an equal initial section of 4 token. How does SimpleParse detect
when it has to use the second production?

[1] http://simpleparse.sourceforge.net/s..._grammars.html
Jun 27 '08 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Karalius, Joseph | last post by:
Can anyone explain what is happening here? I haven't found any useful info on Google yet. Thanks in advance. mmagnet:/home/jkaralius/src/zopeplone/Python-2.3.5 # make gcc -pthread -c...
3
by: Himanshu Garg | last post by:
Hello, I am trying to pinpoint an apparent bug in HTML::Parser. The encoding of the text seems to change incorrectly if the locale isn't set properly. However Parser.pm in the directory...
1
by: CES | last post by:
All, Is their a way of testing if their is an XML parser installed and what the version number is with in JavaScript??? Thanks in advance. CES
11
by: Jorge Godoy | last post by:
Hi! I'm needing a parser to retrieve some information from source code -- including parts of code -- from Fortran, to use in a project with a documentation system. Any recommendations on a...
2
by: MrFredBloggs | last post by:
Hi All, I'm new to XML. My DTD: <?xml version='1.0' encoding='utf-8'?> <!DOCTYPE database > isn't accepted by an XML parser, it complians about the line:
8
by: SL | last post by:
I try to validate against a schema a document stored in several files thanks to external entities. The parseur add a 'xml:base="url"' attribute on the root element of this sub-trees during parsing,...
1
by: Ratbert | last post by:
Hi, I'm searching for a portable library, providing some sort of parsing functionality. The library should help to encode and decodes command phrases. Commands are plain text C-strings,...
15
by: an0047 | last post by:
Hello Would like to develop a simple XML parser with own commands The aproach is first to develop a state machine to later implement it in C. I had a look to some posts relating lexical...
0
by: UncleRic | last post by:
Environment: Mac OS X (10.4.10) on MacBook Pro I'm a Perl Neophyte. I've downloaded the XML::Parser module and am attempting to install it in my working directory (referenced via PERL5LIB env): ...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.