473,856 Members | 1,438 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

PyParsing and Headaches

Hi,

I'm trying to construct a parser, but I'm stuck with some basic
stuff... For example, I want to match the following:

letter = "A"..."Z" | "a"..."z"
literal = letter+
include_bool := "+" | "-"
term = [include_bool] literal

So I defined this as:

literal = Word(alphas)
include_bool = Optional(oneOf( "+ -"))
term = include_bool + literal

The problem is that:

term.parseStrin g("+a") -(['+', 'a'], {}) # OK
term.parseStrin g("+ a") -(['+', 'a'], {}) # KO. It shouldn't
recognize any token since I didn't said the SPACE was allowed between
include_bool and literal.

Can anyone give me an hand here?

Cheers!

Hugo Ferreira

BTW, the following is the complete grammar I'm trying to implement with
pyparsing:

## L ::= expr | expr L
## expr ::= term | binary_expr
## binary_expr ::= term " " binary_op " " term
## binary_op ::= "*" | "OR" | "AND"
## include_bool ::= "+" | "-"
## term ::= ([include_bool] [modifier ":"] (literal | range)) | ("~"
literal)
## modifier ::= (letter | "_")+
## literal ::= word | quoted_words
## quoted_words ::= '"' word (" " word)* '"'
## word ::= (letter | digit | "_")+
## number ::= digit+
## range ::= number (".." | "...") number
## letter ::= "A"..."Z" | "a"..."z"
## digit ::= "0"..."9"

And this is where I got so far:

word = Word(nums + alphas + "_")
binary_op = oneOf("* and or", caseless=True). setResultsName( "operator")
include_bool = oneOf("+ -")
literal = (word | quotedString).s etResultsName(" literal")
modifier = Word(alphas + "_")
rng = Word(nums) + (Literal("..") | Literal("...")) + Word(nums)
term = ((Optional(incl ude_bool) + Optional(modifi er + ":") + (literal |
rng)) | ("~" + literal)).setRe sultsName("Term ")
binary_expr = (term + binary_op + term).setResult sName("binary")
expr = (binary_expr | term).setResult sName("Expr")
L = OneOrMore(expr)
--
GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85

Nov 22 '06 #1
4 1604
On Wed, Nov 22, 2006 at 11:17:52AM -0800, Bytter wrote:
Hi,

I'm trying to construct a parser, but I'm stuck with some basic
stuff... For example, I want to match the following:

letter = "A"..."Z" | "a"..."z"
literal = letter+
include_bool := "+" | "-"
term = [include_bool] literal

So I defined this as:

literal = Word(alphas)
include_bool = Optional(oneOf( "+ -"))
term = include_bool + literal
+ here means that you allow a space. You need to explicitly override this.
Try:

term = Combine(include _bool + literal)
>
The problem is that:

term.parseStrin g("+a") -(['+', 'a'], {}) # OK
term.parseStrin g("+ a") -(['+', 'a'], {}) # KO. It shouldn't
recognize any token since I didn't said the SPACE was allowed between
include_bool and literal.

Can anyone give me an hand here?

Cheers!

Hugo Ferreira

BTW, the following is the complete grammar I'm trying to implement with
pyparsing:

## L ::= expr | expr L
## expr ::= term | binary_expr
## binary_expr ::= term " " binary_op " " term
## binary_op ::= "*" | "OR" | "AND"
## include_bool ::= "+" | "-"
## term ::= ([include_bool] [modifier ":"] (literal | range)) | ("~"
literal)
## modifier ::= (letter | "_")+
## literal ::= word | quoted_words
## quoted_words ::= '"' word (" " word)* '"'
## word ::= (letter | digit | "_")+
## number ::= digit+
## range ::= number (".." | "...") number
## letter ::= "A"..."Z" | "a"..."z"
## digit ::= "0"..."9"

And this is where I got so far:

word = Word(nums + alphas + "_")
binary_op = oneOf("* and or", caseless=True). setResultsName( "operator")
include_bool = oneOf("+ -")
literal = (word | quotedString).s etResultsName(" literal")
modifier = Word(alphas + "_")
rng = Word(nums) + (Literal("..") | Literal("...")) + Word(nums)
term = ((Optional(incl ude_bool) + Optional(modifi er + ":") + (literal |
rng)) | ("~" + literal)).setRe sultsName("Term ")
binary_expr = (term + binary_op + term).setResult sName("binary")
expr = (binary_expr | term).setResult sName("Expr")
L = OneOrMore(expr)
--
GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85

--
http://mail.python.org/mailman/listinfo/python-list
Nov 22 '06 #2
"Bytter" <by****@gmail.c omwrote in message
news:11******** **************@ j44g2000cwa.goo glegroups.com.. .
Hi,

I'm trying to construct a parser, but I'm stuck with some basic
stuff... For example, I want to match the following:

letter = "A"..."Z" | "a"..."z"
literal = letter+
include_bool := "+" | "-"
term = [include_bool] literal

So I defined this as:

literal = Word(alphas)
include_bool = Optional(oneOf( "+ -"))
term = include_bool + literal

The problem is that:

term.parseStrin g("+a") -(['+', 'a'], {}) # OK
term.parseStrin g("+ a") -(['+', 'a'], {}) # KO. It shouldn't
recognize any token since I didn't said the SPACE was allowed between
include_bool and literal.
As Chris pointed out in his post, the most direct way to fix this is to use
Combine. Note that Combine does two things: it requires the expressions to
be adjacent, and it combines the results into a single token. For instance,
when defining the expression for a real number, something like:

realnum = Optional(oneOf( "+ -")) + Word(nums) + "." + Word(nums)

Pyparsing would parse "3.14159" into the separate tokens ['', '3', '.',
'14159']. For this grammar, pyparsing would also accept "2. 23" as ['',
'2', '.', '23'], even though there is a space between the decimal point and
"23". But by wrapping it inside Combine, as in:

realnum = Combine(Optiona l(oneOf("+ -")) + Word(nums) + "." + Word(nums))

we accomplish two things: pyparsing only matches if all the elements are
adjacent, with no whitespace or comments; and the matched token is returned
as ['3.14159']. (Yes, I left off scientific notation, but it is an
extension of the same issue.)

Pyparsing in general does implicit whitespace skipping; it is part of the
zen of pyparsing, and distinguishes it from conventional regexps (although I
think there is a new '?' switch for re's that puts '\s*'s between re terms
for you). This is to simplify the grammar definition, so that it doesn't
need to be littered with "optional whitespace or comments could go here"
expressions; instead, whitespace and comments (or "ignorables " in pyparsing
terminology) are parsed over before every grammar expression. I instituted
this out of recoil from a previous project, in which a co-developer
implemented a boolean parser by first tokenizing by whitespace, then parsing
out the tokens. Unfortunately, this meant that "color=='bl ue' &&
size=='medium'" would not parse successfully, instead requiring "color ==
'blue' && size == 'medium'". It doesn't seem like much, but our support
guys got many calls asking why the boolean clauses weren't matching. I
decided that when I wrote a parser, "y=m*x+b" would be just as parseable as
"y = m * x + b". For that matter, you'd be surprised where whitespace and
comments sneak in to people's source code: spaces after left parentheses and
comments after semicolons, for example, are easily forgotten when spec'ing
out the syntax for a C "for" statement; whitespace inside HTML tags is
another unanticipated surprise.

So looking at your grammar, you say you don't want to have this be a
successful parse:
term.parseStrin g("+ a") -(['+', 'a'], {})

because, "It shouldn't recognize any token since I didn't said the SPACE was
allowed between include_bool and literal." In fact, pyparsing allows spaces
by default, that's why the given parse succeeds. I would turn this question
around, and ask you in terms of your grammar - what SHOULD be allowed
between include_bool and literal? If spaces are not a problem, then your
grammar as-is is sufficient. If spaces are absolutely verboten, then there
are 2 or 3 different techniques in pyparsing to disable the
whitespace-skipping behavior, depending on whether you want all whitespace
skipping disabled, just for literals of a certain type, or just for literals
when following a leading include_bool sign.

Thanks for giving pyparsing a try; if you want further help, you can post
here, or on the pyparsing wiki - the discussion threads on the Home page are
a pretty good support and message log.

-- Paul
Nov 22 '06 #3
(This message has already been sent to the mailing-list, but I don't
have sure this is arriving well since it doesn't come up in the usenet,
so I'm posting it through here now.)

Chris,

Thanks for your quick answer. That changes a lot of stuff, and now I'm
able to do my parsing as I intended to.

Still, there's a remaining problem. By using Combine(), everything is
interpreted as a single token. Though what I need is that
'include_bool' and 'literal' be parsed as separated tokens, though
without a space in the middle...

Paul,

Thanks for your detailed explanation. One of the things I think is
missing from the documentation (or that I couldn't find easy) is the
kind of explanation you give about 'The Way of PyParsing'. For example,
It took me a while to understand that I could easily implement simple
recursions using OneOrMany(Group ()). Or maybe things were out there and
I didn't searched enough...

Still, fwiw, congratulations for the library. PyParsing allowed me to
do in just a couple of hours, including learning about it's API (minus
this little inconvenient) what would have taken me a couple of days
with, for example, ANTLR (in fact, I've already put aside ANTLR more
than once in the past for a built-from-scratch parser).

Cheers,

Hugo Ferreira

On Nov 22, 7:50 pm, Chris Lambacher <c...@kateandch ris.netwrote:
On Wed, Nov 22, 2006 at 11:17:52AM -0800, Bytter wrote:
Hi,
I'm trying to construct a parser, but I'm stuck with some basic
stuff... For example, I want to match the following:
letter = "A"..."Z" | "a"..."z"
literal = letter+
include_bool := "+" | "-"
term = [include_bool] literal
So I defined this as:
literal = Word(alphas)
include_bool = Optional(oneOf( "+ -"))
term = include_bool + literal+ here means that you allow a space. You need to explicitly override this.
Try:

term = Combine(include _bool + literal)
The problem is that:
term.parseStrin g("+a") -(['+', 'a'], {}) # OK
term.parseStrin g("+ a") -(['+', 'a'], {}) # KO. It shouldn't
recognize any token since I didn't said the SPACE was allowed between
include_bool and literal.
Can anyone give me an hand here?
Cheers!
Hugo Ferreira
BTW, the following is the complete grammar I'm trying to implement with
pyparsing:
## L ::= expr | expr L
## expr ::= term | binary_expr
## binary_expr ::= term " " binary_op " " term
## binary_op ::= "*" | "OR" | "AND"
## include_bool ::= "+" | "-"
## term ::= ([include_bool] [modifier ":"] (literal | range)) | ("~"
literal)
## modifier ::= (letter | "_")+
## literal ::= word | quoted_words
## quoted_words ::= '"' word (" " word)* '"'
## word ::= (letter | digit | "_")+
## number ::= digit+
## range ::= number (".." | "...") number
## letter ::= "A"..."Z" | "a"..."z"
## digit ::= "0"..."9"
And this is where I got so far:
word = Word(nums + alphas + "_")
binary_op = oneOf("* and or", caseless=True). setResultsName( "operator")
include_bool = oneOf("+ -")
literal = (word | quotedString).s etResultsName(" literal")
modifier = Word(alphas + "_")
rng = Word(nums) + (Literal("..") | Literal("...")) + Word(nums)
term = ((Optional(incl ude_bool) + Optional(modifi er + ":") + (literal |
rng)) | ("~" + literal)).setRe sultsName("Term ")
binary_expr = (term + binary_op + term).setResult sName("binary")
expr = (binary_expr | term).setResult sName("Expr")
L = OneOrMore(expr)
--
GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85
--
http://mail.python.org/mailman/listinfo/python-list
Nov 23 '06 #4
Heya there,

Ok, found the solution. I just needed to use leaveWhiteSpace () in the
places I want pyparsing to take into consideration the spaces.
Thx for the help.

Cheers!

Hugo Ferreira

On Nov 23, 11:57 am, "Bytter" <byt...@gmail.c omwrote:
(This message has already been sent to the mailing-list, but I don't
have sure this is arriving well since it doesn't come up in the usenet,
so I'm posting it through here now.)

Chris,

Thanks for your quick answer. That changes a lot of stuff, and now I'm
able to do my parsing as I intended to.

Still, there's a remaining problem. By using Combine(), everything is
interpreted as a single token. Though what I need is that
'include_bool' and 'literal' be parsed as separated tokens, though
without a space in the middle...

Paul,

Thanks for your detailed explanation. One of the things I think is
missing from the documentation (or that I couldn't find easy) is the
kind of explanation you give about 'The Way of PyParsing'. For example,
It took me a while to understand that I could easily implement simple
recursions using OneOrMany(Group ()). Or maybe things were out there and
I didn't searched enough...

Still, fwiw, congratulations for the library. PyParsing allowed me to
do in just a couple of hours, including learning about it's API (minus
this little inconvenient) what would have taken me a couple of days
with, for example, ANTLR (in fact, I've already put aside ANTLR more
than once in the past for a built-from-scratch parser).

Cheers,

Hugo Ferreira

On Nov 22, 7:50 pm, Chris Lambacher <c...@kateandch ris.netwrote:
On Wed, Nov 22, 2006 at 11:17:52AM -0800, Bytter wrote:
Hi,
I'm trying to construct a parser, but I'm stuck with some basic
stuff... For example, I want to match the following:
letter = "A"..."Z" | "a"..."z"
literal = letter+
include_bool := "+" | "-"
term = [include_bool] literal
So I defined this as:
literal = Word(alphas)
include_bool = Optional(oneOf( "+ -"))
term = include_bool + literal+ here means that you allow a space. You need to explicitly override this.
Try:
term = Combine(include _bool + literal)
The problem is that:
term.parseStrin g("+a") -(['+', 'a'], {}) # OK
term.parseStrin g("+ a") -(['+', 'a'], {}) # KO. It shouldn't
recognize any token since I didn't said the SPACE was allowed between
include_bool and literal.
Can anyone give me an hand here?
Cheers!
Hugo Ferreira
BTW, the following is the complete grammar I'm trying to implement with
pyparsing:
## L ::= expr | expr L
## expr ::= term | binary_expr
## binary_expr ::= term " " binary_op " " term
## binary_op ::= "*" | "OR" | "AND"
## include_bool ::= "+" | "-"
## term ::= ([include_bool] [modifier ":"] (literal | range)) | ("~"
literal)
## modifier ::= (letter | "_")+
## literal ::= word | quoted_words
## quoted_words ::= '"' word (" " word)* '"'
## word ::= (letter | digit | "_")+
## number ::= digit+
## range ::= number (".." | "...") number
## letter ::= "A"..."Z" | "a"..."z"
## digit ::= "0"..."9"
And this is where I got so far:
word = Word(nums + alphas + "_")
binary_op = oneOf("* and or", caseless=True). setResultsName( "operator")
include_bool = oneOf("+ -")
literal = (word | quotedString).s etResultsName(" literal")
modifier = Word(alphas + "_")
rng = Word(nums) + (Literal("..") | Literal("...")) + Word(nums)
term = ((Optional(incl ude_bool) + Optional(modifi er + ":") + (literal |
rng)) | ("~" + literal)).setRe sultsName("Term ")
binary_expr = (term + binary_op + term).setResult sName("binary")
expr = (binary_expr | term).setResult sName("Expr")
L = OneOrMore(expr)
--
GPG Fingerprint: B0D7 1249 447D F5BB 22C5 5B9B 078C 2615 504B 7B85
--
>http://mail.python.org/mailman/listinfo/python-list
Nov 23 '06 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
2360
by: Lukas Holcik | last post by:
Hi everyone! How can I simply search text for regexps (lets say <a href="(.*?)">(.*?)</a>) and save all URLs(1) and link contents(2) in a dictionary { name : URL}? In a single pass if it could. Or how can I replace the html &entities; in a string "blablabla&amp;blablabal&amp;balbalbal" with the chars they mean using re.sub? I found out they are stored in an dict . I though about this functionality:
4
2085
by: the.theorist | last post by:
Hey, I'm trying my hand and pyparsing a log file (named l.log): FIRSTLINE PROPERTY1 DATA1 PROPERTY2 DATA2 PROPERTYS LIST ID1 data1 ID2 data2
3
2000
by: rh0dium | last post by:
Hi all, I have a file which I need to parse and I need to be able to break it down by sections. I know it's possible but I can't seem to figure this out. The sections are broken by <> with one or more keywords in the <>. What I want to do is to be able to pars a particular section of the file. So for example I need to be able to look at the SYSLIB section. Presumably the sections are
13
2082
by: 7stud | last post by:
To the developer: 1) I went to the pyparsing wiki to download the pyparsing module and try it 2) At the wiki, there was no index entry in the table of contents for Downloads. After searching around a bit, I finally discovered a tiny link buried in some text at the top of the home page. 3) Link goes to sourceforge. At sourceforge, there was a nice, green 'download' button that stood out from the page. 4) I clicked on the download...
1
2656
by: Steve | last post by:
Hi All (especially Paul McGuire!) Could you lend a hand in the grammar and paring of the output from the function win32pdhutil.ShowAllProcesses()? This is the code that I have so far (it is very clumsy at the moment) : import string
1
353
by: Neal Becker | last post by:
I'm just trying out pyparsing. I get stack overflow on my first try. Any help? #/usr/bin/python from pyparsing import Word, alphas, QuotedString, OneOrMore, delimitedList first_line = '' def main():
18
4744
by: Just Another Victim of the Ambient Morality | last post by:
Is pyparsing really a recursive descent parser? I ask this because there are grammars it can't parse that my recursive descent parser would parse, should I have written one. For instance: from pyparsing import * grammar = OneOrMore(Word(alphas)) + Literal('end') grammar.parseString('First Second Third end')
3
1735
by: hubritic | last post by:
I am trying to parse data that looks like this: IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION 2BFA76F6 1208230607 T S SYSPROC SYSTEM SHUTDOWN BY USER A6D1BD62 1215230807 I H Firmware Event My problem is that sometimes there is a RESOURCE_NAME and sometimes not, so I wind up with "Firmware" as my RESOURCE_NAME and "Event" as
5
1497
by: Paul McGuire | last post by:
I've just uploaded to SourceForge and PyPI the latest update to pyparsing, version 1.5.1. It has been a couple of months since 1.5.0 was released, and a number of bug-fixes and enhancements have accumulated in SVN, so time for a release! Here's what's new in Pyparsing 1.5.1: - Added __dir__() methods to ParseBaseException and ParseResults, to support new dir() behavior in Py2.6 and Py3.0. If dir() is called on a ParseResults object,...
0
9922
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
11066
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10702
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10800
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9536
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7098
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5763
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5962
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
3
3203
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.