Gentlemen,
I'm running into a problem whilst testing the parsing of a language I've
created with TPG . It seems that for some reason, TPG balks when I try
to parse an expression whose first letter is 't' (or, in fact, at any
time when 't' is at the beginning of a token). This doesn't happen with
any other letter (as far as I know), nor if the 'T' is capitalised.
My grammar looks like this:
# Tokens
separator space '\s+';
token Num '\d+(.\d+)?';
token Ident '[a-zA-Z]\w*';
token CharList '\'.*\'';
token CatUnOp '~';
token CatOp '[/\^~]';
token MetaOp '[=\+\-!]';
token Date '\d\d-\d\d-\d\d\d\d';
token FileID '(\w+\.\w+)'
;
# Rules
START -> CatExpr '\?' '[' MetaExpr ']'
| CatExpr
| FileID
;
CatExpr -> CatUnOp CatName
| CatName (CatOp CatName)*
| CatName
;
CatName -> Ident
#| '(' CatExpr ')'
;
MetaExpr -> MetaCrit (',' MetaCrit)*
;
MetaCrit -> Ident MetaOp Value
;
Value -> CharList | Num | Date
;
My test script like this:
if __name__ == '__main__':
""" For testing purposes only """
parseTests = ('This/is/a/simple/test', 'another/simple/test',
"a/test/with/[author='drew']")
for line in parseTests:
try:
print "\nParsing: %s \n%s\n" % (line,"="*(len(line)+9))
qp = MFQueryParser()
print qp(line)
except Exception, inst:
print "EXCEPTION: " + str(inst)
The output when using a letter which is not 't':
Parsing: another/simple/test
============================
[ 1][ 2]START.CatExpr: (1,1) Ident another != CatUnOp
[ 2][ 3]START.CatExpr.CatName: (1,1) Ident another == Ident
[ 3][ 2]START.CatExpr: (1,8) CatOp / == CatOp
[ 4][ 3]START.CatExpr.CatName: (1,9) Ident simple == Ident
[ 5][ 2]START.CatExpr: (1,15) CatOp / == CatOp
[ 6][ 3]START.CatExpr.CatName: (1,16) _tok_2 t != Ident
[ 7][ 1]START: (1,15) CatOp / != _tok_1
[ 8][ 2]START.CatExpr: (1,1) Ident another != CatUnOp
[ 9][ 3]START.CatExpr.CatName: (1,1) Ident another == Ident
[ 10][ 2]START.CatExpr: (1,8) CatOp / == CatOp
[ 11][ 3]START.CatExpr.CatName: (1,9) Ident simple == Ident
[ 12][ 2]START.CatExpr: (1,15) CatOp / == CatOp
[ 13][ 3]START.CatExpr.CatName: (1,16) _tok_2 t != Ident
EXCEPTION: SyntacticError at line 1, row 16: Syntax error near t
The output when using 't' as the first letter:
Parsing: tanother/simple/test
=============================
[ 1][ 2]START.CatExpr: (1,1) _tok_2 t != CatUnOp
[ 2][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 3][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 4][ 2]START.CatExpr: (1,1) _tok_2 t != CatUnOp
[ 5][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 6][ 3]START.CatExpr.CatName: (1,1) _tok_2 t != Ident
[ 7][ 1]START: (1,1) _tok_2 t != FileID
EXCEPTION: SyntacticError at line 1, row 1: Syntax error near t
I'm not sure whether this is something I'm doing wrong in my regular
expressions or whether something is being escaped in the TPG code, or
whether.... I just don't know!
I'm going through the TPG code at the moment but am not very hopeful of
finding the problem. Could someone possibly just let me know if I've
made an obvious mistake somewhere?
Many thanks,
Andrew 5 2608
Andrew James wrote: Gentlemen,
I'm running into a problem whilst testing the parsing of a language I've created with TPG . It seems that for some reason, TPG balks when I try to parse an expression whose first letter is 't' (or, in fact, at any time when 't' is at the beginning of a token). This doesn't happen with any other letter (as far as I know), nor if the 'T' is capitalised.
I have no idea what TPG is, but if it's only having trouble with a lower
case t, could it be that the t is somehow getting escaped and TPG thinks
it's a tab character?
Just a thought.
greg
Greg,
Thanks for your response. Thought about this and tried with a lowercase
'n' and a few other letters to check it out. It ran fine, so I'm back to
square one ;-/
TPG is a parser generator for Python (which is really quite cool) and
can be found at: http://christophe.delord.free.fr/en/tpg/
Can anyone else offer any insight?
Regards,
Andrew
On Thu, 2004-11-18 at 13:48 +0000, Greg Krohn wrote: Andrew James wrote: Gentlemen,
I'm running into a problem whilst testing the parsing of a language I've created with TPG . It seems that for some reason, TPG balks when I try to parse an expression whose first letter is 't' (or, in fact, at any time when 't' is at the beginning of a token). This doesn't happen with any other letter (as far as I know), nor if the 'T' is capitalised.
I have no idea what TPG is, but if it's only having trouble with a lower case t, could it be that the t is somehow getting escaped and TPG thinks it's a tab character?
Just a thought.
greg
--
Andrew James <dr**@gremlinhosting.com>
Gentlemen,
Fixed what turned out to be my appalling knowledge of regexps. It turns
out that inline tokens are parsed as regular expressions by TPG and I
hadn't properly escaped the '[' characters. This means the START rule
has been changed to this:
START -> CatExpr ('\[' MetaExpr '\]')?
Now things work as expected, but I'm still unsure as to why this symptom
was only occurring with the uncapitalised character 't'.... can anyone
enlighten me?
Regards,
Andrew
On Thu, 2004-11-18 at 15:36 +0000, Andrew James wrote: Greg, Thanks for your response. Thought about this and tried with a lowercase 'n' and a few other letters to check it out. It ran fine, so I'm back to square one ;-/
TPG is a parser generator for Python (which is really quite cool) and can be found at: http://christophe.delord.free.fr/en/tpg/
Can anyone else offer any insight?
Regards, Andrew
On Thu, 2004-11-18 at 13:48 +0000, Greg Krohn wrote: Andrew James wrote: Gentlemen,
I'm running into a problem whilst testing the parsing of a language I've created with TPG . It seems that for some reason, TPG balks when I try to parse an expression whose first letter is 't' (or, in fact, at any time when 't' is at the beginning of a token). This doesn't happen with any other letter (as far as I know), nor if the 'T' is capitalised.
I have no idea what TPG is, but if it's only having trouble with a lower case t, could it be that the t is somehow getting escaped and TPG thinks it's a tab character?
Just a thought.
greg -- Andrew James <dr**@gremlinhosting.com>
--
Andrew James <dr**@gremlinhosting.com>
"Andrew James" <dr**@gremlinhosting.com> wrote in message
news:ma**************************************@pyth on.org... Gentlemen,
I'm running into a problem whilst testing the parsing of a language I've created with TPG . It seems that for some reason, TPG balks when I try to parse an expression whose first letter is 't' (or, in fact, at any time when 't' is at the beginning of a token). This doesn't happen with any other letter (as far as I know), nor if the 'T' is capitalised.
My grammar looks like this:
# Tokens separator space '\s+'; token Num '\d+(.\d+)?'; token Ident '[a-zA-Z]\w*'; token CharList '\'.*\''; token CatUnOp '~'; token CatOp '[/\^~]'; token MetaOp '[=\+\-!]'; token Date '\d\d-\d\d-\d\d\d\d'; token FileID '(\w+\.\w+)' ; # Rules START -> CatExpr '\?' '[' MetaExpr ']' | CatExpr | FileID ; CatExpr -> CatUnOp CatName | CatName (CatOp CatName)* | CatName ; CatName -> Ident #| '(' CatExpr ')' ; MetaExpr -> MetaCrit (',' MetaCrit)* ; MetaCrit -> Ident MetaOp Value ; Value -> CharList | Num | Date ;
My test script like this:
if __name__ == '__main__': """ For testing purposes only """ parseTests = ('This/is/a/simple/test', 'another/simple/test', "a/test/with/[author='drew']") for line in parseTests: try: print "\nParsing: %s \n%s\n" % (line,"="*(len(line)+9)) qp = MFQueryParser() print qp(line) except Exception, inst: print "EXCEPTION: " + str(inst)
<snip>
FYI, as a comparative data point, here is your parser implemented using
pyparsing. I had to change your last test case because it didn't seem to
match your grammar.
-- Paul
(Download pyparsing at http://pyparsing.sourceforge.net.)
from pyparsing import alphas, nums, alphanums, Word, Optional, oneOf, Group,
\
Literal, Combine, sglQuotedString, Forward, delimitedList, ZeroOrMore,
OneOrMore
integer = Word(nums)
num = Combine(integer + Optional("." + integer))
identChars = alphanums + "_$"
ident = Word(alphas, identChars)
charList = sglQuotedString
unop = Literal("~")
binop = oneOf("/ ^ ~")
metaOp = oneOf("= + - !")
date = Combine( Word(nums,exact=2) + "-" + Word(nums,exact=2) + "-" +
Word(nums,exact=4) )
fileId = Combine( Word(identChars) + "." + Word(identChars) )
value = charList | date | num
metaCrit = ident + metaOp + value
metaExpr = Group(delimitedList( metaCrit ))
expr = Forward()
name = ident | Group( "(" + expr + ")" )
expr << Group( ( unop + name ) | ( name + ZeroOrMore(binop + name) ) )
start = Group( expr + "?" + "[" + metaExpr + "]" ) | expr | fileId
parseTests = (
'This/is/a/simple/test',
'tanother/simple/test',
"a/test/with?[author='drew']"
)
for t in parseTests:
print start.parseString(t)
Output:
=====
[['This', '/', 'is', '/', 'a', '/', 'simple', '/', 'test']]
[['tanother', '/', 'simple', '/', 'test']]
[[['a', '/', 'test', '/', 'with'], '?', '[', ['author', '=', "'drew'"],
']']]
Paul,
Thanks for your detailed response. I've read through the PyParsing site
and I'm considering using it for further development of my project -
wish I'd found it before!
I noted your comments about my last test case; the syntax of the
language I'm implementing is closely related to XPath and the [] at the
end of the query are intended to contain metadata criteria. I have since
revised my parsing rules (see below) and they now parse both the
original test queries and arbitrarily complex boolean expressions.
I'm just about to post a new message with a description of what the
language is aimed at doing and asking for any improvements - I would
value your opinion if you have the time.
# Tokens
separator space '\s+';
token Num '\d+(.\d+)?';
token Ident '[a-zA-Z]\w*';
token CharList '\'.*\'';
token CatUnOp '~';
token CatOp '[/\^]';
token MetaOp '[=\+\-!]';
token Date '\d\d-\d\d-\d\d\d\d';
token FileID '(\w+\.\w+)';
token EmptyLine '^$';
# Rules
START -> CatExpr ('\[' MetaExpr '\]')?
| FileID
| EmptyLine
;
CatExpr -> CatUnOp CatName
| CatName (CatOp CatExpr)*
;
CatName -> Ident
| '\(' CatExpr '\)'
;
MetaExpr -> MetaCrit (',' MetaCrit)*
;
MetaCrit -> Ident MetaOp Value
;
Value -> CharList | Num | Date
;
parseTests = (
"simple",
"this/is/a/simple/test",
"a/test/with/metadata[author='drew',date=10]",
"music/mp3/~jackson/michael",
"docs/latex/~(computer^science)",
"media/video/((comedy/action)^thriller)"
)
Regards,
Andrew This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: David Moore |
last post by:
Hello
I posted a thread about this a while back, but I can't actually find it
again so I can reply to it with the solution I found, so I'm making a new
thread and hoping it goes to the top of...
|
by: mccoyn |
last post by:
I'm porting an old project to use .NET. When I try to link in a static library (.lib) that has a single managed class in it. I get the following errors:
LINK : error LNK2020: unresolved token...
|
by: Ian Griffiths |
last post by:
I have been given a schema, instances of which I'm required to be able to
consume and generate. I'd like to be able to manipulate these instances as
DataSets internally in my application. The...
|
by: Ben Midgley |
last post by:
Please have a look at the code below
My problem is the error message "Nonportable pointer conversion", so far as
I can tell there is no pointer conversion occuring here so I am assuming it
is...
|
by: Edward Mitchell |
last post by:
I have a problem that involves the error I receive when attempting to
complete the asp.net web application example (Walkthrough: Creating a Web
Application Using a Third-Party Business Object). ...
|
by: cranium.2003 |
last post by:
hi,
Here is my code
#include <iostream.h>
int main()
{
cout <<"HI";
return 0;
}
and using following command to compile a C++ program
g++ ex1.cpp -o ex1
|
by: kya2 |
last post by:
I am not able to create following store procedure.
CREATE PROCEDURE DBSAMBA.InsertDeleteBatch(OUT norows INT )
RESULT SETS 1
LANGUAGE SQL
BEGIN part1
DECLARE TOTAL_LEFT INT DEFAULT 0;
...
|
by: mrkafk |
last post by:
So I set out to learn handling three-letter-acronym files in Python,
and SAX worked nicely until I encountered badly formed XMLs, like with
bad characters in it (well Unicode supposed to handle it...
|
by: mrkafk |
last post by:
(this is a repost, for it's been a while since I posted this text via
Google Groups and it plain didn't appear on c.l.py - if it did appear
anyway, apols)
So I set out to learn handling...
|
by: CloudSolutions |
last post by:
Introduction:
For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
|
by: Faith0G |
last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome former...
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
| |