Python parser that records source ranges

[Jonathan Edwards]

The parser library module only records source line numbers for tokens. I
need a parser that records ranges of line and character locations for
each AST node, so I can map back to the source. Does anyone know of such
a thing? Thanks

Jonathan

[Jeff Epler]

The tokenize module will give column information for each token, but
it produces a stream of tokens only, not an AST.

Jeff

[logistix at cathoderaymission.net]

Jonathan Edwards <ed*****@nospam .lcs.mit.edu> wrote in message news:<qRKdb.456 249$Oz4.260848@ rwcrnsc54>...

The parser library module only records source line numbers for tokens. I
need a parser that records ranges of line and character locations for
each AST node, so I can map back to the source. Does anyone know of such
a thing? Thanks

Jonathan

You know there's not going to be a one-to-one relationship, right?
Most ast nodes are symbols and aren't going to match to any tokens.
Python asts also use a lot of intermediate nodes to enforce operator
precidence.

Anyway, I have some rather specialized code in PyXR that syncs tokens
to an ast. You probably won't be able to use it out of the box but it
should give you a good start:

http://www.cathoderaymission.net/~logistix/PyXR/

The source file of particular interest to you would be astToHtml.py:

http://tinyurl.com/p3cn

[Jonathan Edwards]

So the basic idea is to match up the leaves of the AST with the list of
tokens from tokenizer, which do contain location info. I had thought of
that, but was hoping there was a more informative parser out there.
Thanks.

Jonathan
logistix at cathoderaymissi on.net wrote:

Jonathan Edwards <ed*****@nospam .lcs.mit.edu> wrote in message news:<qRKdb.456 249$Oz4.260848@ rwcrnsc54>...

The parser library module only records source line numbers for tokens. I
need a parser that records ranges of line and character locations for
each AST node, so I can map back to the source. Does anyone know of such
a thing? Thanks

Jonathan

You know there's not going to be a one-to-one relationship, right?
Most ast nodes are symbols and aren't going to match to any tokens.
Python asts also use a lot of intermediate nodes to enforce operator
precidence.

Anyway, I have some rather specialized code in PyXR that syncs tokens
to an ast. You probably won't be able to use it out of the box but it
should give you a good start:

http://www.cathoderaymission.net/~logistix/PyXR/

The source file of particular interest to you would be astToHtml.py:

http://tinyurl.com/p3cn

[logistix at cathoderaymission.net]

Jonathan Edwards <ed*****@nospam .lcs.mit.edu> wrote in message news:<3F******* *******@nospam. lcs.mit.edu>...

So the basic idea is to match up the leaves of the AST with the list of
tokens from tokenizer, which do contain location info. I had thought of
that, but was hoping there was a more informative parser out there.
Thanks.

Jonathan

Its really not that bad. The more I think about it, the code
reference I sent you is way overcomplicated . General pseudocode for
walking asts generated via parser.ast2tupl e(parser.suite( code)) is:

def walk_node(node) :
if len(node) == 2 and type(node[1]) is not tuple:
walk_token(node )
else:
return walk_symbol(nod e)

def walk_symbol(nod e):
symbol_type = node[0]
symbol_leaves = node[1:]
for leave in symbol_leaves:
walk_node(nod)

def walk_token(node ):
token_type = node[0]
token_value = node[1]

[Paul Paterson]

"Jonathan Edwards" <ed*****@nospam .lcs.mit.edu> wrote in message
news:qRKdb.4562 49$Oz4.260848@r wcrnsc54...

The parser library module only records source line numbers for tokens. I
need a parser that records ranges of line and character locations for
each AST node, so I can map back to the source. Does anyone know of such
a thing? Thanks

Jonathan

If I understand you correctly, then the Simpleparse parser may be just what
you are looking for:

http://simpleparse.sourceforge.net

It is very powerful but still easy to use. The AST it produces gives the
start and end points of the matching tokens. Below is an example for parsing
a statement (from a VB grammar) ... you will see each node comprises a tuple
of (token_name, start_char, end_char, [sub_node1, sub_node2, ...]).

The example below looks rather complex because of the grammar, but you can
see that most of the sub_node matches all relate to the same characters in
the source. You can easily match each token to the corresponding text in the
source.

Paul

c("a = f(20, val)", verbose=1)

1 15
[('line_body',
0,
15,
[('single_statem ent',
0,
14,
[('assignment_st atement',
0,
14,
[('object', 0, 1, [('primary', 0, 1, [('identifier', 0, 1, [])])]),
('expression',
4,
14,
[('par_expressio n',
4,
14,
[('base_expressi on',
4,
14,
[('simple_expr',
4,
14,
[('call',
4,
14,
[('object',
4,
14,
[('primary',
4,
5,
[('identifier', 4, 5, [])]),
('parameter_lis t',
5,
14,
[('list',
5,
14,
[('bare_list',
6,
13,
[('bare_list_ite m',
6,
8,
[('expression',
6,
8,
[('par_expressio n',
6,
8,
[('base_expressi on',
6,
8,
[('simple_expr',
6,
8,
[('atom',
6,
8,
[('literal',
6,
8,
[('integer',
6,
8,
[('decimalintege r',
6,
8,
None)])])])])])])])]),
('bare_list_ite m',
10,
13,
[('expression',
10,
13,
[('par_expressio n',
10,
13,
[('base_expressi on',
10,
13,
[('simple_expr',
10,
13,
[('call',
10,
13,
[('object',
10,
13,
[('primary',
10,
13,
[('identifier',
10,
13,

[])])])])])])])])])])])])])])])])])])])]),
('line_end', 14, 15, [('NEWLINE', 14, 15, None)])])]