flex: plex?

[Peter Kleiweg]

Hi,

I am looking for a Python equivalent of Flex. After some
browsing, it seems that Plex is my best bet, but I would like to
hear suggestions before I dive in.

I have been using Flex a lot in combination with C programming.
I never use Yacc (or Bison), though Flex and Yacc are supposed
to be used together. But I found Yacc too cumbersome, and
unnecessary. Yacc is about context-free grammars, nice if you
like to describe an input language as a set of context-free
rewrite rules. Flex is about finite-state automata, but can be
extended to push-down automata (equivalent to context-free
grammars) and beyond, as long as it stays deterministic. I found
this to be very productive. I prefer automata over rewrite rules.

So, what I am looking for is a Python version of Flex that
supports the following capabilities, which are also supported by Plex:

- assigning any Python code to a matched input pattern

- input pattern recognition depending on 'states', which can be
changed through actions

Things that seems to be missing from Plex:

- patterns like:
A/B : match A only if followed by B, give A as result, keep B
in the input stream

- actions like:
REJECT: ask the flexer to come up with the next best match for the
same input
LESS(n): push n characters back into the input stream

All these are not really necessary, but they make some things
simpler.

So, are there any other packages that do these things better
than Plex?

--
Peter Kleiweg L:NL,af,da,de,en,ia,nds,no,sv,(fr,it) S:NL,de,en,(da,ia)
info: http://www.let.rug.nl/~kleiweg/ls.html

[Jeff Epler]

If plex uses python regular expressions, you can use the zero-length
lookahead assertion instead of flex's A/B syntax: "A(?=B)"

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBInqKJd01MZaTXX0RAh5dAKCivbanEnAb1N7QShDi5U LE/+32DwCfTG34
cb0k64glgd54ryIoemvZBSo=
=l+Mj
-----END PGP SIGNATURE-----

[Peter Kleiweg]

Jeff Epler schreef:

If plex uses python regular expressions, you can use the zero-length
lookahead assertion instead of flex's A/B syntax: "A(?=B)"

Alas, it doesn't.
--
Peter Kleiweg L:NL,af,da,de,en,ia,nds,no,sv,(fr,it) S:NL,de,en,(da,ia)
info: http://www.let.rug.nl/~kleiweg/ls.html

[David M. Cooke]

At some point, Peter Kleiweg <in*************@nl.invalid> wrote:

Hi,

I am looking for a Python equivalent of Flex. After some
browsing, it seems that Plex is my best bet, but I would like to
hear suggestions before I dive in.

I have been using Flex a lot in combination with C programming.
I never use Yacc (or Bison), though Flex and Yacc are supposed
to be used together. But I found Yacc too cumbersome, and
unnecessary. Yacc is about context-free grammars, nice if you
like to describe an input language as a set of context-free
rewrite rules. Flex is about finite-state automata, but can be
extended to push-down automata (equivalent to context-free
grammars) and beyond, as long as it stays deterministic. I found
this to be very productive. I prefer automata over rewrite rules.

If you've been using Flex, you might want to check out FlexModule at
<http://www.cs.utexas.edu/users/mcguire/software/fbmodule/>
which allows you to use Flex to generate a lexer callable from python.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca

[Peter Kleiweg]

David M. Cooke schreef:

If you've been using Flex, you might want to check out FlexModule at
<http://www.cs.utexas.edu/users/mcguire/software/fbmodule/>
which allows you to use Flex to generate a lexer callable from python.

I have tried it. (Had to edit the Makefile by hand, because
"make -f Makefile.pre.in boot" did not produce a valid Makefile,
but I did manage to build and run the example.) The problem is,
the Flex file (and Yacc file) is compiled into C, and then into
a shared lib. This means I must use C code in the Flex file.

What I want is something that lets me call arbitrary Python code
from the lexer. Ideal would be to have a version of Flex that
produces a Python file instead of a C file. I have kept looking,
and I think I found what I was looking for: PyLly, which is part
of PyGgy. http://www.lava.net/~newsham/pyggy/
--
Peter Kleiweg L:NL,af,da,de,en,ia,nds,no,sv,(fr,it) S:NL,de,en,(da,ia)
info: http://www.let.rug.nl/~kleiweg/ls.html