generic tokenizer

[Angus Mackay]

I remember python having a generic tokenizer in the library. all I want
is to set a list of token seperators and then read tokens out of a
stream, the token seperators should be returned as themselves.

is there anything like this?

cheers, Angus.

[Miki Tebeka]

Hello Angus,

I remember python having a generic tokenizer in the library. all I want
is to set a list of token seperators and then read tokens out of a
stream, the token seperators should be returned as themselves.

is there anything like this?

There are external lex/yacc like packages (like PLY
http://systems.cs.uchicago.edu/ply/). But maybe you're talking about shlex
(http://www.python.org/dev/doc/devel/...ule-shlex.html)

HTH.
--
------------------------------------------------------------------------
Miki Tebeka <mi*********@zo ran.com>
http://tebeka.spymac.net
The only difference between children and adults is the price of the toys

[Alex Martelli]

Angus Mackay <ye**@right.com > wrote:

I remember python having a generic tokenizer in the library. all I want
is to set a list of token seperators and then read tokens out of a
stream, the token seperators should be returned as themselves.

is there anything like this?

Not as such in the standard library: the functions in module tokenizer
do not let you 'set a list of token separators'. If what you're
tokenizing can fit in a string in memory, module re can help:

x=re.compile('( \s+|,|;)')
for w in x.split('a,b, c;d; e'): print repr(w),'+',

....
'a' + ',' + 'b' + ',' + '' + ' ' + 'c' + ';' + 'd' + ';' + '' + ' ' +
'e' +
Note that you get empty-string items when two separators abut.

If the limitations of re.split (stuff must fit in memory, &c) are a
problem, then the lexx-like solutions I see somebody else suggested may
be more appropriate for your needs.
Alex