Good String Tokenizer

[JamesHoward]

I have searched the board and noticed that there isn't really any sort
of good implementation of a string tokenizer that will tokenize based
on a custom set of tokens and return both the tokens and the parts
between the tokens.

For example, if I have the string:

"Hello, World! How are you?"

And my splitting points are comma, and exclamation point then I would
expect to get back.

["Hello", ",", " World", "!", " How are you?"]

Does anyone know of a tokenizer that will allow for this sort of use?

Thanks in advance,
Jim Howard

[Duncan Booth]

JamesHoward <Ja************@gmail.comwrote:

I have searched the board

what board? I don't see any boards here.

>
And my splitting points are comma, and exclamation point then I would
expect to get back.

["Hello", ",", " World", "!", " How are you?"]

Does anyone know of a tokenizer that will allow for this sort of use?

>>import re
re.split("([!,])", "Hello, World! How are you?")

['Hello', ',', ' World', '!', ' How are you?']