Strange re behavior: normal?

[Robin Munn]

How is re.split supposed to work? This wasn't at all what I expected:

[rmunn@localhost ~]$ python
Python 2.2.2 (#1, Jan 12 2003, 12:07:20)
[GCC 3.2] on linux2
Type "help", "copyright" , "credits" or "license" for more information.

import re
re.split(r'\W+' , 'a b c d') ['a', 'b', 'c', 'd'] # Expected result. .... re.split(r'\b', 'a b c d') ['a b c d'] # Huh?
Since \b matches the empty string, but only at the beginning and end of
a word, I would have expected re.split(r'\b', 'a b c d' to produce
either:

['', 'a', ' ', 'b', ' ', 'c', ' ', 'd', '']

or:

['a', ' ', 'b', ' ', 'c', ' ', 'd']

But I didn't expect that re.split(r'\b', 'a b c d') would yield no splits
whatsoever. The module doc says "split(patt ern, string[, maxsplit = 0]):
split string by the occurrences of pattern". re.findall() seems to think
that \b occurs eight times in 'a b c d':
re.findall(r'\b ', 'a b c d')

['', '', '', '', '', '', '', '']

So why doesn't re.split() think so? I'm puzzled.

--
Robin Munn <rm***@pobox.co m> | http://www.rmunn.com/ | PGP key 0x6AFB6838
-----------------------------+-----------------------+----------------------
"Remember, when it comes to commercial TV, the program is not the product.
YOU are the product, and the advertiser is the customer." - Mark W. Schumann

[David C. Fox]

Robin Munn wrote:

How is re.split supposed to work? This wasn't at all what I expected:

[rmunn@localhost ~]$ python
Python 2.2.2 (#1, Jan 12 2003, 12:07:20)
[GCC 3.2] on linux2
Type "help", "copyright" , "credits" or "license" for more information.

import re
re.split(r' \W+', 'a b c d')
['a', 'b', 'c', 'd']
# Expected result.
...
re.split(r' \b', 'a b c d')
['a b c d']
# Huh?

Since \b matches the empty string, but only at the beginning and end of
a word, I would have expected re.split(r'\b', 'a b c d' to produce
either:

['', 'a', ' ', 'b', ' ', 'c', ' ', 'd', '']

or:

['a', ' ', 'b', ' ', 'c', ' ', 'd']

But I didn't expect that re.split(r'\b', 'a b c d') would yield no splits
whatsoever. The module doc says "split(patt ern, string[, maxsplit = 0]):
split string by the occurrences of pattern". re.findall() seems to think
that \b occurs eight times in 'a b c d':

re.findall( r'\b', 'a b c d')

['', '', '', '', '', '', '', '']

So why doesn't re.split() think so? I'm puzzled.

It looks like re.split is not splitting on zero-length matches. I get
similar behavior (in Python 2.2.2) if I try:

re.split('x*', 'a b c d')

or

re.split('(?=c) ', 'a b c d')

I don't have the Python source handy to verify this hypothesis, though.

If this is correct, it should at least be documented.

David

[Mike Rovner]

Fredrik Lundh wrote:

Mike Rovner wrote:

IMHO that split behavior is a bug although technicaly it is not.
(From re manual:
"This module provides regular expression matching operations similar
to those found in Perl.")

is "split" really a matching operation?

fact is, all methods have Python-specific behaviour. it's just the RE
language itself that's based on Perl.

With all due respect to Python and not trying to bend it to any other
language
I believe it trys to do what user expects.

string.split() splits on (clearly documented nonempty) substring.
re.split() splits on RE.
splut(r'\b',... ) clearly means (at least for me) 'split on word boundary'
It doesn't reject it (as in r'\b?') nor provide expected behavior.

I understand why it does what it does, but don't agree with it.

Regards,
Mike

[Fredrik Lundh]

Mike Rovner wrote:

With all due respect to Python and not trying to bend it to any other
language I believe it trys to do what user expects.

it does exactly what I expect it to do.

</F>

[John J. Lee]

"Fredrik Lundh" <fr*****@python ware.com> writes:

Mike Rovner wrote:

With all due respect to Python and not trying to bend it to any other
language I believe it trys to do what user expects.

it does exactly what I expect it to do.

</F>

You don't count. ;-)
John

[Robin Munn]

Michael Janssen <Ja*****@rz.u ni-frankfurt.de> wrote:

What's the good of splitting by boundaries? Someone else wanted this a
few days ago on tutor and I can't figure out a reason by now.

Heh. I bet I know the name of the person who was asking about this on
the tutor list. He's a friend of mine, and I've been helping him learn
Python. He E-mailed me about trying to split on word boundaries with
re.split(r'\b', 'some text'), and it was his E-mail that caused me to
discover that splitting by boundaries didn't do what I expected.

What's the good of it? As someone else pointed out, it allows you to
fetch the words and the separating text, yielding:

['See', ' ', 'Spot', '. ', 'See', ' ', 'Spot', ' ', 'run', '.']

which may be useful in certain English-language-parsing situations,
since it would allow you to look "ahead" or "back" from a word to see
what punctuation precedes or follows it.

Anyway, the re.split behavior I described isn't particularly bothering
me, but I do think it should be better documented. Time to submit a doc
patch, methinks...

--
Robin Munn <rm***@pobox.co m> | http://www.rmunn.com/ | PGP key 0x6AFB6838
-----------------------------+-----------------------+----------------------
"Remember, when it comes to commercial TV, the program is not the product.
YOU are the product, and the advertiser is the customer." - Mark W. Schumann