Strange re behavior: normal?

[Robin Munn]

How is re.split supposed to work? This wasn't at all what I expected:

[rmunn@localhost ~]$ python
Python 2.2.2 (#1, Jan 12 2003, 12:07:20)
[GCC 3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import re
re.split(r'\W+', 'a b c d') ['a', 'b', 'c', 'd'] # Expected result. .... re.split(r'\b', 'a b c d') ['a b c d'] # Huh?
Since \b matches the empty string, but only at the beginning and end of
a word, I would have expected re.split(r'\b', 'a b c d' to produce
either:

['', 'a', ' ', 'b', ' ', 'c', ' ', 'd', '']

or:

['a', ' ', 'b', ' ', 'c', ' ', 'd']

But I didn't expect that re.split(r'\b', 'a b c d') would yield no splits
whatsoever. The module doc says "split(pattern, string[, maxsplit = 0]):
split string by the occurrences of pattern". re.findall() seems to think
that \b occurs eight times in 'a b c d':
re.findall(r'\b', 'a b c d')

['', '', '', '', '', '', '', '']

So why doesn't re.split() think so? I'm puzzled.

--
Robin Munn <rm***@pobox.com> | http://www.rmunn.com/ | PGP key 0x6AFB6838
-----------------------------+-----------------------+----------------------
"Remember, when it comes to commercial TV, the program is not the product.
YOU are the product, and the advertiser is the customer." - Mark W. Schumann

[David C. Fox]

Robin Munn wrote:

How is re.split supposed to work? This wasn't at all what I expected:

[rmunn@localhost ~]$ python
Python 2.2.2 (#1, Jan 12 2003, 12:07:20)
[GCC 3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import re
re.split(r'\W+', 'a b c d')
['a', 'b', 'c', 'd']
# Expected result.
...
re.split(r'\b', 'a b c d')
['a b c d']
# Huh?

Since \b matches the empty string, but only at the beginning and end of
a word, I would have expected re.split(r'\b', 'a b c d' to produce
either:

['', 'a', ' ', 'b', ' ', 'c', ' ', 'd', '']

or:

['a', ' ', 'b', ' ', 'c', ' ', 'd']

But I didn't expect that re.split(r'\b', 'a b c d') would yield no splits
whatsoever. The module doc says "split(pattern, string[, maxsplit = 0]):
split string by the occurrences of pattern". re.findall() seems to think
that \b occurs eight times in 'a b c d':

re.findall(r'\b', 'a b c d')

['', '', '', '', '', '', '', '']

So why doesn't re.split() think so? I'm puzzled.

It looks like re.split is not splitting on zero-length matches. I get
similar behavior (in Python 2.2.2) if I try:

re.split('x*', 'a b c d')

or

re.split('(?=c)', 'a b c d')

I don't have the Python source handy to verify this hypothesis, though.

If this is correct, it should at least be documented.

David

[Mike Rovner]

Fredrik Lundh wrote:

Mike Rovner wrote:

IMHO that split behavior is a bug although technicaly it is not.
(From re manual:
"This module provides regular expression matching operations similar
to those found in Perl.")

is "split" really a matching operation?

fact is, all methods have Python-specific behaviour. it's just the RE
language itself that's based on Perl.

With all due respect to Python and not trying to bend it to any other
language
I believe it trys to do what user expects.

string.split() splits on (clearly documented nonempty) substring.
re.split() splits on RE.
splut(r'\b',...) clearly means (at least for me) 'split on word boundary'
It doesn't reject it (as in r'\b?') nor provide expected behavior.

I understand why it does what it does, but don't agree with it.

Regards,
Mike

[Fredrik Lundh]

Mike Rovner wrote:

With all due respect to Python and not trying to bend it to any other
language I believe it trys to do what user expects.

it does exactly what I expect it to do.

</F>

[John J. Lee]

"Fredrik Lundh" <fr*****@pythonware.com> writes:

Mike Rovner wrote:

With all due respect to Python and not trying to bend it to any other
language I believe it trys to do what user expects.

it does exactly what I expect it to do.

</F>

You don't count. ;-)
John

[Robin Munn]

Michael Janssen <Ja*****@rz.uni-frankfurt.de> wrote:

What's the good of splitting by boundaries? Someone else wanted this a
few days ago on tutor and I can't figure out a reason by now.

Heh. I bet I know the name of the person who was asking about this on
the tutor list. He's a friend of mine, and I've been helping him learn
Python. He E-mailed me about trying to split on word boundaries with
re.split(r'\b', 'some text'), and it was his E-mail that caused me to
discover that splitting by boundaries didn't do what I expected.

What's the good of it? As someone else pointed out, it allows you to
fetch the words and the separating text, yielding:

['See', ' ', 'Spot', '. ', 'See', ' ', 'Spot', ' ', 'run', '.']

which may be useful in certain English-language-parsing situations,
since it would allow you to look "ahead" or "back" from a word to see
what punctuation precedes or follows it.

Anyway, the re.split behavior I described isn't particularly bothering
me, but I do think it should be better documented. Time to submit a doc
patch, methinks...

--
Robin Munn <rm***@pobox.com> | http://www.rmunn.com/ | PGP key 0x6AFB6838
-----------------------------+-----------------------+----------------------
"Remember, when it comes to commercial TV, the program is not the product.
YOU are the product, and the advertiser is the customer." - Mark W. Schumann