Does Python optimize regexes?

[Jason Smith]

Hi. I just have a question about optimizations Python does when
converting to bytecode.

import re
for someString in someListOfStrin gs:
if re.match('foo', someString):
print someString, "matched!"

Does Python notice that re.match is called with the same expression, and
thus lift it out of the loop? Or do I need to always optimize by hand
using re.compile? I suspect so because the Python bytecode generator
would hardly know about a library function like re.compile, unlike e.g.
Perl, with builtin REs.

Thanks much for any clarification or advice.

--
Jason Smith
Open Enterprise Systems
Bangkok, Thailand
http://oes.co.th

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA4VGMm5q EoSbpT3kRAsYpAK ClQmFeamBfDx0vg TVpMc+utgmD/QCcDjpi
edwYF0cRA1V2Bvl qV6y4/l4=
=YMB7
-----END PGP SIGNATURE-----

[Peter Otten]

Jason Smith wrote:

Hi. I just have a question about optimizations Python does when
converting to bytecode.

import re
for someString in someListOfStrin gs:
if re.match('foo', someString):
print someString, "matched!"

Does Python notice that re.match is called with the same expression, and
thus lift it out of the loop? Or do I need to always optimize by hand
using re.compile? I suspect so because the Python bytecode generator
would hardly know about a library function like re.compile, unlike e.g.
Perl, with builtin REs.

Thanks much for any clarification or advice.

Python puts the compiled regular expressions into a cache. The relevant code
is in sre.py:

def match(pattern, string, flags=0):
return _compile(patter n, flags).match(st ring)

....

def _compile(*key):
p = _cache.get(key)
if p is not None:
return p
....

So not explicitly calling compile() in advance only costs you two function
calls and a dictionary lookup - and maybe some clarity in your code.

Peter

[Peter Otten]

Peter Otten wrote:

Python puts the compiled regular expressions into a cache. The relevant

By the way, re.compile() uses that cache, too:

import re
r1 = re.compile("abc ")
r2 = re.compile("abc ")
r1 is r2

True

Peter

[Michael Geary]

Peter Otten wrote:

Python puts the compiled regular expressions into a cache. The relevant
code is in sre.py:

def match(pattern, string, flags=0):
return _compile(patter n, flags).match(st ring)

...

def _compile(*key):
p = _cache.get(key)
if p is not None:
return p
...

So not explicitly calling compile() in advance only costs you two function
calls and a dictionary lookup - and maybe some clarity in your code.

That cost can be significant. Here's a test case where not precompiling the
regular expression increased the run time by more than 50%:

http://groups.google.com/groups?selm....supernews.com

-Mike

[Jason Smith]

Thanks much to Peter and Michael for the clarification.

Peter Otten wrote:

So not explicitly calling compile() in advance only costs you two function
calls and a dictionary lookup - and maybe some clarity in your code.

The reason I asked is because I felt that re.compile() was less clear:

someRegex = re.compile('sea rchforme')
while something:
theString = getTheString()
if someRegex.searc h(theString):
celebrate()

I wanted to remove someRegex since I can shave a line of code and some
confusion, but I was worried about re.search() in a loop.

The answer is this is smartly handled in Python, as opposed to bytecode
optimizations. Great!

--
Jason Smith
Open Enterprise Systems
Bangkok, Thailand
http://oes.co.th

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA4jN0m5q EoSbpT3kRAry2AJ 9RQQnHiGiR2S5bv 2CdOpOhMNXOdACe KfyO
a3iZduUZ5qmkOco BOkV3XEQ=
=D+ea
-----END PGP SIGNATURE-----

[Aahz]

In article <ma************ *************** ***********@pyt hon.org>,
Jason Smith <jh*@oes.co.t h> wrote:

The reason I asked is because I felt that re.compile() was less clear:

someRegex = re.compile('sea rchforme')
while something:
theString = getTheString()
if someRegex.searc h(theString):
celebrate()

I wanted to remove someRegex since I can shave a line of code and some
confusion, but I was worried about re.search() in a loop.

My reasoning is slightly different. I'm always forgetting with
re.search whether the pattern or string goes first; with re.compile, you
can't fail. Yesterday I fixed a couple of bugs where someone else made
the same error....
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"Typing is cheap. Thinking is expensive." --Roy Smith, c.l.py