By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
444,100 Members | 2,892 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 444,100 IT Pros & Developers. It's quick & easy.

Regexp optimization question

P: n/a
I'm working on a project (Atox) where I need to match quite a few
regular expressions (several hundred) in reasonably large text files.
I've found that this can easily get rather slow. (There are many
things that slow Atox down -- it hasn't been designed for speed, and
any optimizations will entail quite a bit of refactoring.)

I've tried to speed this up by using the same trick as SPARK, putting
all the regexps into a single or-group in a new regexp. That helped a
*lot* -- but now I have to find out which one of them matched at a
certain location. I haven't yet looked at the performance of the code
for checking this, because I encountered a problem before that: Using
named groups will only work for 100 patterns -- not a terrible
problem, since I can create several 100-group patterns -- and using
named groups slows down the matching *a lot*. As far as I could tell,
using named groups actually was slower than simply matching the
patterns one by one.

So: What can I do? Is there any way of getting more speed here, except
implementing the matching code (i.e. the code right around the calls
to _re) in C or Pyrex? (I've tried using Psyco, but that didn't help;
I guess it might help if I implemented things differently...)

Any ideas?

--
Magnus Lie Hetland "Oppression and harassment is a small price to pay
http://hetland.org to live in the land of the free." -- C. M. Burns
Jul 18 '05 #1
Share this Question
Share on Google+
19 Replies


P: n/a
Magnus Lie Hetland wrote:
I've tried to speed this up by using the same trick as SPARK, putting
all the regexps into a single or-group in a new regexp. That helped a
*lot* -- but now I have to find out which one of them matched at a
certain location.


Are you using the .lastindex attribute of match objects yet?

Martin

Jul 18 '05 #2

P: n/a
Magnus Lie Hetland <ml*@furu.idi.ntnu.no> wrote:
Any ideas?


Few concrete examples, perhaps? Sadly, my telepathetic power is not
what it used to be...

--
William Park, Open Geometry Consulting, <op**********@yahoo.ca>
Linux solution/training/migration, Thin-client
Jul 18 '05 #3

P: n/a

"Magnus Lie Hetland" <ml*@furu.idi.ntnu.no> schrieb im Newsbeitrag
news:slrnc8gal3.9da.ml*@furu.idi.ntnu.no...

Any ideas?


Maybe Plex is helpful. I did not use it already, but it seems to adress your
problem
The author of Plex is Greg Ewing. He build Pyrex on top of Plex

The documentation
http://www.cosc.canterbury.ac.nz/~gr...doc/index.html
contains

"""
Plex is designed to fill a need that is left wanting by the existing Python
regular expression modules. If you've ever tried to use one of them for
implementing a scanner, you will have found that they're not really suited
to the task. You can define a bunch of regular expressions which match your
tokens all right, but you can only match one of them at a time against your
input. To match all of them at once, you have to join them all together into
one big r.e., but then you've got no easy way to tell which one matched.
This is the problem that Plex is designed to solve.
"""

Hope I could help you
Guenter
Jul 18 '05 #4

P: n/a
Magnus Lie Hetland wrote:
but now I have to find out which one of them matched at a
certain location.


the techniques discussed in this article may be helpful:

http://effbot.org/zone/xml-scanner.htm

</F>


Jul 18 '05 #5

P: n/a
In article <c6*************@news.t-online.com>, Martin v. Löwis wrote:
Magnus Lie Hetland wrote:
I've tried to speed this up by using the same trick as SPARK, putting
all the regexps into a single or-group in a new regexp. That helped a
*lot* -- but now I have to find out which one of them matched at a
certain location.
Are you using the .lastindex attribute of match objects yet?


Dang! I thought I had read the docs ;)

Seems very useful. I actually need to know all the matches, because
which one gets selected depends on the current priorities (ordering)
in the parser, and cannot be fixed for the scanner. (I could put the
patterns in the or-group in reverse prioritized order, but that would
have to be fixed...)

Thanks for the tip, anyway.
Martin


--
Magnus Lie Hetland "Wake up!" - Rage Against The Machine
http://hetland.org "Shut up!" - Linkin Park
Jul 18 '05 #6

P: n/a
In article <c6************@ID-99293.news.uni-berlin.de>, William Park wrote:
Magnus Lie Hetland <ml*@furu.idi.ntnu.no> wrote:
Any ideas?


Few concrete examples, perhaps? Sadly, my telepathetic power is not
what it used to be...


Well... Hard to give specific examples, as the specifics will be
user-specified.

But in order to explain what I'm doing, I could give a rather generic
example.

I might have a bunch of regexps like 'foo1', 'foo2', 'foo3', ...,
'foo500' (these would be more complex, of course).

Now I can do something like this:

hits = []
for pat in pats:
hits.extend(pat.finditer(text))
# Maybe sort them in order of occurrence

*Or* I can do something like this:

bigPat = '(' + '|'.join(pats) + ')'
hits = list(bigPat.finditer(text))

The last one is *much* faster -- but only if I forego named groups.
And without them, I can't find out which patterns matched at which
locations. (I can, as suggested, use .lastindex to find *one* of them,
but not all, as I need.)

I'm sure there are plenty of bottlenecks in my code, but this seems to
be one of them, and it's slow even if I run it alone (without any of
the rest of my parsing code).

--
Magnus Lie Hetland "Wake up!" - Rage Against The Machine
http://hetland.org "Shut up!" - Linkin Park
Jul 18 '05 #7

P: n/a
In article <c6**********@newsreader2.netcologne.de>, Günter Jantzen wrote:

"Magnus Lie Hetland" <ml*@furu.idi.ntnu.no> schrieb im Newsbeitrag
news:slrnc8gal3.9da.ml*@furu.idi.ntnu.no...

Any ideas?

Maybe Plex is helpful. I did not use it already, but it seems to adress your
problem


Ahah.
The author of Plex is Greg Ewing. He build Pyrex on top of Plex
Yeah, I know -- I just never looked at Plex in detail.
The documentation
http://www.cosc.canterbury.ac.nz/~gr...doc/index.html
contains [snip]

Yeah, I looked at the docs, and it looks very promising!

One of the reasons I've avoided existing lexers is that I don't do
standard tokenization -- I don't partition all of the text into regexp
tokens. I allow the lexer to skip over text -- somewhat like how
whitespace is normally handled, except that this can be *any* text --
and to return the next token that is of any use to the current parsing
rule.

But who knows -- I may be able to use Plex anyway.

One problem might be that the regexp format seems to be quite
stripped-down (although, of course, a regexp is a regexp,
theoretically speaking ;)

No non-greedy matching, no lookahead/lookback etc.

But if Plex gives a real performance boost, I may be able to live with
that. (The regexp part is functionality that is available to the user,
in my case.)
Hope I could help you
It might. Thanks.
Guenter


--
Magnus Lie Hetland "Wake up!" - Rage Against The Machine
http://hetland.org "Shut up!" - Linkin Park
Jul 18 '05 #8

P: n/a
In article <ma**************************************@python.o rg>,
Fredrik Lundh wrote:
Magnus Lie Hetland wrote:
but now I have to find out which one of them matched at a
certain location.
the techniques discussed in this article may be helpful:

http://effbot.org/zone/xml-scanner.htm


Yes, this is indeed useful. The most useful thing, I guess, is the
lastindex attribute (which was pointed out earlier) -- but I suspect I
can't use it, as I have to know all the patterns that were matched at
a given location (and let the parser choose which one to use, based
on its context-dependent priorities). Or, at least, that's how it
seems to me now.

I *could*, of course, say that ambiguity is not allowed -- or that the
effects aren't defined. Or maybe I could order the patterns in terms
of specificity (but analyzing regexps for specificity is no mean feat,
of course ;)

Thanks for the pointer, though -- a nice tutorial!
</F>


--
Magnus Lie Hetland "Wake up!" - Rage Against The Machine
http://hetland.org "Shut up!" - Linkin Park
Jul 18 '05 #9

P: n/a
Magnus Lie Hetland <ml*@furu.idi.ntnu.no> wrote:
Now I can do something like this:

hits = []
for pat in pats:
hits.extend(pat.finditer(text))
# Maybe sort them in order of occurrence

*Or* I can do something like this:

bigPat = '(' + '|'.join(pats) + ')'
hits = list(bigPat.finditer(text))

The last one is *much* faster -- but only if I forego named groups.
And without them, I can't find out which patterns matched at which
locations. (I can, as suggested, use .lastindex to find *one* of them,
but not all, as I need.)


Since you want both the matched strings and their locations in file, you
pretty much have to this manually, one by one.

--
William Park, Open Geometry Consulting, <op**********@yahoo.ca>
Linux solution/training/migration, Thin-client
Jul 18 '05 #10

P: n/a
In article <c6************@ID-99293.news.uni-berlin.de>, William Park wrote:
[snip]

Since you want both the matched strings and their locations in file, you
pretty much have to this manually, one by one.


Well -- that's more or less what I'm doing. (Or -- I can get the match
objects all at once, of course, using finditer.)

I guess I'll have to look elsewhere for performance improvements. Hm.

--
Magnus Lie Hetland "Wake up!" - Rage Against The Machine
http://hetland.org "Shut up!" - Linkin Park
Jul 18 '05 #11

P: n/a
Magnus Lie Hetland <ml*@furu.idi.ntnu.no> wrote:
In article <c6************@ID-99293.news.uni-berlin.de>, William Park wrote:
[snip]

Since you want both the matched strings and their locations in file, you
pretty much have to this manually, one by one.


Well -- that's more or less what I'm doing. (Or -- I can get the match
objects all at once, of course, using finditer.)

I guess I'll have to look elsewhere for performance improvements. Hm.


You can write up something in C, as standalone or as patch to Bash
shell. Essentially,
- locate start of string match
- print location (and the string)
- move pointer past the end of string.
- repeat.

Closest you can do using standary tools is strings and byte-offset of
the lines the strings occurs.

--
William Park, Open Geometry Consulting, <op**********@yahoo.ca>
Linux solution/training/migration, Thin-client
Jul 18 '05 #12

P: n/a
> One of the reasons I've avoided existing lexers is that I don't do
standard tokenization -- I don't partition all of the text into regexp
tokens. I allow the lexer to skip over text -- somewhat like how
whitespace is normally handled, except that this can be *any* text --
and to return the next token that is of any use to the current parsing
rule.


pyparsing supports this kind of text skipping, using scanString() instead of
parseString(). scanString() is actually a generator, yielding for each
match a tuple consisting of:
- matched tokens (returned as a ParseResults object - sort of super-list,
supporting simple list semantics, but some tokens can be named and accessed
as in a dictionary or as attributes)
- start location in source string
- end location in source string

Download pyparsing at http://pyparsing.sourceforge.net .

-- Paul
Jul 18 '05 #13

P: n/a
In article <kb******************@fe2.texas.rr.com>, Paul McGuire
wrote: [snip]

pyparsing supports this kind of text skipping, using scanString()
instead of parseString().


I already have a working implementation in Python -- if this isn't
more efficient (I'm just talking about the tokenization part) I don't
think there would be much gain in switching.

(IIRC, the pyparsing docs say that pyparsing is slow for complex
grammars, at least.)

BTW: I have not done some experiments with Plex with lots of regular
expressiosn; simply compiling a pattern with 500 alternatives took
forever, whereas re.compile was quite fast.

So... If I can somehow be content with only getting one match per
position, I guess re is the best solution.

Or I could implement something in C (Pyrex)... (Or use something like
pybison.)

--
Magnus Lie Hetland "Wake up!" - Rage Against The Machine
http://hetland.org "Shut up!" - Linkin Park
Jul 18 '05 #14

P: n/a
ml*@furu.idi.ntnu.no (Magnus Lie Hetland) wrote in message news:<slrnc8ieul.7j7.ml*@furu.idi.ntnu.no>...
One problem might be that the regexp format seems to be quite
stripped-down (although, of course, a regexp is a regexp,
theoretically speaking ;)

No non-greedy matching, no lookahead/lookback etc.


It's important to note that many of these features turn the
expressions that represent their use into decidedly non-regular
expressions, regardless of what Pythoneers and Perlers call them :)

Jeremy
Jul 18 '05 #15

P: n/a
Magnus Lie Hetland wrote:
One of the reasons I've avoided existing lexers is that I don't do
standard tokenization -- I don't partition all of the text into regexp
tokens. I allow the lexer to skip over text -- somewhat like how
whitespace is normally handled, except that this can be *any* text --
and to return the next token that is of any use to the current parsing
rule.

But who knows -- I may be able to use Plex anyway.


You should be able to do this with Plex by defining a
fall-back rule that matches any single character and
ignores it.

If you make it the last rule, and make sure it only
matches one character, any other rule that also matches
will take precedence over it.

If you need to match different sets of tokens in
different circumstances, you can do this with states,
and have the parser switch states according to what
it's currently doing.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg

Jul 18 '05 #16

P: n/a
Magnus Lie Hetland wrote:
BTW: I have not done some experiments with Plex with lots of regular
expressiosn; simply compiling a pattern with 500 alternatives took
forever, whereas re.compile was quite fast.


Yeow, that's really thrashing Plex to death. Did you wait
for it to finish? I'm curious how long it actually took.

Plex is really designed for parsing programming languages
and the like, where the number of patterns is small enough
to be written by hand, and some common sense used in the
process. For example, if you have a lot of patterns with a
regularity to them such as "foo1", "foo2", ..., "foo500"
it will be much faster to compile a pattern such as

Str("foo") + (Range("19") | (Range("09") + Range("09"))
| (Range("14") + Range("09") + Range("09")) | Str("500"))

than to compile a huge expression with 500 alternatives.
The resulting state machine will likely be a lot smaller,
too.

On the plus side, scanning will be just as fast however
you write the REs, which is not true for most RE
implementations of the Perl variety.

One more tip: If you're careful what sort of things you
use for actions, you can pickle the Lexicon, so that it
only needs to be generated once rather than each time the
program starts up. I'm currently doing this in Pyrex, and
it works quite well. Let me know if you want any more info
about doing this.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg

Jul 18 '05 #17

P: n/a
Jeremy Fincher wrote:
It's important to note that many of these features turn the
expressions that represent their use into decidedly non-regular
expressions, regardless of what Pythoneers and Perlers call them :)


Indeed, which is one reason Plex doesn't do them. Plex
converts all the REs into a big DFA, which means that
Plex REs must truly be regular.

This is the price you pay for scanning time independent
of the complexity of the REs.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg

Jul 18 '05 #18

P: n/a
Jeremy Fincher wrote:
It's important to note that many of these features turn the
expressions that represent their use into decidedly non-regular
expressions, regardless of what Pythoneers and Perlers call them :)


the use of "regular" in this context refers to religion, not computer
science.

</F>


Jul 18 '05 #19

P: n/a
On Fri, 23 Apr 2004 22:56:38 +0000 (UTC), ml*@furu.idi.ntnu.no (Magnus Lie Hetland) wrote:
In article <c6************@ID-99293.news.uni-berlin.de>, William Park wrote:
[snip]

Since you want both the matched strings and their locations in file, you
pretty much have to this manually, one by one.


Well -- that's more or less what I'm doing. (Or -- I can get the match
objects all at once, of course, using finditer.)

I guess I'll have to look elsewhere for performance improvements. Hm.

Have you tried something like this?
s = 'asdad foobaz rhubarb pie'
import re
rxo = re.compile(r'(foo|bar|baz)')
rxo.split(s)[1::2]

['foo', 'baz', 'bar']

Regards,
Bengt Richter
Jul 18 '05 #20

This discussion thread is closed

Replies have been disabled for this discussion.