embedding executable code in a regular expression in Python

[Avi Kak]

Folks,

Does regular expression processing in Python allow for executable
code to be embedded inside a regular expression?

For example, in Perl the following two statements

$regex = qr/hello(?{print "saw hello\n"})mello (?{print "saw
mello\n"})/;
"jellohellomell o" =~ /$regex/;

will produce the output

saw hello
saw mello

Is it possible to do the same in Python with any modules that come
with the standard distribution, or with any other modules?

Thanks in advance for any help,

Avi Kak (ka*@purdue.edu )

[Anthra Norell]

----- Original Message -----
From: "Avi Kak" <ka*@purdue.edu >
Newsgroups: comp.lang.pytho n
To: <py*********@py thon.org>
Sent: Sunday, July 16, 2006 11:05 PM
Subject: embedding executable code in a regular expression in Python

Folks,

Does regular expression processing in Python allow for executable
code to be embedded inside a regular expression?

For example, in Perl the following two statements

$regex = qr/hello(?{print "saw hello\n"})mello (?{print "saw
mello\n"})/;
"jellohellomell o" =~ /$regex/;

will produce the output

saw hello
saw mello

Is it possible to do the same in Python with any modules that come
with the standard distribution, or with any other modules?

Thanks in advance for any help,

Avi Kak (ka*@purdue.edu )

--
http://mail.python.org/mailman/listinfo/python-list

There's a new module out that can do it like this:

import SE
Stream_Editor = SE.SE ('<EAT"~[hm]ello~=print \'saw =\'\n" ')
for line in Stream_Editor ('yellomellojel lohello').split ('\n'):
exec ("%s" % line)

saw mello
saw hello

You'll find SE in the Cheese Shop. It has been commented favorably. I
wouldn't comment it, because I wrote it.

Regards

Frederic

[Ben C]

On 2006-07-16, Avi Kak <ka*@purdue.edu wrote:

Folks,

Does regular expression processing in Python allow for executable
code to be embedded inside a regular expression?

For example, in Perl the following two statements

$regex = qr/hello(?{print "saw hello\n"})mello (?{print "saw
mello\n"})/;
"jellohellomell o" =~ /$regex/;

will produce the output

saw hello
saw mello

Is it possible to do the same in Python with any modules that come
with the standard distribution, or with any other modules?

You can use sub and make the replacement pattern a function (or any
"callable" thing) and it gets called back with the match object:

import re

def f(mo):
if "hello" in mo.groups():
print "saw hello"
if "mello" in mo.groups():
print "saw mello"

re.sub(r'(hello )(mello)', f, "jellohellomell o")

Actually I didn't know you could do that in Perl. The time I've found
this useful in Python is substitutions to convert e.g.
"background-color" into "backgroundColo r"; a function turns the c into
C. I always assumed in Perl you would need to use eval for this, but
perhaps there is another way.

[Stefan Behnel]

Avi Kak wrote:

Does regular expression processing in Python allow for executable
code to be embedded inside a regular expression?

For example, in Perl the following two statements

$regex = qr/hello(?{print "saw hello\n"})mello (?{print "saw
mello\n"})/;
"jellohellomell o" =~ /$regex/;

will produce the output

saw hello
saw mello

Is it possible to do the same in Python with any modules that come
with the standard distribution, or with any other modules?

Just in case you were referring to security concerns: Sufficiently complex REs
can take ages to compile and run and eat tons of memory, so there are always
issues involved here.

Stefan

[Paul McGuire]

Avi Kak wrote:

Folks,

Does regular expression processing in Python allow for executable
code to be embedded inside a regular expression?

For example, in Perl the following two statements

$regex = qr/hello(?{print "saw hello\n"})mello (?{print "saw
mello\n"})/;
"jellohellomell o" =~ /$regex/;

will produce the output

saw hello
saw mello

Not nearly so terse, but perhaps easier to follow, here is a pyparsing
version. Pyparsing parse actions are intended to do just what you ask.
Parse actions may be defined to take no arguments, just one argument
(which will be passed the list of matching token strings), 2 arguments
(the match location and the matching tokens), or 3 arguments (the
original source string, the match location, and the tokens). Parse
actions are very good for transforming input text into modified output
form, such as the "background-color" to "backgroundColo r" transform -
the BoaConstructor team used pyparsing to implement a version upgrade
that transformed user source to a new version of wx (involving a
variety of suh changes).

Here is your jello/mello program, with two variations of parse actions.

-- Paul

from pyparsing import *

instr = "jellorelohello mellofellowbell owmello"
searchTerm = oneOf( ["jello","me llo"] )

# simple parse action, just echoes matched text
def echoMatchedText (tokens):
print "saw", tokens[0]

searchTerm.setP arseAction( echoMatchedText )
searchTerm.sear chString(instr)

# modified parse action, prints location too
def echoMatchedText (loc,tokens):
print "saw", tokens[0], "at locn", loc

searchTerm.setP arseAction( echoMatchedText )
searchTerm.sear chString(instr)

Prints out:
saw jello
saw mello
saw mello
saw jello at locn 0
saw mello at locn 14
saw mello at locn 31

[Anthra Norell]

Hi, see below ...

----- Original Message -----
From: "Paul McGuire" <pt***@austin.r r.com>
Newsgroups: comp.lang.pytho n
To: <py*********@py thon.org>
Sent: Monday, July 17, 2006 10:09 AM
Subject: Re: embedding executable code in a regular expression in Python

Avi Kak wrote:

Folks,

Does regular expression processing in Python allow for executable
code to be embedded inside a regular expression?

For example, in Perl the following two statements

$regex = qr/hello(?{print "saw hello\n"})mello (?{print "saw
mello\n"})/;
"jellohellomell o" =~ /$regex/;

will produce the output

saw hello
saw mello

Not nearly so terse, but perhaps easier to follow, here is a pyparsing
version. Pyparsing parse actions are intended to do just what you ask.
Parse actions may be defined to take no arguments, just one argument
(which will be passed the list of matching token strings), 2 arguments
(the match location and the matching tokens), or 3 arguments (the
original source string, the match location, and the tokens). Parse
actions are very good for transforming input text into modified output
form, such as the "background-color" to "backgroundColo r" transform -
the BoaConstructor team used pyparsing to implement a version upgrade
that transformed user source to a new version of wx (involving a
variety of suh changes).

Here is your jello/mello program, with two variations of parse actions.

-- Paul

from pyparsing import *

instr = "jellorelohello mellofellowbell owmello"
searchTerm = oneOf( ["jello","me llo"] )

# simple parse action, just echoes matched text
def echoMatchedText (tokens):
print "saw", tokens[0]

searchTerm.setP arseAction( echoMatchedText )
searchTerm.sear chString(instr)

# modified parse action, prints location too
def echoMatchedText (loc,tokens):
print "saw", tokens[0], "at locn", loc

searchTerm.setP arseAction( echoMatchedText )
searchTerm.sear chString(instr)

Prints out:
saw jello
saw mello
saw mello
saw jello at locn 0
saw mello at locn 14
saw mello at locn 31

--
http://mail.python.org/mailman/listinfo/python-list

One fine example of a pyparse solution!

On my part I had a second thought on the SE solution I proposed. It was
needlessly complicated. Let me try again:

>>sentence = 'On each occurrence of the word "square" in this text the

variable n will be squared. When it says "double" it will be doubled. At the
end print n % 11.'

>>def square_n (): global n; n *= n
def double_n (): global n; n += n
def n_mod_11 (): global n; return n % 11
se_definition s = """<EAT>

.... "square=pri nt 'Calling square_n () - ',; square_n (); print 'n is now
%d' % n \n" # A piece of code for 'square'
... "double=pri nt 'Calling double_n () - ',; double_n (); print 'n is now
%d' % n \n" # Another piece of code for 'double'
... ".=\nprint 'n %% 11 is: %d' % n_mod_11 ()\n" # Another piece of code
for the final dot
.... """

>>from SE import *
Stream_Edit or = SE (definitions)
n = 9; exec (Stream_Editor (sentence))

Calling square_n () - n is now 81
Calling square_n () - n is now 6561
n % 11 is: 5
Calling double_n () - n is now 13122
Calling double_n () - n is now 26244
n % 11 is: 9
n % 11 is: 9

Suppose we now realize that the quoted words "square" and "double" should
not trigger the respective function and neither should the ending dot of
each sentence, except the last one. We fix the defintions in seconds, adding
the exceptions as deletions. The order is immaterial. Targets may be regular
expressions. (The last one is.)

se_definitions = """<EAT # Deletes everything except the defined
substitutions
.... "square=pri nt 'Calling square_n () - ',; square_n (); print 'n is now
%d' % n \n" # A piece of code for 'square'
.... "double=pri nt 'Calling double_n () - ',; double_n (); print 'n is now
%d' % n \n" # Another piece of code for 'double'
.... ".=\nprint 'n %% 11 is: %d' % n_mod_11 ()\n" # Another piece of code
for the final dot
.... \""square"=" \""double"=" # Deletes the quoted words
.... "~\.\s~=" # Deletes dots followed by white space.
.... """

>>n = 9; exec (SE (se_definitions )(sentence))

Calling square_n () - n is now 81
Calling double_n () - n is now 162
n % 11 is: 8

Note 1: SE is no match for pyparse on many accounts. Conversely SE has its
own strengths in the realm of its specialty, in particular ease and speed of
use, versatility and modularity.
Most any problem can be solved in a variety of different ways. Finding
good approaches is one of the appeals of programming.

Note 2: The SE solution rests entirely on Python's 'exec' functionality and
so here we have another testimony to Python's excellence.

Note 3: I shall include the example in the manual. I hadn't thought of it.
My thanks to the OP for the inspiration. Perhaps he might want to explain
his purpose. I don't quite see the solution's problem.

Frederic