Regexes: How to handle escaped characters

[Torsten Bronger]

Hallöchen!

I need some help with finding matches in a string that has some
characters which are marked as escaped (in a separate list of
indices). Escaped means that they must not be part of any match.

My current approach is to look for matches in substrings with the
escaped characters as boundaries between the substrings. However,
then ^ and $ in the patterns are treated wrongly. (Although I use
startpos and endpos parameters for this and no slicing.)

Another idea was to have a special unicode character that never
takes part in a match. The docs are not very promising regarding
such a thing, or did I miss something?

Any other ideas?

Tschö,
Torsten.

--
Torsten Bronger, aquisgrana, europa vetus
Jabber ID: br*****@jabber. org
(See http://ime.webhop.org for ICQ, MSN, etc.)

[James Stroud]

Torsten Bronger wrote:

Hallöchen!

I need some help with finding matches in a string that has some
characters which are marked as escaped (in a separate list of
indices). Escaped means that they must not be part of any match.

My current approach is to look for matches in substrings with the
escaped characters as boundaries between the substrings. However,
then ^ and $ in the patterns are treated wrongly. (Although I use
startpos and endpos parameters for this and no slicing.)

Another idea was to have a special unicode character that never
takes part in a match. The docs are not very promising regarding
such a thing, or did I miss something?

Any other ideas?

Tschö,
Torsten.

You should probably provide examples of what you are trying to do or you
will likely get a lot of irrelevant answers.

James

[Torsten Bronger]

Hallöchen!

James Stroud writes:

Torsten Bronger wrote:

>I need some help with finding matches in a string that has some
characters which are marked as escaped (in a separate list of
indices). Escaped means that they must not be part of any match.

[...]

You should probably provide examples of what you are trying to do
or you will likely get a lot of irrelevant answers.

Example string: u"Hollo", escaped positions: [4]. Thus, the second
"o" is escaped and must not be found be the regexp searches.

Instead of re.search, I call the function guarded_search( pattern,
text, offset) which takes care of escaped caracters. Thus, while

re.search("o$", string)

will find the second "o",

guarded_search( "o$", string, 0)

won't find anything. But how to program "guarded_search "?
Actually, it is about changing the semantics of the regexp syntax:
"." doesn't mean anymore "any character except newline" but "any
character except newline and characters marked as escaped". And so
on, for all syntax elements of regular expressions. Escaped
characters must spoil any match, however, the regexp machine should
continue to search for other matches.

Tschö,
Torsten.

--
Torsten Bronger, aquisgrana, europa vetus
Jabber ID: br*****@jabber. org
(See http://ime.webhop.org for ICQ, MSN, etc.)

[James Stroud]

Torsten Bronger wrote:

Hallöchen!

James Stroud writes:

>>Torsten Bronger wrote:

>>>I need some help with finding matches in a string that has some
characters which are marked as escaped (in a separate list of
indices). Escaped means that they must not be part of any match.

[...]

You should probably provide examples of what you are trying to do
or you will likely get a lot of irrelevant answers.

Example string: u"Hollo", escaped positions: [4]. Thus, the second
"o" is escaped and must not be found be the regexp searches.

Instead of re.search, I call the function guarded_search( pattern,
text, offset) which takes care of escaped caracters. Thus, while

re.search("o$", string)

will find the second "o",

guarded_search( "o$", string, 0)

won't find anything. But how to program "guarded_search "?
Actually, it is about changing the semantics of the regexp syntax:
"." doesn't mean anymore "any character except newline" but "any
character except newline and characters marked as escaped". And so
on, for all syntax elements of regular expressions. Escaped
characters must spoil any match, however, the regexp machine should
continue to search for other matches.

Tschö,
Torsten.

You will probably need to implement your own findall, etc., but this
seems to do it for search:

def guarded_search( rgx, astring, escaped):
m = re.search(rgx, astring)
if m:
s = m.start()
e = m.end()
for i in escaped:
if s <= i <= e:
m = None
break
return m
Here it is in use:

pydef guarded_search( rgx, astring, escaped):
.... m = re.search(rgx, astring)
.... if m:
.... s = m.start()
.... e = m.end()
.... for i in escaped:
.... if s <= i <= e:
.... m = None
.... break
.... return m
....
pyimport re
pyescaped = [1, 5, 15]
pyprint guarded_search( 'abc', 'xyzabcxyz', escaped)
None
pyprint guarded_search( 'abc', 'xyzxyzabcxyz', escaped)
<_sre.SRE_Mat ch object at 0x40379720>

James

[John Machin]

On May 18, 6:00 am, Torsten Bronger <bron...@physik .rwth-aachen.de>
wrote:

Hallöchen!

James Stroud writes:

Torsten Bronger wrote:

I need some help with finding matches in a string that has some
characters which are marked as escaped (in a separate list of
indices). Escaped means that they must not be part of any match.

[...]

You should probably provide examples of what you are trying to do
or you will likely get a lot of irrelevant answers.

Example string: u"Hollo", escaped positions: [4]. Thus, the second
"o" is escaped and must not be found be the regexp searches.

Instead of re.search, I call the function guarded_search( pattern,
text, offset) which takes care of escaped caracters. Thus, while

re.search("o$", string)

will find the second "o",

guarded_search( "o$", string, 0)

Huh? Did you mean 4 instead of zero?

>
won't find anything.

Quite apart from the confusing use of "escape", your requirements are
still as clear as mud. Try writing up docs for your "guarded_search "
function. Supply test cases showing what you expect to match and what
you don't expect to match. Is "offset" the offset in the text? If so,
don't you really want a set of "forbidden" offsets, not just one?

But how to program "guarded_search "?
Actually, it is about changing the semantics of the regexp syntax:
"." doesn't mean anymore "any character except newline" but "any
character except newline and characters marked as escaped".

Make up your mind whether you are "escaping" characters [likely to be
interpreted by many people as position-independent] or "escaping"
positions within the text.

And so
on, for all syntax elements of regular expressions. Escaped
characters must spoil any match, however, the regexp machine should
continue to search for other matches.

Whatever your exact requirement, it would seem unlikely to be so
wildly popularly demanded as to warrant inclusion in the "regexp
machine". You would have to write your own wrapper, something like the
following totally-untested example of one possible implementation of
one possible guess at what you mean:

import re
def guarded_search( pattern, text, forbidden_offse ts, overlap=False):
regex = re.compile(patt ern)
pos = 0
while True:
m = regex.search(te xt, pos)
if not m:
return
start, end = m.span()
for bad_pos in forbidden_offse ts:
if start <= bad_pos < end:
break
else:
yield m
if overlap:
pos = start + 1
else:
pos = end
8<-------

HTH,
John

[John Machin]

On May 18, 6:50 am, James Stroud <[e-mail] la.eduwrote:

def guarded_search( rgx, astring, escaped):
m = re.search(rgx, astring)
if m:
s = m.start()
e = m.end()
for i in escaped:
if s <= i <= e:

Did you mean to write

if s <= i < e:

?

m = None
break
return m

Your guarded search fails if there is a match after the rightmost bad
position i.e. it gives up at the first bad position.

My "guarded_search " (see separated post) needs the following done to
it:
1. make a copy
2. change name of copy to "guarded_search all" or something similar
3. change "yield" to "return" in the original

Cheers,
John

[Paul McGuire]

On May 17, 4:06 pm, John Machin <sjmac...@lexic on.netwrote:

On May 18, 6:00 am, Torsten Bronger <bron...@physik .rwth-aachen.de>
wrote:

Hallöchen!

James Stroud writes:

Torsten Bronger wrote:

>I need some help with finding matches in a string that has some
>characters which are marked as escaped (in a separate list of
>indices). Escaped means that they must not be part of any match.

>[...]

You should probably provide examples of what you are trying to do
or you will likely get a lot of irrelevant answers.

Example string: u"Hollo", escaped positions: [4]. Thus, the second
"o" is escaped and must not be found be the regexp searches.

Instead of re.search, I call the function guarded_search( pattern,
text, offset) which takes care of escaped caracters. Thus, while

re.search("o$", string)

will find the second "o",

guarded_search( "o$", string, 0)

Huh? Did you mean 4 instead of zero?

won't find anything.

Quite apart from the confusing use of "escape", your requirements are
still as clear as mud. Try writing up docs for your "guarded_search "
function. Supply test cases showing what you expect to match and what
you don't expect to match. Is "offset" the offset in the text? If so,
don't you really want a set of "forbidden" offsets, not just one?

But how to program "guarded_search "?
Actually, it is about changing the semantics of the regexp syntax:
"." doesn't mean anymore "any character except newline" but "any
character except newline and characters marked as escaped".

Make up your mind whether you are "escaping" characters [likely to be
interpreted by many people as position-independent] or "escaping"
positions within the text.

And so
on, for all syntax elements of regular expressions. Escaped
characters must spoil any match, however, the regexp machine should
continue to search for other matches.

Whatever your exact requirement, it would seem unlikely to be so
wildly popularly demanded as to warrant inclusion in the "regexp
machine". You would have to write your own wrapper, something like the
following totally-untested example of one possible implementation of
one possible guess at what you mean:

import re
def guarded_search( pattern, text, forbidden_offse ts, overlap=False):
regex = re.compile(patt ern)
pos = 0
while True:
m = regex.search(te xt, pos)
if not m:
return
start, end = m.span()
for bad_pos in forbidden_offse ts:
if start <= bad_pos < end:
break
else:
yield m
if overlap:
pos = start + 1
else:
pos = end
8<-------

HTH,
John- Hide quoted text -

- Show quoted text -

Here are two pyparsing-based routines, guardedSearch and
guardedSearchBy Column. The first uses a pyparsing parse action to
reject matches at a given string location, and returns a list of
tuples containing the string location and matched text. The second
uses an enhanced version of guardedSearch that uses the pyparsing
built-ins col and lineno to filter matches by column instead of by raw
string location, and returns a list of tuples of line and column of
the match location, and the matching text. (Note that string
locations are zero-based, while line and column numbers are 1-based.)

-- Paul
from pyparsing import Regex,ParseExce ption,col,linen o

def guardedSearch(p attern, text, forbidden_offse ts):

def offsetValidator (strng,locn,tok ens):
if locn in forbidden_offse ts:
raise ParseException, "can't match at offset %d" % locn

regex = Regex(pattern). setParseAction( offsetValidator )
return [ (tokStart,toks[0]) for toks,tokStart,t okEnd in
regex.scanStrin g(text) ]

print guardedSearch(u "o", u"Hollo how are you", [4,])
def guardedSearchBy Column(pattern, text, forbidden_colum ns):

def offsetValidator (strng,locn,tok ens):
if col(locn,strng) in forbidden_colum ns:
raise ParseException, "can't match at offset %d" % locn

regex = Regex(pattern). setParseAction( offsetValidator )
return [ (lineno(tokStar t,text),col(tok Start,text),tok s[0])
for toks,tokStart,t okEnd in regex.scanStrin g(text) ]

text = """\
alksjdflasjf;sa
a;sljflsjlaj
;asjflasfja;sf
aslfj;asfj;dsf
aslf;lajdf;ajsf
aslfj;afsj;sd
"""
print guardedSearchBy Column(";", text, [1,6,11,])

Prints:
[(1, 'o'), (7, 'o'), (15, 'o')]
[(1, 13, ';'), (2, 2, ';'), (3, 12, ';'), (5, 5, ';')]

[John Machin]

On May 18, 8:16 am, Paul McGuire <[e-mail] .comwrote:

On May 17, 4:06 pm, John Machin <sjmac...@lexic on.netwrote:

On May 18, 6:00 am, Torsten Bronger <bron...@physik .rwth-aachen.de>
wrote:

Hallöchen!

James Stroud writes:
Torsten Bronger wrote:

I need some help with finding matches in a string that has some
characters which are marked as escaped (in a separate list of
indices). Escaped means that they must not be part of any match.

Note: "must not be *part of* any match" [my emphasis]

[big snip]

>
Here are two pyparsing-based routines, guardedSearch and
guardedSearchBy Column. The first uses a pyparsing parse action to
reject matches at a given string location

Seems to be somewhat less like what the OP might have in mind ...

While we're waiting for clarification from the OP, there's a chicken-
and-egg thought that's been nagging me: if the OP knows so much about
the searched string that he can specify offsets which search patterns
should not span, why does he still need to search it?

Cheers,
John

[Paul McGuire]

On May 17, 6:12 pm, John Machin <sjmac...@lexic on.netwrote:

>
Note: "must not be *part of* any match" [my emphasis]

Ooops, my bad. See this version:

from pyparsing import Regex,ParseExce ption,col,linen o,getTokensEndL oc

# fake (and inefficient) version of any if not yet upgraded to Py2.5
any = lambda lst : sum(list(lst)) 0

def guardedSearch(p attern, text, forbidden_offse ts):

def offsetValidator (strng,locn,tok ens):
start,end = locn,getTokensE ndLoc()-1
if any( start <= i <= end for i in forbidden_offse ts ):
raise ParseException, "can't match at offset %d" % locn

regex = Regex(pattern). setParseAction( offsetValidator )
return [ (tokStart,toks[0]) for toks,tokStart,t okEnd in
regex.scanStrin g(text) ]

print guardedSearch(u r"o\S", u"Hollo how are you", [8,])
def guardedSearchBy Column(pattern, text, forbidden_colum ns):

def offsetValidator (strng,locn,tok ens):
start,end = col(locn,strng) , col(getTokensEn dLoc(),strng)-1
if any( start <= i <= end for i in forbidden_colum ns ):
raise ParseException, "can't match at col %d" % start

regex = Regex(pattern). setParseAction( offsetValidator )
return [ (lineno(tokStar t,text),col(tok Start,text),tok s[0])
for toks,tokStart,t okEnd in regex.scanStrin g(text) ]

text = """\
alksjdflasjf;sa
a;sljflsjlaj
;asjflasfja;sf
aslfj;asfj;dsf
aslf;lajdf;ajsf
aslfj;afsj;sd
"""
print guardedSearchBy Column("[fa];", text, [4,12,13,])

Prints:
[(1, 'ol'), (15, 'ou')]
[(2, 1, 'a;'), (5, 10, 'f;')]

>
While we're waiting for clarification from the OP, there's a chicken-
and-egg thought that's been nagging me: if the OP knows so much about
the searched string that he can specify offsets which search patterns
should not span, why does he still need to search it?

I suspect that this is column/tabular data (a log file perhaps?), and
some columns are not interesting, but produce many false hits for the
search pattern.

-- Paul

[John Machin]

On May 18, 9:46 am, Paul McGuire <[e-mail] .comwrote:

On May 17, 6:12 pm, John Machin <sjmac...@lexic on.netwrote:

Note: "must not be *part of* any match" [my emphasis]

While we're waiting for clarification from the OP, there's a chicken-
and-egg thought that's been nagging me: if the OP knows so much about
the searched string that he can specify offsets which search patterns
should not span, why does he still need to search it?

I suspect that this is column/tabular data (a log file perhaps?), and
some columns are not interesting, but produce many false hits for the
search pattern.

If so, why not split the record into fields and look only at the
interesting fields? Smells to me of yet another case of re abuse/
misuse ...