I forget how to find multiple instances of stuff between tags using
regular expressions. Specifically I want to find all the text between a
series of begin/end pairs in a multiline file.
I tried: p = 'begin(.*)end' m = re.search(p,s,re.DOTALL)
and got everything between the first begin and last end. I guess
because of a greedy match. What I want to do is a list where each
element is the text between another begin/end pair.
TIA
David Lees 7 2605
David Lees wrote: I forget how to find multiple instances of stuff between tags using regular expressions. Specifically I want to find all the text between a series of begin/end pairs in a multiline file.
I tried: >>> p = 'begin(.*)end' >>> m = re.search(p,s,re.DOTALL)
and got everything between the first begin and last end. I guess because of a greedy match. What I want to do is a list where each element is the text between another begin/end pair.
people will tell you to use non-greedy matches, but that's often a
bad idea in cases like this: the RE engine has to store lots of back-
tracking information, and your program will consume a lot more
memory than it has to (and may run out of stack and/or memory).
a better approach is to do two searches: first search for a "begin",
and once you've found that, look for an "end"
import re
pos = 0
START = re.compile("begin")
END = re.compile("end")
while 1:
m = START.search(text, pos)
if not m:
break
start = m.end()
m = END.search(text, start)
if not m:
break
end = m.start()
process(text[start:end])
pos = m.end() # move forward
at this point, it's also obvious that you don't really have to use
regular expressions:
pos = 0
while 1:
start = text.find("begin", pos)
if start < 0:
break
start += 5
end = text.find("end", start)
if end < 0:
break
process(text[start:end])
pos = end # move forward
</F>
<!-- (the eff-bot guide to) the python standard library (redux): http://effbot.org/zone/librarybook-index.htm
-->
On Thu, 17 Jul 2003 04:27:23 GMT, David Lees <ab***************@verizon.net> wrote: I forget how to find multiple instances of stuff between tags using regular expressions. Specifically I want to find all the text between a series of begin/end pairs in a multiline file.
I tried: p = 'begin(.*)end' m = re.search(p,s,re.DOTALL) and got everything between the first begin and last end. I guess because of a greedy match. What I want to do is a list where each element is the text between another begin/end pair.
You were close. For non-greedy add the question mark after the greedy expression: import re s = """
... begin first end
... begin
... second
... end
... begin problem begin nested end end
... begin last end
... """ p = 'begin(.*?)end' rx =re.compile(p,re.DOTALL) rx.findall(s)
[' first ', '\nsecond\n', ' problem begin nested ', ' last ']
Notice what happened with the nested begin-ends. If you have nesting, you
will need more than a simple regex approach.
Regards,
Bengt Richter
Fredrik,
Not sure about the original poster, but I can use that. Thanks!
--Alan
"Fredrik Lundh" <fr*****@pythonware.com> wrote in message news:<ma**********************************@python. org>... David Lees wrote:
I forget how to find multiple instances of stuff between tags using regular expressions. Specifically I want to find all the text between a series of begin/end pairs in a multiline file.
I tried: >>> p = 'begin(.*)end' >>> m = re.search(p,s,re.DOTALL)
and got everything between the first begin and last end. I guess because of a greedy match. What I want to do is a list where each element is the text between another begin/end pair.
people will tell you to use non-greedy matches, but that's often a bad idea in cases like this: the RE engine has to store lots of back- tracking information, and your program will consume a lot more memory than it has to (and may run out of stack and/or memory).
a better approach is to do two searches: first search for a "begin", and once you've found that, look for an "end"
import re
pos = 0
START = re.compile("begin") END = re.compile("end")
while 1: m = START.search(text, pos) if not m: break start = m.end() m = END.search(text, start) if not m: break end = m.start() process(text[start:end]) pos = m.end() # move forward
at this point, it's also obvious that you don't really have to use regular expressions:
pos = 0
while 1: start = text.find("begin", pos) if start < 0: break start += 5 end = text.find("end", start) if end < 0: break process(text[start:end]) pos = end # move forward
</F>
<!-- (the eff-bot guide to) the python standard library (redux): http://effbot.org/zone/librarybook-index.htm -->
On Thu, 17 Jul 2003 08:44:50 +0200, "Fredrik Lundh" <fr*****@pythonware.com> wrote: David Lees wrote:
I forget how to find multiple instances of stuff between tags using regular expressions. Specifically I want to find all the text between a series of begin/end pairs in a multiline file.
I tried: >>> p = 'begin(.*)end' >>> m = re.search(p,s,re.DOTALL) and got everything between the first begin and last end. I guess because of a greedy match. What I want to do is a list where each element is the text between another begin/end pair.
people will tell you to use non-greedy matches, but that's often a bad idea in cases like this: the RE engine has to store lots of back-
would you say so for this case? Or how like this case?
tracking information, and your program will consume a lot more memory than it has to (and may run out of stack and/or memory).
For the above case, wouldn't the regex compile to a state machine
that just has a few states to recognize e out of .* and then revert to .*
if the next is not n, and if it is, then look for d similarly, and if not,
revert to .*, etc or finish? For a short terminating match, it would seem
relatively cheap?
at this point, it's also obvious that you don't really have to use regular expressions:
pos = 0
while 1: start = text.find("begin", pos) if start < 0: break start += 5 end = text.find("end", start) if end < 0: break process(text[start:end]) pos = end # move forward
</F>
Or breaking your loop with an exception instead of tests: text = """begin s1 end
... sdfsdf
... begin s2 end
... """
def process(s): print 'processing(%r)'%s
... try:
... end = 0 # end of previous search
... while 1:
... start = text.index("begin", end) + 5
... end = text.index("end", start)
... process(text[start:end])
... except ValueError:
... pass
...
processing(' s1 ')
processing(' s2 ')
Or if you're guaranteed that every begin has an end, you could also write
for begxxx in text.split('begin')[1:]:
... process(begxxx.split('end')[0])
...
processing(' s1 ')
processing(' s2 ')
Regards,
Bengt Richter
Andrew Bennetts wrote: On Thu, Jul 17, 2003 at 04:27:23AM +0000, David Lees wrote:
I forget how to find multiple instances of stuff between tags using regular expressions. Specifically I want to find all the text between a
^^^^^^^^
How about re.findall?
E.g.:
>>> re.findall('BEGIN(.*?)END', 'BEGIN foo END BEGIN bar END') [' foo ', ' bar ']
-Andrew.
Actually this fails with the multi-line type of file I was asking about. re.findall('BEGIN(.*?)END', 'BEGIN foo\nmumble END BEGIN bar END')
[' bar ']
On Fri, 18 Jul 2003 04:31:32 GMT, David Lees <ab***************@verizon.net> wrote: Andrew Bennetts wrote: On Thu, Jul 17, 2003 at 04:27:23AM +0000, David Lees wrote:
I forget how to find multiple instances of stuff between tags using regular expressions. Specifically I want to find all the text between a
^^^^^^^^
How about re.findall?
E.g.:
>>> re.findall('BEGIN(.*?)END', 'BEGIN foo END BEGIN bar END') [' foo ', ' bar ']
-Andrew.
Actually this fails with the multi-line type of file I was asking about.
re.findall('BEGIN(.*?)END', 'BEGIN foo\nmumble END BEGIN bar END')[' bar ']
It works if you include the DOTALL flag (?s) at the beginning, which makes
.. also match \n: (BTW, (?si) would make it case-insensitive). import re re.findall('(?s)BEGIN(.*?)END', 'BEGIN foo\nmumble END BEGIN bar END')
[' foo\nmumble ', ' bar ']
Regards,
Bengt Richter
Bengt Richter wrote: On Fri, 18 Jul 2003 04:31:32 GMT, David Lees <ab***************@verizon.net> wrote:
Andrew Bennetts wrote:
On Thu, Jul 17, 2003 at 04:27:23AM +0000, David Lees wrote:
I forget how to find multiple instances of stuff between tags using regular expressions. Specifically I want to find all the text between a
^^^^^^^^
How about re.findall?
E.g.:
>>> re.findall('BEGIN(.*?)END', 'BEGIN foo END BEGIN bar END') [' foo ', ' bar ']
-Andrew.
Actually this fails with the multi-line type of file I was asking about.
>re.findall('BEGIN(.*?)END', 'BEGIN foo\nmumble END BEGIN bar END')
[' bar ']
It works if you include the DOTALL flag (?s) at the beginning, which makes . also match \n: (BTW, (?si) would make it case-insensitive).
>>> import re >>> re.findall('(?s)BEGIN(.*?)END', 'BEGIN foo\nmumble END BEGIN bar END')
[' foo\nmumble ', ' bar ']
Regards, Bengt Richter
I just tried to benchmark both Fredrik's suggestions along with Bengt's
using the same input file. The results (looping 200 times over the 400k
file) are:
Fredrik, regex = 1.74003930667
Fredrik, no regex = 0.434207978947
Bengt, regex = 1.45420158149
Interesting how much faster the non-regex approach is.
Thanks again.
David Lees
The code (which I have not carefully checked) is:
import re, time
def timeBengt(s,N):
p = 'begin msc(.*?)end msc'
rx =re.compile(p,re.DOTALL)
t0 = time.clock()
for i in xrange(N):
x = x = rx.findall(s)
t1 = time.clock()
return t1-t0
def timeFredrik1(text,N):
t0 = time.clock()
for i in xrange(N):
pos = 0
START = re.compile("begin")
END = re.compile("end")
while 1:
m = START.search(text, pos)
if not m:
break
start = m.end()
m = END.search(text, start)
if not m:
break
end = m.start()
pass
pos = m.end() # move forward
t1 = time.clock()
return t1-t0
def timeFredrik(text,N):
t0 = time.clock()
for i in xrange(N):
pos = 0
while 1:
start = text.find("begin msc", pos)
if start < 0:
break
start += 9
end = text.find("end msc", start)
if end < 0:
break
pass
pos = end # move forward
t1 = time.clock()
return t1-t0
fh = open('scu.cfg','rb')
s = fh.read()
fh.close()
N = 200
print 'Fredrik, regex = ',timeFredrik1(s,N)
print 'Fredrik, no regex = ',timeFredrik(s,N)
print 'Bengt, regex = ',timeBengt(s,N) This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Bradley Plett |
last post by:
I'm hopeless at regular expressions (I just don't use them often
enough to gain/maintain knowledge), but I need one now and am looking
for help. I need to parse through a document to find a URL,...
|
by: Neri |
last post by:
Some document processing program I write has to deal with documents
that have headers and footers that are unnecessary for the main
processing part. Therefore, I'm using a regular expression to go...
|
by: Lee Kuhn |
last post by:
I am trying the create a regular expression that will essentially match
characters in the middle of a fixed-length string. The string may be any
characters, but will always be the same length. In...
|
by: James D. Marshall |
last post by:
The issue at hand, I believe is my comprehension of using regular
expression, specially to assist in replacing the expression with other text.
using regular expression (\s*) my understanding is...
|
by: Billa |
last post by:
Hi,
I am replaceing a big string using different regular expressions (see
some example at the end of the message). The problem is whenever I
apply a "replace" it makes a new copy of string and I...
|
by: Pete Davis |
last post by:
I'm using regular expressions to extract some data and some links from some
web pages. I download the page and then I want to get a list of certain
links.
For building regular expressions, I use...
|
by: Zach |
last post by:
Hello,
Please forgive if this is not the most appropriate newsgroup for this
question. Unfortunately I didn't find a newsgroup specific to regular
expressions.
I have the following regular...
|
by: Mike |
last post by:
I have a regular expression (^(.+)(?=\s*).*\1 ) that results in
matches. I would like to get what the actual regular expression is.
In other words, when I apply ^(.+)(?=\s*).*\1 to " HEART...
|
by: Mr.Steskal |
last post by:
Posted: Wed Jul 11, 2007 7:01 am Post subject: Regular Expression
Help
--------------------------------------------------------------------------------
I need help writing a regular...
|
by: Lit |
last post by:
Hi,
I am looking for a Regular expression for a password for my RegExp
ValidationControl
Requirements are,
At least 8 characters long.
At least one digit
At least one upper case character
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |