473,322 Members | 1,522 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

using re.finditer()

I am still fairly new to Python and trying to learn to put RE's to good
use. I am a little confused about the finditer() method. It is documented
like so:

finditer( pattern, string)

Return an iterator over all non-overlapping matches for the RE pattern in
string. For each match, the iterator returns a match object. Empty matches
are included in the result unless they touch the beginning of another match.
New in version 2.2.
I would say this documentation is not quite right (or incomplete at
best) because it doesn't document any restriction about multiline matching,
but it certainly seems to have one. This would seem to be an ideal
application for finditer()...

#! /usr/bin/python

import re

html = """
<table>
<tr>
<td>Data 1-1</td>
<td>Data 1-2</td>
<td>Data 1-3</td>
</tr>
<tr>
<td>Data 2-1</td>
<td>Data 2-2
</td>
<td>Data 2-3</td>
</tr>
</table>
"""

pat = r'<td.*?>(.*?)</td>'
for match in re.finditer(pat, html):
print match.group(1)
The iterator returned seems to work fine to step through items that
happen to be contained within one line. That is, you can step through flat,
one-line td's, but if you want to step through tr's, this doesn't work (run
this code and notice Data 2-2 is not there). finditer() doesn't accept a
flag like re.DOTALL, as re.match() and re.search() do. It seems a shame not
to be able to put an otherwise smart design to use.

One work around to this is by applying a regualr RE like
'<tr.*?>(.*?)</tr>', finding the first match, and then chopping off the
found part as you go. Another is to change the pattern to something like:
pat = r'<td.*?>([\n\w\s\d-]*?)</td>' I guess I will get one of those
implemented and get this task done, but I am still interested in learning to
use RE's better.
Interestingly, using pat = r'<td.*?>([\r\n.]*?)</td>' does NOT work, and I
don't understand why - can someone explain that?
What exactly would be the equivalent set for dot with re.DOTALL turned on
([\w\W\r\n] works here, but does that really cover it)? Other ideas?

I can think of split & join substituion tricks & the like to replace \n
with something else, then put it back in, but that get's kinda messy and
requires you to find some special substitution character that's not
elsewhere. I want to apply this to dynamically generated text that I don't
control and keep this as general as possible. I'm wondering if I'm not
missing something here - is there no way to make finditer() work
(straightforwardly) on multilines using just a simple dot RE?

Thanks for taking the time to read my post! :)

-ej
Jul 18 '05 #1
4 7618
Erik Johnson wrote:
pat = r'<td.*?>(.*?)</td>'
for match in re.finditer(pat, html):
printÂ*match.group(1)
TheÂ*iteratorÂ*returnedÂ*seemsÂ*toÂ*workÂ*fineÂ*to Â*stepÂ*throughÂ*itemsÂ*that
happen to be contained within one line. That is, you can step through
flat, one-line td's, but if you want to step through tr's, this doesn't
work (run this code and notice Data 2-2 is not there). finditer() doesn't
accept a flag like re.DOTALL, as re.match() and re.search() do. It seems a
shame not to be able to put an otherwise smart design to use.


There was a discussion on python-dev recently concerning "missing arguments"
in re.findall() and re.finditer(), see

http://mail.python.org/pipermail/pyt...er/048662.html

I think no change was made as there is already an alternative spelling:

r = re.compile(r'<td.*?>(.*?)</td>', re.DOTALL)
for match in r.finditer(html):
print match.group(1)

(or two, I didn't know about the option to embed flags in the string until
Robert Brewer's post).

Peter
Jul 18 '05 #2
Robert Brewer wrote:
Embed the flag(s) you desire in the regex itself. For example, to
include DOTALL, change r'<td.*?>(.*?)</td>' to r'(?s)<td.*?>(.*?)</td>'
Ahhhh! :) Sorry, my bad. Its right there in the docs, but I missed it -
haven't fully comprehended all of re yet. :)
Peter Otten wrote:
r = re.compile(r'<td.*?>(.*?)</td>', re.DOTALL)
for match in r.finditer(html):
print match.group(1)


Good - perhaps a more obvious way to do it.

So there's two good work-arounds.
Thank you both for your helpful replies! :)

I am still left puzzled though, why this won't work:

pat = r'<td.*?>([\n.]*?)</td>'
for match in re.finditer(pat, html):
print match.group(1)

but this will:

pat = r'<td.*?>([\w\W]*?)</td>'
for match in re.finditer(pat, html):
print match.group(1)

Thanks,
-ej
Jul 18 '05 #3
Erik Johnson wrote:
I am still left puzzled though, why this won't work:

pat = r'<td.*?>([\n.]*?)</td>'
for match in re.finditer(pat, html):
printÂ*match.group(1)

re.findall(r"[.\n]", "\nx\n") ['\n', '\n'] re.findall(r"[.\n]", "\n.\n")

['\n', '.', '\n']

It seems a dot inside [] means a dot rather than "any character".

Peter
Jul 18 '05 #4

"Peter Otten" wrote in message news:cl*************@news.t-online.com...
re.findall(r"[.\n]", "\nx\n") ['\n', '\n'] re.findall(r"[.\n]", "\n.\n")

['\n', '.', '\n']

It seems a dot inside [] means a dot rather than "any character".


DOH! That's right in the docs too. <blush>

[]
Used to indicate a set of characters. Characters can be listed individually,
or a range of characters can be indicated by giving two characters and
separating them by a "-". Special characters are not active inside sets. For
example, [akm$] will match any of the characters "a", "k", "m", or "$";
[a-z] will match any lowercase letter, and [a-zA-Z0-9] matches any letter or
digit. Character classes such as \w or \S (defined below) are also
acceptable inside a range. If you want to include a "]" or a "-" inside a
set, precede it with a backslash, or place it as the first character. The
pattern []] will match ']', for example.
Like I said, I'm learning. Nothing like experience! :)
Thanks for your help!

-ej
Jul 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Robert Oschler | last post by:
I figured I'd utter some words of praise for re.finditer(). It's such a great feeling, when you come across a single statement that does everything you want in two lines of code: for g in...
8
by: Chris Lasher | last post by:
Hello, I really like the finditer() method of the re module. I'm having difficulty at the moment, however, because finditer() still creates a callable-iterator oject, even when no match is found....
6
by: Erick | last post by:
Hello, I've been looking for a while for an answer, but so far I haven't been able to turn anything up yet. Basically, what I'd like to do is to use re.finditer to search a large file (or a file...
2
by: rawCoder | last post by:
Hi All, I have a *.cer file, a public key of some one and I want to encrypt some thing using this public key. Can someone point me to a sample code for Encrypting some file using...
15
by: could ildg | last post by:
In re, the punctuation "^" can exclude a single character, but I want to exclude a whole word now. for example I have a string "hi, how are you. hello", I want to extract all the part before the...
8
by: John Pye | last post by:
Hi all I have a file with a bunch of perl regular expressions like so: /(^|)\*(.*?)\*(|$)/$1'''$2'''$3/ # bold /(^|)\_\_(.*?)\_\_(|$)/$1''<b>$2<\/ b>''$3/ # italic bold...
3
by: Gilles Ganault | last post by:
Hello I'd like to make sure there isn't an easier way to extract all the occurences found with re.finditer: ======================= req = urllib2.Request(url, None, headers) response =...
3
by: JDeats | last post by:
I have some .NET 1.1 code that utilizes this technique for encrypting and decrypting a file. http://support.microsoft.com/kb/307010 In .NET 2.0 this approach is not fully supported (a .NET 2.0...
2
by: Fabian Braennstroem | last post by:
Hi, I would like to delete a region on a log file which has this kind of structure: #------flutest------------------------------------------------------------ 498 1.0086e-03 2.4608e-04...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.