467,074 Members | 918 Online
Bytes | Developer Community
Ask Question

Home New Posts Topics Members FAQ

Post your question to a community of 467,074 developers. It's quick & easy.

Pyparsing: Non-greedy matching?

I'm trying to use pyparsing write a screenscraper. I've got some
arbitrary HTML text I define as opener & closer. In between is the HTML
data I want to extract. However, the data may contain the same
characters as used in the closer (but not the exact same text,
obviously). I'd like to get the *minimal* amount of data between these.

Here's an example (whitespace may differ):

from pyparsing import *

test=r"""<tr class="tableTopSpace"><td></td></tr>
<tr class="tableTitleDark"><td class="tableTitleDark">Job
Information</td></tr><tr><td><table width="100%" border="0"
cellspacing="3"><tr>
<td width="110" valign="top"><div align="right"><strong>Job Title:
</strong></div></td>
<td class="ccDisplayCell">Big Old <B
STYLE="background-color:#FFEF95">Head Honcho</B> Boss Man</td></tr>
<tr>
<td width="110" valign="top"><div align="right"><strong>Employer:
</strong></div></td>
<td width="200" nowrap class="ccDisplayCell"><table><tr><td colspan="2"
valign="top">Global Megacorp</td></tr></table></td><td>
<script>
function escapecomp(){
}
"""

data=Combine(OneOrMore(Word(printables)), adjacent=False,
joinString=" ")

title_open=Literal(r"""<td width="110" valign="top"><div
align="right"><strong>Job Title: </strong></div></td>
<td class="ccDisplayCell">""")
title_open.suppress()

title_close=Literal(r"""</td>""")
title_close.suppress()

title=title_open + data + title_close
title2=title_open + (data | title_close)
title.scanString(test).next() Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
title2.scanString(test).next() ((['<td width="110" valign="top"><div align="right"><strong>Job Title:\n
</strong></div></td>\n<td class="ccDisplayCell">', 'Big Old <B
STYLE="background-color:#FFEF95">Head Honcho</B> Boss Man</td> </tr>
<tr> <td width="110" valign="top"><div align="right"><strong>Employer:
</strong></div></td> <td width="200" nowrap
class="ccDisplayCell"><table><tr><td colspan="2" valign="top">Global
Megacorp</td></tr></table></td> <td> <script> function escapecomp(){
}'], {}), 182, 656)


I'd expected title to work, but it doesn't match at all. ;( In other
test variants, title2 gives extra stuff at the end though not
necessarily to the end of the string (due to unprintable characters,
perhaps).

I want a ParseResult more like:
['<td width="110" valign="top"><div align="right"><strong>Job Title:\n
</strong></div></td>\n<td class="ccDisplayCell">', 'Big Old <B
STYLE="background-color:#FFEF95">Head Honcho</B> Boss Man, '</td>']

I sort of understand why title2 works as it does (the OneOrMore just
slurps up everything), but for the life of me I can't figure out how to
fix it. ;) Is there a way of writing something similar to RE's ".*?" ?

--Pete

--
Peter Fein pf***@pobox.com 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman
Jul 18 '05 #1
  • viewed: 2141
Share:
2 Replies
"Peter Fein" <pf***@pobox.com> wrote in message
news:ma**************************************@pyth on.org...
I'm trying to use pyparsing write a screenscraper. I've got some
arbitrary HTML text I define as opener & closer. In between is the HTML
data I want to extract. However, the data may contain the same
characters as used in the closer (but not the exact same text,
obviously). I'd like to get the *minimal* amount of data between these.

Here's an example (whitespace may differ):

from pyparsing import *

test=r"""<tr class="tableTopSpace"><td></td></tr>
<tr class="tableTitleDark"><td class="tableTitleDark">Job
Information</td></tr><tr><td><table width="100%" border="0"
cellspacing="3"><tr>
<td width="110" valign="top"><div align="right"><strong>Job Title:
</strong></div></td>
<td class="ccDisplayCell">Big Old <B
STYLE="background-color:#FFEF95">Head Honcho</B> Boss Man</td></tr>
<tr>
<td width="110" valign="top"><div align="right"><strong>Employer:
</strong></div></td>
<td width="200" nowrap class="ccDisplayCell"><table><tr><td colspan="2"
valign="top">Global Megacorp</td></tr></table></td><td>
<script>
function escapecomp(){
}
"""

data=Combine(OneOrMore(Word(printables)), adjacent=False,
joinString=" ")

title_open=Literal(r"""<td width="110" valign="top"><div
align="right"><strong>Job Title: </strong></div></td>
<td class="ccDisplayCell">""")
title_open.suppress()

title_close=Literal(r"""</td>""")
title_close.suppress()

title=title_open + data + title_close
title2=title_open + (data | title_close)
title.scanString(test).next() Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
title2.scanString(test).next() ((['<td width="110" valign="top"><div align="right"><strong>Job Title:\n
</strong></div></td>\n<td class="ccDisplayCell">', 'Big Old <B
STYLE="background-color:#FFEF95">Head Honcho</B> Boss Man</td> </tr>
<tr> <td width="110" valign="top"><div align="right"><strong>Employer:
</strong></div></td> <td width="200" nowrap
class="ccDisplayCell"><table><tr><td colspan="2" valign="top">Global
Megacorp</td></tr></table></td> <td> <script> function escapecomp(){
}'], {}), 182, 656)


I'd expected title to work, but it doesn't match at all. ;( In other
test variants, title2 gives extra stuff at the end though not
necessarily to the end of the string (due to unprintable characters,
perhaps).

I want a ParseResult more like:
['<td width="110" valign="top"><div align="right"><strong>Job Title:\n
</strong></div></td>\n<td class="ccDisplayCell">', 'Big Old <B
STYLE="background-color:#FFEF95">Head Honcho</B> Boss Man, '</td>']

I sort of understand why title2 works as it does (the OneOrMore just
slurps up everything), but for the life of me I can't figure out how to
fix it. ;) Is there a way of writing something similar to RE's ".*?" ?

--Pete

--
Peter Fein pf***@pobox.com 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman


Peter -

Well you are correct, OneOrMore just keeps on slurping as long as it
continues to find matching text. Unlike RE's, it does not look ahead in the
RE to treat the next literal as a terminating expression.

In the examples that come with pyparsing, there is an HTML extractor
(getNTPservers.py) that uses a CharsNotIn("<") expression for the body of an
HTML tag. That works for the given case, but wont work for you - the body
of your tag also includes other HTML tags, such as <B>, so a CharsNotIn
would terminate before the complete body were extracted.

Assuming that your <td> tag wont contain any nested <td> tag, you could
define your data content as "everything up until I find '</td>'". For this
you can use pyparsing's SkipTo element. I think if you define data as:
data = SkipTo("</td>")
then your code should start working better.

There are a couple of other points on your sample code. Note that
suppress() is *not* a mutator, but actually a factory method - in
expr.suppress(), expr is not modified by suppress, but returns a Suppress
object wrapped around an expr. So in place of:
title_close=Literal(r"""</td>""")
title_close.suppress()
you should do
title_close=Literal(r"""</td>""").suppress()
or
title_close=Literal(r"""</td>""")
title_close = title_close.suppress()

-- Paul
Jul 18 '05 #2
On 12/31/04 03:00 AM CST, "Paul McGuire" <pt***@austin.rr._bogus_.com>
sayeth:
Assuming that your <td> tag wont contain any nested <td> tag, you
could define your data content as "everything up until I find
'</td>'". For this you can use pyparsing's SkipTo element. I think
if you define data as:
data = SkipTo("</td>")
then your code should start working better.


Hey! It does! ;) I just worked up (note to googlers- don't do this):
goodchars=printables.replace("<", "")
good_ab="<" + (~Literal(r"""/td>""") + SkipTo(">", include=True))
good_ab.setDebug(True)
simple=Word(goodchars)
complex=Combine(simple | good_ab, adjacent=False, joinString="")
data=Combine(OneOrMore(complex), adjacent=False, joinString=" ")
title4=title_open+data

But that would break for closers with more than one ">". Need to stop
thinking like these are regexps. Thanks - this is a great tool. ;)

--
Peter Fein pf***@pobox.com 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman
Jul 18 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

5 posts views Thread by Lukas Holcik | last post: by
reply views Thread by could ildg | last post: by
7 posts views Thread by Steven Bethard | last post: by
4 posts views Thread by the.theorist | last post: by
3 posts views Thread by rh0dium | last post: by
13 posts views Thread by 7stud | last post: by
18 posts views Thread by Just Another Victim of the Ambient Morality | last post: by
3 posts views Thread by hubritic | last post: by
5 posts views Thread by Paul McGuire | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.