473,834 Members | 1,879 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Pyparsing: Non-greedy matching?

I'm trying to use pyparsing write a screenscraper. I've got some
arbitrary HTML text I define as opener & closer. In between is the HTML
data I want to extract. However, the data may contain the same
characters as used in the closer (but not the exact same text,
obviously). I'd like to get the *minimal* amount of data between these.

Here's an example (whitespace may differ):

from pyparsing import *

test=r"""<tr class="tableTop Space"><td></td></tr>
<tr class="tableTit leDark"><td class="tableTit leDark">Job
Information</td></tr><tr><td><tab le width="100%" border="0"
cellspacing="3" ><tr>
<td width="110" valign="top"><d iv align="right">< strong>Job Title:
</strong></div></td>
<td class="ccDispla yCell">Big Old <B
STYLE="backgrou nd-color:#FFEF95"> Head Honcho</B> Boss Man</td></tr>
<tr>
<td width="110" valign="top"><d iv align="right">< strong>Employer :
</strong></div></td>
<td width="200" nowrap class="ccDispla yCell"><table>< tr><td colspan="2"
valign="top">Gl obal Megacorp</td></tr></table></td><td>
<script>
function escapecomp(){
}
"""

data=Combine(On eOrMore(Word(pr intables)), adjacent=False,
joinString=" ")

title_open=Lite ral(r"""<td width="110" valign="top"><d iv
align="right">< strong>Job Title: </strong></div></td>
<td class="ccDispla yCell">""")
title_open.supp ress()

title_close=Lit eral(r"""</td>""")
title_close.sup press()

title=title_ope n + data + title_close
title2=title_op en + (data | title_close)
title.scanStrin g(test).next() Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
title2.scanStri ng(test).next() ((['<td width="110" valign="top"><d iv align="right">< strong>Job Title:\n
</strong></div></td>\n<td class="ccDispla yCell">', 'Big Old <B
STYLE="backgrou nd-color:#FFEF95"> Head Honcho</B> Boss Man</td> </tr>
<tr> <td width="110" valign="top"><d iv align="right">< strong>Employer :
</strong></div></td> <td width="200" nowrap
class="ccDispla yCell"><table>< tr><td colspan="2" valign="top">Gl obal
Megacorp</td></tr></table></td> <td> <script> function escapecomp(){
}'], {}), 182, 656)


I'd expected title to work, but it doesn't match at all. ;( In other
test variants, title2 gives extra stuff at the end though not
necessarily to the end of the string (due to unprintable characters,
perhaps).

I want a ParseResult more like:
['<td width="110" valign="top"><d iv align="right">< strong>Job Title:\n
</strong></div></td>\n<td class="ccDispla yCell">', 'Big Old <B
STYLE="backgrou nd-color:#FFEF95"> Head Honcho</B> Boss Man, '</td>']

I sort of understand why title2 works as it does (the OneOrMore just
slurps up everything), but for the life of me I can't figure out how to
fix it. ;) Is there a way of writing something similar to RE's ".*?" ?

--Pete

--
Peter Fein pf***@pobox.com 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman
Jul 18 '05 #1
2 2593
"Peter Fein" <pf***@pobox.co m> wrote in message
news:ma******** *************** *************** @python.org...
I'm trying to use pyparsing write a screenscraper. I've got some
arbitrary HTML text I define as opener & closer. In between is the HTML
data I want to extract. However, the data may contain the same
characters as used in the closer (but not the exact same text,
obviously). I'd like to get the *minimal* amount of data between these.

Here's an example (whitespace may differ):

from pyparsing import *

test=r"""<tr class="tableTop Space"><td></td></tr>
<tr class="tableTit leDark"><td class="tableTit leDark">Job
Information</td></tr><tr><td><tab le width="100%" border="0"
cellspacing="3" ><tr>
<td width="110" valign="top"><d iv align="right">< strong>Job Title:
</strong></div></td>
<td class="ccDispla yCell">Big Old <B
STYLE="backgrou nd-color:#FFEF95"> Head Honcho</B> Boss Man</td></tr>
<tr>
<td width="110" valign="top"><d iv align="right">< strong>Employer :
</strong></div></td>
<td width="200" nowrap class="ccDispla yCell"><table>< tr><td colspan="2"
valign="top">Gl obal Megacorp</td></tr></table></td><td>
<script>
function escapecomp(){
}
"""

data=Combine(On eOrMore(Word(pr intables)), adjacent=False,
joinString=" ")

title_open=Lite ral(r"""<td width="110" valign="top"><d iv
align="right">< strong>Job Title: </strong></div></td>
<td class="ccDispla yCell">""")
title_open.supp ress()

title_close=Lit eral(r"""</td>""")
title_close.sup press()

title=title_ope n + data + title_close
title2=title_op en + (data | title_close)
title.scanStrin g(test).next() Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
title2.scanStri ng(test).next() ((['<td width="110" valign="top"><d iv align="right">< strong>Job Title:\n
</strong></div></td>\n<td class="ccDispla yCell">', 'Big Old <B
STYLE="backgrou nd-color:#FFEF95"> Head Honcho</B> Boss Man</td> </tr>
<tr> <td width="110" valign="top"><d iv align="right">< strong>Employer :
</strong></div></td> <td width="200" nowrap
class="ccDispla yCell"><table>< tr><td colspan="2" valign="top">Gl obal
Megacorp</td></tr></table></td> <td> <script> function escapecomp(){
}'], {}), 182, 656)


I'd expected title to work, but it doesn't match at all. ;( In other
test variants, title2 gives extra stuff at the end though not
necessarily to the end of the string (due to unprintable characters,
perhaps).

I want a ParseResult more like:
['<td width="110" valign="top"><d iv align="right">< strong>Job Title:\n
</strong></div></td>\n<td class="ccDispla yCell">', 'Big Old <B
STYLE="backgrou nd-color:#FFEF95"> Head Honcho</B> Boss Man, '</td>']

I sort of understand why title2 works as it does (the OneOrMore just
slurps up everything), but for the life of me I can't figure out how to
fix it. ;) Is there a way of writing something similar to RE's ".*?" ?

--Pete

--
Peter Fein pf***@pobox.com 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman


Peter -

Well you are correct, OneOrMore just keeps on slurping as long as it
continues to find matching text. Unlike RE's, it does not look ahead in the
RE to treat the next literal as a terminating expression.

In the examples that come with pyparsing, there is an HTML extractor
(getNTPservers. py) that uses a CharsNotIn("<") expression for the body of an
HTML tag. That works for the given case, but wont work for you - the body
of your tag also includes other HTML tags, such as <B>, so a CharsNotIn
would terminate before the complete body were extracted.

Assuming that your <td> tag wont contain any nested <td> tag, you could
define your data content as "everything up until I find '</td>'". For this
you can use pyparsing's SkipTo element. I think if you define data as:
data = SkipTo("</td>")
then your code should start working better.

There are a couple of other points on your sample code. Note that
suppress() is *not* a mutator, but actually a factory method - in
expr.suppress() , expr is not modified by suppress, but returns a Suppress
object wrapped around an expr. So in place of:
title_close=Lit eral(r"""</td>""")
title_close.sup press()
you should do
title_close=Lit eral(r"""</td>""").suppres s()
or
title_close=Lit eral(r"""</td>""")
title_close = title_close.sup press()

-- Paul
Jul 18 '05 #2
On 12/31/04 03:00 AM CST, "Paul McGuire" <pt***@austin.r r._bogus_.com>
sayeth:
Assuming that your <td> tag wont contain any nested <td> tag, you
could define your data content as "everything up until I find
'</td>'". For this you can use pyparsing's SkipTo element. I think
if you define data as:
data = SkipTo("</td>")
then your code should start working better.


Hey! It does! ;) I just worked up (note to googlers- don't do this):
goodchars=print ables.replace(" <", "")
good_ab="<" + (~Literal(r"""/td>""") + SkipTo(">", include=True))
good_ab.setDebu g(True)
simple=Word(goo dchars)
complex=Combine (simple | good_ab, adjacent=False, joinString="")
data=Combine(On eOrMore(complex ), adjacent=False, joinString=" ")
title4=title_op en+data

But that would break for closers with more than one ">". Need to stop
thinking like these are regexps. Thanks - this is a great tool. ;)

--
Peter Fein pf***@pobox.com 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman
Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
2359
by: Lukas Holcik | last post by:
Hi everyone! How can I simply search text for regexps (lets say <a href="(.*?)">(.*?)</a>) and save all URLs(1) and link contents(2) in a dictionary { name : URL}? In a single pass if it could. Or how can I replace the html &entities; in a string "blablabla&amp;blablabal&amp;balbalbal" with the chars they mean using re.sub? I found out they are stored in an dict . I though about this functionality:
0
1144
by: could ildg | last post by:
pyparsing is very convenient to use. But I want to find some a py tool to parse non-English strings. Does pyparsing support UNICODE strings? If not, can someone tell me what py tool can do it? Thanks in advance.
7
2218
by: Steven Bethard | last post by:
How do I make sure that my entire string was parsed when I call a pyparsing element's parseString method? Here's a dramatically simplified version of my problem: py> import pyparsing as pp py> match = pp.Word(pp.nums) py> def parse_num(s, loc, toks): .... n, = toks .... return int(n) + 10 ....
4
2081
by: the.theorist | last post by:
Hey, I'm trying my hand and pyparsing a log file (named l.log): FIRSTLINE PROPERTY1 DATA1 PROPERTY2 DATA2 PROPERTYS LIST ID1 data1 ID2 data2
3
1999
by: rh0dium | last post by:
Hi all, I have a file which I need to parse and I need to be able to break it down by sections. I know it's possible but I can't seem to figure this out. The sections are broken by <> with one or more keywords in the <>. What I want to do is to be able to pars a particular section of the file. So for example I need to be able to look at the SYSLIB section. Presumably the sections are
13
2076
by: 7stud | last post by:
To the developer: 1) I went to the pyparsing wiki to download the pyparsing module and try it 2) At the wiki, there was no index entry in the table of contents for Downloads. After searching around a bit, I finally discovered a tiny link buried in some text at the top of the home page. 3) Link goes to sourceforge. At sourceforge, there was a nice, green 'download' button that stood out from the page. 4) I clicked on the download...
1
2653
by: Steve | last post by:
Hi All (especially Paul McGuire!) Could you lend a hand in the grammar and paring of the output from the function win32pdhutil.ShowAllProcesses()? This is the code that I have so far (it is very clumsy at the moment) : import string
18
4736
by: Just Another Victim of the Ambient Morality | last post by:
Is pyparsing really a recursive descent parser? I ask this because there are grammars it can't parse that my recursive descent parser would parse, should I have written one. For instance: from pyparsing import * grammar = OneOrMore(Word(alphas)) + Literal('end') grammar.parseString('First Second Third end')
3
1733
by: hubritic | last post by:
I am trying to parse data that looks like this: IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION 2BFA76F6 1208230607 T S SYSPROC SYSTEM SHUTDOWN BY USER A6D1BD62 1215230807 I H Firmware Event My problem is that sometimes there is a RESOURCE_NAME and sometimes not, so I wind up with "Firmware" as my RESOURCE_NAME and "Event" as
5
1497
by: Paul McGuire | last post by:
I've just uploaded to SourceForge and PyPI the latest update to pyparsing, version 1.5.1. It has been a couple of months since 1.5.0 was released, and a number of bug-fixes and enhancements have accumulated in SVN, so time for a release! Here's what's new in Pyparsing 1.5.1: - Added __dir__() methods to ParseBaseException and ParseResults, to support new dir() behavior in Py2.6 and Py3.0. If dir() is called on a ParseResults object,...
0
9799
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9646
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10510
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10548
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
1
7758
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6954
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5794
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4427
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3978
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.