python/xpath question...

bruce

for guys with python/xpath expertise..

i'm playing with xpath.. and i'm trying to solve an issue...

i have the following kind of situation where i'm trying to get certain data.

i have a bunch of tr/td...

i can create an xpath, that gets me all of the tr.. i only want to get the
sibling tr up until i hit a 'tr' that has a 'th' anybody have an idea as to
how this query might be created?..
the idea would be to start at the "Summer B", to skip the 1st "tr", to get
the next "tr"s until you get to the next "Summer" section...

sample data.....

<tr<Th colspan=14 class="soc_comm ent"Summer B </th</tr>

<tr>
<td nowrap valign="bottom" class="colhelp" >
<a href="#">Course 
Course
 Course number and suffix, if applicable.
 C = combined lecture and lab course
 L = laboratory course
</a></td>
</tr>

<tr>
<td valign="top" nowrap><a href="javascrip t:crsdescunderp op('AST1002');" >AST
1002</a></td>
</tr>
<tr>
<td valign="top" nowrap><a
href="javascrip t:crsdescunderp op('AST1022L'); ">AST 1022L</a></td>
</tr>
<tr>
<td valign="top" nowrap><a
href="javascrip t:crsdescunderp op('AST1022L'); ">AST 1022L</a></td>
</tr>
<tr>
<td valign="top" nowrap><a
href="javascrip t:crsdescunderp op('AST1022L'); ">AST 1022L</a></td>
</tr>
<tr<Th colspan=14 class="soc_comm ent"Summer C </th</tr>

<tr>
<td nowrap valign="bottom" class="colhelp" >
<a href="#">Course 
..
..
..

thanks...

-bruce

Jul 6 '06 #1

Subscribe Reply

2486

Stefan Behnel

bruce wrote:

for guys with python/xpath expertise..

i'm playing with xpath.. and i'm trying to solve an issue...

i have the following kind of situation where i'm trying to get certain data.

i have a bunch of tr/td...

i can create an xpath, that gets me all of the tr.. i only want to get the
sibling tr up until i hit a 'tr' that has a 'th' anybody have an idea as to
how this query might be created?..

I'm not quite sure how this is supposed to be related to Python, but if you're
trying to find a sibling, what about using the "sibling" axis in XPath?

Stefan

Jul 6 '06 #2

John J. Lee

(Damn gmane's authorizor, I think I lost four postings because the
auth messages went to my work email address (and I thought the
authorization was supposed to be one-time only per group anyway??). I
deleted them as spam since I hadn't posted from there for days :-(
Grrr. At least I could reconstruct this one...)

"bruce" <be*******@eart hlink.netwrites :

for guys with python/xpath expertise..

i'm playing with xpath.. and i'm trying to solve an issue...

i have the following kind of situation where i'm trying to get certain data.

i have a bunch of tr/td...

i can create an xpath, that gets me all of the tr.. i only want to get the
sibling tr up until i hit a 'tr' that has a 'th' anybody have an idea as to
how this query might be created?..

[...]

((//tr/th)[2]/../following-sibling::tr/td/..)[count(.|((//tr/th)[3]/../preceding-sibling::*))=co unt((//tr/th)[3]/../preceding-sibling::*)]
which makes use of the following idiom for writing an intersection:

$set1[count(.|$set2)= count($set2)]
and gets the second group in the sequence you describe. IMHO, this
illustrates what happens when XPath is pushed too far ;-) I don't see
an easier way, but perhaps I missed one.

Example code:

(Note that the expression used here doesn't get any trailing group of
tr elements if there's no terminating tr/th -- that fits your
specification, but may not be what you really wanted. To fix that,
meditate on the above expression for an hour or two <0.8 wink>.)

#---------------------------------------------------------
def xpath(path, source):
import StringIO
import pprint
from lxml import etree
f = StringIO.String IO(source)
tree = etree.parse(f)
r = tree.xpath(path )
#return "\n".join(etree .tostring(el) for el in r)
return pprint.pformat([etree.tostring( el) for el in r])

simple = """\
<html>
<tr><th>A</th></tr>
<tr><td>B</td></tr>
<tr><td>C</td></tr>
<tr><th>D</th></tr>
<tr><td>E</td></tr>
<tr><td>F</td></tr>
<tr><th>G</th></tr>
<tr><td>H</td></tr>
<tr><td>I</td></tr>
</html>
"""

for i in range(3):
expr = '((//tr/th)[%s]/../following-sibling::tr/td/..)[count(.|((//tr/th)[%s]/../preceding-sibling::*))=co unt((//tr/th)[%s]/../preceding-sibling::*)]' % (i+1, i+2, i+2)
print "---------------------"
print xpath(expr, simple)
#---------------------------------------------------------
john[0]$ tst.py
---------------------
['<tr><td>B</td></tr>\n', '<tr><td>C</td></tr>\n']
---------------------
['<tr><td>E</td></tr>\n', '<tr><td>F</td></tr>\n']
---------------------
[]
Knowing what you're doing, though, you'd probably be better off with
BeautifulSoup than XPath. Also note that mechanize (which I know
you're using) only supports BeautifulSoup 2 at present. You can't use
BeautifulSoup 3 yet (I hope to fix that 'RSN').
John

Jul 9 '06 #3

John J. Lee

Stefan Behnel <st************ ******@web.dewr ites:
[...]

I'm not quite sure how this is supposed to be related to Python, but if you're
trying to find a sibling, what about using the "sibling" axis in XPath?

<nit>
There's no "sibling" axis in XPath. I'm sure you meant
"following-sibling" and/or "preceding-sibling".
</nit>
John

Jul 9 '06 #4

Similar topics

6925

XPath and XQuery in Python?

by: Nelson Minar | last post by:

Could someone help me get started using XPath or XQuery in Python? I'm overwhelmed by all the various options and am lacking guidance on what the simplest way to go is. What library do I need to enable three line Python programs to extract data with XPath expressions? I have this problem a lot with Python and XML. Even with Uche's excellent yearly roundups I have a hard time finding how to do fancy things with XML in Python. I think it's...

Python

2408

Python xml.dom, help reading attribute data

by: Thierry Lam | last post by:

Let's say I have the following xml tag: <para role="success">1</para> I can't figure out what kind of python xml.dom codes I should invoke to read the data 1? Any help please? Thanks Thierry

Python

2074

xpath support in python 2.4

by: And80 | last post by:

Hi, I would like to use xpath modules in python2.4.... In my local machine I am running python2.3.5 and on the server I run python2.4. I have seen that while on my computer i am able to import xml.xpath, on the server the module seems to not exist. Is it still part of the standard library? if not, what should I use? Thank you in advance, Andrea Fiore

Python

382

python/libxml2dom questions

by: bruce | last post by:

hi paul... in playing around with the test python app (see below) i've got a couple of basic questions. i can't seem to find the answers via google, and when i've looked in the libxml2dom stuff that i downloaded i didn't see answers either... for the list in the "for label in d.xpath" how can i find out the size of the list???? a simple/basic question, but it's driving me up a wall!!!

Python

1729

RE: python screen scraping/parsing

by: bruce | last post by:

Hi Paul... Thanks for the reply. Came to the same conclusion a few minutes before I saw your email. Another question: tr=d.xpath(foo) gets me an array of nodes.

Python

2213

python - firefox dom/xpath question/issue

by: bruce | last post by:

Hi. Got a test web page, that basically has two "<html" tags in it. Examining the page via Firefox/Dom Inspector, I can create a test xpath query "/html/body/form" which gets the target form for the test. The issue comes when I examine the page's source html. It looks like: <html> <body> </body>

Python

1934

python mechanize/libxml2dom question

by: bruce | last post by:

hi... i've got the following situation, with the following test url: "http://schedule.psu.edu/soc/fall/Alloz/a-c/acctg.html#". i can generate a list of the tables i want for the courses on the page. however, when i try to create the xpath query, and plug it into the xpath within python, i'm missing something. if i have a parent xpath query, that generates a list of results/nodes... how can i then use the individual parent node, and...

Python

1375

python/xpath question..

by: bruce | last post by:

morning.... i apologize up front as this is really more of an xpath question.. in my python, i'm using the xpath function to iterate/parse some html. i can do something like s=d.xpath("//tr/td/text()") count=len(s)

Python

1057

Re: python/xpath question..

by: John Krukoff | last post by:

On Wed, 2008-09-03 at 13:36 -0700, bruce wrote: Well, you could just do the test (and the count!) in the xpath expression: count( //tr/td ) It sounds like you're not familiar with xpath? I would recommend the O'Reilly XSLT book, it has an excellent introduction to xpath in chapter 3.

Python

8471

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

8388

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...

Windows Server

8907

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

8817

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...

Online Marketing

8593

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

7423

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

5687

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

4215

Trying to create a lan-to-lan vpn between two differents networks

by: TSSRALBI | last post by:

Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...

Networking - Hardware / Configuration

1799

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

General