i have some html which looks like this where i want to scrape out the
href stuff (the www.cnn.com part)
<div class="noFood"> Cheese</div>
<div class="food">Bl ue</div>
<a class="btn" href = "http://www.cnn.com">
so i wrote this code which scrapes it perfectly:
for incident in row('div', {'class':'noFoo d'}):
b = incident.findNe xtSibling('div' , {'class': 'food'})
print b
n = b.findNextSibli ng('a', {'class': 'btn'})
print n
link = n['href'] + "','"
problem is that sometimes the 2nd tag , the <div class="food"> tag , is
sometimes called food, sometimes called drink. so sometimes it looks
like this:
<div class="noFood"> Cheese</div>
<div class="drink">P epsi</div>
<a class="btn" href = "http://www.cnn.com">
how do i alter my script to take into account the fact that i will
sometimes have food and sometimes have drink as the class name? is
there a way to say "look for food or drink" or a way to say "look for
this incident and then find not the next sibling but the 2nd next
sibling" if that makes any sense?
thanks 11 2160 lo************@ gmail.com wrote: i have some html which looks like this where i want to scrape out the href stuff (the www.cnn.com part)
<div class="noFood"> Cheese</div> <div class="food">Bl ue</div> <a class="btn" href = "http://www.cnn.com">
so i wrote this code which scrapes it perfectly:
for incident in row('div', {'class':'noFoo d'}): b = incident.findNe xtSibling('div' , {'class': 'food'}) print b n = b.findNextSibli ng('a', {'class': 'btn'}) print n link = n['href'] + "','"
problem is that sometimes the 2nd tag , the <div class="food"> tag , is sometimes called food, sometimes called drink.
Apparently you are using Beautiful Soup. The value in the attribute
dictionary can be a callable; try this:
def isFoodOrDrink(a ttr):
return attr in ['food', 'drink']
b = incident.findNe xtSibling('div' , {'class': isFoodOrDrink})
Alternately you could omit the class spec and check for it in code.
Kent
i actually realized there are 3 potentials for class names. either
food or drink or dessert. so my question is whether or not i can alter
your function to look like this?
def isFoodOrDrinkOr Desert(attr):
return attr in ['food', 'drink', 'desert']
thanks in advance for the help
Kent Johnson wrote: lo************@ gmail.com wrote: i have some html which looks like this where i want to scrape out the href stuff (the www.cnn.com part)
<div class="noFood"> Cheese</div> <div class="food">Bl ue</div> <a class="btn" href = "http://www.cnn.com">
so i wrote this code which scrapes it perfectly:
for incident in row('div', {'class':'noFoo d'}): b = incident.findNe xtSibling('div' , {'class': 'food'}) print b n = b.findNextSibli ng('a', {'class': 'btn'}) print n link = n['href'] + "','"
problem is that sometimes the 2nd tag , the <div class="food"> tag , is sometimes called food, sometimes called drink.
Apparently you are using Beautiful Soup. The value in the attribute dictionary can be a callable; try this:
def isFoodOrDrink(a ttr): return attr in ['food', 'drink']
b = incident.findNe xtSibling('div' , {'class': isFoodOrDrink})
Alternately you could omit the class spec and check for it in code.
Kent lo************@ gmail.com wrote: i actually realized there are 3 potentials for class names. either food or drink or dessert. so my question is whether or not i can alter your function to look like this?
def isFoodOrDrinkOr Desert(attr): return attr in ['food', 'drink', 'desert']
what happens when you try that ?
</F> lo************@ gmail.com wrote: i actually realized there are 3 potentials for class names. either food or drink or dessert. so my question is whether or not i can alter your function to look like this?
def isFoodOrDrinkOr Desert(attr): return attr in ['food', 'drink', 'desert']
Check the spelling of 'dessert' and give it a try.
Kent
ok i found something that works. instead of using the def i did this:
for incident in row('div', {'class': 'food' or 'drink' }):
and it worked!
only thing is that i think i am messing up the logic and here is why
So when i run my script i get results, meaning it scrapes some stuff
out,
but then i get errors and where i am told:
TypeError: unsupported operand type(s) for +: 'NullType' and 'str'
Is this because of the logic in my code? i mean what i want the script
to
do is look for the <Tr> tag and then find the first div tag named food
or
drink, find its sibling named food, drink or dessert and then find the
button tag which is the following sibling and THEN scrape out the href.
is
it possible that after it finds that first div and looks for the next
sibling and then the next siblings href that it then tries to run the
same
process all over again starting with the 2nd div tag and being that it
can't
find the another div tag after the 2nd div tag that it trips up?
know what i mean?
here is my code:
for row in bs('tr'):
for incident in row('div', {'class': 'food' or 'drink'}):
b = incident.findNe xtSibling('div' , {'class': 'food' or 'drink'
or
'dessert'})
n = b.findNextSibli ng('a', {'class': 'btn'})
link= n['href'] + "','" ho***********@g mail.com wrote: ok i found something that works. instead of using the def i did this:
for incident in row('div', {'class': 'food' or 'drink' }):
and it worked!
'food' or 'drink' doesn't do what you think it does: 'food' or 'drink'
'food'
{'class': 'food' or 'drink'}
{'class': 'food'}
</F>
hey fredrik,
i don't understand what you are saying
Fredrik Lundh wrote: ho***********@g mail.com wrote:
ok i found something that works. instead of using the def i did this:
for incident in row('div', {'class': 'food' or 'drink' }):
and it worked!
'food' or 'drink' doesn't do what you think it does:
>>> 'food' or 'drink' 'food' >>> {'class': 'food' or 'drink'}
{'class': 'food'}
</F> lo************@ gmail.com wrote: hey fredrik,
i don't understand what you are saying
Do what he showed in the Python interactive shell,
Fredrik Lundh wrote: 'food' or 'drink' doesn't do what you think it does:
>>> 'food' or 'drink' 'food'
>>> {'class': 'food' or 'drink'} {'class': 'food'}
"or" returns the first true element, anything but False or None, I
think... so 'food' (a string) is true, and always will return in that
code. http://diveintopython.org/power_of_i...on/and_or.html
Brett
Brett Hoerner wrote: lo************@ gmail.com wrote:
[...] "or" returns the first true element, anything but False or None, I think... so 'food' (a string) is true, and always will return in that code.
Just in case newbies are reading: in Python several different values are
considered false in the context of an "if" statement. These include
False # Boolean False
0 # The integer zero
0.0 # Floating-point zero
(0+0j) # Complex zero
None # The None object
[] # The empty list
() # The empty tuple
{} # The empty dictionary
This is mainly to allow the convenience of writing
if thing:
...
However, one has to be careful that code that needs to treat None
differently from [] uses explicit testing such as
if thing is None:
...
You can also construct your own classes so their instances evaluate to
True or False according to your needs.
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/ This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Peter Rohleder |
last post by:
Hi,
I'm using a style-sheet where I make use of the
XPATH-"following-sibling"-expression.
The part which makes problems looks similar to the following code:
---------------------------
<xsl:for-each select="headdata/extension/person">
|
by: John Smith |
last post by:
I have more than 4,000 pieces of email in my Yahoo account. I
would like to write a program to download them all. In order to
write such a program, I need to know a few things.
1. After I fill out userid, password, and click on "login", what
exactly is sent to Yahoo?
2. When I click on "next" to fetch the next mail, what exactly is
sent to Yahoo?
|
by: Dmitry Jouravlev |
last post by:
Hi,
I have a number of C++ solutions in Visual Studio .NET and when i compile them using "Whole Program Optimization", certain projects report a LNK1171 error saying that c2.dll could not be loaded. The error contains the correct path to c2.dll (and it is definately there).
This only happens on some projects and only when "whole program optimization" option is turned on. If i turn off this option, the problem goes away. I have other...
|
by: Rudy Ray Moore |
last post by:
Hi guys,
I often use the "Find" (aka CTRL-F) tool in "Visual Studio .net 2003 7.1
c++".
I noticed that that dialog box does not go away when I perform a search (it
stays on the screen). This is different from the behavior of VS6.
I find this frusterating because a mouse click is required to close the
window (I cannot ALT-TAB back to the find window to press ESC).
|
by: Lenonardo |
last post by:
Hi.
I'm writing a VB.Net application to update multiple Excel Worksheets.
I'm using late binding (i.e. all variables are objects + use
createobject)
I develop the application on an XP machine with Excel 2002 (version 10.0)
installed - and the code works fine.
I then test the code on a laptop running Excel 2000 (version 9.0) and it
| |
by: ntsNews |
last post by:
Hi,
Using the built in command "Find Next" is there a way to have the Match Drop
Down Menu to default to "Start of Field"... or can it be the only option?
GCM
|
by: Sion Arrowsmith |
last post by:
I have a module which needs to know what directory it's in, and to
refer to files in a sibling directory, something like App/src/foo.py
wants to read App/data/conf.xml . But I have no idea in what context
foo.py is going to be run -- it could be being run as a script, it
could be being imported as a module by another script from anywhere in
the directory structure, it's even possible someone will have called
execfile on it. The following...
|
by: apurvaG |
last post by:
Hi All,
I am facing a problem while fetching a preceding sibling with some specific name from current node.
Following is the sample XML which I am using
<Root>
<document>
<tid>100</tid>
<location>SB</location>
|
by: =?ISO-8859-1?Q?Andreas_M=FCller?= |
last post by:
Hi!
(Python 2.2.3 if this is relevant :-)
I have a list of objects with, lets say, the attributes "ID", "x" and
"y". Now I want to find the index of list element with ID=10.
Of course I can loop through the list manually, but is there a
construct like
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |