By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
425,666 Members | 1,784 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 425,666 IT Pros & Developers. It's quick & easy.

how to find not the next sibling but the 2nd sibling or find sibling "a" OR sinbling "b"

P: n/a
i have some html which looks like this where i want to scrape out the
href stuff (the www.cnn.com part)

<div class="noFood">Cheese</div>
<div class="food">Blue</div>
<a class="btn" href = "http://www.cnn.com">
so i wrote this code which scrapes it perfectly:

for incident in row('div', {'class':'noFood'}):
b = incident.findNextSibling('div', {'class': 'food'})
print b
n = b.findNextSibling('a', {'class': 'btn'})
print n
link = n['href'] + "','"

problem is that sometimes the 2nd tag , the <div class="food"> tag , is
sometimes called food, sometimes called drink. so sometimes it looks
like this:

<div class="noFood">Cheese</div>
<div class="drink">Pepsi</div>
<a class="btn" href = "http://www.cnn.com">

how do i alter my script to take into account the fact that i will
sometimes have food and sometimes have drink as the class name? is
there a way to say "look for food or drink" or a way to say "look for
this incident and then find not the next sibling but the 2nd next
sibling" if that makes any sense?

thanks

Jan 18 '06 #1
Share this Question
Share on Google+
11 Replies


P: n/a
lo************@gmail.com wrote:
i have some html which looks like this where i want to scrape out the
href stuff (the www.cnn.com part)

<div class="noFood">Cheese</div>
<div class="food">Blue</div>
<a class="btn" href = "http://www.cnn.com">
so i wrote this code which scrapes it perfectly:

for incident in row('div', {'class':'noFood'}):
b = incident.findNextSibling('div', {'class': 'food'})
print b
n = b.findNextSibling('a', {'class': 'btn'})
print n
link = n['href'] + "','"

problem is that sometimes the 2nd tag , the <div class="food"> tag , is
sometimes called food, sometimes called drink.


Apparently you are using Beautiful Soup. The value in the attribute
dictionary can be a callable; try this:

def isFoodOrDrink(attr):
return attr in ['food', 'drink']

b = incident.findNextSibling('div', {'class': isFoodOrDrink})

Alternately you could omit the class spec and check for it in code.

Kent
Jan 19 '06 #2

P: n/a
i actually realized there are 3 potentials for class names. either
food or drink or dessert. so my question is whether or not i can alter
your function to look like this?

def isFoodOrDrinkOrDesert(attr):
return attr in ['food', 'drink', 'desert']
thanks in advance for the help

Kent Johnson wrote:
lo************@gmail.com wrote:
i have some html which looks like this where i want to scrape out the
href stuff (the www.cnn.com part)

<div class="noFood">Cheese</div>
<div class="food">Blue</div>
<a class="btn" href = "http://www.cnn.com">
so i wrote this code which scrapes it perfectly:

for incident in row('div', {'class':'noFood'}):
b = incident.findNextSibling('div', {'class': 'food'})
print b
n = b.findNextSibling('a', {'class': 'btn'})
print n
link = n['href'] + "','"

problem is that sometimes the 2nd tag , the <div class="food"> tag , is
sometimes called food, sometimes called drink.


Apparently you are using Beautiful Soup. The value in the attribute
dictionary can be a callable; try this:

def isFoodOrDrink(attr):
return attr in ['food', 'drink']

b = incident.findNextSibling('div', {'class': isFoodOrDrink})

Alternately you could omit the class spec and check for it in code.

Kent


Jan 19 '06 #3

P: n/a
lo************@gmail.com wrote:
i actually realized there are 3 potentials for class names. either
food or drink or dessert. so my question is whether or not i can alter
your function to look like this?

def isFoodOrDrinkOrDesert(attr):
return attr in ['food', 'drink', 'desert']


what happens when you try that ?

</F>

Jan 19 '06 #4

P: n/a
lo************@gmail.com wrote:
i actually realized there are 3 potentials for class names. either
food or drink or dessert. so my question is whether or not i can alter
your function to look like this?

def isFoodOrDrinkOrDesert(attr):
return attr in ['food', 'drink', 'desert']


Check the spelling of 'dessert' and give it a try.

Kent
Jan 19 '06 #5

P: n/a
ok i found something that works. instead of using the def i did this:

for incident in row('div', {'class': 'food' or 'drink' }):

and it worked!

only thing is that i think i am messing up the logic and here is why

So when i run my script i get results, meaning it scrapes some stuff
out,
but then i get errors and where i am told:

TypeError: unsupported operand type(s) for +: 'NullType' and 'str'
Is this because of the logic in my code? i mean what i want the script
to
do is look for the <Tr> tag and then find the first div tag named food
or
drink, find its sibling named food, drink or dessert and then find the
button tag which is the following sibling and THEN scrape out the href.
is
it possible that after it finds that first div and looks for the next
sibling and then the next siblings href that it then tries to run the
same
process all over again starting with the 2nd div tag and being that it
can't
find the another div tag after the 2nd div tag that it trips up?

know what i mean?
here is my code:

for row in bs('tr'):
for incident in row('div', {'class': 'food' or 'drink'}):
b = incident.findNextSibling('div', {'class': 'food' or 'drink'
or
'dessert'})
n = b.findNextSibling('a', {'class': 'btn'})
link= n['href'] + "','"

Jan 19 '06 #6

P: n/a
ho***********@gmail.com wrote:
ok i found something that works. instead of using the def i did this:

for incident in row('div', {'class': 'food' or 'drink' }):

and it worked!


'food' or 'drink' doesn't do what you think it does:
'food' or 'drink' 'food'
{'class': 'food' or 'drink'}

{'class': 'food'}

</F>

Jan 19 '06 #7

P: n/a
hey fredrik,

i don't understand what you are saying

Fredrik Lundh wrote:
ho***********@gmail.com wrote:
ok i found something that works. instead of using the def i did this:

for incident in row('div', {'class': 'food' or 'drink' }):

and it worked!


'food' or 'drink' doesn't do what you think it does:
>>> 'food' or 'drink' 'food'
>>> {'class': 'food' or 'drink'}

{'class': 'food'}

</F>


Jan 19 '06 #8

P: n/a
lo************@gmail.com wrote:
hey fredrik,

i don't understand what you are saying
Do what he showed in the Python interactive shell,
Fredrik Lundh wrote:
'food' or 'drink' doesn't do what you think it does:
>>> 'food' or 'drink'

'food'
>>> {'class': 'food' or 'drink'}

{'class': 'food'}


"or" returns the first true element, anything but False or None, I
think... so 'food' (a string) is true, and always will return in that
code.

http://diveintopython.org/power_of_i...on/and_or.html

Brett

Jan 20 '06 #9

P: n/a
Brett Hoerner wrote:
lo************@gmail.com wrote: [...] "or" returns the first true element, anything but False or None, I
think... so 'food' (a string) is true, and always will return in that
code.


Just in case newbies are reading: in Python several different values are
considered false in the context of an "if" statement. These include

False # Boolean False
0 # The integer zero
0.0 # Floating-point zero
(0+0j) # Complex zero
None # The None object
[] # The empty list
() # The empty tuple
{} # The empty dictionary

This is mainly to allow the convenience of writing

if thing:
...

However, one has to be careful that code that needs to treat None
differently from [] uses explicit testing such as

if thing is None:
...

You can also construct your own classes so their instances evaluate to
True or False according to your needs.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Jan 20 '06 #10

P: n/a
well actually all i want it to do is find the first thing that shows up
whether its class:food or class: drink so that works for me. only
thing is that after it finds class:food i think it runs through the
html again and finds the following class:drink and being that there is
not class tag after that class: drink tag it fails.

Fredrik Lundh wrote:
ho***********@gmail.com wrote:
ok i found something that works. instead of using the def i did this:

for incident in row('div', {'class': 'food' or 'drink' }):

and it worked!


'food' or 'drink' doesn't do what you think it does:
>>> 'food' or 'drink' 'food'
>>> {'class': 'food' or 'drink'}

{'class': 'food'}

</F>


Jan 23 '06 #11

P: n/a
lo************@gmail.com wrote:
well actually all i want it to do is find the first thing that shows up
whether its class:food or class: drink so that works for me.


what makes you think that looking "food" only will find either "food"
or "drink" ?

</F>

Jan 23 '06 #12

This discussion thread is closed

Replies have been disabled for this discussion.