By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
448,470 Members | 970 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 448,470 IT Pros & Developers. It's quick & easy.

Python loop and append data to a master

P: 2
Hi everyone, I've created a web scraper that scrapes the information I need from a table on a website. It clicks through 6 pages, but it ends up only including the last page at the end of my loop rather than appending as it goes. I'm new to for loops and Python and hoping for some guidance. Here is my code:

Expand|Select|Wrap|Line Numbers
  1.  
  2. while True:
  3.     driver.implicitly_wait(30)
  4.     table = driver.find_element_by_id('preblockBody')
  5.     information = []
  6.     job_elems = table.find_elements_by_xpath("//*[contains(@class,'pbListingTable')]")
  7.     for value in job_elems:
  8.         information.append(value.text)
  9.  
  10.     try:
  11.         driver.find_element_by_partial_link_text('Next').click()
  12.     except:
  13.         break
  14.  
  15. driver.quit()
  16. print(information) 
  17.  
Where should I put the append so it properly runs?
1 Week Ago #1
Share this Question
Share on Google+
1 Reply


P: 2
I was able to figure this out, but not sure how to mark as solved. Had a moment of clarity after posting!
1 Week Ago #2

Post your reply

Sign in to post your reply or Sign up for a free account.