473,222 Members | 1,813 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,222 software developers and data experts.

Retrieve HTML from site using cookies

Greetings gents. I'm a Railser working on a django app that needs to do
some scraping to gather its data.

I need to programatically access a site that requires a username and
password. Once I post to the login.php page, there seems to be a
redirect and it seems that the site is using a session (perhaps a
cookie) to determine whether the user is logged in. So I need to log in
and then have cookies and or sessions maintained as I access the page
that contains the content that I am actually interested in.

What is the simplest way to post data to a form, accept a cookie to
maintain the session (and support redirects) and then (now logged into
the site) retrieve the content of a page on the site?

Is their a library or technique that makes this simple?

Thanks!

Sep 21 '06 #1
1 2544
on*************@gmail.com wrote:
Greetings gents. I'm a Railser working on a django app that needs to do
some scraping to gather its data.

I need to programatically access a site that requires a username and
password. Once I post to the login.php page, there seems to be a
redirect and it seems that the site is using a session (perhaps a
cookie) to determine whether the user is logged in. So I need to log in
and then have cookies and or sessions maintained as I access the page
that contains the content that I am actually interested in.

What is the simplest way to post data to a form, accept a cookie to
maintain the session (and support redirects) and then (now logged into
the site) retrieve the content of a page on the site?

Is their a library or technique that makes this simple?

Thanks!
I submit to you, in its entirety a, a script I wrote to do this. I think
its simple enough to figure out the important parts. I left some
debugging code in. Sorry for no explanation, etc., but I'm still playing
with the new 2.5 distro! This program worked for me, but there may be
comment on whether its good code from others. Names have been changed to
protect the innocent.
James

#! /usr/bin/env python

import sys
import os.path
import time
import random
import urllib
import urllib2
import cookielib

class DummyError(Exception): pass
################################################## #####################
# some constants
################################################## #####################
COOKIEFILE = 'cookies.lwp'

pda = "http://www.somemapcompany.com/"

signin_params = urllib.urlencode({'email_address':'y***@email.addr ess',
'password':'yourpasshere',
'action':'log_in',
'Submit':'Sign In'})

signout_params = urllib.urlencode({'action':'sign_out'})

download_dict = {'screen':'map_details', 'action':'download'}

info_dict = {'screen':'map_details',
'back':'find_maps',
'view':'state'}

################################################## #####################
# setup for cookies
################################################## #####################
cj = cookielib.LWPCookieJar()

if os.path.isfile(COOKIEFILE):
cj.load(COOKIEFILE)

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(c j))
urllib2.install_opener(opener)
################################################## #####################
# make the signin request
################################################## #####################
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
txheaders = {'User-agent' : user_agent}
req = urllib2.Request(pda, signin_params, txheaders)

################################################## #####################
# download_map
################################################## #####################
def download_map(map_id, urlopen=urllib2.urlopen):

################################################## ###################
# download the map
################################################## ###################
download_dict['map_id'] = map_id
download_params = urllib.urlencode(download_dict)

mapin = urlopen(pda + ("?%s" % download_params))

mapout = open('map-%s' % map_id, "wb")
mapout.write(mapin.read())
mapout.close()

################################################## ###################
# download the map info
################################################## ###################
info_dict['map_id'] = map_id
info_params = urllib.urlencode(info_dict)

infoin = urlopen(pda + ("?%s" % info_params))

infoout = open('info-%s' % map_id, "wb")
infoout.write(infoin.read())
infoout.close()
################################################## #####################
# signin and print info
################################################## #####################
try:
signin = urllib2.urlopen(req)
except IOError, e:
print 'We failed to open "%s".' % pda
if hasattr(e, 'code'):
print 'We failed with error code - %s.' % e.code
else:
print
print 'Here are the headers of the page :'
print signin.info()
afile = open("signin.html", "w")
afile.write(signin.read())
afile.close()
################################################## #####################
# report and save cookies
################################################## #####################
print
for index, cookie in enumerate(cj):
print index, ' : ', cookie

cj.save(COOKIEFILE, True, True)

print
print "cookies===>"
os.system('cat %s' % COOKIEFILE)
print "<===cookies"

for map_id in xrange(1001, 1507):
try:
download_map(map_id)
wait = 7.5 + random.randint(0,5)
print "=====waiting %s seconds..." % wait
time.sleep(wait)
except urllib2.HTTPError, e:
# except DummyError, e:
print "%s: failed to download" % map_id
print " HTTP ERROR:", e
else:
print "# Downloaded map %s successfully." % map_id

signout = urllib.urlopen(pda + ("?%s" % signout_params))

afile = open("signout.html", "w")
afile.write(signout.read())
afile.close()
--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Sep 21 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: David Rasmussen | last post by:
Some sites seem to be session driven in the sense that if I visit the homepage and do a few clicks I can navigate anywhere I want, but if I paste the current location into a new browser window...
7
by: Eric | last post by:
I am certain the answer will be 'NO', but I wanted to ask anyway just incase I have missed something. Everyone knows that one pass data to a page in an anchor tag by using the GET Method: <a...
2
by: Jim | last post by:
Hi, I'm working on an intranet site for a large client who has a search routine that runs on another intranet site on a different server. This search routine saves its searches in cookies for...
3
by: Chuck Renner | last post by:
Please help! This MIGHT even be a bug in PHP! I'll provide version numbers and site specific information (browser, OS, and kernel versions) if others cannot reproduce this problem. I'm...
2
by: * Tong * | last post by:
Hi, I'm wondering if you can show me an example php page that allows me to retrieve and display 3rd party pages. I.e., when my php script is called as http://site/path/getit.php?file_key it...
4
by: Jeff | last post by:
hey ASP.NET 2.0 I'm preparing for a certification exam on asp.net 2.0 and yesterday I took a skill assessment test on microsoft.com. One of the questions was about creating cookies. This was...
1
by: GpOscar | last post by:
Hey whats up.. i have created a form which does calculations i have installed it on the server and run on it all the other workstations. Now on this form there are some rates that change...
0
by: apondu | last post by:
I'm trying to screen scrape a site that requires a password. I am using C#.Net, i am new to this and with the information available around on the internet i just put tht information into the...
1
by: snitu | last post by:
Hello I have a problem with how to use alternate stylesheet in my site home page using cookies. Normal html file is working with out using cookies.But i have to give authority to customer to...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: veera ravala | last post by:
ServiceNow is a powerful cloud-based platform that offers a wide range of services to help organizations manage their workflows, operations, and IT services more efficiently. At its core, ServiceNow...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.