473,241 Members | 1,430 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,241 software developers and data experts.

Handling cookies without urllib2 and cookielib

Standard disclaimer: read, googled, read some more. If you have a link,
please free free to point me there.

I'm using HTTPlib to construct some functional tests for a web app we're
writing. We're not using urllib2 because we need support for PUT and
DELETE methods, which urllib2 does not do.

We also need client-side cookie handling. So, I start reading about
cookielib and run across a problem. It's cookie handling is tied quite
tightly to urllib2's request object. httplib has somewhat different
semantics in its request object. So, you can use cookielib with httplib.
And cookie lib has no simple function (that I could find) for passing in a
set-cookie header and getting back a CookieJar object (or even a list of
Cookie objects).

I'm sure I'm not the first to have to deal with httplib and cookies. Anyone
have suggestions or pointers?

j

Dec 15 '07 #1
2 3051
On 14 dic, 23:44, Joshua Kugler <jkug...@bigfoot.comwrote:
I'm using HTTPlib to construct some functional tests for a web app we're
writing. We're not using urllib2 because we need support for PUT and
DELETE methods, which urllib2 does not do.

We also need client-side cookie handling. So, I start reading about
cookielib and run across a problem. It's cookie handling is tied quite
tightly to urllib2's request object. httplib has somewhat different
semantics in its request object. So, you can use cookielib with httplib.
And cookie lib has no simple function (that I could find) for passing in a
set-cookie header and getting back a CookieJar object (or even a list of
Cookie objects).
What about correcting the first thing, making urllib2 support HEAD/PUT/
DELETE?

import urllib2

class Request(urllib2.Request):

def __init__(self, url, data=None, headers={},
origin_req_host=None, unverifiable=False,
method=None):
urllib2.Request.__init__(self, url, data, headers,
origin_req_host, unverifiable)
self.method = method

def get_method(self):
if self.method is None:
if self.data is not None:
return "POST"
else:
return "GET"
return self.method

pyf = urllib2.urlopen(Request("http://www.python.org/",
method="HEAD"))
pyprint f.info()
Date: Sun, 16 Dec 2007 00:03:43 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_ssl/2.2.3 OpenSSL/
0.9.8c
Last-Modified: Sat, 15 Dec 2007 16:25:58 GMT
ETag: "60193-3e6a-a24fb180"
Accept-Ranges: bytes
Content-Length: 15978
Connection: close
Content-Type: text/html

pyprint len(f.read())
0

Notes:
a) Instead of urlopen(url,...) you must use urlopen(Request(url,...))
b) Redirection is not handled correctly in HTTPRedirectHandler (the
request method should be copied over)
c) I've not verified PUT / DELETE methods
d) I'll try to make a proper patch later

--
Gabriel Genellina
Dec 16 '07 #2
Gabriel Genellina wrote:
On 14 dic, 23:44, Joshua Kugler <jkug...@bigfoot.comwrote:
>I'm using HTTPlib to construct some functional tests for a web app we're
writing. We're not using urllib2 because we need support for PUT and
DELETE methods, which urllib2 does not do.

We also need client-side cookie handling. So, I start reading about
cookielib and run across a problem. It's cookie handling is tied quite
tightly to urllib2's request object. httplib has somewhat different
semantics in its request object. So, you can use cookielib with httplib.
And cookie lib has no simple function (that I could find) for passing in
a set-cookie header and getting back a CookieJar object (or even a list
of Cookie objects).

What about correcting the first thing, making urllib2 support HEAD/PUT/
DELETE?
<SNIP>

We may have to do that, and then hack on the Redirect handler too so it will
properly keep the request method. But that's not our preference, for
obvious reasons. :)

I just find it hard to believe that no one has ever needed to do cookie
handling in a generic way (i.e. input: set-cookie header, output: cookie
objects) before. May have to write my own. Or sublcass/extend cookielib.

j

Dec 18 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Alex Hunsley | last post by:
I'm using urllib to post data to a web form by issuing a command similar to this: filename, headers = urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.XmlFeed",...
0
by: C. Titus Brown | last post by:
Hi all, just spent some time playing with cookielib in Python 2.4, trying to get the cookielib example to work with my mailman admindb page. The problem was that cookies weren't getting saved....
1
by: Alex Hunsley | last post by:
I'm writing a test script in python for pulling web pages from a web server using urllib2 and cookielib. Since the main thing I am testing is what happens when concurrent requests are made to the...
2
by: john.lehmann | last post by:
Attacked is a piece of code which first hits the login page successfully and receives back login cookies. But then when I attempt to hit a page which is restricted to logged in users only, I fail....
1
by: onceuponapriori | last post by:
Greetings gents. I'm a Railser working on a django app that needs to do some scraping to gather its data. I need to programatically access a site that requires a username and password. Once I...
2
by: Gilles Ganault | last post by:
Hello I need to write a script to automate fetching data from a web site: 1. using the POST method, log on, with login/password saved as cookies 2. download page and extract relevent information...
2
by: Devraj | last post by:
Hi everyone, I have been battling to make my code work with a HTTPS proxy, current my code uses urllib2 to to most things and works well, except that urllib2 doesn't handle HTTPS proxies. ...
2
by: Larry Bates | last post by:
I'm struggling with a project using mechanize and cookies to screen scape a website. The site requires a client created cookie for authentication. Below is the code I'm attempting to use with the...
3
by: trihaitran | last post by:
Hi I am trying to pull some data from a Web site: http://schoolfinder.com The issue is that I want to use the advanced search feature which requires logging into the Web site. I have a username...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.