urlib - automatic cookie handling

[Alex Hunsley]

I'm using urllib to post data to a web form by issuing a command similar to this:
filename, headers =
urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.XmlFeed",
"content.txt", None, urllib.urlencode({"aParameter": "theValue"}))

Now, the problem is that the above fails, since I am not sending a session
cookie. Visitors to the web sites' html submission form are sent a session
cookie which is given back to the server when they submit a search via the
browser, as often happens.
Now, I could use urllib to get the form page and read the cookie from the
headers that are returned and then manually put that cookie in my submission to
the servlet, but my question is: is there a way to tell urllib or some other
part of HTTP handling in python that I want to remember any cookie that is
given to me, and give it back to that site if I send requests later on?

thanks
alex

[Pierre-Frédéric Caillaud]

Yes.

First fetch the URL which gives you the cookie. Parse the HTTP headers to
get the cookie (use the header parsing function and get the Set-Cookie
header). Then send the cookie in your headers along your next request, and
you're in. For this you must use urllib2 which allows custom headers (use
request.headers or something).

I'm using urllib to post data to a web form by issuing a command similar
to this:
filename, headers =
urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.XmlFeed",
"content.txt", None, urllib.urlencode({"aParameter": "theValue"}))

Now, the problem is that the above fails, since I am not sending a
session cookie. Visitors to the web sites' html submission form are sent
a session cookie which is given back to the server when they submit a
search via the browser, as often happens.
Now, I could use urllib to get the form page and read the cookie from
the headers that are returned and then manually put that cookie in my
submission to the servlet, but my question is: is there a way to tell
urllib or some other part of HTTP handling in python that I want to
remember any cookie that is given to me, and give it back to that site
if I send requests later on?

thanks
alex

[Alex Hunsley]

Pierre-Frédéric Caillaud wrote:

Yes.

First fetch the URL which gives you the cookie. Parse the HTTP
headers to get the cookie (use the header parsing function and get the
Set-Cookie header). Then send the cookie in your headers along your
next request, and you're in. For this you must use urllib2 which allows
custom headers (use request.headers or something).

thanks for your response!
An even better answer, which I've just found:
http://wwwsearch.sourceforge.net/ClientCookie

Can I ask that you please don't top-post? It makes threads hard to follow...

thanks
alex

[Michael Geary]

Alex Hunsley wrote:

I'm using urllib to post data to a web form by issuing a command similar
to this:

filename, headers =
urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.
XmlFeed", "content.txt", None, urllib.urlencode({"aParameter": "theValue"}))

Now, the problem is that the above fails, since I am not sending a session
cookie. Visitors to the web sites' html submission form are sent a session
cookie which is given back to the server when they submit a search via the
browser, as often happens.
Now, I could use urllib to get the form page and read the cookie from the
headers that are returned and then manually put that cookie in my
submission to the servlet, but my question is: is there a way to tell urllib or some other part of HTTP handling in python that I want to remember
any cookie that is given to me, and give it back to that site if I send
requests later on?

You're looking for ClientCookie:

http://wwwsearch.sourceforge.net/ClientCookie/

http://wwwsearch.sourceforge.net/ClientCookie/doc.html

-Mike

[Alex Hunsley]

Michael Geary wrote:

Alex Hunsley wrote:

I'm using urllib to post data to a web form by issuing a command similar
to this:

filename, headers =

urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.
XmlFeed",

"content.txt", None, urllib.urlencode({"aParameter": "theValue"}))

Now, the problem is that the above fails, since I am not sending a session
cookie. Visitors to the web sites' html submission form are sent a session
cookie which is given back to the server when they submit a search via the
browser, as often happens.
Now, I could use urllib to get the form page and read the cookie from the
headers that are returned and then manually put that cookie in my
submission to the servlet, but my question is: is there a way to tell

urllib

or some other part of HTTP handling in python that I want to remember
any cookie that is given to me, and give it back to that site if I send
requests later on?

You're looking for ClientCookie:

http://wwwsearch.sourceforge.net/ClientCookie/

http://wwwsearch.sourceforge.net/ClientCookie/doc.html

-Mike

hi there!
Snap! I think we both posted at about the same time. Good to know someone was
suggesting it anyway!

thanks
alex

[John J. Lee]

Alex Hunsley <la**@tardis.ed.ac.molar.uk> writes:
[...]

http://wwwsearch.sourceforge.net/ClientCookie/

[...] Snap! I think we both posted at about the same time. Good to know
someone was suggesting it anyway!

You might care to know that the cookie-handling part of ClientCookie
has found its way into Python 2.4, as module cookielib. So, the
development version of CC (which is compatible with old versions of
Python) is currently moving towards being as-compatible-as-possible
with cookielib and 2.4 urllib2.

Actually, if you're using Python 2.3 and want to be
forwards-compatible with 2.4 here, I'm 95% sure you can just grab
cookielib and urllib2 from Python CVS and use those. cookielib
doesn't import urllib2, so you can leave 2.3's urllib2 intact and use
a local copy of 2.4's urllib2, eg. by keeping it somewhere like
~/lib/python and renaming it urllib2_24.py. cookielib doesn't include
stuff like HTTPEquivProcessor that are in ClientCookie, but I'll make
those available in a form that works with 2.4 in a separate package
soon (that package will probably be ClientCookie itself, in a 1.0.x or
late 0.9.x version).

I should mention these things on the web page...
John