473,701 Members | 2,769 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

urllib2/cookies - surely there's a better way ?

Hi - I'm writing a script which fetches a page from a web server and
takes note of any set-cookies which are served in the headers so that
when I next request a page I can send those cookies back to the
server. This is so that the usage analysis software on the server
(based on cookies) will take account of the scripts activities.

Now the thing is I'm halfway through doing this but I'm thinking there
must be a more refined mechanism than the one I'm using (see below).

I'm not really asking if there's a way to smarten up the rather clunky
splits (although if necessary that would be welcome) I'm more asking
is there not a more refined interface to the whole area of cookies.

Strangely enought the doco says that f.info() "return the
meta-information of the page, as a dictionary-like object" - well as
far as I can see it's a string and the f.info().header s is a list. I
don't usually find errors in the doco so this makes me wonder if
there's something I'm doing fundamentally wrong ?

Anyway any ideas would be welcome. Here goes with the work in progress
import urllib2
from string import split
from string import upper
req_headers = {
'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows

TARGETURL = 'http://www.somedomain. com/a/b/'

req = urllib2.Request (TARGETURL, None, req_headers)
f = urllib2.urlopen (req)
lstHeaders=f.in fo().headers

for h in lstHeaders:

lstHeaderConten ts = split(h,":",1)
if upper(lstHeader Contents[0]) == "SET-COOKIE":
print lstHeaderConten ts[1]
#get the keyword value pair to the left of the first
lstYetAnother = split(lstHeader Contents[1],";",1)
#put the keyword value pair into a list
lstOneMore = split(lstYetAno ther[0],"=",1)
print "Keyword=" + lstOneMore[0] + ". Value = " +
lstOneMore[1] + "."

That's the script - thanks for readind this far.


Jul 18 '05 #1
2 2107
Richard Shea wrote:

(about urllib2 and cookies)

Search for "ClientCookie". ..
Jul 18 '05 #2
Peter Hansen <pe***@engcorp. com> wrote in message news:<84******* *************@p owergate.ca>...
Richard Shea wrote:

(about urllib2 and cookies)

Search for "ClientCookie". ..

That's great ! I actually laughed when I read the doco - I was only
looking for something to parse cookies with but this does the whole
thing ! I haven't yet used it but I've had a few wriggles getting to
where the 'import ClientCookie' works so I thought I might tell the
newsgroup what I did to make it work (although that might be fairly
obvious to many).

First of all I was installing on a W98 machine. I tried using the
install procedure "python setup.py build" but I got the message
"error: package directory 'ClientCookie' does not exist". I do have a
slightly weird setup so I wasn't all that surprised.

Anyway I then took the fallback option of copying the 'ClientCookie'
directory from the .ZIP manually. In order to make this work you need
to ensure that sys.path contains a path to ClientCookie before it
finds the standard libraries. I chose to do that by editing the
registry at


and adding a new entry there with a value which pointed at the
clientcookie directory (ie C:/a/b/ClientCookie-0.4.18/ClientCookie)
however although I got an extra entry at the 'right' place in sys.path
this didn't allow me to "import ClientCookie" and eventually I
modified the Registry entry to read C:/a/b/ClientCookie-0.4.18 and now
everthing seems to be fine.

This is probably pretty straightforward stuff for most pepole but I
still find some aspects of 'import' a dark art so I thoguht I was
worth sticking it into the archives.

Thanks again for the tip.


Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: Doug Gray | last post by:
Python2.3 on Redhat Linux 8.0 here is the code. import urllib2 import ClientCookie request = urllib2.Request("http://fantasygames.sportingnews.com/crs/home_check_reg.htm
by: Josef Cihal | last post by:
Hi, I get an error, when I am trying to download URL with using Cookies. Where is the Problem? Thank u very much for all ideas!!! sincerely Josef
by: tomazi75-nospam(at)gmail.com | last post by:
Hello all, I've a problem using urllib2 with a proxy which need authentication. I've tested the 'simple way' : -- code -- import urllib # example values for the post
by: dmbkiwi | last post by:
I've been using urllib2 to try and automate logging into the google adsense page. I want to download the csv report files, so that I can do some analysis of them. However, I don't really know how web forms work, and the examples on the python.org/doc site aren't really helpful. I've found working scripted login code using javascript, but I don't speak javascript. These are the relevant functions:
by: itay_k | last post by:
Hi, I dont understand why this is so complicated, just to add one line of cookie header on the GET request. This is my unworking code: import time import Cookie import cookielib, urllib2
by: Ant | last post by:
Hello all, I am using urllib2 as a part of a web testing tool. One of the things I am testing is the effect of two different people perforing the same actions on the website - do they interfer with each other or not. So to emulate this, I essentially have the following function: def get_opener(): policy = cookielib.DefaultCookiePolicy(rfc2965=True)
by: Chaos | last post by:
For the Program I Am Making I Make Multiple HTTP Request to My Server. I found that using urllib2 it was pretty slow, when I activated the program and tested it it would hang from 2secs-5secs since I am doing it multiple times I wanted to speed it up by using pycurl. But I got the samething. Here is my code: import urllib import os.path import cookielib import pycurl
by: cp.finances.gouv | last post by:
Hello all, I'm facing a strange behavior of urllib2 trying to access gmail account behind a proxy (Squid). The following works perfectly : wget --save-cookies cookies --keep-session-cookies --post- data="Email=externe.finances.gouv@gmail.com&Passwd=cpfinances" --no- check-certificate -O - https://www.google.com/accounts/ServiceLoginBoxAuth
by: Joshua Kugler | last post by:
Standard disclaimer: read, googled, read some more. If you have a link, please free free to point me there. I'm using HTTPlib to construct some functional tests for a web app we're writing. We're not using urllib2 because we need support for PUT and DELETE methods, which urllib2 does not do. We also need client-side cookie handling. So, I start reading about cookielib and run across a problem. It's cookie handling is tied quite...
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.