By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
448,563 Members | 1,241 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 448,563 IT Pros & Developers. It's quick & easy.

urllib - changing the user agent

P: n/a
I'm writing a function that will query the comp.lang.python newsgroup
via google groups....... (I haven't got nntp access from work..)

I'm using urllib (for the first time)..... and google don't seem very
keen to let me search the group from within a program - the returned
pages all tell me 'you're not allowed to do that' :-)

I read in the urllib manual pages :

class URLopener( [proxies[, **x509]])

Base class for opening and reading URLs. Unless you need to support
opening objects using schemes other than http:, ftp:, gopher: or
file:, you probably want to use FancyURLopener.
By default, the URLopener class sends a User-Agent: header of
"urllib/VVV", where VVV is the urllib version number. Applications can
define their own User-Agent: header by subclassing URLopener or
FancyURLopener and setting the instance attribute version to an
appropriate string value before the open() method is called.
Could anyone tell me how to subclass this correctly with the version
attribute set and what text string I should use to mimic Internet
explorer and/or mozilla ?
Ta

Fuzzy
Jul 18 '05 #1
Share this Question
Share on Google+
7 Replies


P: n/a
mi*****@foord.net (Fuzzyman) writes:
[...]
I'm using urllib (for the first time)..... and google don't seem very
keen to let me search the group from within a program - the returned
pages all tell me 'you're not allowed to do that' :-)
Don't do it, then. Use the google API (mind you, not certain that
google groups is part of that).

[...] Could anyone tell me how to subclass this correctly with the version
attribute set and what text string I should use to mimic Internet
explorer and/or mozilla ?


IIRC,

opener.addheaders = [("User-agent", "whatever")]

Use a program like ethereal to find what your browser sends for
"whatever".
John
Jul 18 '05 #2

P: n/a
"Fuzzyman" <mi*****@foord.net> wrote in message
news:80**************************@posting.google.c om...
I'm writing a function that will query the comp.lang.python newsgroup
via google groups....... (I haven't got nntp access from work..)

I'm using urllib (for the first time)..... and google don't seem very
keen to let me search the group from within a program - the returned
pages all tell me 'you're not allowed to do that' :-)

I read in the urllib manual pages :

class URLopener( [proxies[, **x509]])

Base class for opening and reading URLs. Unless you need to support
opening objects using schemes other than http:, ftp:, gopher: or
file:, you probably want to use FancyURLopener.
By default, the URLopener class sends a User-Agent: header of
"urllib/VVV", where VVV is the urllib version number. Applications can
define their own User-Agent: header by subclassing URLopener or
FancyURLopener and setting the instance attribute version to an
appropriate string value before the open() method is called.
Could anyone tell me how to subclass this correctly with the version
attribute set and what text string I should use to mimic Internet
explorer and/or mozilla ?
Ta

Fuzzy


I normally use something like this for crawling webpages.

import urllib
urllib.URLopener.version = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT
5.0; T312461)'
urllib.FancyURLopener.prompt_user_passwd = lambda self, host, realm: (None,
None)

Anthony McDonald
Jul 18 '05 #3

P: n/a
On 9 Jan 2004 05:09:41 -0800, mi*****@foord.net (Fuzzyman) wrote:

[ wants to change the user-agent in HTTP request from urllib ]

Fuzzy --

I take the easy way out, and use urllib2, instead:

url = "http//www.spam.com/eggs.html"
req_headers = {
'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)',
}
req = urllib2.Request(url, None, req_headers)
Jul 18 '05 #4

P: n/a
On 09 Jan 2004 14:12:56 +0000, jj*@pobox.com (John J. Lee) wrote:
Don't do it, then. Use the google API (mind you, not certain that
google groups is part of that).


Can Google APIs be used to access Google Groups? ...

No. The Google Web APIs service can only be used to search
Google's main index of 3 billion web pages.

http://www.google.com/apis/api_faq.html#ini2

Jul 18 '05 #5

P: n/a
Terry Carroll <ca*****@tjc.com> writes:
On 9 Jan 2004 05:09:41 -0800, mi*****@foord.net (Fuzzyman) wrote:

[ wants to change the user-agent in HTTP request from urllib ]

Fuzzy --

I take the easy way out, and use urllib2, instead: [...] req = urllib2.Request(url, None, req_headers)


or again, you can set .addheaders on OpenerDirector (which will cause
those headers to be added to all requests).
John
Jul 18 '05 #6

P: n/a
|Thus Spake John J. Lee On the now historical date of Fri, 09 Jan 2004
20:16:54 +0000|
or again, you can set .addheaders on OpenerDirector (which will cause
those headers to be added to all requests).


This, however, does not stop the original User-agent header to be sent,
and google still filters out the request. Instead, it just causes a
second user-agent to be sent.

Here is the very bad code I used to solve this problem. There are better
ways, I assure you, but it should point you in the right direction. You
should probably do this with inheritance, but for my quick script, this is
what I did.

----
#Override the default OpenerDirector Class Init.
#OpenerDirector *insists* on adding a User-Agent
#That some websites don't like.
foo = OpenerDirector.__init__
def bar(self, Agent='None'):
foo(self)
self.addheaders = [('User-Agent',Agent)]

OpenerDirector.__init__ = bar
-----

HTH

Sam Walters.

--
Never forget the halloween documents.
http://www.opensource.org/halloween/
""" Where will Microsoft try to drag you today?
Do you really want to go there?"""

Jul 18 '05 #7

P: n/a
Samuel Walters <sw*************@yahoo.com> writes:
|Thus Spake John J. Lee On the now historical date of Fri, 09 Jan 2004
20:16:54 +0000|
or again, you can set .addheaders on OpenerDirector (which will cause
those headers to be added to all requests).

(just to clarify, I meant an OpenerDirector instance, not the class
itself)

This, however, does not stop the original User-agent header to be sent,
and google still filters out the request. Instead, it just causes a
second user-agent to be sent.
No, it does stop them being sent. Perhaps you mutated the base class
..addheaders by mistake (using .append(("User-agent", "blah")), for
example)? Don't do that! Mutating class attributes is a bad idea.

Here is the very bad code I used to solve this problem. There are better
ways, I assure you, but it should point you in the right direction. You

[...]

No need to assure me of *that* <wink>. You can call a base class
constructor directly, you know. And clobbering the base class'
constructor is another very bad idea.
John
Jul 18 '05 #8

This discussion thread is closed

Replies have been disabled for this discussion.