By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,768 Members | 2,004 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,768 IT Pros & Developers. It's quick & easy.

webspider getting stuck

P: n/a
i am writing a simple webspider .

how do i avoid getting stuck at something like this:
Enter username for W3CACL at www.w3.org:

?

i can obv add an if-clause for the specific site but since i guess
there will be more of the same thats ov not a viable approach in the
long run.
Jun 27 '08 #1
Share this Question
Share on Google+
1 Reply


P: n/a
no**********@yahoo.se wrote:
i am writing a simple webspider .

how do i avoid getting stuck at something like this:
Enter username for W3CACL at www.w3.org:

?

It's a silly feature of urllib. See

http://docs.python.org/lib/module-urllib.html

where it says:

"Note: When performing basic authentication, a FancyURLopener instance calls its
prompt_user_passwd() method. The default implementation asks the users for the
required information on the controlling terminal. A subclass may override this
method to support more appropriate behavior if needed."

Yes, the default behavior when faced with a site that wants authentication
is to to ask for a user name and password on standard input. This is
seldom what you want.

So subclass and overrride.

John Nagle
Jun 27 '08 #2

This discussion thread is closed

Replies have been disabled for this discussion.