web spider and password protected pages

[jdonnell]

I've been writing a simple web spider for fun, and I've run into a
problem I can't figure out. The spider hangs (waits for username and
pass) when I hit a page that requires .htaccess authentication.

self.f = urllib.urlopen( 'http://blogbloc.com/~jay/test/')
#nothing below here gets executed
print self.f.info()
....

It hangs as soon as I call urllib.urlopen( ). I was going to try to read
the info and break for pages that require authentication, but it hangs
before I can call self.f.info()

Any ideas?

[Peter Hansen]

jdonnell wrote:

I've been writing a simple web spider for fun, and I've run into a
problem I can't figure out. The spider hangs (waits for username and
pass) when I hit a page that requires .htaccess authentication.

self.f = urllib.urlopen( 'http://blogbloc.com/~jay/test/')
#nothing below here gets executed
print self.f.info()
...

It hangs as soon as I call urllib.urlopen( ). I was going to try to read
the info and break for pages that require authentication, but it hangs
before I can call self.f.info()

Any ideas?

I tried Google. First I looked for "python urlopen authentication" .
I scanned the top page for the word "authentication " and found a
few references, then something called FancyURLOpener. Adding that
to my search, skipping down a couple of links, I quickly found
a page that starts "Here is an explanation about how to handle password
protected sites."

Another approach that often works is to throw in the word
"recipe", hoping perhaps to get a hit in the Python Cookbook
page: try "python http authentication recipe", for example.

I hope that teaches you a bit about how to fish, rather than
just giving you one. ;-)

-Peter

[jdonnell]

"I quickly found
a page that starts "Here is an explanation about how to handle password
protected sites."

....

I hope that teaches you a bit about how to fish, rather than
just giving you one. ;-) "

Actually, I found a much easier solution, but since you know how to
fish I don't need to tell you what it is ;)

[Peter Hansen]

jdonnell wrote:

"I quickly found
a page that starts "Here is an explanation about how to handle password
protected sites."

...

I hope that teaches you a bit about how to fish, rather than
just giving you one. ;-) "

Actually, I found a much easier solution, but since you know how to
fish I don't need to tell you what it is ;)

Nevertheless, perhaps you'll still post the answer here so
that others who come along later can benefit from your
experience in the same way that you benefited from reading
whatever page you found (even if you didn't benefit from
my suggestions...) .

That's the way this forum works best -- thanks,
-Peter

[jdonnell]

"Neverthele ss, perhaps you'll still post the answer here so
that others who come along later can benefit from your
experience in the same way that you benefited from reading
whatever page you found (even if you didn't benefit from
my suggestions...) . "

Your funny :) Perhaps you should take your own advice. My guess is that
the google search you described will return different results in a few
months. Your first post won't benefit those who "come along later".

"The spider hangs (waits for username and
pass) when I hit a page that requires .htaccess authentication. "

I was using urllib.
urllib2 doesn't have this problem. Simply switching urllib to urllib2
fixed the problem

[Peter Hansen]

jdonnell wrote:

"Neverthele ss, perhaps you'll still post the answer here so
that others who come along later can benefit from your
experience in the same way that you benefited from reading
whatever page you found (even if you didn't benefit from
my suggestions...) . "

Your funny :) Perhaps you should take your own advice. My guess is that
the google search you described will return different results in a few
months. Your first post won't benefit those who "come along later".

Sure it will, for those with the wits to understand that I
was trying to show someone how he can *search* for the
information himself, rather than having to beg for others
to do his work for his. Whatever results Google shows in
a few months is irrelevant... it's the technique that mattered.

If there'd been the slightest sign that you'd actually tried
to find the answer yourself before you asked, your criticism
would be far more effective.

-Peter