473,288 Members | 1,726 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,288 software developers and data experts.

webspider getting stuck

i am writing a simple webspider .

how do i avoid getting stuck at something like this:
Enter username for W3CACL at www.w3.org:

?

i can obv add an if-clause for the specific site but since i guess
there will be more of the same thats ov not a viable approach in the
long run.
Jun 27 '08 #1
1 1440
no**********@yahoo.se wrote:
i am writing a simple webspider .

how do i avoid getting stuck at something like this:
Enter username for W3CACL at www.w3.org:

?

It's a silly feature of urllib. See

http://docs.python.org/lib/module-urllib.html

where it says:

"Note: When performing basic authentication, a FancyURLopener instance calls its
prompt_user_passwd() method. The default implementation asks the users for the
required information on the controlling terminal. A subclass may override this
method to support more appropriate behavior if needed."

Yes, the default behavior when faced with a site that wants authentication
is to to ask for a user name and password on standard input. This is
seldom what you want.

So subclass and overrride.

John Nagle
Jun 27 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

303
by: mike420 | last post by:
In the context of LATEX, some Pythonista asked what the big successes of Lisp were. I think there were at least three *big* successes. a. orbitz.com web site uses Lisp for algorithms, etc. b....
5
by: Stuart Palmer | last post by:
Hi, I am trying to get cdo.message working on my home development machine, however, when I send it it appears to get stuck in the queue and never actually gets sent. If I don't have a domain...
4
by: Andrew DeFaria | last post by:
I'm trying to use the onKeyDown event on a text field. I intersept the keystroke, interpret it and then do certain actions based on certain keys stuck. If a special key is stuck I replace the value...
22
by: PeteCresswell | last post by:
I've been to http://msdn.microsoft.com/library/default.asp?url=/library/en-us/mschrt/html/vbobjtitleobject.asp, but still don't have a clue. For example, I've got a chart object namde...
10
by: Yitzik | last post by:
I have IBM ThinkPad with 2.2GHz and 512MB RAM and I have problems when I try to debug a project I'm working on it. The computer suddenly getting stuck in the middle of the debug, it doesn't...
9
by: Microsoft News Server | last post by:
Hi, I am currently having a problem with random, intermittent lock ups in my ASP.net application on our production server (99% CPU usage by 3 threads, indefinately). I currently use IIS Debug...
4
by: traineeirishprogrammer | last post by:
I am at this stage an intermediate programmer at php. Well I am to some extent. See I know a good amount of theory behind php but I never put it into practicality. I never really made a useful...
7
by: alphasahoo | last post by:
Hi I am working on a program which writes the output a SQL select statements from number of source tables first to a load matrix and then writes to a load.dat file. But while writing to the...
2
by: notnorwegian | last post by:
url = re.compile(r"^((ht|f)tp(s?)\:\/\/|~/|/)?(+:\w+@)?( {1} (+\.)+ ({2,5}))(:{1,5})?((/?\w+/)+|/?)(\w+\.{3,4})?((\?\w+=\w+)? (& \w+=\w+)*)?") why isnt this url catching something like: ...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.