A problem while using urllib

[Johnny Lee]

Hi,
I was using urllib to grab urls from web. here is the work flow of
my program:

1. Get base url and max number of urls from user
2. Call filter to validate the base url
3. Read the source of the base url and grab all the urls from "href"
property of "a" tag
4. Call filter to validate every url grabbed
5. Continue 3-4 until the number of url grabbed gets the limit

In filter there is a method like this:

--------------------------------------------------
# check whether the url can be connected
def filteredByConne ction(self, url):
assert url

try:
webPage = urllib2.urlopen (url)
except urllib2.URLErro r:
self.logGenerat or.log("Error: " + url + " <urlopen error timed
out>")
return False
except urllib2.HTTPErr or:
self.logGenerat or.log("Error: " + url + " not found")
return False
self.logGenerat or.log("Connect ing " + url + " successed")
webPage.close()
return True
----------------------------------------------------

But every time when I ran to the 70 to 75 urls (that means 70-75
urls have been tested via this way), the program will crash and all the
urls left will raise urllib2.URLErro r until the program exits. I tried
many ways to work it out, using urllib, set a sleep(1) in the filter (I
thought it was the massive urls crashed the program). But none works.
BTW, if I set the url from which the program crashed to base url, the
program will still crashed at the 70-75 url. How can I solve this
problem? thanks for your help

Regards,
Johnny

[Steve Holden]

John J. Lee wrote:

Steve Holden <st***@holdenwe b.com> writes:
[...]

File "/usr/lib/python2.4/urllib2.py", line 996, in do_open
raise URLError(err)
urllib2.URLEr ror: <urlopen error (120, 'Operation already in progress')>

Looking at that part of the course of urrllib2 we see:

headers["Connection "] = "close"
try:
h.request(req.g et_method(), req.get_selecto r(), req.data,
headers)
r = h.getresponse()
except socket.error, err: # XXX what error?
raise URLError(err)

So my conclusion is that there's something in the Cygwin socket module
that causes problems not seen under other platforms.

I couldn't find any obviously-related error in the Python bug tracker,
and I have copied this message to the Cygwin list in case someone there
knows what the problem is.

[...]

I don't *think* this is related, but just in case:

http://python.org/sf/1208304

Good catch, John, I suspect this is a possibility so I've added the
following note:

"""The Windows 2.4.1 build doesn't show this error, but the Cygwin 2.4.1
build does still have uncollectable objects after a urllib2.urlopen (),
so there may be a platform dependency here. No 2.4.2 on Cygwin yet, so
nothing conclusive as lsof isn't available."""

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Johnny Lee]

Steve Holden 写�：

Good catch, John, I suspect this is a possibility so I've added the
following note:

"""The Windows 2.4.1 build doesn't show this error, but the Cygwin 2.4.1
build does still have uncollectable objects after a urllib2.urlopen (),
so there may be a platform dependency here. No 2.4.2 on Cygwin yet, so
nothing conclusive as lsof isn't available."""

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Maybe it's really a problem of platform dependency. Take a look at this
brief example, (not using urllib, but just want to show the platform
dependency of python):

Here is the snapshot from dos:
-----------------------
D:\>python
ActivePython 2.4.1 Build 247 (ActiveState Corp.) based on
Python 2.4.1 (#65, Jun 20 2005, 17:01:55) [MSC v.1310 32 bit (Intel)]
on win32
Type "help", "copyright" , "credits" or "license" for more information.

f = open("t", "r")
f.tell() 0L f.readline() 'http://cn.realestate.y ahoo.com\n' f.tell() 28L

--------------------------

Here is the a snapshot from cygwin:
---------------------------
Johnny Lee@esmcn-johnny /cygdrive/d
$ python
Python 2.4.1 (#1, May 27 2005, 18:02:40)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright" , "credits" or "license" for more information. f = open("t", "r")
f.tell() 0L f.readline() 'http://cn.realestate.y ahoo.com\n' f.tell()

31L

--------------------------------