Why does this fail?

[Dave Murray]

New to Python question, why does this fail?

Thanks,
Dave

---testcase.py---
import sys, urllib, htmllib
def Checkit(URL):
try:
print "Opening", URL
f = urllib.open(URL )
f.close()
return 1
except:
return 0

rtfp = Checkit("http://www.python.org/doc/Summary.html")
if rtfp == 1:
print "OK"
else:
print "Fail"
python testcase.py

[Mel Wilson]

In article <vv************ @corp.supernews .com>,
"Dave Murray" <dl******@mic ro-net.com> wrote:

New to Python question, why does this fail?

---testcase.py---
import sys, urllib, htmllib
def Checkit(URL):
try:
print "Opening", URL
f = urllib.open(URL )
f.close()
return 1
except: Here, try
except Exception, details
print "Exception: ", details return 0

rtfp = Checkit("http://www.python.org/doc/Summary.html")
if rtfp == 1:
print "OK"
else:
print "Fail"

Then you'll see

Opening http://www.python.org/doc/Summary.html
Exception: 'module' object has no attribute 'open'
Fail
You probably mean urlopen .

Regards. Mel.

[Mark McEahern]

On Sun, 2004-01-04 at 18:58, Dave Murray wrote:

New to Python question, why does this fail?
[...] try: [...] except:

[...]

Because you're treating all errors as if they're what you expect. You
should be more specific in your except clause. Do this and you'll see
what I mean:

try:
whatever
except:
raise # raise whatever exception occurred
return 0

In other words, you should be explicit about the errors you silence.

Also, it's not clear what Checkit() is actually supposed to do. Is it
supposed to verify the URL actually exists? urllib doesn't raise an
error for 404 not found--urllib2 does. Try that instead.

Cheers,

// m

[Isaac To]

>>>>> "Dave" == Dave Murray <dl******@mic ro-net.com> writes:

Dave> New to Python question, why does this fail? Thanks, Dave

Dave> f = urllib.open(URL )

urllib does not have an open function. Instead, it has a constructor called
URLopener, which creates an object with such a method. So instead, you have
to say

opener = urllib.URLopene r()
f = opener(URL)

Regards,
Isaac.

[Dave Murray]

Thank you all, this is a hell of a news group. The diversity of answers
helped me with some unasked questions, and provided more elegant solutions
to what I thought that I had figured out on my own. I appreciate it.

It's part of a spider that I'm working on to verify my own (and friends) web
page and check for broken links. Looks like making it follow robot rules
(robots.txt and meta field exclusions) is what's left.

I have found the library for html/sgml to be not very robust. Big .php and
..html with lot's of cascades and external references break it very
ungracefully (sgmllib.SGMLPa rseError: expected name token). I'd like to be
able to trap that stuff and just move on to the next file, accepting the
error. I'm reading in the external links and printing the title as a sanity
check in addition to collecting href anchors. This problem that I asked
about reared it's head when I started testing for a robots.txt file, which
may or may not exist.

The real point is to learn the language. When a new grad wrote a useful
utility at work in Python faster than I could have written it in C I decided
that I needed to learn Python. He's very sharp but he sold me on the
language too. Since I often must write utilities, Python seems to be a very
good thing since I normally don't have much time to kill on them.

Dave

[Mark McEahern]

On Sun, 2004-01-04 at 20:58, Dave Murray wrote:
[...]

I have found the library for html/sgml to be not very robust. Big .php and
.html with lot's of cascades and external references break it very
ungracefully (sgmllib.SGMLPa rseError: expected name token).

I'd suggest using htmllib.

// m

[Anand Pillai]

I could not help replying to this thread...

There are already quite a lot of spider programs existing
in Python. I am the author of one of the first programs of
the kind, called HarvestMan. It is multithreaded and has
many features for downloading websites, checking links etc.
You can get it from the HarvestMan homepage at
http://harvestman.freezope.org. HarvestMan is quite
comprehensive and is a bit more than a link checker or
web crawler. My feeling is that it is not easy to understand
for a Python beginner though the program is distributed
as source code in true Python tradition.

If you want something simpler, try spider.py. You can get
information on it from the PyPi pages.

My point was that, there is nothing to gain from re-inventing
the wheel again and again. Spider programs have been written in
Python, so you should try to use them rather than writing code
from scratch. If you think that you are having new ideas, then
take the code of HarvestMan(or spider) and customize it or
improve on it. I will be happy to merge the changes back in the
code if I think they improve the program, if it is for HarvestMan.

This is the main reason why developers release programs as
opensource. Help the community, and help yourselves. Re-inventing
the wheel is perhaps not the way to go.

best regards

-Anand
"Dave Murray" <dl******@mic ro-net.com> wrote in message news:<vv******* *****@corp.supe rnews.com>...

Thank you all, this is a hell of a news group. The diversity of answers
helped me with some unasked questions, and provided more elegant solutions
to what I thought that I had figured out on my own. I appreciate it.

It's part of a spider that I'm working on to verify my own (and friends) web
page and check for broken links. Looks like making it follow robot rules
(robots.txt and meta field exclusions) is what's left.

I have found the library for html/sgml to be not very robust. Big .php and
.html with lot's of cascades and external references break it very
ungracefully (sgmllib.SGMLPa rseError: expected name token). I'd like to be
able to trap that stuff and just move on to the next file, accepting the
error. I'm reading in the external links and printing the title as a sanity
check in addition to collecting href anchors. This problem that I asked
about reared it's head when I started testing for a robots.txt file, which
may or may not exist.

The real point is to learn the language. When a new grad wrote a useful
utility at work in Python faster than I could have written it in C I decided
that I needed to learn Python. He's very sharp but he sold me on the
language too. Since I often must write utilities, Python seems to be a very
good thing since I normally don't have much time to kill on them.

Dave

[Dave Murray]

Thank you for the information. I will check them out after I finish my
effort. My purpose isn't to obtain a spider program, it is to learn Python
by doing. If the exercise will result in something that I can use, it gives
me incentive to not abandon the effort because the exercise is interesting
to me. The sources that you pointed out should be rich in information on how
I could have done it better if I had been more experienced in Python
(knowledgeable about it's libraries, etc.)

Whenever I learn something new I like to work at it, get help if I'm stuck
on something silly (why waste time?), assess what I did against a higher
standard, repeat. It's just the way that I learn. I can see that this forum
will be just what I need for a chunk of that process. I appreciate it.

Regards,
Dave

----- Original Message -----
From: "Anand Pillai" <py*******@Hotp op.com>

I could not help replying to this thread...

There are already quite a lot of spider programs existing
in Python. --
This is the main reason why developers release programs as
opensource. Help the community, and help yourselves. Re-inventing
the wheel is perhaps not the way to go.

[Dave Murray]

After re-reading this part, I can see that it is an idea that I like. How
does participating in open source work for someone (me) who has signed the
customary intellectual property agreement with the corporation that they
work for? Since programming is part of my job, developing test solutions
implemented on automatic test equipment (the hardware too) I don't know if I
would/could be poison to an open source project. How does that work? I've
never participated. If all the work is done on someone's own time, not using
company resources, yadda-yadda-hadda-hadda, do corporate lawwwyaahhhs have a
history of trying to dispute that and stake a claim? No doubt, many of you
are in the same position.

Regards,
Dave
"Anand Pillai" <py*******@Hotp op.com> wrote in message
news:84******** *************** ***@posting.goo gle.com...

This is the main reason why developers release programs as
opensource. Help the community, and help yourselves. Re-inventing
the wheel is perhaps not the way to go.

[Skip Montanaro]

Dave> How does participating in open source work for someone (me) who
Dave> has signed the customary intellectual property agreement with the
Dave> corporation that they work for? Since programming is part of my
Dave> job, developing test solutions implemented on automatic test
Dave> equipment (the hardware too) I don't know if I would/could be
Dave> poison to an open source project. How does that work?

Only your corporate counsel knows for sure. <wink> Seriously, the degree to
which you are allowed to release code to an open source project and the
manner in which is released is probably a matter best taken up with your
company's legal department. Some companies are fairly enlightened. Some
are not. You may need very little review to release bug fixes or test cases
(my guess is you might be pretty good at writing test cases ;-), more review
to release a new module or package, and considerable participation by
management and the legal eagles if you want to release a sophisticated
application into the wild.

In any case, if you make large contributions to an open source project such
as Python, I'm pretty sure a release form for substantial amounts of code
will be required at the Python end of things. See here

http://www.python.org/psf/psf-contri...agreement.html

for more details. Note that it hasn't been updated in a couple years. I
don't know if MAL has something which is more up-to-date.

Skip