Bad links looking like good ones, why?: Link Checkers

[Steve]

Hi;

I had a big link checking job to do and it has been years since I have
done anything like that so I found a test page to use that I knew had
bad links on it( a friends site ) and I decided to test the various
free services out.

I tried about 5 different link checkers on the test page I had ,
including Xenu and NetMechanic. I got 5 sets of identical results.

All of these link checkers reported a bad link as good. It is the
first link on this sample of the test page I used. The link is called
"Mini Pigs":

http://beforewisdom.com/dcorgs.html

My question is, what would cause a link checker to report a bad link as
good and aside from manual checks is there anyway around this problem?

It looks like the link is being redirected to a porn site and then
redirected to a 404 page. Do redirects bollucks up link checkers?

I also used a link checker extension to firefox that deals with
"forwarded or forbidden" links, which also reported the afforemention
link as good.

I am curious :).

[Daniel R. Tobias]

Steve wrote:

All of these link checkers reported a bad link as good. It is the
first link on this sample of the test page I used. The link is called
"Mini Pigs":

http://beforewisdom.com/dcorgs.html

My question is, what would cause a link checker to report a bad link as
good and aside from manual checks is there anyway around this problem?

It looks like the link is being redirected to a porn site and then
redirected to a 404 page. Do redirects bollucks up link checkers?

When I tried it, that link went to a page claiming to have "DC
Vegetarian Organizations", which in turn had two links, one of which was
"404 Not Found". Not a very useful page, but it was an actual Web page,
so link checkers properly reported it as a good link. No redirects (to
porn or otherwise) were present when I looked.

--
== Dan ==
Dan's Mail Format Site: http://mailformat.dan.info/
Dan's Web Tips: http://webtips.dan.info/
Dan's Domain Site: http://domains.dan.info/

[Stephen Poley]

On Sun, 20 Aug 2006 11:33:59 -0400, "Daniel R. Tobias" <da*@tobias.name>
wrote:

>Steve wrote:

>All of these link checkers reported a bad link as good. It is the
first link on this sample of the test page I used. The link is called
"Mini Pigs":

http://beforewisdom.com/dcorgs.html

My question is, what would cause a link checker to report a bad link as
good and aside from manual checks is there anyway around this problem?

It looks like the link is being redirected to a porn site and then
redirected to a 404 page. Do redirects bollucks up link checkers?

When I tried it, that link went to a page claiming to have "DC
Vegetarian Organizations", which in turn had two links, one of which was
"404 Not Found". Not a very useful page, but it was an actual Web page,
so link checkers properly reported it as a good link. No redirects (to
porn or otherwise) were present when I looked.

If you look at the link history in Opera, there is indeed a page there
with a title suggesting porn. All but a bit odd but, as you say, there
is a page there so the link checkers are correct.

--
Stephen Poley

http://www.xs4all.nl/~sbpoley/webmatters/

[Steve]

Daniel R. Tobias wrote:

Steve wrote:

All of these link checkers reported a bad link as good. It is the
first link on this sample of the test page I used. The link is called
"Mini Pigs":

http://beforewisdom.com/dcorgs.html

My question is, what would cause a link checker to report a bad link as
good and aside from manual checks is there anyway around this problem?

It looks like the link is being redirected to a porn site and then
redirected to a 404 page. Do redirects bollucks up link checkers?

When I tried it, that link went to a page claiming to have "DC
Vegetarian Organizations", which in turn had two links, one of which was
"404 Not Found". Not a very useful page, but it was an actual Web page,
so link checkers properly reported it as a good link. No redirects (to
porn or otherwise) were present when I looked.

It is not a real page. I parsed it down from a real page figuring it
would be easier to tell people to go to the "mini pigs" link at the top
of a page with only 3 links.

[Stephen Poley]

On 20 Aug 2006 09:18:07 -0700, "Steve" <st**********@yahoo.comwrote:

>It is not a real page. I parsed it down from a real page figuring it
would be easier to tell people to go to the "mini pigs" link at the top
of a page with only 3 links.

It is most certainly a real page. The source starts as follows (I'll
spare you the rest):

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>Arabic sex story in arabic in real sex action.</title>
</head>
<SCRIPT language=JavaScript src="/menu2.js"></SCRIPT>
....
--
Stephen Poley

http://www.xs4all.nl/~sbpoley/webmatters/

[Nikita the Spider]

In article <11*********************@b28g2000cwb.googlegroups. com>,
"Steve" <st**********@yahoo.comwrote:

Hi;

I had a big link checking job to do and it has been years since I have
done anything like that so I found a test page to use that I knew had
bad links on it( a friends site ) and I decided to test the various
free services out.

I tried about 5 different link checkers on the test page I had ,
including Xenu and NetMechanic. I got 5 sets of identical results.

All of these link checkers reported a bad link as good. It is the
first link on this sample of the test page I used. The link is called
"Mini Pigs":

http://beforewisdom.com/dcorgs.html

My question is, what would cause a link checker to report a bad link as
good and aside from manual checks is there anyway around this problem?

It looks like the link is being redirected to a porn site and then
redirected to a 404 page. Do redirects bollucks up link checkers?

Steve,
Your definition of "good" and "bad" for links is different from that of
the average link checker. All of the link checkers that I'm aware of
merely test that a GET or HEAD request to the destination of a link
generates a successful response. It'd be a pretty special link checker
if it evaluated the content of the link target. How is the link checker
supposed to know what you consider "good" content? For all it knows,
maybe you *want* to link to a site featuring live nude zebras. To each
his own.

My site crawler (see my sig) primarily performs HTML validation but it
also does link checks. As you can see on this page, I'm very specific
about which response codes I consider "bad" and "good":
http://nikitathespider.com/reports/sample/HotLinks.html

Note that 301 (Moved Permanently) is on the "bad" list but 302 (Moved
Temporarily) is not. That's a judgment call on my part; other link
checkers may have a different opinion.

HTH

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more

[Nick Kew]

Nikita the Spider wrote:

Steve,
Your definition of "good" and "bad" for links is different from that of
the average link checker. All of the link checkers that I'm aware of
merely test that a GET or HEAD request to the destination of a link
generates a successful response. It'd be a pretty special link checker
if it evaluated the content of the link target. How is the link checker
supposed to know what you consider "good" content? For all it knows,
maybe you *want* to link to a site featuring live nude zebras. To each
his own.

Yep. But what a link checker can do (and what Valet does) is to detect
updated links, so it alerts you if the target of a link has been updated
since you last checked. It's up to you whether you use that information
to check that the link is still what you thought it was.

Note that 301 (Moved Permanently) is on the "bad" list but 302 (Moved
Temporarily) is not. That's a judgment call on my part; other link
checkers may have a different opinion.

I'd say that's an oversimplistic approach. Redirects are not
inherently either good or bad. Valet flags them as redirects,
with the suggestion that you might want to update them.

--
Nick Kew

[Rik]

Nick Kew wrote:

Nikita the Spider wrote:

>Steve,
Your definition of "good" and "bad" for links is different from that
of the average link checker. All of the link checkers that I'm aware
of merely test that a GET or HEAD request to the destination of a
link generates a successful response. It'd be a pretty special link
checker if it evaluated the content of the link target. How is the
link checker supposed to know what you consider "good" content? For
all it knows, maybe you *want* to link to a site featuring live nude
zebras. To each his own.

Yep. But what a link checker can do (and what Valet does) is to
detect updated links, so it alerts you if the target of a link has
been updated since you last checked. It's up to you whether you use
that information to check that the link is still what you thought it
was.

>Note that 301 (Moved Permanently) is on the "bad" list but 302 (Moved
Temporarily) is not. That's a judgment call on my part; other link
checkers may have a different opinion.

I'd say that's an oversimplistic approach. Redirects are not
inherently either good or bad. Valet flags them as redirects,
with the suggestion that you might want to update them.

Well, that's exactly the case here: if you want to test you link for
validity, and get an answer it's permanently moved, it will mean the link
is not valid anymore, and should be updated. A temporarily moved means your
specific link is still valid.

Grtz,
--
Rik Wasmus

[Nikita the Spider]

In article <bc************@asgard.webthing.com>,
Nick Kew <ni**@asgard.webthing.comwrote:

Nikita the Spider wrote:

Steve,
Your definition of "good" and "bad" for links is different from that of
the average link checker. All of the link checkers that I'm aware of
merely test that a GET or HEAD request to the destination of a link
generates a successful response. It'd be a pretty special link checker
if it evaluated the content of the link target. How is the link checker
supposed to know what you consider "good" content? For all it knows,
maybe you *want* to link to a site featuring live nude zebras. To each
his own.

Yep. But what a link checker can do (and what Valet does) is to detect
updated links, so it alerts you if the target of a link has been updated
since you last checked. It's up to you whether you use that information
to check that the link is still what you thought it was.

That's a nice feature, although I think it would be noisy for some sites.

Note that 301 (Moved Permanently) is on the "bad" list but 302 (Moved
Temporarily) is not. That's a judgment call on my part; other link
checkers may have a different opinion.

I'd say that's an oversimplistic approach. Redirects are not
inherently either good or bad. Valet flags them as redirects,
with the suggestion that you might want to update them.

I agree. I don't use the good/bad terminology myself, I was just trying
to stay consistent in the context of this posting. I just call them
"hot" links which "are the links on your site that are most likely to
need attention". One gets the information one needs to decide whether or
not the link is "broken" by one's own standards.

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more