Link Checking

[Steve]

Hi Guys;

You have been giving me a lot of useful information in the other two
threads. Thanks! Very interesting.

Here is my situation. My friend is writing a book. He has 3100
citations, 1225 of which have URLs.
He wants to Q/C the urls. He removed the http:// from the urls when he
prepared his materials. I wrote a script that extracted the citation
number and the urls from his citation list, prepended an "http", and
printed it on an HTML page as a hyperlink. I used a firefox extension
( http://www.kevinfreitas.net/extensions/linkchecker/ )to check the
links which reports on bad links, skipped (by the checker) links, and
forwarded/forbidden links. It color codes the links it checks.

At this point I am wondering if there is anything better I can do as
far as Q/Cing links short of doing manual checks?

>From the other threads I know that the link checkers are not perfect

and that I should tell my friend that. Is there a more accurate
description I can give him( 90% accurate, mostly good, coin toss etc)

Thanks

[ken]

Hi Steve,

You could use LinkScan/QuickCheck which is a free web service to check
for broken links. It is located at

http://www.elsop.com/linkscan/quickcheck.html

Or download a free trial of LinkScan good for 15 days that is more
comprehensive and very powerful. That can be found at

http://www.elsop.com/linkscan/dleval.cgi

Ken

[Jukka K. Korpela]

Steve <st**********@yahoo.comscripsit:

Here is my situation. My friend is writing a book. He has 3100
citations, 1225 of which have URLs.

In a printed book? Even if you check all the links just before the book
starts printing, many of them will have stopped working by the time
customers get the book. So what is the purpose of the URLs? If the book is
scientific, citations and URLs might be needed. In that case, remember to
include both the date of checking and the title or main heading or some
other content identification for the page. That way, people will have a
sporting chance of visiting, now or after 10 years, the page as it was at
the time of citation (assuming they know how to use www.archive.org and it
will remain avaulable).

>From the other threads I know that the link checkers are not perfect

and that I should tell my friend that. Is there a more accurate
description I can give him( 90% accurate, mostly good, coin toss etc)

Throwing percentages isn't useful; 97.5 of all percentages have just been
made up, and the remaining 3.5 % have been miscalculated.

The point is that no link checker can check what the link really points to.
The checkers won't notice anything if the server sends a normal OK response.
Yet the page may contain porn instead of the expected content or, more
typically, a page that tells that the domain is for sale and/or contains
lots of links to pages that someone wants to advertize.

--
Jukka K. Korpela ("Yucca")
http://www.cs.tut.fi/~jkorpela/

[Steve]

Jukka K. Korpela wrote:

The point is that no link checker can check what the link really points to.
The checkers won't notice anything if the server sends a normal OK response.
Yet the page may contain porn instead of the expected content or, more
typically, a page that tells that the domain is for sale and/or contains
lots of links to pages that someone wants to advertize.

That is what I thought and what I have been advising my friend. I
asked what I did as one last chance of their being something more I can
do for him.

He has volunteers standing by to manually check the URLs. The link
checker reports and HTML pages I generated should give them a good
start.

He is in good shape. The book will be printed but will also be online.
He has real citations, the urls are redundant so he is in a good
spot.

[ken]

Steve wrote:

Jukka K. Korpela wrote:

The point is that no link checker can check what the link really pointsto.
The checkers won't notice anything if the server sends a normal OK response.
Yet the page may contain porn instead of the expected content or, more
typically, a page that tells that the domain is for sale and/or contains
lots of links to pages that someone wants to advertize.

Not true. LinkScan Profiler™ detects adult content links or any other
type of content the user profiles. It enables webmasters and content
managers to identify pages with "inappropriate" (e.g. adult) content
linked to from their site so they can determine if they wish to
continue to link to a specific site. Furthermore, the product has the
capability of e-mailing an alarm to content managers when such a page
is found. This capability was developed when users discovered that a
link from a medical/education site to a page on breast cancer was
hijacked and replaced with a porno page. More information on this
capability can be found at:

http://www.elsop.com/linkscan/overview.html

Ken

[Alexander Huber]

Xenu has lots of useful options to make link checking easier:
http://home.snafu.de/tilman/xenulink.html#Download

Alexander.
--
Alexander Huber
mailto:al*****@gmx.net

[Jukka K. Korpela]

ke*@elsop.com <ke*@elsop.comscripsit:

LinkScan ProfilerT detects adult content links or any other
type of content the user profiles.

And you are selling it, right? It would be fair to say that explicitly,
wouldn't it?

What you saying is seriously half-truth, or quarter-truth, or something like
that.

Of course you can set up some purported content recognition for the linked
resources, or use filters based on some previous recognition or settings.

Attempts at recognizing porn, which you prefer calling "adult content" using
the common euphemistic and misleading phrase, have been failures. They can
easily be defeated, and they easily filter out non-porn content. But of
course people who sell software for it will hardly admit that.

It enables webmasters and content
managers to identify pages with "inappropriate" (e.g. adult) content
linked to from their site so they can determine if they wish to
continue to link to a specific site.

Even if that worked, and it doesn't, it would matter rather little.
Actually, the _only_ situation where any of my links (and I have lots of
them) became a porn link was about six or seven years ago, I guess, when a
reputable validator's domain was lost and bought by someone who wanted to
sell porn. I have seen hundreds if not thousands of my links turn to dust
when site administrators redesign a site breaking all links (sometimes
making old URLs work but point to completely different resources), just mess
things up, replace some content by something else, etc., or change domain so
that the old address keeps "working" but now points e.g. to a domain name
salesman's page.

--
Jukka K. Korpela ("Yucca")
http://www.cs.tut.fi/~jkorpela/

[Nick Kew]

Jukka K. Korpela wrote:

ke*@elsop.com <ke*@elsop.comscripsit:

>LinkScan ProfilerT detects adult content links or any other
type of content the user profiles.

And you are selling it, right? It would be fair to say that explicitly,
wouldn't it?

ISTR ken@elsop was a regular spammer around here some years ago,
and I was pleased to note this looked distinctly less spammy.

What you saying is seriously half-truth, or quarter-truth, or something
like that.

It looked to me more like marketing department spin. And less
harmful than many precisely because ...

Attempts at recognizing porn, which you prefer calling "adult content"
using the common euphemistic and misleading phrase, have been failures.
They can easily be defeated, and they easily filter out non-porn
content. But of course people who sell software for it will hardly admit
that.

.... the claim was sufficiently outlandish that buyers are
unlikely to take it as anything more than marketing crap.

Actually, the _only_ situation where any of my links (and I have lots of
them) became a porn link was about six or seven years ago, I guess, when
a reputable validator's domain was lost and bought by someone who wanted
to sell porn.

Yeah, I got caught by that one.

I have seen hundreds if not thousands of my links turn to

dust when site administrators redesign a site breaking all links
(sometimes making old URLs work but point to completely different
resources),

That's why detecting changes in contents is what really matters.
And using metrics to detect when change is *significant*.

--
Nick Kew

[Andy Dingley]

Nick Kew wrote:

... the claim was sufficiently outlandish that buyers are
unlikely to take it as anything more than marketing crap.

The claim might be outlandish to you and I, but I'm more concerned
about the less technically savvy who might fall for it, such as those
with managerial oversight of public bodies such as schools or
libraries. "Pr0n filtering" is a tempting technology to be selling
them, and it's easy to set up demos that do indeed show it "working".
Apart from the fact it simply doesn't work that well, I'm also not keen
on My Tax Dollars (tm) being wasted on snake oil from spammers.