How to limit the number of web pages downloaded from a site?

[Nad]

I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

[dorayme]

In article <g7**********@a ioe.org>, na*@invalid.com (Nad) wrote:

I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

Password protect folders or pages, make users register to get the
passwords, that would slow them down a bit. But really, if you make
stuff available publicly...

--
dorayme

[Adrienne Boswell]

Gazing into my crystal ball I observed na*@invalid.com (Nad) writing in
news:g7******** **@aioe.org:

I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

You could store their IP address in a session, and check to see the length
of time between requests.

--
Adrienne Boswell at Home
Arbpen Web Site Design Services
http://www.cavalcade-of-coding.info
Please respond to the group so others can share

[Neredbojias]

On 08 Aug 2008, na*@invalid.com (Nad) wrote:

I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

Change the articles' text to Olde Englishe.

--
Neredbojias
http://www.neredbojias.net/
Public Website

[Nad]

In article <Xn************ *************** *@69.16.185.247 >, Adrienne Boswell
<ar****@yahoo.c omwrote:

>Gazing into my crystal ball I observed na*@invalid.com (Nad) writing in
news:g7******* ***@aioe.org:

>I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

You could store their IP address in a session, and check to see the length
of time between requests.

Well, something along those lines.
The problem is the server side support.
Some servers do not allow cgi, php, javascript, or even ssi
executable commands, and I'd like it to work on ANY server.

[Nad]

In article <Xn************ *************** **@194.177.96.7 8>, Neredbojias
<Sc********@gma il.comwrote:

>On 08 Aug 2008, na*@invalid.com (Nad) wrote:

>I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

Change the articles' text to Olde Englishe.

:--}

I like that!!!

[Lars Eighner]

In our last episode, <g7**********@a ioe.org>, the lovely and talented Nad
broadcast on alt.html:

I have a very large site with valuable information. Is there any way to
prevent downloading a large number of articles. Some people want to
download the entire site.

It depends upon what you mean by 'articles.' If put you html documents on a
web server. you are pretty much inviting the public to view/download as much
of it as they want. If it is 'valuable', why are you giving it away? And
if you are giving it away valuable stuff, what did you expect? What is your
real concern here?

If you only worried about server load, why not zip or tar and gzip it up
and put it on an FTP server? This is most practical for related documents,
such as parts of a tutorial or parts of a spec. If you are a philanthropist
who is giving away valuable stuff, you can give it away in big chunks so
the nickel and dime requests don't bug you.

Well-behaved download-the-whole-site spiders will obey robots.txt, but that
is pretty much a courtesy thing, and it won't stop anyone who is manually
downloading a page at a time, and it won't stop rogue or altered spiders.
Likewise, you can block nice spiders which send a true user-agent ID, but
not so nice spiders can spoof their ID. That's kind of pointless, because
most of the nice spiders will obey robots.txt anyway.

You can make pages available through php or cgi which keeps track of the
number of documents with hidden controls. This is easily defeated by
anyone determined to do so, and like a cheap lock, will only keep the honest
people out. Beyond that, you can go to various user account schemes up to
putting your documents on a secure server.

But I think what you are asking is 'Can I keep my documents public and still
limit public access?' And the answer to that is, of course not because
there is a fundamental contradiction in what you want.

Any hints or pointers would be appreciated.

--
Lars Eighner <http://larseighner.com/us****@larseigh ner.com
War hath no fury like a noncombatant.
- Charles Edward Montague

[Neredbojias]

On 08 Aug 2008, na*@invalid.com (Nad) wrote:

In article <Xn************ *************** **@194.177.96.7 8>, Neredbojias
<Sc********@gma il.comwrote:

>>On 08 Aug 2008, na*@invalid.com (Nad) wrote:

>>I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

Change the articles' text to Olde Englishe.

:--}

I like that!!!

<grin>

Seriously, I don't think there's much you can do that is practical. With
server-side support, you could impliment some kind of time limit and/or p/w
but you indicated you didn't want to rely on that. An off-the-wall "non-
solution" would be to use reasonably long meta page redirects, but the user
could always come back with a new time limit.

--
Neredbojias
http://www.neredbojias.net/
Public Website

[Nad]

In article <do************ *************** *******@news-vip.optusnet.co m.au>,
dorayme <do************ @optusnet.com.a uwrote:

>In article <g7**********@a ioe.org>, na*@invalid.com (Nad) wrote:

>I have a very large site with valuable information.
Is there any way to prevent downloading a large number
of articles. Some people want to download the entire site.

Any hints or pointers would be appreciated.

Password protect folders or pages, make users register to get the
passwords, that would slow them down a bit.

It doesn't work. For example, Teleport Pro (a program to download
the entire sites) allows you to specify login/passwd.
So, once they register, they can enter this info and boom...

>But really, if you make
stuff available publicly...

Well, the site is 150 megs, over 20k articles.
And there are plenty of people who would LOVE to have
the entire site on their own box.
Then you have a problem. Providers usually charge for
the amount of traffic. In one month, you'd have to shell
out some bux, just to give the information to the
"gimme free Coce" zombies.
That does not make sense.

[Nad]

In article <sl************ *******@debrand ed.larseighner. com>, Lars Eighner
<us****@larseig hner.comwrote:

>In our last episode, <g7**********@a ioe.org>, the lovely and talented Nad
broadcast on alt.html:

>I have a very large site with valuable information. Is there any way to
prevent downloading a large number of articles. Some people want to
download the entire site.

It depends upon what you mean by 'articles.' If put you html documents on a
web server. you are pretty much inviting the public to view/download as much
of it as they want. If it is 'valuable', why are you giving it away? And
if you are giving it away valuable stuff, what did you expect? What is your
real concern here?

Downloading the entire 150+ meg site, which translates into
all sorts of things.

>If you only worried about server load, why not zip or tar and gzip it up
and put it on an FTP server? This is most practical for related documents,
such as parts of a tutorial or parts of a spec. If you are a philanthropist
who is giving away valuable stuff, you can give it away in big chunks so
the nickel and dime requests don't bug you.

Well-behaved download-the-whole-site spiders will obey robots.txt,

That doesn't work. Some random user may come and download the
entire site. By the time you put him into robots.txt, it is too late.

>but that
is pretty much a courtesy thing, and it won't stop anyone who is manually
downloading a page at a time,

That is not a problem. They can manually download as much as they want.
But no automated downloads.

>and it won't stop rogue or altered spiders.
Likewise, you can block nice spiders which send a true user-agent ID, but
not so nice spiders can spoof their ID. That's kind of pointless, because
most of the nice spiders will obey robots.txt anyway.

>You can make pages available through php or cgi which keeps track of the
number of documents with hidden controls. This is easily defeated by
anyone determined to do so,

How?

>and like a cheap lock, will only keep the honest
people out. Beyond that, you can go to various user account schemes up to
putting your documents on a secure server.

Well, no account schemes, no user verification, no limits beyond
trying to automatically download the entire site pretty much.

>But I think what you are asking is 'Can I keep my documents public and still
limit public access?'

Not really. AUTOMATED download.

>And the answer to that is, of course not because
there is a fundamental contradiction in what you want.

I do not see it at the moment.

>Any hints or pointers would be appreciated.