"proper" way to download several hundred text files from a web-site ?

[Bill Woodruff]

As an exercise I wrote a small C# program to download several hundred text files of guitar tab music by parsing the home page of the
site and retrieving all the links to the music text files and using 'WebRequest and a StreamReader to download them.

I followed up the call to WebRequest.Create with a call to Application.DoEvents, and called Application.DoEvents() again after the
call to ReadToEnd on the StreamReader.

What I was looking for was some kind of delay mechanism to ensure that I was not making requests to the site too frequently. Is
there any way to ask a site something like : "Can I have another helping please, or would you rather I come back later ?."

Appreciate any clues.

thanks, Bill

[Mark]

"Bill Woodruff" <bi***@dotscience.com> wrote in
news:#x**************@TK2MSFTNGP11.phx.gbl:

What I was looking for was some kind of delay mechanism to ensure that
I was not making requests to the site too frequently. Is there any way
to ask a site something like : "Can I have another helping please, or
would you rather I come back later ?."

Appreciate any clues.

thanks, Bill

The RFC says no more than 4 connections to any given HTTP 1.0 server at any
given moment, and no more than 2 to any given HTTP 1.1 server at any
moment. As long as you don't break that rule (and I think that the
WebRequest class respects IE's settings, so if you haven't changed them
with some sort of "accellerator", those are defaults) you should be fine.

If you want to introduce more delay, then Thread.Sleep() is always good, as
long as you don't do it in UI threads (but you wouldn't do work in UI
threads, would you? that would be bad).

Mark