473,325 Members | 2,872 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,325 software developers and data experts.

Iterating and downloading a URL that lists files?

Is there a way to open a URL, one that shows nothing other than a directory
listing, and iteratively go through the list to download the files?
Aug 2 '07 #1
5 1102
On Aug 2, 5:15 am, "Jim Bancroft" <sd...@sfd.comwrote:
Is there a way to open a URL, one that shows nothing other than a directory
listing, and iteratively go through the list to download the files?
Hi Jim,

you can use a HttpWebRequest with a StreamReader to get the page from
the URL, then the content of the page can be parsed with a regular
expression to get the list of all links and then again use
HttpWebRequest to get the files...

Hope it helps

Aug 2 '07 #2
Hi,

Alexey Smirnov wrote:
On Aug 2, 5:15 am, "Jim Bancroft" <sd...@sfd.comwrote:
>Is there a way to open a URL, one that shows nothing other than a directory
listing, and iteratively go through the list to download the files?

Hi Jim,

you can use a HttpWebRequest with a StreamReader to get the page from
the URL, then the content of the page can be parsed with a regular
expression to get the list of all links and then again use
HttpWebRequest to get the files...

Hope it helps
I did that a long time ago with a Java applet. I then discovered that
the page format was different in Netscape (back then) and IE. I hope
things are more standard now!

HTH,
Laurent
--
Laurent Bugnion [MVP ASP.NET]
Software engineering, Blog: http://www.galasoft.ch
PhotoAlbum: http://www.galasoft.ch/pictures
Support children in Calcutta: http://www.calcutta-espoir.ch
Aug 2 '07 #3
Hi,

Alexey Smirnov wrote:
On Aug 2, 10:50 am, "Laurent Bugnion, MVP" <galasoft...@bluewin.ch>
wrote:
>the page format was different in Netscape (back then) and IE. I hope
things are more standard now!

Laurent, do you mean that the default directory listing depends on the
client browser? It depends on a web server and not on a browser. I
have no example for IIS, but here's the one for Apache
Back then, the client browser used to add content to the directory
listing page. I tested again right now in Firefox and IE, and apparently
this is not the case anymore, so you're lucky!

I am talking about Netspace 4 and IE4, that was back in... not sure...
97 maybe?

Anyway, good luck with your project!
Laurent
--
Laurent Bugnion [MVP ASP.NET]
Software engineering, Blog: http://www.galasoft.ch
PhotoAlbum: http://www.galasoft.ch/pictures
Support children in Calcutta: http://www.calcutta-espoir.ch
Aug 2 '07 #4
On Aug 2, 1:22 pm, "Laurent Bugnion, MVP" <galasoft...@bluewin.ch>
wrote:
>
I am talking about Netspace 4 and IE4, that was back in... not sure...
97 maybe?
Huh? We are talking about ASP.NET :-)

Aug 2 '07 #5
Hi,

Alexey Smirnov wrote:
On Aug 2, 1:22 pm, "Laurent Bugnion, MVP" <galasoft...@bluewin.ch>
wrote:
>I am talking about Netspace 4 and IE4, that was back in... not sure...
97 maybe?

Huh? We are talking about ASP.NET :-)
Yeah I was rambling. Your post awoke old nostalgies, sorry :-) I'll shut
up now. :-)

Laurent
--
Laurent Bugnion [MVP ASP.NET]
Software engineering, Blog: http://www.galasoft.ch
PhotoAlbum: http://www.galasoft.ch/pictures
Support children in Calcutta: http://www.calcutta-espoir.ch
Aug 2 '07 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Luke StClair | last post by:
Only marginally belonging in this newsgroup... but oh well. I've just started writing in python, and I want to make the files available on the web. So I did the standard <a...
6
by: Gustaf Liljegren | last post by:
I ran into this problem today: I got an array with Account objects. I need to iterate through this array to supplement the accounts in the array with more data. But the compiler complains when I...
2
by: Bala | last post by:
Hi I am trying to download the PDF files from my webserver using ASP.Net. All my files are stored at F Drive on webserver. Like this F:\Main Folder\Sub Folder\Files\File1.pdf I am...
4
by: Jeremy Chapman | last post by:
I have build a web page that lists files in a database. When the user clicks on the file the page streams the file contents to the browser with code like:...
1
by: just.starting | last post by:
Hi, My dot net client downloads files and checks for any new files time to time. The server is apache2.0.53 server. So what happens is that my file download thing works fine if I dont try to call...
0
by: just.starting | last post by:
I am having problem while downloading files from an apache server2.0.53 with php4.3.10.While downloading some files it generally stops after downloading some specific amount and then stops...
17
by: Gal Diskin | last post by:
Hi, I am writing a code that needs to iterate over 3 lists at the same time, i.e something like this: for x1 in l1: for x2 in l2: for x3 in l3: print "do something with", x1, x2, x3 What I...
7
by: Ehsan | last post by:
I foundd this code in ASPN Python Cookbook for downloading files in python but when it finished downloading files the files became corrupted and didn't open, the files in internet havn't any...
4
RMWChaos
by: RMWChaos | last post by:
The next episode in the continuing saga of trying to develop a modular, automated DOM create and remove script asks the question, "Where should I put this code?" Alright, here's the story: with a...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.