Reading Url source with c# problem

[brightlight]

Hi

I am trying to read source of a web page (ASP) to grab pictures. I am coding in C# and I can succesfully do it with the following code for html pages.

WebRequest url = WebRequest.Create(url2);
WebResponse response = url.GetResponse();
StreamReader input = new StreamReader(response.GetResponseStream());
recieve = input.ReadToEnd().ToString();


input.Close();
response.Close();

But it fails in ASP pages. as an example...when url2=http://www.pearsoned.co.uk/bookshop/detail.asp?item=305606 it only grabs the source for http://www.pearsoned.co.uk/bookshop/detail.asp and since function did not completed Icannot see the location for the picture"9780201882605.jpg". However the location of the pic is visible in IExplorer->View->Source.

any help will be apreciated ...

[balabaster]

Are you sure your URL2 variable has the full address in it?

url2=http://xx.com/item.asp?no=1701

I just ran your code against a website I wrote (well actually, I translated it to VB first, but it's the same code) that uses querystrings and I substituted the address in place of URL2 in your code and it worked just fine for me.
VB

Dim url As WebRequest = WebRequest.Create("http://localhost/OnCall/OnCallView.aspx?loc=23")
Dim response As WebResponse = url.GetResponse()
Dim input As StreamReader = New StreamReader(response.GetResponseStream())
Dim receive As String = input.ReadToEnd().ToString()
System.IO.File.WriteAllText("C:\Test.htm", receive)

C#

WebRequest url = WebRequest.Create(http://localhost/OnCall/OnCallView.aspx?loc=23);
WebResponse response = url.GetResponse();
StreamReader input = new StreamReader(response.GetResponseStream());
String receive = input.ReadToEnd().ToString();
System.IO.File.WriteAllText("C:\\Test2.htm", receive);

Both work just fine at my end...

[brightlight]

well thanks for the quick answer. But NOW it is working for me too.. But I need to try the code at work for our local pages on monday. I am confused :)

[brightlight]

hello again

No it is not working in my example. But the code works. The thing that I am working is local so I have no chance to give you the link. But I can tell you the problem.

In my example threre is an item_search.asp with few checkbox options. After submit button pressed (with item no in textbox) it returns item.asp?no=1701. If I enter direct link as url2=http://xx.com/item.asp?no=1701 it returns me the source of http://xx.com/item_search.asp. (Guess there are sub functions running and since i didnt enter any it gives me the home page)

I did use another solution as



HTMLDocument myDoc = new HTMLDocumentClass();
myDoc = (HTMLDocument) axWebBrowser1.Document;
HTMLDocument myDoc2 = new HTMLDocumentClass();
HTMLInputElement otxtSearchBox = (HTMLInputElement) myDoc.all.item("item_no", 0);

otxtSearchBox.value =itembox.Text;
otxtSearchBox.focus();
SendKeys.Send("{ENTER}");
System.IO.StreamWriter str = null;

if ( ( (int)axWebBrowser1.ReadyState >= 4) & ( ! axWebBrowser1.Busy ))
{
str = new StreamWriter("C:\\saved.txt");
str.Write( myDoc2.documentElement.innerHTML) ;
}

but guess.......did not work...I am completely out of ideas. AND please share any ideas if you have..

[Plater]

Well, my first suggestion is to use HttpWebReques over WebRequest (if you have it, might be new in .NET2.0).

Then I guess I would try to simulate the selection system that goes on, possibly with a series of HttpWebRequest(or WebRequest).

[balabaster]

hello again

No it is not working in my example. But the code works. The thing that I am working is local so I have no chance to give you the link. But I can tell you the problem.

In my example threre is an item_search.asp with few checkbox options. After submit button pressed (with item no in textbox) it returns item.asp?no=1701. If I enter direct link as url2=http://xx.com/item.asp?no=1701 it returns me the source of http://xx.com/item_search.asp. (Guess there are sub functions running and since i didnt enter any it gives me the home page)

I did use another solution as



HTMLDocument myDoc = new HTMLDocumentClass();
myDoc = (HTMLDocument) axWebBrowser1.Document;
HTMLDocument myDoc2 = new HTMLDocumentClass();
HTMLInputElement otxtSearchBox = (HTMLInputElement) myDoc.all.item("item_no", 0);

otxtSearchBox.value =itembox.Text;
otxtSearchBox.focus();
SendKeys.Send("{ENTER}");
System.IO.StreamWriter str = null;

if ( ( (int)axWebBrowser1.ReadyState >= 4) & ( ! axWebBrowser1.Busy ))
{
str = new StreamWriter("C:\\saved.txt");
str.Write( myDoc2.documentElement.innerHTML) ;
}

but guess.......did not work...I am completely out of ideas. AND please share any ideas if you have..

Okay - what are you trying to achieve?

My original understanding is that given a URL with parameter extensions on the address, you wish to be able to parse the source code and provide a list of images on that page. Given that the parameter extensions may change, the list of images within that page may change too.

For instance pictures for the address with extension "?no=1701" would be different pictures for the address with extension "?no=1702".

Also, from my understanding of what you wrote above: If you add the address directly by: URL2 = "item.asp?no=1701" it works, but if you populate URL2 dynamically it's not working.

If you put these addresses into Internet Explorer directly, the pictures load and are displayed correctly. If you view the source from Internet Explorer, can you tell me if the picture locations are fully qualified paths to locations of the pictures on the remote host or are they relative to the path of the asp page you are viewing? i.e. do they appear as http://xx.com/images/image1.gif or ./images/image1.gif? If these picture files are relative, you will obviously need to add the address in your application. I suspect that they're fully qualified paths or it wouldn't work when you hard code the address. What I suspect is that the address isn't being populated correctly when you assign it dynamically.

URL2 = "http://xx.com/item.asp?no="& CurrentItem

Get the page for URL2... is this how you're loading URL2 or something?