How to retrieve data from a website

[rinu mariam johnson]

I want to retrieve data from a website based on a file number.But i have to do this programmatically using vb.net and finally have to store the results in a csv file.
In my web application,i will give the file number in a textbox(through GUI) and the site address(through vb.net code) and when i click the search button,the csv output should be obtained.
I am fed up with this problem for the last 2 weeks..can anybody pls help me in this situation.

Thanks & Regards,
rinu

[anijos]

You can use a datatable to save the data first and later convert it into .csv format.

[rinu mariam johnson]

hi anijos,
Thanx for ur quick reply..but i didnt get the first part itself..i.e,i am not able to access the site through vb.net code(i dont want to open the site in my page)and also i am not getting the search results of the particular file no. in my page

Hope u got it..pls help me

[anijos]

Rinu,

what do u mean by serach result for file no? i dont understand that part.. can you explain a bit more?

[rinu mariam johnson]

hi anijos,
surely..i'll explain that in detail..my intention is to give one traffic file no(e.g.vehiclenumber)in one textbox in my webapplication...on clicking search button it should give the same results as the results from the website(in the third party website...there is a provision for entering the file no..and on search..it will display the results in that site )

I dont want to take the website each and everytime,instead,i'm giving the website address in my vb.net code..and based on the file no. which i'm changing in the textbox,the output should vary..

Did u get it?

[anijos]

Hi Rinu,

I still dont under stand your application. you said something about a third party website rt? can u send me the url for that? may be that will allow me to understand your problem better.

what i've understood till now is,

In a web site when you give a parameter like vehicle no and search, you are getting the details. you want to implement that locally. is my understanding correct?

Thanks
AniJos

[rinu mariam johnson]

Hi anijos
wat u understood is right.yes,i hav to get search results based on the input vehicle no(file no) and finally i hav to store the results in a csv file(i want to do this programmatically in my web applicationusing vb.net or c#).
Pls pardon me for not exposing the url bcos i dont hav the permission for that.Hope u understood my situation.

If any idea is there,pls help me.
regards
rinu

[Frinavale]

Are you having problems retrieving the information?
Or are you having problems storing the information retrieved into a CVS file?

-Frinny

[rinu mariam johnson]

Hi,
I didnt get the first part itself..(retrieving the information from the website)
I tried a lot with datascraping code,but it didnt work.
I got the html of the website,after that, am not able to get the input tag of the concerned textbox(using getElementsByTagName)
i'm stuck from that onwards.
hope u guys can help me

regards
rinu

[Frinavale]

Go to the website and look at the HTML code to find out what the name of the input field is.

You're going to have to use the HttpWebRequest to request the page's content (using the GetResponse() method). From there you'll have to parse through the HTML (returned to you in a HttpWebResponse object) and find the input field.

Please look into how to retrieve web content and investigate how to parse that information before continuing to attempt to write anything to a CVS file.

Taking smaller steps will keep you from getting overwhelmed by the problem...and it will make it easier for us to answer your questions.

-Frinny

[rinu mariam johnson]

Hi,
I used HttpWebRequest to request the content of the website and is returned in a HttpWebResponse object.
I got the html of the website,but my problem is,i'm not able to parse through the html of the website(not getting the match of the concerned textbox' input tag).

This code is working properly for website address like "http://www.abc.com"
but can you give me solution for such like (http://www.abc.com/index.php?name=abc&id=123) for user authentication as u give? how can I get the contents from such types of pages.. (I am not able to expose the exact website address..www.abc.com is just given for example)

I am attaching the code :

 Dim str As String = ""
        Dim uri As New Uri("http://www.abc.com/index.php?name=abc&id=123")
        Dim request As HttpWebRequest = DirectCast(WebRequest.Create(uri.AbsoluteUri), HttpWebRequest)
        request.AllowAutoRedirect = True
        request.MaximumAutomaticRedirections = 5
        request.KeepAlive = True
        request.Timeout = &H7530
 
        Dim response As HttpWebResponse = Nothing
        Try
           response = DirectCast(request.GetResponse, HttpWebResponse)
        Catch exception1 As WebException
            literal1.Text = exception1.Message
        End Try
 
 
        Dim reader As StreamReader = Nothing
        Try
            Dim encoding As New UTF8Encoding
            reader = New StreamReader(response.GetResponseStream, encoding)
            str = reader.ReadToEnd
            literal1.Text = str
        Catch exception As Exception
            literal1.Text = exception.Message
        End Try
 

Thanx
Rinu

[anijos]

Try

request.Credentials = new NetworkCredential(username, password);

AniJos

[Frinavale]

@anijos
:) This has already been covered, from what I understand NetworkCredentials don't work with Forms Authentication.

[rinu mariam johnson]

Hi..
I am still fed up with the problem,as I said earlier,i got the html of the website,but i'm not able to get the match for that input tag from this html,I tried the code below:

HTMLDoc = wbBrowser.Document
 
        Dim iHTMLCol As IHTMLElementCollection
        Dim iHTMLEle As IHTMLElement
       ''HTMLDoc.focus()
 
        iHTMLCol = HTMLDoc.getElementsByTagName("input")        ' Type the Traffic file no in  the TrfNo text box  
 
'(when i am trying to get HTMLDoc.getElementsByTagName("input").length..it is returning 0,eventhough so many input tags r there)
        For Each iHTMLEle In iHTMLCol
            If Not iHTMLEle.getAttribute("name") Is Nothing Then                             str = iHTMLEle.getAttribute("name").ToString
                If str = "TrfNo" Then 
                    iHTMLEle.setAttribute("value", 1234)
                End If
                Exit For
            End If
        Next         
 
 
' Press the submit button
 
         For Each iHTMLEle In iHTMLCol
            If Not iHTMLEle.getAttribute("name") Is Nothing Then
 
               If iHTMLEle.outerHTML = "<INPUT type=SUBMIT" & _
                                        "name=btnG" & _
                                  "value=Search> " Then
                   'document.forms[0].submitBttn.click()
                   iHTMLEle.click()
                 Exit For
              End If
          End If
        Next
 
       Do
        Loop Until Not wbBrowser.Busy

[Ramk]

Plz use [code] tags for when you are posting code.

Try using this:

 
HtmlDocument.GetFilteredHtmlTags()