VB.NET - Passing the login screen

[martin.lanny]

I need to grab a html source code from the second page after I hit this
login screen with inserted variables:

example:
https://www.mypage.com/login.php?use...assword=joepsw

will lead to:

https://www.mypage.com/hello.php (this is the page I need a source
code from)
, it cannot be hit directly....

So, just to explain it more... If I hit the first url in browser it
logs me into another page and that is the page I need to get the html
source from.

I wrote this code:

Dim finalurl As String =
"https://www.mypage.com/login.php?username=joe&password=joepsw"

Dim webreq As System.Net.WebRequest =
System.Net.WebRequest.Create(finalurl)

Dim webres As System.Net.WebResponse = webreq.GetResponse()
Dim enc As System.Text.Encoding = _
System.Text.Encoding.GetEncoding("latin1")
Dim st As System.IO.Stream = webres.GetResponseStream()
Dim sr As New System.IO.StreamReader(st, enc)
Dim html As String = sr.ReadToEnd()

RichTextBox1.Text = html
sr.Close()

So it should log me in and get the code, but this is not happening.
It will get the source code of the first login page, not the second
one.

Why is it happening. Is it too fast? Is it not posting the variables
for username and password?

If that is the case, how can I first post the username and password to
php page and then grab a resulting source code?

Thanks a lot.

Martin

[martin.lanny]

So I tried to play with it little more and wrote this....


Try

Dim req As System.Net.HttpWebRequest
Dim res As System.Net.HttpWebResponse
Dim sr As System.IO.StreamReader
Dim s As System.IO.Stream
Dim b As Byte()

Dim myPostData As String = "username=joe&password=joepsw"
Dim myPostURL As String =
"http://www.mywebsite.com/login.php"

req = System.Net.WebRequest.Create(myPostURL)

b = System.Text.Encoding.ASCII.GetBytes(myPostData)

req.Method = "post"
req.ContentType = "application/x-www-form-urlencoded"
req.ContentLength = b.Length

'send the data
s = req.GetRequestStream()
s.Write(b, 0, b.Length)
s.Close()
res = req.GetResponse()

sr = New System.IO.StreamReader(res.GetResponseStream())
RichTextBox1.Text = sr.ReadToEnd()

sr.Close()
res.Close()

Catch ex As Exception
MessageBox.Show(ex.ToString)

End Try

This doesn't work either.
I can't get pass by first screen, it always shows me (in richtextbox)
the source code of the login screen.

Can someone help me, how I can get the code from the second page???

Martin.

[martin.lanny]

Can anyone help, please?

[Neil]

Martin,

I had to do something similar, and found the following
tool invaluable - ieHTTPHeaders from www.blunck.info. It
lets you see exactly what is being sent to the server.

I also found that after two https requests, no matter how
I changed the code, the CLR would throw an exception. so I
used the IPWorks controls. They're pretty good, although I
thought the help was a bit minimal. (MethodX - performs
Method X. That sort of utterly useless stuff.) They had
good simple support for certificates, which, being simple
myself, was very handy :-)

Neil

-----Original Message-----
So I tried to play with it little more and wrote this....


Try

Dim req As System.Net.HttpWebRequest
Dim res As System.Net.HttpWebResponse
Dim sr As System.IO.StreamReader
Dim s As System.IO.Stream
Dim b As Byte()

Dim myPostData As String = "username=joe&password=joepsw" Dim myPostURL As String =
"http://www.mywebsite.com/login.php"

req = System.Net.WebRequest.Create(myPostURL)

b = System.Text.Encoding.ASCII.GetBytes (myPostData)
req.Method = "post"
req.ContentType = "application/x-www-form- urlencoded" req.ContentLength = b.Length

'send the data
s = req.GetRequestStream()
s.Write(b, 0, b.Length)
s.Close()
res = req.GetResponse()

sr = New System.IO.StreamReader (res.GetResponseStream()) RichTextBox1.Text = sr.ReadToEnd()

sr.Close()
res.Close()

Catch ex As Exception
MessageBox.Show(ex.ToString)

End Try

This doesn't work either.
I can't get pass by first screen, it always shows me (in richtextbox)the source code of the login screen.

Can someone help me, how I can get the code from the second page???
Martin.

.

[martin.lanny]

Neil,
how would this solve my problem?
martin

[Neil]

Martin,

My suspicion would be that you are not sending the correct
information to the server, and so you are getting returned
to the logon screen.

I presumed that you must have tried to login using
internet explorer, and that worked. If you try again via
IE using the toolbar I recommended, then it will tell you
exactly what IE is sending to the server. Worth checking,
as it's freeware.

As for IPWorks - it doesn't crash - that's how it'll
help ;-)

Neil

-----Original Message-----
Neil,
how would this solve my problem?
martin

.

[martin.lanny]

This is what is shows for headers... Makes no sense to me.

request headers:
Connection: Keep-Alive
Host: www.mywebsite.com

Response headers:
Set-Cookie:SubId=; Expires=Tue, 31-May-05 14:34:18 GMT;
Path=/,SourceId=; Expires=Tue, 31-May-05 14:34:18 GMT; Path=/,ClickId=;
Expires=Tue, 31-May-05 14:34:18 GMT; Path=/,I=; Expires=Tue, 01-Mar-05
14:34:18 GMT; Path=/,S=s=bo; Domain=.mywebsite.com; Path=/
Content-Type: text/html; charset=UTF-8
Cache-control: private
Transfer-Encoding: chunked
Date: Wed, 02 Mar 2005 14:34:18 GMT

[Neil]

Mmmm - I would have expected to see a GET statement to
start. Forgive me if I'm being patronising, but it hadn't
scrolled past the beginning had it.

here's a sample of what I get if I go to
https://www.microsoft.com

GET / HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, application/x-shockwave-flash,
application/vnd.ms-excel, application/vnd.ms-powerpoint,
application/msword, */*
Accept-Language: en-gb
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)
Host: www.microsoft.com
Connection: Keep-Alive
Cookie:
MC1=GUID=5c1fef65e083a54598c659daddbdfbfe&HASH=65e f&LV=2004
7&V=3

do you get the same thing for the ms website>?

Neil

-----Original Message-----
This is what is shows for headers... Makes no sense to me.

request headers:
Connection: Keep-Alive
Host: www.mywebsite.com

Response headers:
Set-Cookie:SubId=; Expires=Tue, 31-May-05 14:34:18 GMT;
Path=/,SourceId=; Expires=Tue, 31-May-05 14:34:18 GMT; Path=/,ClickId=;Expires=Tue, 31-May-05 14:34:18 GMT; Path=/,I=; Expires=Tue, 01-Mar-0514:34:18 GMT; Path=/,S=s=bo; Domain=.mywebsite.com; Path=/
Content-Type: text/html; charset=UTF-8
Cache-control: private
Transfer-Encoding: chunked
Date: Wed, 02 Mar 2005 14:34:18 GMT

.

[MARTIN LANNY]

Neil,
this never worked...and I found it might not be possible to get it to
work.

Now I am trying to do this through axwebbrowser, problem is here:

http://groups-beta.google.com/group/...9445b44100a298

Can you look at it.

Martin

[Neil]

Martin,

I'm sorry - I can't help with that control. If you have
the money to spend, I would really recommend the IPWorks
controls. I wrote something using them to log on to a site
and download some reports which we run on a nightly basis
with no problems. I suspect you are trying to do something
similar? They have a demo version, so it might be worth
you trying. BTW - I don't work for them!

I understand your frustration. I would have thought the
internet-based portions of .Net would have been tested and
work like a charm.
Neil

-----Original Message-----
Neil,
this never worked...and I found it might not be possible to get it towork.

Now I am trying to do this through axwebbrowser, problem is here:
http://groups- beta.google.com/group/microsoft.public.dotnet.languages.vb/
browse_frm/thread/1d517619b21fbf7d/769445b44100a298
Can you look at it.

Martin

.

[MARTIN LANNY]

This problem was all because no one could get axwebbrowser to refresh
properly.
I found this function:

Public Function GetPageHTML(ByVal URL As String, Optional ByVal
TimeoutSeconds As Integer = 10) As String
' Retrieves the HTML from the specified URL,
' using a default timeout of 10 seconds
Dim objRequest As Net.WebRequest
Dim objResponse As Net.WebResponse
Dim objStreamReceive As System.IO.Stream
Dim objEncoding As System.Text.Encoding
Dim objStreamRead As System.IO.StreamReader

Try
' Setup our Web request
objRequest = Net.WebRequest.Create(URL)
objRequest.Timeout = TimeoutSeconds * 1000
' Retrieve data from request
objResponse = objRequest.GetResponse
objStreamReceive = objResponse.GetResponseStream
objEncoding = System.Text.Encoding.GetEncoding( _
"utf-8")
objStreamRead = New System.IO.StreamReader( _
objStreamReceive, objEncoding)
' Set function return value
GetPageHTML = objStreamRead.ReadToEnd()
' Check if available, then close response
If Not objResponse Is Nothing Then
objResponse.Close()
End If
Catch
' Error occured grabbing data, simply return nothing
Return ""
End Try
End Function

To grab html code from the page and it is always a new and refreshed
page.
So this solved my troubles and because I do not have to use
axwebbrowser anymore, my application went down in size by 7 mb.
Amazing ...
Anyway, thanks for trying to help me.

Martin

[Neil]

No problem.

It's late on a Friday afternoon, so you'll have to forgive
me, but how is the new code fundamentally different from
what you had?

Neil

-----Original Message-----
This problem was all because no one could get axwebbrowser to refreshproperly.
I found this function:

Public Function GetPageHTML(ByVal URL As String, Optional ByValTimeoutSeconds As Integer = 10) As String
' Retrieves the HTML from the specified URL,
' using a default timeout of 10 seconds
Dim objRequest As Net.WebRequest
Dim objResponse As Net.WebResponse
Dim objStreamReceive As System.IO.Stream
Dim objEncoding As System.Text.Encoding
Dim objStreamRead As System.IO.StreamReader

Try
' Setup our Web request
objRequest = Net.WebRequest.Create(URL)
objRequest.Timeout = TimeoutSeconds * 1000
' Retrieve data from request
objResponse = objRequest.GetResponse
objStreamReceive = objResponse.GetResponseStream objEncoding = System.Text.Encoding.GetEncoding ( _ "utf-8")
objStreamRead = New System.IO.StreamReader( _
objStreamReceive, objEncoding)
' Set function return value
GetPageHTML = objStreamRead.ReadToEnd()
' Check if available, then close response
If Not objResponse Is Nothing Then
objResponse.Close()
End If
Catch
' Error occured grabbing data, simply return nothing Return ""
End Try
End Function

To grab html code from the page and it is always a new and refreshedpage.
So this solved my troubles and because I do not have to useaxwebbrowser anymore, my application went down in size by 7 mb.Amazing ...
Anyway, thanks for trying to help me.

Martin

.

[MARTIN LANNY]

Yeah it is, but that was the only way to get it to work.