473,623 Members | 3,366 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

screen scrape + login

n8
Hi,

Hi have to do the followign and have been racking my brain with
various solutions that have had no so great results.

I want to use the System.Net.WebC lient to submit data to a form (log a
user in) and then redirect to the correct article.

Here is the scenerio.
If you are not logged into the site for certain articles you are
redirected to a shtml login page. The login.shtml page posts to
another url for authentication and then lets you in. If have clicked
on an article that you have to log in to, then you are sent to the
login page with an appeneded URL,
http://www.domainname.com?orq:http:/..._2653091.shtml.
I have tried setting a webclient request to the url that the above
login form posts too, but I keep getting Method Not Allowed.

Any Ideas?
Nov 18 '05 #1
14 7877
more info required, but here is typical login

1) you request a page with webclient
2) you are returned a redirect header to the login page.
3) you code detects the login redirect, then post the required form data to
the login page (manually view the login page to get the form fields required
and method).

note: an asp.net login site requires that you actually do a get to the
login page to get valid viewstate to postback. other systems may also
require scaping of the get data to before doing the actual post.

4) a successful post to the login will return a cookie value you must send
on subsequent requests, and a redirect header to the originally requested
page.
-- bruce (sqlwork.com)

"n8" <na********@yah oo.com> wrote in message
news:6a******** *************** ***@posting.goo gle.com...
| Hi,
|
| Hi have to do the followign and have been racking my brain with
| various solutions that have had no so great results.
|
| I want to use the System.Net.WebC lient to submit data to a form (log a
| user in) and then redirect to the correct article.
|
| Here is the scenerio.
| If you are not logged into the site for certain articles you are
| redirected to a shtml login page. The login.shtml page posts to
| another url for authentication and then lets you in. If have clicked
| on an article that you have to log in to, then you are sent to the
| login page with an appeneded URL,
|
http://www.domainname.com?orq:http:/...a_2653091.shtm
l.
| I have tried setting a webclient request to the url that the above
| login form posts too, but I keep getting Method Not Allowed.
|
| Any Ideas?
Nov 18 '05 #2
I have an exampe of this here:

http://odetocode.com/Articles/162.aspx

It's basically posting to the login form, getting the cookie back, and
then making sure to send the cookie along when requesting the
protected content.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On 24 Nov 2004 13:55:23 -0800, na********@yaho o.com (n8) wrote:
Hi,

Hi have to do the followign and have been racking my brain with
various solutions that have had no so great results.

I want to use the System.Net.WebC lient to submit data to a form (log a
user in) and then redirect to the correct article.

Here is the scenerio.
If you are not logged into the site for certain articles you are
redirected to a shtml login page. The login.shtml page posts to
another url for authentication and then lets you in. If have clicked
on an article that you have to log in to, then you are sent to the
login page with an appeneded URL,
http://www.domainname.com?orq:http:/..._2653091.shtml.
I have tried setting a webclient request to the url that the above
login form posts too, but I keep getting Method Not Allowed.

Any Ideas?


Nov 18 '05 #3
Scott,
FYI - that was one of the best articles on the subject I ever read.
I was completely stuck on this issue about 6 months ago and I implemented it
straight away using the concepts you presented here.

Excellent work and explanation.
--
Joe Fallon

"Scott Allen" <bitmask@[nospam].fred.net> wrote in message
news:av******** *************** *********@4ax.c om...
I have an exampe of this here:

http://odetocode.com/Articles/162.aspx

It's basically posting to the login form, getting the cookie back, and
then making sure to send the cookie along when requesting the
protected content.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On 24 Nov 2004 13:55:23 -0800, na********@yaho o.com (n8) wrote:
Hi,

Hi have to do the followign and have been racking my brain with
various solutions that have had no so great results.

I want to use the System.Net.WebC lient to submit data to a form (log a
user in) and then redirect to the correct article.

Here is the scenerio.
If you are not logged into the site for certain articles you are
redirected to a shtml login page. The login.shtml page posts to
another url for authentication and then lets you in. If have clicked
on an article that you have to log in to, then you are sent to the
login page with an appeneded URL,
http://www.domainname.com?orq:http:/..._2653091.shtml.
I have tried setting a webclient request to the url that the above
login form posts too, but I keep getting Method Not Allowed.

Any Ideas?

Nov 18 '05 #4
Thanks, Joe. I appreciate the feedback.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Wed, 24 Nov 2004 20:48:24 -0500, "Joe Fallon"
<jf******@nospa mtwcny.rr.com> wrote:
Scott,
FYI - that was one of the best articles on the subject I ever read.
I was completely stuck on this issue about 6 months ago and I implemented it
straight away using the concepts you presented here.

Excellent work and explanation.


Nov 18 '05 #5
n8
Thanks for the example. I had seen your example earlier and had tried
it and always get to one particular point where I cannot seem to get
beyond. There are two hidden fields both called web.fixed_value s that
appear to be something like a view state but the page is shtml. I am
and have been able to pull down the site, etc. but everytime I try and
post my data (with or without the web.fixed_value s) I always get the
response Method Not Allowed. Below is the code I am using along with
the sire I am trying to access with my account. ANy further help on
this would be greatly appreciated.

private void Page_Load(objec t sender, System.EventArg s e)
{
string LOGIN_URL = "http://augustachronicl e.com/login.shtml";
string cookieAge = "31536000";

try
{
HttpWebRequest webRequest = WebRequest.Crea te(LOGIN_URL) as
HttpWebRequest;

StreamReader responseReader = new
StreamReader(we bRequest.GetRes ponse().GetResp onseStream());

string responseData = responseReader. ReadToEnd();
responseReader. Close();

// get the web fixed values
string fixedvalue1 = ExtractFixedVal ues1(responseDa ta);

string fixedvalue2 = ExtractFixedVal ues2(responseDa ta);

string postData = String.Format(" web.fixed_value s={0}&web.fixed _values={1}&ACT ION=Login&USER= {2}&PASS={3}&co okie_age={4}",f ixedvalue1,fixe dvalue2,userNam e,
password, cookieAge);

// have a cookie container ready to receive the forms auth cookie
CookieContainer cookies = new CookieContainer ();

// now post to the login form
webRequest = WebRequest.Crea te(LOGIN_URL) as HttpWebRequest;
webRequest.Meth od = "POST";
webRequest.Cont entType = "applicatio n/x-www-form-urlencoded";
webRequest.Cook ieContainer = cookies;

// write the form values into the request message
StreamWriter requestWriter = new
StreamWriter(we bRequest.GetReq uestStream());
requestWriter.W rite(postData);
requestWriter.C lose();

// we don't need the contents of the response, just the cookie it
issues
webRequest.GetR esponse().Close ();

// now we can send out cookie along with a request for the protected
page
webRequest = WebRequest.Crea te("http://augustachronicl e.com/stories/112404/usc_FBC--SpurrierProfile .shtml")
as HttpWebRequest;
webRequest.Cook ieContainer = cookies;
responseReader = new
StreamReader(we bRequest.GetRes ponse().GetResp onseStream());

// and read the response
responseData = responseReader. ReadToEnd();
responseReader. Close();

Response.Write( responseData);
}
catch (Exception ex)
{
Response.Write( ex.ToString());
}

}

private string ExtractFixedVal ues1(string s)
{
string viewStateNameDe limiter = "web.fixed_valu es";
string valueDelimiter = "value=\"";

int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
int viewStateValueP osition = s.IndexOf(
valueDelimiter, viewStateNamePo sition
);

int viewStateStartP osition = viewStateValueP osition +
valueDelimiter. Length;
int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);

return HttpUtility.Url EncodeUnicode(
s.Substring(vie wStateStartPosi tion,
viewStateEndPos ition - viewStateStartP osition
)
);
}
private string ExtractFixedVal ues2(string s)
{
string viewStateNameDe limiter = "web.fixed_valu es";
string valueDelimiter = "value=\"";

int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
int viewStateValueP osition = s.IndexOf(value Delimiter,
viewStateNamePo sition
);

int viewStateStartP osition = viewStateValueP osition +
valueDelimiter. Length;
int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);

string sTemp = s.Remove(0,view StateEndPositio n);

viewStateNamePo sition = sTemp.IndexOf(v iewStateNameDel imiter);
viewStateValueP osition = sTemp.IndexOf(
valueDelimiter, viewStateNamePo sition
);

viewStateStartP osition = viewStateValueP osition +
valueDelimiter. Length;
viewStateEndPos ition = sTemp.IndexOf(" \"", viewStateStartP osition);

return HttpUtility.Url EncodeUnicode(
sTemp.Substring (
viewStateStartP osition,
viewStateEndPos ition - viewStateStartP osition
)
);
}
Scott Allen <bitmask@[nospam].fred.net> wrote in message news:<k8******* *************** **********@4ax. com>...
Thanks, Joe. I appreciate the feedback.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Wed, 24 Nov 2004 20:48:24 -0500, "Joe Fallon"
<jf******@nospa mtwcny.rr.com> wrote:
Scott,
FYI - that was one of the best articles on the subject I ever read.
I was completely stuck on this issue about 6 months ago and I implemented it
straight away using the concepts you presented here.

Excellent work and explanation.

Nov 18 '05 #6
Everything looks like it is in order, Nathan. I'd examine the HTTP
traffic between your program and the server to make sure it all
matches exactly, even little things like the Agent header. I had one
financial site reject HttpWebRequests until I set the UserAgent
property to look just like IE. I guess it was a weak attempt at
preventing screen scraping programs.

--
Scott
http://www.OdeToCode.com/blogs/scott/

n 27 Nov 2004 12:39:42 -0800, na********@yaho o.com (n8) wrote:
Thanks for the example. I had seen your example earlier and had tried
it and always get to one particular point where I cannot seem to get
beyond. There are two hidden fields both called web.fixed_value s that
appear to be something like a view state but the page is shtml. I am
and have been able to pull down the site, etc. but everytime I try and
post my data (with or without the web.fixed_value s) I always get the
response Method Not Allowed. Below is the code I am using along with
the sire I am trying to access with my account. ANy further help on
this would be greatly appreciated.

private void Page_Load(objec t sender, System.EventArg s e)
{
string LOGIN_URL = "http://augustachronicl e.com/login.shtml";
string cookieAge = "31536000";

try
{
HttpWebReque st webRequest = WebRequest.Crea te(LOGIN_URL) as
HttpWebRequest ;

StreamReader responseReader = new
StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());

string responseData = responseReader. ReadToEnd();
responseReader .Close();

// get the web fixed values
string fixedvalue1 = ExtractFixedVal ues1(responseDa ta);

string fixedvalue2 = ExtractFixedVal ues2(responseDa ta);

string postData = String.Format(" web.fixed_value s={0}&web.fixed _values={1}&ACT ION=Login&USER= {2}&PASS={3}&co okie_age={4}",f ixedvalue1,fixe dvalue2,userNam e,
password, cookieAge);

// have a cookie container ready to receive the forms auth cookie
CookieContaine r cookies = new CookieContainer ();

// now post to the login form
webRequest = WebRequest.Crea te(LOGIN_URL) as HttpWebRequest;
webRequest.Met hod = "POST";
webRequest.Con tentType = "applicatio n/x-www-form-urlencoded";
webRequest.Coo kieContainer = cookies;

// write the form values into the request message
StreamWriter requestWriter = new
StreamWriter(w ebRequest.GetRe questStream());
requestWriter. Write(postData) ;
requestWriter. Close();

// we don't need the contents of the response, just the cookie it
issues
webRequest.Get Response().Clos e();

// now we can send out cookie along with a request for the protected
page
webRequest = WebRequest.Crea te("http://augustachronicl e.com/stories/112404/usc_FBC--SpurrierProfile .shtml")
as HttpWebRequest;
webRequest.Coo kieContainer = cookies;
responseRead er = new
StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());

// and read the response
responseData = responseReader. ReadToEnd();
responseReader .Close();

Response.Write (responseData);
}
catch (Exception ex)
{
Response.Write (ex.ToString()) ;
}

}

private string ExtractFixedVal ues1(string s)
{
string viewStateNameDe limiter = "web.fixed_valu es";
string valueDelimiter = "value=\"";

int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
int viewStateValueP osition = s.IndexOf(
valueDelimiter , viewStateNamePo sition
);

int viewStateStartP osition = viewStateValueP osition +
valueDelimiter .Length;
int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);

return HttpUtility.Url EncodeUnicode(
s.Substring(vi ewStateStartPos ition,
viewStateEndPos ition - viewStateStartP osition
)
);
}
private string ExtractFixedVal ues2(string s)
{
string viewStateNameDe limiter = "web.fixed_valu es";
string valueDelimiter = "value=\"";

int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
int viewStateValueP osition = s.IndexOf(value Delimiter,
viewStateNameP osition
);

int viewStateStartP osition = viewStateValueP osition +
valueDelimiter .Length;
int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);

string sTemp = s.Remove(0,view StateEndPositio n);

viewStateNameP osition = sTemp.IndexOf(v iewStateNameDel imiter);
viewStateValue Position = sTemp.IndexOf(
valueDelimiter , viewStateNamePo sition
);

viewStateStart Position = viewStateValueP osition +
valueDelimiter .Length;
viewStateEndPo sition = sTemp.IndexOf(" \"", viewStateStartP osition);

return HttpUtility.Url EncodeUnicode(
sTemp.Substrin g(
viewStateStart Position,
viewStateEndPo sition - viewStateStartP osition
)
);
}
Scott Allen <bitmask@[nospam].fred.net> wrote in message news:<k8******* *************** **********@4ax. com>...
Thanks, Joe. I appreciate the feedback.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Wed, 24 Nov 2004 20:48:24 -0500, "Joe Fallon"
<jf******@nospa mtwcny.rr.com> wrote:
>Scott,
>FYI - that was one of the best articles on the subject I ever read.
>I was completely stuck on this issue about 6 months ago and I implemented it
>straight away using the concepts you presented here.
>
>Excellent work and explanation.


Nov 18 '05 #7
n8
Scott,

Thanks for the information. I added a useragent to make it look like
IE, but I still get the 405 Method not allowed error message. What is
the best way to monitor the HTTP Traffic between my application and
the remote site? Are there any tools i can download to show me what
is going back and forth?

Thanks in advance,

n8

Scott Allen <bitmask@[nospam].fred.net> wrote in message news:<2b******* *************** **********@4ax. com>...
Everything looks like it is in order, Nathan. I'd examine the HTTP
traffic between your program and the server to make sure it all
matches exactly, even little things like the Agent header. I had one
financial site reject HttpWebRequests until I set the UserAgent
property to look just like IE. I guess it was a weak attempt at
preventing screen scraping programs.

--
Scott
http://www.OdeToCode.com/blogs/scott/

n 27 Nov 2004 12:39:42 -0800, na********@yaho o.com (n8) wrote:
Thanks for the example. I had seen your example earlier and had tried
it and always get to one particular point where I cannot seem to get
beyond. There are two hidden fields both called web.fixed_value s that
appear to be something like a view state but the page is shtml. I am
and have been able to pull down the site, etc. but everytime I try and
post my data (with or without the web.fixed_value s) I always get the
response Method Not Allowed. Below is the code I am using along with
the sire I am trying to access with my account. ANy further help on
this would be greatly appreciated.

private void Page_Load(objec t sender, System.EventArg s e)
{
string LOGIN_URL = "http://augustachronicl e.com/login.shtml";
string cookieAge = "31536000";

try
{
HttpWebReque st webRequest = WebRequest.Crea te(LOGIN_URL) as
HttpWebRequest ;

StreamReader responseReader = new
StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());

string responseData = responseReader. ReadToEnd();
responseReader .Close();

// get the web fixed values
string fixedvalue1 = ExtractFixedVal ues1(responseDa ta);

string fixedvalue2 = ExtractFixedVal ues2(responseDa ta);

string postData = String.Format(" web.fixed_value s={0}&web.fixed _values={1}&ACT ION=Login&USER= {2}&PASS={3}&co okie_age={4}",f ixedvalue1,fixe dvalue2,userNam e,
password, cookieAge);

// have a cookie container ready to receive the forms auth cookie
CookieContaine r cookies = new CookieContainer ();

// now post to the login form
webRequest = WebRequest.Crea te(LOGIN_URL) as HttpWebRequest;
webRequest.Met hod = "POST";
webRequest.Con tentType = "applicatio n/x-www-form-urlencoded";
webRequest.Coo kieContainer = cookies;

// write the form values into the request message
StreamWriter requestWriter = new
StreamWriter(w ebRequest.GetRe questStream());
requestWriter. Write(postData) ;
requestWriter. Close();

// we don't need the contents of the response, just the cookie it
issues
webRequest.Get Response().Clos e();

// now we can send out cookie along with a request for the protected
page
webRequest = WebRequest.Crea te("http://augustachronicl e.com/stories/112404/usc_FBC--SpurrierProfile .shtml")
as HttpWebRequest;
webRequest.Coo kieContainer = cookies;
responseRead er = new
StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());

// and read the response
responseData = responseReader. ReadToEnd();
responseReader .Close();

Response.Write (responseData);
}
catch (Exception ex)
{
Response.Write (ex.ToString()) ;
}

}

private string ExtractFixedVal ues1(string s)
{
string viewStateNameDe limiter = "web.fixed_valu es";
string valueDelimiter = "value=\"";

int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
int viewStateValueP osition = s.IndexOf(
valueDelimiter , viewStateNamePo sition
);

int viewStateStartP osition = viewStateValueP osition +
valueDelimiter .Length;
int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);

return HttpUtility.Url EncodeUnicode(
s.Substring(vi ewStateStartPos ition,
viewStateEndPos ition - viewStateStartP osition
)
);
}
private string ExtractFixedVal ues2(string s)
{
string viewStateNameDe limiter = "web.fixed_valu es";
string valueDelimiter = "value=\"";

int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
int viewStateValueP osition = s.IndexOf(value Delimiter,
viewStateNameP osition
);

int viewStateStartP osition = viewStateValueP osition +
valueDelimiter .Length;
int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);

string sTemp = s.Remove(0,view StateEndPositio n);

viewStateNameP osition = sTemp.IndexOf(v iewStateNameDel imiter);
viewStateValue Position = sTemp.IndexOf(
valueDelimiter , viewStateNamePo sition
);

viewStateStart Position = viewStateValueP osition +
valueDelimiter .Length;
viewStateEndPo sition = sTemp.IndexOf(" \"", viewStateStartP osition);

return HttpUtility.Url EncodeUnicode(
sTemp.Substrin g(
viewStateStart Position,
viewStateEndPo sition - viewStateStartP osition
)
);
}
Scott Allen <bitmask@[nospam].fred.net> wrote in message news:<k8******* *************** **********@4ax. com>...
Thanks, Joe. I appreciate the feedback.

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Wed, 24 Nov 2004 20:48:24 -0500, "Joe Fallon"
<jf******@nospa mtwcny.rr.com> wrote:

>Scott,
>FYI - that was one of the best articles on the subject I ever read.
>I was completely stuck on this issue about 6 months ago and I implemented it
>straight away using the concepts you presented here.
>
>Excellent work and explanation.

Nov 18 '05 #8
You might try a program called httplook. I think it is
http://www.httplook.com if not, google for it...

"n8" <na********@yah oo.com> wrote in message
news:6a******** *************** ***@posting.goo gle.com...
Scott,

Thanks for the information. I added a useragent to make it look like
IE, but I still get the 405 Method not allowed error message. What is
the best way to monitor the HTTP Traffic between my application and
the remote site? Are there any tools i can download to show me what
is going back and forth?

Thanks in advance,

n8

Scott Allen <bitmask@[nospam].fred.net> wrote in message
news:<2b******* *************** **********@4ax. com>...
Everything looks like it is in order, Nathan. I'd examine the HTTP
traffic between your program and the server to make sure it all
matches exactly, even little things like the Agent header. I had one
financial site reject HttpWebRequests until I set the UserAgent
property to look just like IE. I guess it was a weak attempt at
preventing screen scraping programs.

--
Scott
http://www.OdeToCode.com/blogs/scott/

n 27 Nov 2004 12:39:42 -0800, na********@yaho o.com (n8) wrote:
>Thanks for the example. I had seen your example earlier and had tried
>it and always get to one particular point where I cannot seem to get
>beyond. There are two hidden fields both called web.fixed_value s that
>appear to be something like a view state but the page is shtml. I am
>and have been able to pull down the site, etc. but everytime I try and
>post my data (with or without the web.fixed_value s) I always get the
>response Method Not Allowed. Below is the code I am using along with
>the sire I am trying to access with my account. ANy further help on
>this would be greatly appreciated.
>
>private void Page_Load(objec t sender, System.EventArg s e)
>{
>string LOGIN_URL = "http://augustachronicl e.com/login.shtml";
>string cookieAge = "31536000";
>
>try
>{
>HttpWebReque st webRequest = WebRequest.Crea te(LOGIN_URL) as
>HttpWebRequest ;
>
>StreamReader responseReader = new
>StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());
>
>string responseData = responseReader. ReadToEnd();
>responseReader .Close();
>
>// get the web fixed values
>string fixedvalue1 = ExtractFixedVal ues1(responseDa ta);
>
>string fixedvalue2 = ExtractFixedVal ues2(responseDa ta);
>
>string postData =
>String.Format( "web.fixed_valu es={0}&web.fixe d_values={1}&AC TION=Login&USER ={2}&PASS={3}&c ookie_age={4}", fixedvalue1,fix edvalue2,userNa me,
>password, cookieAge);
>
>// have a cookie container ready to receive the forms auth cookie
>CookieContaine r cookies = new CookieContainer ();
>
>// now post to the login form
>webRequest = WebRequest.Crea te(LOGIN_URL) as HttpWebRequest;
>webRequest.Met hod = "POST";
>webRequest.Con tentType = "applicatio n/x-www-form-urlencoded";
>webRequest.Coo kieContainer = cookies;
>
>// write the form values into the request message
>StreamWriter requestWriter = new
>StreamWriter(w ebRequest.GetRe questStream());
>requestWriter. Write(postData) ;
>requestWriter. Close();
>
>// we don't need the contents of the response, just the cookie it
>issues
>webRequest.Get Response().Clos e();
>
>// now we can send out cookie along with a request for the protected
>page
>webRequest =
>WebRequest.Cre ate("http://augustachronicl e.com/stories/112404/usc_FBC--SpurrierProfile .shtml")
>as HttpWebRequest;
>webRequest.Coo kieContainer = cookies;
>responseRead er = new
>StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());
>
>// and read the response
>responseData = responseReader. ReadToEnd();
>responseReader .Close();
>
>Response.Write (responseData);
>}
>catch (Exception ex)
>{
>Response.Write (ex.ToString()) ;
>}

>}
>
>private string ExtractFixedVal ues1(string s)
>{
>string viewStateNameDe limiter = "web.fixed_valu es";
>string valueDelimiter = "value=\"";
>
>int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
>int viewStateValueP osition = s.IndexOf(
>valueDelimiter , viewStateNamePo sition
>);
>
>int viewStateStartP osition = viewStateValueP osition +
>valueDelimiter .Length;
>int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);
>
>return HttpUtility.Url EncodeUnicode(
>s.Substring(vi ewStateStartPos ition,
> viewStateEndPos ition - viewStateStartP osition
>)
>);
>}
>
>
>private string ExtractFixedVal ues2(string s)
>{
>string viewStateNameDe limiter = "web.fixed_valu es";
>string valueDelimiter = "value=\"";
>
>int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
>int viewStateValueP osition = s.IndexOf(value Delimiter,
>viewStateNameP osition
> );
>
>int viewStateStartP osition = viewStateValueP osition +
>valueDelimiter .Length;
>int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);
>
>string sTemp = s.Remove(0,view StateEndPositio n);
>
>viewStateNameP osition = sTemp.IndexOf(v iewStateNameDel imiter);
>viewStateValue Position = sTemp.IndexOf(
>valueDelimiter , viewStateNamePo sition
>);
>
>viewStateStart Position = viewStateValueP osition +
>valueDelimiter .Length;
>viewStateEndPo sition = sTemp.IndexOf(" \"", viewStateStartP osition);
>
>return HttpUtility.Url EncodeUnicode(
>sTemp.Substrin g(
>viewStateStart Position,
>viewStateEndPo sition - viewStateStartP osition
>)
>);
>}
>
>
>Scott Allen <bitmask@[nospam].fred.net> wrote in message
>news:<k8****** *************** ***********@4ax .com>...
>> Thanks, Joe. I appreciate the feedback.
>>
>> --
>> Scott
>> http://www.OdeToCode.com/blogs/scott/
>>
>> On Wed, 24 Nov 2004 20:48:24 -0500, "Joe Fallon"
>> <jf******@nospa mtwcny.rr.com> wrote:
>>
>> >Scott,
>> >FYI - that was one of the best articles on the subject I ever read.
>> >I was completely stuck on this issue about 6 months ago and I
>> >implemented it
>> >straight away using the concepts you presented here.
>> >
>> >Excellent work and explanation.

Nov 18 '05 #9
Also, if you get a fix - please let us know.

"n8" <na********@yah oo.com> wrote in message
news:6a******** *************** ***@posting.goo gle.com...
Scott,

Thanks for the information. I added a useragent to make it look like
IE, but I still get the 405 Method not allowed error message. What is
the best way to monitor the HTTP Traffic between my application and
the remote site? Are there any tools i can download to show me what
is going back and forth?

Thanks in advance,

n8

Scott Allen <bitmask@[nospam].fred.net> wrote in message
news:<2b******* *************** **********@4ax. com>...
Everything looks like it is in order, Nathan. I'd examine the HTTP
traffic between your program and the server to make sure it all
matches exactly, even little things like the Agent header. I had one
financial site reject HttpWebRequests until I set the UserAgent
property to look just like IE. I guess it was a weak attempt at
preventing screen scraping programs.

--
Scott
http://www.OdeToCode.com/blogs/scott/

n 27 Nov 2004 12:39:42 -0800, na********@yaho o.com (n8) wrote:
>Thanks for the example. I had seen your example earlier and had tried
>it and always get to one particular point where I cannot seem to get
>beyond. There are two hidden fields both called web.fixed_value s that
>appear to be something like a view state but the page is shtml. I am
>and have been able to pull down the site, etc. but everytime I try and
>post my data (with or without the web.fixed_value s) I always get the
>response Method Not Allowed. Below is the code I am using along with
>the sire I am trying to access with my account. ANy further help on
>this would be greatly appreciated.
>
>private void Page_Load(objec t sender, System.EventArg s e)
>{
>string LOGIN_URL = "http://augustachronicl e.com/login.shtml";
>string cookieAge = "31536000";
>
>try
>{
>HttpWebReque st webRequest = WebRequest.Crea te(LOGIN_URL) as
>HttpWebRequest ;
>
>StreamReader responseReader = new
>StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());
>
>string responseData = responseReader. ReadToEnd();
>responseReader .Close();
>
>// get the web fixed values
>string fixedvalue1 = ExtractFixedVal ues1(responseDa ta);
>
>string fixedvalue2 = ExtractFixedVal ues2(responseDa ta);
>
>string postData =
>String.Format( "web.fixed_valu es={0}&web.fixe d_values={1}&AC TION=Login&USER ={2}&PASS={3}&c ookie_age={4}", fixedvalue1,fix edvalue2,userNa me,
>password, cookieAge);
>
>// have a cookie container ready to receive the forms auth cookie
>CookieContaine r cookies = new CookieContainer ();
>
>// now post to the login form
>webRequest = WebRequest.Crea te(LOGIN_URL) as HttpWebRequest;
>webRequest.Met hod = "POST";
>webRequest.Con tentType = "applicatio n/x-www-form-urlencoded";
>webRequest.Coo kieContainer = cookies;
>
>// write the form values into the request message
>StreamWriter requestWriter = new
>StreamWriter(w ebRequest.GetRe questStream());
>requestWriter. Write(postData) ;
>requestWriter. Close();
>
>// we don't need the contents of the response, just the cookie it
>issues
>webRequest.Get Response().Clos e();
>
>// now we can send out cookie along with a request for the protected
>page
>webRequest =
>WebRequest.Cre ate("http://augustachronicl e.com/stories/112404/usc_FBC--SpurrierProfile .shtml")
>as HttpWebRequest;
>webRequest.Coo kieContainer = cookies;
>responseRead er = new
>StreamReader(w ebRequest.GetRe sponse().GetRes ponseStream());
>
>// and read the response
>responseData = responseReader. ReadToEnd();
>responseReader .Close();
>
>Response.Write (responseData);
>}
>catch (Exception ex)
>{
>Response.Write (ex.ToString()) ;
>}

>}
>
>private string ExtractFixedVal ues1(string s)
>{
>string viewStateNameDe limiter = "web.fixed_valu es";
>string valueDelimiter = "value=\"";
>
>int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
>int viewStateValueP osition = s.IndexOf(
>valueDelimiter , viewStateNamePo sition
>);
>
>int viewStateStartP osition = viewStateValueP osition +
>valueDelimiter .Length;
>int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);
>
>return HttpUtility.Url EncodeUnicode(
>s.Substring(vi ewStateStartPos ition,
> viewStateEndPos ition - viewStateStartP osition
>)
>);
>}
>
>
>private string ExtractFixedVal ues2(string s)
>{
>string viewStateNameDe limiter = "web.fixed_valu es";
>string valueDelimiter = "value=\"";
>
>int viewStateNamePo sition = s.IndexOf(viewS tateNameDelimit er);
>int viewStateValueP osition = s.IndexOf(value Delimiter,
>viewStateNameP osition
> );
>
>int viewStateStartP osition = viewStateValueP osition +
>valueDelimiter .Length;
>int viewStateEndPos ition = s.IndexOf("\"", viewStateStartP osition);
>
>string sTemp = s.Remove(0,view StateEndPositio n);
>
>viewStateNameP osition = sTemp.IndexOf(v iewStateNameDel imiter);
>viewStateValue Position = sTemp.IndexOf(
>valueDelimiter , viewStateNamePo sition
>);
>
>viewStateStart Position = viewStateValueP osition +
>valueDelimiter .Length;
>viewStateEndPo sition = sTemp.IndexOf(" \"", viewStateStartP osition);
>
>return HttpUtility.Url EncodeUnicode(
>sTemp.Substrin g(
>viewStateStart Position,
>viewStateEndPo sition - viewStateStartP osition
>)
>);
>}
>
>
>Scott Allen <bitmask@[nospam].fred.net> wrote in message
>news:<k8****** *************** ***********@4ax .com>...
>> Thanks, Joe. I appreciate the feedback.
>>
>> --
>> Scott
>> http://www.OdeToCode.com/blogs/scott/
>>
>> On Wed, 24 Nov 2004 20:48:24 -0500, "Joe Fallon"
>> <jf******@nospa mtwcny.rr.com> wrote:
>>
>> >Scott,
>> >FYI - that was one of the best articles on the subject I ever read.
>> >I was completely stuck on this issue about 6 months ago and I
>> >implemented it
>> >straight away using the concepts you presented here.
>> >
>> >Excellent work and explanation.

Nov 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
1364
by: Jason Steeves | last post by:
I have one .aspx form that my users fill out and this then takes that information and populates a second .aspx form via session variables. I need to screen scrape the second .aspx form and e-mail that off. I have figured out how to do the screen scrape but it e-mails a blank form with none of the session variables set. Is there a way to screen scrapte the second .aspx form with the variables set from the first .aspx form? Thanks in...
3
3998
by: Ollie | last post by:
I know you can screen scrape a website using the System.Net.HttpWebResponse & System.Net.HttpWebRequest classes. But how do you screen scrape a secured website (https) that takes a username & password, I guess what I am asking where does the username & password go and where do you store any returned token for further requests... Cheers Ollie
2
3224
by: Rob Lauer | last post by:
I have written two completely separate web applications that cannot talk directly to one another (applications "A" and "B"). Application "A" has a form that takes some input (radio buttons, whatever) and when that form is submitted via a POST, I get back some data. Simple enough. The trick is that I need to write a web service on application "B" that will post data to that same form and then do a screen scrape of the results returned. I...
0
3645
by: Steve | last post by:
I am working on an application to screen scrape information from a web page. I have the base code working but the problem is I have to login before I can get the info I need. The page is hosted on my Router. When I go to the IP of the router I get the following page. <HTML> <head> <meta http-equiv="content-type" content="text/html;charset=iso-8859-1"> <title>Login</title> </head>
5
1653
by: crjunk | last post by:
I have a screen scrape page that allows the user to submit a url. When they hit submit, the page is returned back to them on my screen scrape page. Which computer actuall connects to the url to pull in the information -the user's computer or the webserver? Thanks, CR Junk
7
2160
by: Swanand Mokashi | last post by:
Hi all -- I would like to create an application(call it Application "A") that I would like to mimic exactly as a form on a foreign system (Application "F"). Application "F" is on the web (so basically I can not control it). I will have a form exactly on Application "A" as that of Application "F". Application "A" will submit to the url of the application "F". I would like to do a screen scrape of the confirmation obtained after submitting...
7
3550
by: ljr2600 | last post by:
Hello, I'm very new to python and still familiarizing myself with the language, sorry if the post seems moronic or simple. For a side project I'm working on I need to be able to scrape a modern computer desktop. Is there any basic material already available to do this? I'd rather not need to write my own to interact with hardware. Thanks!
3
4016
by: Gregory A Greenman | last post by:
I'm trying to screen scrape a site that requires a password. If I access the site's login page in my browser and view the source, I see that it does not contain a viewstate. When my program posts the login information, the response I get is the same page as if I had logged in using my browser. In the page it says "Welcome" followed by my name. The cookie collection returned doesn't contain any cookies (response.cookies.count = 0).
1
3513
by: newdev | last post by:
Hi All, Can somebody maybe please help me? - how do i screen scrape data from a dos application / window to .net application by using c#? - how do i screen scrape data from a dos application / window to sql database? - how do i screen scrape data from a dos application / window to sql database and insert other data back to the dos application / window? Thanks
0
8662
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8603
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8317
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8463
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
4067
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4154
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2593
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1769
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
2
1468
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.