submit form via HttpWebRequest or WebClient

[John A Grandy]

has anyone successfully used HttpWebRequest or WebClient class to simulate
submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a button.
when the button is clicked the form is submitted with the textbox contents.

could you please post some sample code? thanks.

[Joerg Jooss]

John A Grandy wrote:

has anyone successfully used HttpWebRequest or WebClient class to
simulate submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a
button. when the button is clicked the form is submitted with the
textbox contents.

could you please post some sample code? thanks.

That's a full blown WebRequest/WebResponse sample including URL encoding.
public void PostForm(string url, string formData, string encodingName) {
// URL encode the post data, applying the character encoding given by
// encodingName.
MemoryStream content = new MemoryStream(fo rmData.Length * 2);
Encoding encoding = Encoding.GetEnc oding(encodingN ame);
string[] keyValuePairs = formData.Split( '&', '=');
int numberOfPairs = keyValuePairs.L ength;
bool isKey = true;
foreach (string keyValue in keyValuePairs) {
byte[] bytes = HttpUtility.Url EncodeToBytes(k eyValue, encoding);
content.Write(b ytes, 0, bytes.Length);
if (isKey) {
content.WriteBy te((byte) '=');
}
else {
content.WriteBy te((byte) '&');
}
isKey = !isKey;
}
// We'll end up with one surplus '&' -- shouldn't hurt, but we cut it
anyway
content.SetLeng th(content.Leng th - 1);

// That's the actual HTTP part of the sample
HttpWebRequest request = (HttpWebRequest ) WebRequest.Crea te(url);
request.Method = "POST";
request.Content Type = String.Format(
"applicatio n/x-www-form-urlencoded; charset={0}", encodingName);
request.Content Length = content.Length;
using (Stream requestStream = request.GetRequ estStream()) {
content.WriteTo (requestStream) ;
}

HttpWebResponse response = (HttpWebRespons e) request.GetResp onse();
using (Stream responseStream = response.GetRes ponseStream()) {
byte[] buffer = new byte[response.Conten tLength];
responseStream. Read(buffer, 0, buffer.Length);
// Process buffer... e.g. write to file, decode to string, etc.
}
}

Cheers,

--
Joerg Jooss
jo*********@gmx .net

[Scott Allen]

Hi John:

I have an example here:

Screen Scraping, ViewState, and Authentication using ASP.Net
http://odetocode.com/Articles/162.aspx

This also shows how to extract the viewstate out of an asp.net page to
post back.

HTH,

--
Scott
http://www.OdeToCode.com

On Mon, 23 Aug 2004 23:23:31 -0700, "John A Grandy"
<johnagrandy-at-yahoo.com> wrote:

has anyone successfully used HttpWebRequest or WebClient class to simulate
submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a button.
when the button is clicked the form is submitted with the textbox contents.

could you please post some sample code? thanks.

[John A Grandy]

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textb ox1=thedata"

but how do i know what encodingName should be ?
"Joerg Jooss" <jo*********@gm x.net> wrote in message
news:ua******** ******@tk2msftn gp13.phx.gbl...

John A Grandy wrote:

has anyone successfully used HttpWebRequest or WebClient class to
simulate submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a
button. when the button is clicked the form is submitted with the
textbox contents.

could you please post some sample code? thanks.

That's a full blown WebRequest/WebResponse sample including URL encoding.
public void PostForm(string url, string formData, string encodingName) {
// URL encode the post data, applying the character encoding given by
// encodingName.
MemoryStream content = new MemoryStream(fo rmData.Length * 2);
Encoding encoding = Encoding.GetEnc oding(encodingN ame);
string[] keyValuePairs = formData.Split( '&', '=');
int numberOfPairs = keyValuePairs.L ength;
bool isKey = true;
foreach (string keyValue in keyValuePairs) {
byte[] bytes = HttpUtility.Url EncodeToBytes(k eyValue, encoding);
content.Write(b ytes, 0, bytes.Length);
if (isKey) {
content.WriteBy te((byte) '=');
}
else {
content.WriteBy te((byte) '&');
}
isKey = !isKey;
}
// We'll end up with one surplus '&' -- shouldn't hurt, but we cut it
anyway
content.SetLeng th(content.Leng th - 1);

// That's the actual HTTP part of the sample
HttpWebRequest request = (HttpWebRequest ) WebRequest.Crea te(url);
request.Method = "POST";
request.Content Type = String.Format(
"applicatio n/x-www-form-urlencoded; charset={0}", encodingName);
request.Content Length = content.Length;
using (Stream requestStream = request.GetRequ estStream()) {
content.WriteTo (requestStream) ;
}

HttpWebResponse response = (HttpWebRespons e) request.GetResp onse();
using (Stream responseStream = response.GetRes ponseStream()) {
byte[] buffer = new byte[response.Conten tLength];
responseStream. Read(buffer, 0, buffer.Length);
// Process buffer... e.g. write to file, decode to string, etc.
}
}

Cheers,

--
Joerg Jooss
jo*********@gmx .net

[Joerg Jooss]

John A Grandy wrote:

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textb ox1=thedata"
Right.
but how do i know what encodingName should be ?

That's the tricky part, because this depends on the web application. But
there's a simple heuristic that should work for most sites. Use the same
encoding that the site uses as character encoding. Open the web page and
check out what character encoding your browser applies (in IE
View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western Europe
(Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->

Unless you're working with a non US or non Western European web site, those
are the ones you'll most likely encounter. In case of doubt, use ISO-8859-1.

Cheers,

--
Joerg Jooss
jo*********@gmx .net

[John A Grandy]

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website.

i'm communicating over vpn with a private web-server meant for
data-provision to intermediaries.

the web-server does not validate credentials. it just immediately returns
data (interesting, as an xml doc)

the only working tool i have is a java applet written by someone else
(someone who is long gone).

the java applet is very simple. just a textbox and a submit button.

a long alphanumeric string is entered into the textbox (it ends with "^").

clicking the button fires a js function that calls form.submit and the form
is posted.

"view source" on the IE browseer hosting the java applet reveals only a very
simple block of html

a <textarea> element is used for data-entry.

i can't disclose the actual url -- but its similar to
"https://www.somecompany .com:5555/someservice.asm x/somefunction"

.... so apparently the web-server is hosting a .net web-service.
i tried your method. i constructed using the data-string successfully used
in the Java applet :

postData = "myTextArea=dat aString";

encodingName = "utf-8";

request.GetResp onse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you think
good possibilities are ?

or could the problem be in some other area ?

"Joerg Jooss" <jo*********@gm x.net> wrote in message
news:en******** ******@TK2MSFTN GP09.phx.gbl...

John A Grandy wrote:

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textb ox1=thedata"
Right.

but how do i know what encodingName should be ?

That's the tricky part, because this depends on the web application. But
there's a simple heuristic that should work for most sites. Use the same
encoding that the site uses as character encoding. Open the web page and
check out what character encoding your browser applies (in IE
View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western

Europe (Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->

Unless you're working with a non US or non Western European web site, those are the ones you'll most likely encounter. In case of doubt, use ISO-8859-1.
Cheers,

--
Joerg Jooss
jo*********@gmx .net

[John A Grandy]

also, the MSDN docs for Encoding.GetEnc oding(name) do not list "ISO-8859-1"

"Joerg Jooss" <jo*********@gm x.net> wrote in message
news:en******** ******@TK2MSFTN GP09.phx.gbl...

John A Grandy wrote:

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textb ox1=thedata"
Right.

but how do i know what encodingName should be ?

That's the tricky part, because this depends on the web application. But
there's a simple heuristic that should work for most sites. Use the same
encoding that the site uses as character encoding. Open the web page and
check out what character encoding your browser applies (in IE
View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western

Europe (Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->

Unless you're working with a non US or non Western European web site, those are the ones you'll most likely encounter. In case of doubt, use ISO-8859-1.
Cheers,

--
Joerg Jooss
jo*********@gmx .net

[Joerg Jooss]

John A Grandy wrote:

also, the MSDN docs for Encoding.GetEnc oding(name) do not list
"ISO-8859-1"

They just show a few names in the .NET docs. The list of all supported
encodings is hidden deeply within MSDN, and of course it's at some place
you'd never expect it to be ;-)

Start here:

http://www.microsoft.com/globaldev/g..._codepage.mspx

Cheers,

--
Joerg Jooss
jo*********@gmx .net

[Joerg Jooss]

John A Grandy wrote:

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website. [...] i can't disclose the actual url -- but its similar to
"https://www.somecompany .com:5555/someservice.asm x/somefunction"

... so apparently the web-server is hosting a .net web-service.
Yes.
i tried your method. i constructed using the data-string
successfully used in the Java applet :

postData = "myTextArea=dat aString";

encodingName = "utf-8";

request.GetResp onse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you
think good possibilities are ?

or could the problem be in some other area ?

I guess the whole approrach could be wrong then. My sample code showed a
programmatic web form submission, but you probably need a SOAP client.
Actually, if it is a "proper" Web Service, VS .NET should be able to
generate the necessary client-side code for you. The test code you've posted
doesn't fit into the picture at all, though.

I guess you'll need to talk to the Web Service guys to find out what
requests their web service can handle.

Cheers,

--
Joerg Jooss
jo*********@gmx .net

[John A Grandy]

Hi Joerg ....

I don't think the approach you suggested is wrong.

I have a simple HTML page that submits its form to the web-server, and
successfully receives a response. The HTML form odes not communicate in any
way with any entity other than the following:

The JavaScript function triggered by the button click is very very simple :
all is does is a form.submit. There is no other client-side functionality
occurring.

Whatever method this company is using to process information on their side ,
they do accept posting of a simple form with

"InputData=0123 456789abcdefghi jklmnopqrstuvwx yz^" (for example)

Is there any problem with URL encoding the "^" character?

I think the encoding-type is probably the villian. I'll have to try various
possibilities.

Also, do you know various common headers that could be added? I see in
many of the postings to this newsgroup that quite a different number of
headers are being used.

"Joerg Jooss" <jo*********@gm x.net> wrote in message
news:uf******** ******@TK2MSFTN GP11.phx.gbl...

John A Grandy wrote:

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website. [...]

i can't disclose the actual url -- but its similar to
"https://www.somecompany .com:5555/someservice.asm x/somefunction"

... so apparently the web-server is hosting a .net web-service.

Yes.

i tried your method. i constructed using the data-string
successfully used in the Java applet :

postData = "myTextArea=dat aString";

encodingName = "utf-8";

request.GetResp onse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you
think good possibilities are ?

or could the problem be in some other area ?

I guess the whole approrach could be wrong then. My sample code showed a
programmatic web form submission, but you probably need a SOAP client.
Actually, if it is a "proper" Web Service, VS .NET should be able to
generate the necessary client-side code for you. The test code you've

posted doesn't fit into the picture at all, though.

I guess you'll need to talk to the Web Service guys to find out what
requests their web service can handle.

Cheers,

--
Joerg Jooss
jo*********@gmx .net