submit form via HttpWebRequest or WebClient

[John A Grandy]

has anyone successfully used HttpWebRequest or WebClient class to simulate
submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a button.
when the button is clicked the form is submitted with the textbox contents.

could you please post some sample code? thanks.

[Joerg Jooss]

John A Grandy wrote:

has anyone successfully used HttpWebRequest or WebClient class to
simulate submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a
button. when the button is clicked the form is submitted with the
textbox contents.

could you please post some sample code? thanks.

That's a full blown WebRequest/WebResponse sample including URL encoding.
public void PostForm(string url, string formData, string encodingName) {
// URL encode the post data, applying the character encoding given by
// encodingName.
MemoryStream content = new MemoryStream(formData.Length * 2);
Encoding encoding = Encoding.GetEncoding(encodingName);
string[] keyValuePairs = formData.Split('&', '=');
int numberOfPairs = keyValuePairs.Length;
bool isKey = true;
foreach (string keyValue in keyValuePairs) {
byte[] bytes = HttpUtility.UrlEncodeToBytes(keyValue, encoding);
content.Write(bytes, 0, bytes.Length);
if (isKey) {
content.WriteByte((byte) '=');
}
else {
content.WriteByte((byte) '&');
}
isKey = !isKey;
}
// We'll end up with one surplus '&' -- shouldn't hurt, but we cut it
anyway
content.SetLength(content.Length - 1);

// That's the actual HTTP part of the sample
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
request.ContentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encodingName);
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream()) {
content.WriteTo(requestStream);
}

HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream()) {
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// Process buffer... e.g. write to file, decode to string, etc.
}
}

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[Scott Allen]

Hi John:

I have an example here:

Screen Scraping, ViewState, and Authentication using ASP.Net
http://odetocode.com/Articles/162.aspx

This also shows how to extract the viewstate out of an asp.net page to
post back.

HTH,

--
Scott
http://www.OdeToCode.com

On Mon, 23 Aug 2004 23:23:31 -0700, "John A Grandy"
<johnagrandy-at-yahoo.com> wrote:

has anyone successfully used HttpWebRequest or WebClient class to simulate
submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a button.
when the button is clicked the form is submitted with the textbox contents.

could you please post some sample code? thanks.

[John A Grandy]

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textbox1=thedata"

but how do i know what encodingName should be ?
"Joerg Jooss" <jo*********@gmx.net> wrote in message
news:ua**************@tk2msftngp13.phx.gbl...

John A Grandy wrote:

has anyone successfully used HttpWebRequest or WebClient class to
simulate submission of a simple HTML form?

for example: a very simple plain-vanilla form with a textbox and a
button. when the button is clicked the form is submitted with the
textbox contents.

could you please post some sample code? thanks.

That's a full blown WebRequest/WebResponse sample including URL encoding.
public void PostForm(string url, string formData, string encodingName) {
// URL encode the post data, applying the character encoding given by
// encodingName.
MemoryStream content = new MemoryStream(formData.Length * 2);
Encoding encoding = Encoding.GetEncoding(encodingName);
string[] keyValuePairs = formData.Split('&', '=');
int numberOfPairs = keyValuePairs.Length;
bool isKey = true;
foreach (string keyValue in keyValuePairs) {
byte[] bytes = HttpUtility.UrlEncodeToBytes(keyValue, encoding);
content.Write(bytes, 0, bytes.Length);
if (isKey) {
content.WriteByte((byte) '=');
}
else {
content.WriteByte((byte) '&');
}
isKey = !isKey;
}
// We'll end up with one surplus '&' -- shouldn't hurt, but we cut it
anyway
content.SetLength(content.Length - 1);

// That's the actual HTTP part of the sample
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
request.ContentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encodingName);
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream()) {
content.WriteTo(requestStream);
}

HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream()) {
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// Process buffer... e.g. write to file, decode to string, etc.
}
}

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[Joerg Jooss]

John A Grandy wrote:

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textbox1=thedata"
Right.
but how do i know what encodingName should be ?

That's the tricky part, because this depends on the web application. But
there's a simple heuristic that should work for most sites. Use the same
encoding that the site uses as character encoding. Open the web page and
check out what character encoding your browser applies (in IE
View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western Europe
(Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->

Unless you're working with a non US or non Western European web site, those
are the ones you'll most likely encounter. In case of doubt, use ISO-8859-1.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[John A Grandy]

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website.

i'm communicating over vpn with a private web-server meant for
data-provision to intermediaries.

the web-server does not validate credentials. it just immediately returns
data (interesting, as an xml doc)

the only working tool i have is a java applet written by someone else
(someone who is long gone).

the java applet is very simple. just a textbox and a submit button.

a long alphanumeric string is entered into the textbox (it ends with "^").

clicking the button fires a js function that calls form.submit and the form
is posted.

"view source" on the IE browseer hosting the java applet reveals only a very
simple block of html

a <textarea> element is used for data-entry.

i can't disclose the actual url -- but its similar to
"https://www.somecompany.com:5555/someservice.asmx/somefunction"

.... so apparently the web-server is hosting a .net web-service.
i tried your method. i constructed using the data-string successfully used
in the Java applet :

postData = "myTextArea=dataString";

encodingName = "utf-8";

request.GetResponse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you think
good possibilities are ?

or could the problem be in some other area ?

"Joerg Jooss" <jo*********@gmx.net> wrote in message
news:en**************@TK2MSFTNGP09.phx.gbl...

John A Grandy wrote:

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textbox1=thedata"
Right.

but how do i know what encodingName should be ?

That's the tricky part, because this depends on the web application. But
there's a simple heuristic that should work for most sites. Use the same
encoding that the site uses as character encoding. Open the web page and
check out what character encoding your browser applies (in IE
View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western

Europe (Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->

Unless you're working with a non US or non Western European web site, those are the ones you'll most likely encounter. In case of doubt, use ISO-8859-1.
Cheers,

--
Joerg Jooss
jo*********@gmx.net

[John A Grandy]

also, the MSDN docs for Encoding.GetEncoding(name) do not list "ISO-8859-1"

"Joerg Jooss" <jo*********@gmx.net> wrote in message
news:en**************@TK2MSFTNGP09.phx.gbl...

John A Grandy wrote:

so whatever data the user would enter into the text box "textbox1", i
simulate by composing the string

formData="textbox1=thedata"
Right.

but how do i know what encodingName should be ?

That's the tricky part, because this depends on the web application. But
there's a simple heuristic that should work for most sites. Use the same
encoding that the site uses as character encoding. Open the web page and
check out what character encoding your browser applies (in IE
View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western

Europe (Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->

Unless you're working with a non US or non Western European web site, those are the ones you'll most likely encounter. In case of doubt, use ISO-8859-1.
Cheers,

--
Joerg Jooss
jo*********@gmx.net

[Joerg Jooss]

John A Grandy wrote:

also, the MSDN docs for Encoding.GetEncoding(name) do not list
"ISO-8859-1"

They just show a few names in the .NET docs. The list of all supported
encodings is hidden deeply within MSDN, and of course it's at some place
you'd never expect it to be ;-)

Start here:

http://www.microsoft.com/globaldev/g..._codepage.mspx

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[Joerg Jooss]

John A Grandy wrote:

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website. [...] i can't disclose the actual url -- but its similar to
"https://www.somecompany.com:5555/someservice.asmx/somefunction"

... so apparently the web-server is hosting a .net web-service.
Yes.
i tried your method. i constructed using the data-string
successfully used in the Java applet :

postData = "myTextArea=dataString";

encodingName = "utf-8";

request.GetResponse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you
think good possibilities are ?

or could the problem be in some other area ?

I guess the whole approrach could be wrong then. My sample code showed a
programmatic web form submission, but you probably need a SOAP client.
Actually, if it is a "proper" Web Service, VS .NET should be able to
generate the necessary client-side code for you. The test code you've posted
doesn't fit into the picture at all, though.

I guess you'll need to talk to the Web Service guys to find out what
requests their web service can handle.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[John A Grandy]

Hi Joerg ....

I don't think the approach you suggested is wrong.

I have a simple HTML page that submits its form to the web-server, and
successfully receives a response. The HTML form odes not communicate in any
way with any entity other than the following:

The JavaScript function triggered by the button click is very very simple :
all is does is a form.submit. There is no other client-side functionality
occurring.

Whatever method this company is using to process information on their side ,
they do accept posting of a simple form with

"InputData=0123456789abcdefghijklmnopqrstuvwxy z^" (for example)

Is there any problem with URL encoding the "^" character?

I think the encoding-type is probably the villian. I'll have to try various
possibilities.

Also, do you know various common headers that could be added? I see in
many of the postings to this newsgroup that quite a different number of
headers are being used.

"Joerg Jooss" <jo*********@gmx.net> wrote in message
news:uf**************@TK2MSFTNGP11.phx.gbl...

John A Grandy wrote:

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website. [...]

i can't disclose the actual url -- but its similar to
"https://www.somecompany.com:5555/someservice.asmx/somefunction"

... so apparently the web-server is hosting a .net web-service.

Yes.

i tried your method. i constructed using the data-string
successfully used in the Java applet :

postData = "myTextArea=dataString";

encodingName = "utf-8";

request.GetResponse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you
think good possibilities are ?

or could the problem be in some other area ?

I guess the whole approrach could be wrong then. My sample code showed a
programmatic web form submission, but you probably need a SOAP client.
Actually, if it is a "proper" Web Service, VS .NET should be able to
generate the necessary client-side code for you. The test code you've

posted doesn't fit into the picture at all, though.

I guess you'll need to talk to the Web Service guys to find out what
requests their web service can handle.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[John A Grandy]

"Joerg Jooss" <jo*********@gmx.net> wrote in message
news:uf**************@TK2MSFTNGP11.phx.gbl...

John A Grandy wrote:

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website. [...]

i can't disclose the actual url -- but its similar to
"https://www.somecompany.com:5555/someservice.asmx/somefunction"

... so apparently the web-server is hosting a .net web-service.

Yes.

i tried your method. i constructed using the data-string
successfully used in the Java applet :

postData = "myTextArea=dataString";

encodingName = "utf-8";

request.GetResponse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you
think good possibilities are ?

or could the problem be in some other area ?

I guess the whole approrach could be wrong then. My sample code showed a
programmatic web form submission, but you probably need a SOAP client.
Actually, if it is a "proper" Web Service, VS .NET should be able to
generate the necessary client-side code for you. The test code you've

posted doesn't fit into the picture at all, though.

I guess you'll need to talk to the Web Service guys to find out what
requests their web service can handle.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[John A Grandy]

are the web.config <webServices> settings related to successfully posting to
a remote web-service ?

or do these settings only relate to the hosting of web-services of
web-servers ... ?
for example, by default , the HttpPost setting is not present in web.config
.....

web.config file

<webServices>

<protocols>

<add name="HttpPost" />

</protocols>

</webServices>

"Joerg Jooss" <jo*********@gmx.net> wrote in message
news:uf**************@TK2MSFTNGP11.phx.gbl...

John A Grandy wrote:

very interesting ....

unfortunately, i'm in a stickier situation than communicating with a
public-facing website. [...]

i can't disclose the actual url -- but its similar to
"https://www.somecompany.com:5555/someservice.asmx/somefunction"

... so apparently the web-server is hosting a .net web-service.

Yes.

i tried your method. i constructed using the data-string
successfully used in the Java applet :

postData = "myTextArea=dataString";

encodingName = "utf-8";

request.GetResponse();

triggered a runtime error : "500 : internal server error"
do you think the problem might be with encodingName? what do you
think good possibilities are ?

or could the problem be in some other area ?

I guess the whole approrach could be wrong then. My sample code showed a
programmatic web form submission, but you probably need a SOAP client.
Actually, if it is a "proper" Web Service, VS .NET should be able to
generate the necessary client-side code for you. The test code you've

posted doesn't fit into the picture at all, though.

I guess you'll need to talk to the Web Service guys to find out what
requests their web service can handle.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[Joerg Jooss]

John A Grandy wrote:

are the web.config <webServices> settings related to successfully
posting to a remote web-service ?

or do these settings only relate to the hosting of web-services of
web-servers ... ?
for example, by default , the HttpPost setting is not present in
web.config ....

web.config file

<webServices>

<protocols>

<add name="HttpPost" />

</protocols>

</webServices>

You're right, these lines control whether an ASP.NET based web service
accepts "pure" (non SOAP) requests. The line <add name="httPost"/> should
enable posting. Note this is *disabled* in ASP.NET 1.1 by default.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[John A Grandy]

so, for my purposes of communicating with a remote web service (hosted by a
commercial company) ...

it is *their* responsibility to make sure that their web.config (or
machine.config) is properly setup to accept HTTPPost requests ... ?
"Joerg Jooss" <jo*********@gmx.net> wrote in message
news:eL*************@TK2MSFTNGP12.phx.gbl...

John A Grandy wrote:

are the web.config <webServices> settings related to successfully
posting to a remote web-service ?

or do these settings only relate to the hosting of web-services of
web-servers ... ?
for example, by default , the HttpPost setting is not present in
web.config ....

web.config file

<webServices>

<protocols>

<add name="HttpPost" />

</protocols>

</webServices>

You're right, these lines control whether an ASP.NET based web service
accepts "pure" (non SOAP) requests. The line <add name="httPost"/> should
enable posting. Note this is *disabled* in ASP.NET 1.1 by default.

Cheers,

--
Joerg Jooss
jo*********@gmx.net

[Joerg Jooss]

John A Grandy wrote:

so, for my purposes of communicating with a remote web service
(hosted by a commercial company) ...

it is *their* responsibility to make sure that their web.config (or
machine.config) is properly setup to accept HTTPPost requests ... ?

*Technically* yes.

But in the end, this is decision between business partners ("We want POST!"
"We have SOAP!") and of course of IT security. If I was concerned: They
offer the service, they make they rules, and they need to tell you what you
can do and what you should do.

Cheers,

--
Joerg Jooss
jo*********@gmx.net