Making posts to an ASP.NET webform.

[Bernard]

hiya everyone,

I've made this little webcrawler using BeautifulSoup, urllib and
urllib2. I've been encountering ASP.NET Forms recently and I can't seem
to make a proper post to some of them. My post function has been doing
great until this particular website.
Here's some explanations first so that you guys understands my
problem...

Usually there are 8 hidden inputs spread all over the web page:
__EVENTTARGET --that is the id of the control that is assigned by the
ASP.NET engine. It is usually in this particular form
"dg_Result$_ctl2$linkbtn1" The javascript function DoPostback usually
replaces the '$' for a ':' before doing the real post. Some other
ASP.NET page doesn't.
__EVENTARGUMENT --nothing's here usually.
__ScrollTop --its value is often 0
__ScrollLeft --same as __ScrollTop
__ValidationSummary --usually empty
__VIEWSTATEENCRYPTED --an encrypted string. I don't know what it
means nor what it does.
__EVENTVALIDATION --an encrypted string. I don't know what it means
nor what it does.
__VIEWSTATE --the encrypted state of the current control. the state
of a datagrid for example.

I extract all those values using regexes on the web page, build a
dictionnary out of all those values and encrypt the whole thing using
urllib.urlencode(). Afterwards I extract the form action and post the
whole thing with cookies enabled. This procedure works like a charm.

The website I've just encounted has only 3 of the 8 hidden inputs.
(EVENTTARGET, EVENTARGUMENT & VIEWSTATE ).
I tried posting the whole thing using only these 3 values & nada.
I then tried posting the whole thing using all empty values for the 5
remaining values...still nothing.

Has anyone tried what I'm doing? and if you tried how have you
succeeded getting the data back after the post action?

thanks for any tips!

cP

[Max M]

Bernard skrev:

Has anyone tried what I'm doing? and if you tried how have you
succeeded getting the data back after the post action?

Most likely you get assigned a cookie that you then need to return.

Try the cookielib which automates all this for you.

[Bernard]

Max M wrote:

Bernard skrev:

Has anyone tried what I'm doing? and if you tried how have you
succeeded getting the data back after the post action?

Most likely you get assigned a cookie that you then need to return.

Try the cookielib which automates all this for you.

Yea I'm already using it. I set the cookies before making the server
request. its name is ASP.NET_SessionId.

I've tried setting it on & off during my various tests but I still get
the "urllib2.HTTPError: HTTP Error 500: Internal Server Error" message.

thanks for the response by the way :)

[Gabriel Genellina]

At Monday 2/10/2006 13:05, Bernard wrote:

Has anyone tried what I'm doing? and if you tried how have you
succeeded getting the data back after the post action?

Use a packet sniffer or something to look at an actual POST that
works, this way you could see what's missing in your code.
Gabriel Genellina
Softlab SRL

__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[Bernard]

Gabriel Genellina wrote:

At Monday 2/10/2006 13:05, Bernard wrote:

Has anyone tried what I'm doing? and if you tried how have you
succeeded getting the data back after the post action?

Use a packet sniffer or something to look at an actual POST that
works, this way you could see what's missing in your code.
Gabriel Genellina
Softlab SRL

Thanks for that suggestion Gabriel. It's really a great idea to see
what's going on behind the scene but it still doesn't work even though
the post made by my function has the same data as a real browser post.

here's the HTTP Header from my sniffer:
-------------------------------------------------------------
http://www.newportrealty.com/Search/...t.aspx?Area=WC

POST /Search/Default.aspx?Area=WC HTTP/1.1
Host: www.newportrealty.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.6)
Gecko/20060728 Firefox/1.5.0.6
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.newportrealty.com/Search/...t.aspx?Area=WC
Cookie: ASPSESSIONIDQQBTTDAT=JMBNNNNBNNJELLBBNCCDMNCK;
ASP.NET_SessionId=hzdjizvca2zid3f205s5kd45
Content-Type: application/x-www-form-urlencoded
Content-Length: 4649
__EVENTTARGET=dg_Result%3A_ctl1%3A_ctl1&__EVENTARG UMENT=&__VIEWSTATE=dDw0MD(...)
HTTP/1.x 200 OK
Cache-Control: private
Content-Length: 37519
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Date: Wed, 04 Oct 2006 15:15:37 GMT
-------------------------------------------------------------

and here's the output of the encoded data as well as the cookies in my
function:

-------------------------------------------------------------
cookies:
<Cookie ASP.NET_SessionId=lddb3iruopvpnjiar0dlytvi for
www.newportrealty.com/>
<Cookie ASPSESSIONIDQQBTTDAT=OMBNNNNBMLDDAIHNDHJGLFKP for
www.newportrealty.com/>

encoded post data:
__EVENTTARGET=dg_Result%3A_ctl14%3A_ctl2&__EVENTAR GUMENT=&__VIEWSTATE=dDw0MDAyOTcw(...)
-------------------------------------------------------------

My function has already worked on other asp.net websites and I'm pretty
sure this would work out just fine. is there something I'm missing here?

[Bernard]

Bernard wrote:

Gabriel Genellina wrote:

At Monday 2/10/2006 13:05, Bernard wrote:

Has anyone tried what I'm doing? and if you tried how have you
succeeded getting the data back after the post action?

Use a packet sniffer or something to look at an actual POST that
works, this way you could see what's missing in your code.
Gabriel Genellina
Softlab SRL

Thanks for that suggestion Gabriel. It's really a great idea to see
what's going on behind the scene but it still doesn't work even though
the post made by my function has the same data as a real browser post.

here's the HTTP Header from my sniffer:
-------------------------------------------------------------
http://www.newportrealty.com/Search/...t.aspx?Area=WC

POST /Search/Default.aspx?Area=WC HTTP/1.1
Host: www.newportrealty.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.6)
Gecko/20060728 Firefox/1.5.0.6
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.newportrealty.com/Search/...t.aspx?Area=WC
Cookie: ASPSESSIONIDQQBTTDAT=JMBNNNNBNNJELLBBNCCDMNCK;
ASP.NET_SessionId=hzdjizvca2zid3f205s5kd45
Content-Type: application/x-www-form-urlencoded
Content-Length: 4649
__EVENTTARGET=dg_Result%3A_ctl1%3A_ctl1&__EVENTARG UMENT=&__VIEWSTATE=dDw0MD(...)
HTTP/1.x 200 OK
Cache-Control: private
Content-Length: 37519
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Date: Wed, 04 Oct 2006 15:15:37 GMT
-------------------------------------------------------------

and here's the output of the encoded data as well as the cookies in my
function:

-------------------------------------------------------------
cookies:
<Cookie ASP.NET_SessionId=lddb3iruopvpnjiar0dlytvi for
www.newportrealty.com/>
<Cookie ASPSESSIONIDQQBTTDAT=OMBNNNNBMLDDAIHNDHJGLFKP for
www.newportrealty.com/>

encoded post data:
__EVENTTARGET=dg_Result%3A_ctl14%3A_ctl2&__EVENTAR GUMENT=&__VIEWSTATE=dDw0MDAyOTcw(...)
-------------------------------------------------------------

My function has already worked on other asp.net websites and I'm pretty
sure this would work out just fine. is there something I'm missing here?

ok my bad I've just found out that I didn't build properly the form
action url!! It works now!

thanks for all the responses!!