httpwebrequest, cookies, session

[Susan Van Houen]

Hi Everybody,

I have a problem that is driving me crazy and I am hoping for some help
here.

I am building an application (vb.net, vs2003) that crawls selected websites.
The application works fine except for one thing: Some sites rely on session
cookies and I don't seem to be returning the cookies.

The IIS5 log looks like this when I access the same page with

IE:
12:13:15 192.168.0.2 CLIENT GET /pm4/aboutus.asp - 200 71
Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.N ET+CLR+1.0.3705;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.4 0607)
signinfrom=;+ASPSESSIONIDCAABTABS=EMFHLDECGHEEOCJB FMLLMDFD;+ASPSESSIONIDACACQBBS=NMONBFECJPICHEGFGAC FEIJG
-

My Application:
11:40:42 192.168.0.2 CLIENT GET /pm4/aboutus.asp - 200 40
Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Q312461 ;+.NET+CLR+1.0.3705)
- -

As you see the session cookies are missing.

The relevant part of my code looks like this:

Dim req As HttpWebRequest = CType(WebRequest.Create(url), HttpWebRequest)
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461;
..NET CLR 1.0.3705)"
req.CookieContainer = New CookieContainer
req.KeepAlive = True
req.Headers.Set("Pragma", "no-cache")
req.Timeout = 30000
req.ContentType = "application/x-www-form-urlencoded"
Dim ReceiveStream As Stream
Dim encode As Encoding
Dim sr As StreamReader
result = req.GetResponse()
ReceiveStream = result.GetResponseStream()
encode = System.Text.Encoding.GetEncoding("utf-8")
sr = New StreamReader(ReceiveStream, encode)
etc...

If I access a .aspx page it confirms that my application supports cookies,
javascript, etc. It basically accepts that I emulate an IE.
But the cookie thing doesn't work.

What I am doing wrong?

Yours,