469,917 Members | 1,479 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,917 developers. It's quick & easy.

Empty HttpWebResponse length for some url (with queries?)

For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...=WO2005028634),
the content length for the HttpWebResponse I get with request.GetResponse in
empty. The response.GetResponseStream() also empty. However, I am able to
open the URL in the browser (the URL address remains the same; it is not
redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1" and the
HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?

Thanks

Jason

Nov 17 '05 #1
13 8984
Hi,

You are using this:

StreamReader sr = new StreamReader(resp.GetResponseStream());

Shouldn't you be using this:

StreamReader sr = new StreamReader(resp.GetResponse().GetResponseStream( ));

The rest seens fine to me!
Hope it helps...
o.f

Nov 17 '05 #2

Is the length empty or is the object empty? Streams from webrequests by
default dont allow seeking so it won't know the length off hand.. if you
put a break in the code after it gets the stream you should be able to
hover over the stream and see if it's null or not, it shouldn't be null
unless it got an error, but it may not know the length...

Jason Manfield wrote:
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...=WO2005028634),
the content length for the HttpWebResponse I get with request.GetResponse in
empty. The response.GetResponseStream() also empty. However, I am able to
open the URL in the browser (the URL address remains the same; it is not
redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1" and the
HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?

Thanks

Jason

Nov 17 '05 #3
The object is not empty. The length of pageData (created with
StreamReader.readToEnd()) is 0. And the resp.ContentLength is also 0. But it
does show info like ContentType (which is "text/html; charset=iso-8859-1").
The HttpStatusCode is also OK. This problem occurs for only this URL. Is
there a special requirement in the WebRequest for URLs with query strings in
it?

"Michael McCarthy" wrote:

Is the length empty or is the object empty? Streams from webrequests by
default dont allow seeking so it won't know the length off hand.. if you
put a break in the code after it gets the stream you should be able to
hover over the stream and see if it's null or not, it shouldn't be null
unless it got an error, but it may not know the length...

Jason Manfield wrote:
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...=WO2005028634),
the content length for the HttpWebResponse I get with request.GetResponse in
empty. The response.GetResponseStream() also empty. However, I am able to
open the URL in the browser (the URL address remains the same; it is not
redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1" and the
HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?

Thanks

Jason

Nov 17 '05 #4
I also found in the response header info like:

Set-Cookie: JSESSIONID=0000AQWFK7gpN1TDbm0IKxJ94Tu:10dg77lr6;P ath=/

"Jason Manfield" wrote:
The object is not empty. The length of pageData (created with
StreamReader.readToEnd()) is 0. And the resp.ContentLength is also 0. But it
does show info like ContentType (which is "text/html; charset=iso-8859-1").
The HttpStatusCode is also OK. This problem occurs for only this URL. Is
there a special requirement in the WebRequest for URLs with query strings in
it?

"Michael McCarthy" wrote:

Is the length empty or is the object empty? Streams from webrequests by
default dont allow seeking so it won't know the length off hand.. if you
put a break in the code after it gets the stream you should be able to
hover over the stream and see if it's null or not, it shouldn't be null
unless it got an error, but it may not know the length...

Jason Manfield wrote:
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...=WO2005028634),
the content length for the HttpWebResponse I get with request.GetResponse in
empty. The response.GetResponseStream() also empty. However, I am able to
open the URL in the browser (the URL address remains the same; it is not
redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1" and the
HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?

Thanks

Jason

Nov 17 '05 #5
Probably you need to send that cookie on the webrequest you are doing.

o.f
Nov 17 '05 #6
Jason Manfield wrote:
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...5028634&F=0&QP
N=WO2005028634), the content length for the HttpWebResponse I get
with request.GetResponse in empty. The response.GetResponseStream()
also empty. However, I am able to open the URL in the browser (the
URL address remains the same; it is not redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1"
and the HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?


The character encoding for example. You're using a UTF-8 StreamReader
to read ISO-8859-1 which will cause problems. Create a ISO-8859-1
StreamReader to decode the response properly:

Encoding enc = Encoding.GetEncoding(28591);
StreamReader reader = new StreamReader(reponseStream, enc);

But you're right about the Content-Length: The server sends
"Content-Length: 0", which is a disastrous error. It is quite possible
that this breaks HttpWebResponse.

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de
Nov 17 '05 #7
I looked at this in some detail. I think the server is looking for a
User-Agent header in the request. If it finds one, it sends a response page.
Otherwise it doesnt.

You can take a look at the attached program. Set the proxy if needed. Run
the program with the UserAgent set. You will see the page getting
downloaded. Then rerun it with that line commented out. And the response
will have Zero content-length.


using System;
using System.IO;
using System.Text;
using System.Net;
using System.Net.Sockets;

public class EP {
public static void Main(string [] args)
{
string us =
"http://v3.espacenet.com/origdoc?DB=EPODOC&IDX=WO2005028634&F=0&QPN=WO20050 28634";
HttpWebRequest req = WebRequest.Create(us) as HttpWebRequest;
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.7.6)
Gecko/20";
//req.Proxy = new WebProxy("http://my-proxy");

try {
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;

foreach(string h in resp.Headers) {
Console.WriteLine(h + ": " + resp.Headers[h]);
}

Stream rs = resp.GetResponseStream();
MemoryStream ms = new MemoryStream();

byte [] data = new byte[1024];
int read = rs.Read(data,0,data.Length);
while(read > 0) {
ms.Write(data,0,read);
read = rs.Read(data,0,data.Length);
}

resp.Close();

ms.Seek(0,SeekOrigin.Begin);

StreamReader sr = new StreamReader(ms);
String s = sr.ReadToEnd();
sr.Close();

Console.WriteLine("====");
Console.WriteLine(s);

} catch(Exception e) {
Console.WriteLine(e);
}
}
}

--
feroze

-----------------
This posting is provided as-is. It offers no warranties and assigns no
rights.

See http://weblogs.asp.net/feroze_daud for System.Net related posts.
----------------

"Jason Manfield" <Ja***********@discussions.microsoft.com> wrote in message
news:F3**********************************@microsof t.com...
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...=WO2005028634),
the content length for the HttpWebResponse I get with request.GetResponse
in
empty. The response.GetResponseStream() also empty. However, I am able to
open the URL in the browser (the URL address remains the same; it is not
redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1" and
the
HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?

Thanks

Jason

Nov 17 '05 #8
I looked at this in some detail. I think the server is looking for a
User-Agent header in the request. If it finds one, it sends a response page.
Otherwise it doesnt.

You can take a look at the attached program. Set the proxy if needed. Run
the program with the UserAgent set. You will see the page getting
downloaded. Then rerun it with that line commented out. And the response
will have Zero content-length.


using System;
using System.IO;
using System.Text;
using System.Net;
using System.Net.Sockets;

public class EP {
public static void Main(string [] args)
{
string us =
"http://v3.espacenet.com/origdoc?DB=EPODOC&IDX=WO2005028634&F=0&QPN=WO20050 28634";
HttpWebRequest req = WebRequest.Create(us) as HttpWebRequest;
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.7.6)
Gecko/20";
//req.Proxy = new WebProxy("http://my-proxy");

try {
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;

foreach(string h in resp.Headers) {
Console.WriteLine(h + ": " + resp.Headers[h]);
}

Stream rs = resp.GetResponseStream();
MemoryStream ms = new MemoryStream();

byte [] data = new byte[1024];
int read = rs.Read(data,0,data.Length);
while(read > 0) {
ms.Write(data,0,read);
read = rs.Read(data,0,data.Length);
}

resp.Close();

ms.Seek(0,SeekOrigin.Begin);

StreamReader sr = new StreamReader(ms);
String s = sr.ReadToEnd();
sr.Close();

Console.WriteLine("====");
Console.WriteLine(s);

} catch(Exception e) {
Console.WriteLine(e);
}
}
}

--
feroze

-----------------
This posting is provided as-is. It offers no warranties and assigns no
rights.

See http://weblogs.asp.net/feroze_daud for System.Net related posts.
----------------

"Jason Manfield" <Ja***********@discussions.microsoft.com> wrote in message
news:F3**********************************@microsof t.com...
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...=WO2005028634),
the content length for the HttpWebResponse I get with request.GetResponse
in
empty. The response.GetResponseStream() also empty. However, I am able to
open the URL in the browser (the URL address remains the same; it is not
redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1" and
the
HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?

Thanks

Jason

Nov 17 '05 #9
Realized (by experimenting) that setting the UserAgent in HttpWebRequest
solved the problem. But, hey, I could access most of websites without setting
it.

req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";
resp = (HttpWebResponse)req.GetResponse();

"Joerg Jooss" wrote:
Jason Manfield wrote:
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...5028634&F=0&QP
N=WO2005028634), the content length for the HttpWebResponse I get
with request.GetResponse in empty. The response.GetResponseStream()
also empty. However, I am able to open the URL in the browser (the
URL address remains the same; it is not redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1"
and the HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?


The character encoding for example. You're using a UTF-8 StreamReader
to read ISO-8859-1 which will cause problems. Create a ISO-8859-1
StreamReader to decode the response properly:

Encoding enc = Encoding.GetEncoding(28591);
StreamReader reader = new StreamReader(reponseStream, enc);

But you're right about the Content-Length: The server sends
"Content-Length: 0", which is a disastrous error. It is quite possible
that this breaks HttpWebResponse.

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de

Nov 17 '05 #10
Realized (by experimenting) that setting the UserAgent in HttpWebRequest
solved the problem. But, hey, I could access most of websites without setting
it.

req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";
resp = (HttpWebResponse)req.GetResponse();

"Joerg Jooss" wrote:
Jason Manfield wrote:
For some URLs
(e.g.http://v3.espacenet.com/origdoc?DB=E...5028634&F=0&QP
N=WO2005028634), the content length for the HttpWebResponse I get
with request.GetResponse in empty. The response.GetResponseStream()
also empty. However, I am able to open the URL in the browser (the
URL address remains the same; it is not redirected)

Here is the code snippet:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageAddress);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string pageData = sr.ReadToEnd();

The Content Type for the response is "text/html; charset=iso-8859-1"
and the HttpStatusCode was OK. The pageData length is 0.

Should the URL be modified for it to work (e.g. some substitution)?

What am I missing?


The character encoding for example. You're using a UTF-8 StreamReader
to read ISO-8859-1 which will cause problems. Create a ISO-8859-1
StreamReader to decode the response properly:

Encoding enc = Encoding.GetEncoding(28591);
StreamReader reader = new StreamReader(reponseStream, enc);

But you're right about the Content-Length: The server sends
"Content-Length: 0", which is a disastrous error. It is quite possible
that this breaks HttpWebResponse.

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de

Nov 17 '05 #11
Joerg Jooss wrote:
But you're right about the Content-Length: The server sends
"Content-Length: 0", which is a disastrous error. It is quite possible
that this breaks HttpWebResponse.


But the content length would only be a specific length when using keep
alive with http 1.1+... which we can assume, using .NET that we don't
care about, since we can use a common connection pool that deprecates
keep alive.

think so?

~Michael.
Nov 17 '05 #12
Joerg Jooss wrote:
But you're right about the Content-Length: The server sends
"Content-Length: 0", which is a disastrous error. It is quite possible
that this breaks HttpWebResponse.


But the content length would only be a specific length when using keep
alive with http 1.1+... which we can assume, using .NET that we don't
care about, since we can use a common connection pool that deprecates
keep alive.

think so?

~Michael.
Nov 17 '05 #13
Michael McCarthy wrote:
Joerg Jooss wrote:
But you're right about the Content-Length: The server sends
"Content-Length: 0", which is a disastrous error. It is quite
possible that this breaks HttpWebResponse.


But the content length would only be a specific length when using
keep alive with http 1.1+... which we can assume, using .NET that we
don't care about, since we can use a common connection pool that
deprecates keep alive.

think so?


In HTTP 1.1 you either

a) specify a Content-Length
b) use Transfer-Encoding: chunked
c) use Connection: close

using c) as request header disables the use of persistent connections
("keep-alive" is an outdated HTTP 1.0+ term).

I don't get the part with the connecion pooling. A connection pool only
makes sense to, um, pool connections to a well known end communication
end points, e.g. a database or an EIS. HTTP clients in general do
communicate with a lot of *different* end points, so what exactly do
want to pool? And regardless of pooling, "Connection: close" must
terminate the TCP connection:

"Once a close has been signaled, the client MUST NOT send any more
requests on that connection." (§8.1.2)

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de
Nov 17 '05 #14

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Michael Evanchik | last post: by
3 posts views Thread by Karsten Grombach | last post: by
3 posts views Thread by Sean Chapman | last post: by
2 posts views Thread by Nuno Magalhaes | last post: by
reply views Thread by XML newbie: Urgent pls help! | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.