By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,584 Members | 1,774 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,584 IT Pros & Developers. It's quick & easy.

Download a webpage that contains javascript

P: n/a
I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat
Jul 17 '05 #1
Share this Question
Share on Google+
5 Replies


P: n/a
Have you checked its behavious whenyou try opening on Browser..? it
redirects you to someother page and checks if you are logged in.. etc etc..

The way you are doing is not the proper way you have to follow the RFC
for HTTP1.1 for the same... the procedure used by you might work for
some sites but not for all the sites....

Cheers

http://www.ziksa.net
email: sa***@ziksa.net

Patrick wrote:
I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat

Jul 17 '05 #2

P: n/a
I had a mistake in my previous post, it should have read:

If I navigate to
http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
And click on a link which calls the follow javascript function
"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
A new page pops up, the browser say its url is

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
Now with my code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

I can download the page at

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
But I cannot download the page at

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
That the javascript function

"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
In the page

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

calls.

When I try to download

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

With my code, all i get is

<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>


So how do i go about downloading the contents of the html page that is
in the window that is displayed when the javascript function above is
called?
regards,

pat

Chander <sa***@ziksa.net> wrote in message news:<7_*******************@news20.bellglobal.com> ...
Have you checked its behavious whenyou try opening on Browser..? it
redirects you to someother page and checks if you are logged in.. etc etc..

The way you are doing is not the proper way you have to follow the RFC
for HTTP1.1 for the same... the procedure used by you might work for
some sites but not for all the sites....

Cheers

http://www.ziksa.net
email: sa***@ziksa.net

Patrick wrote:
I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat

Jul 17 '05 #3

P: n/a
As I menatined earlier read the rfc for http 1.1 when you are requesting
a page firsdt you get is the header you get information for size of the
data to follow, if server has responded as error, what type of error..
or the page is to be redirected, etc.. as per these information you are
to read the following lines or resend the request to the other page as
direction in the header..

Cheers
Patrick wrote:
I had a mistake in my previous post, it should have read:

If I navigate to
http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
And click on a link which calls the follow javascript function
"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
A new page pops up, the browser say its url is

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
Now with my code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

I can download the page at

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
But I cannot download the page at

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
That the javascript function

"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
In the page

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

calls.

When I try to download

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

With my code, all i get is

<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>


So how do i go about downloading the contents of the html page that is
in the window that is displayed when the javascript function above is
called?
regards,

pat

Chander <sa***@ziksa.net> wrote in message news:<7_*******************@news20.bellglobal.com> ...
Have you checked its behavious whenyou try opening on Browser..? it
redirects you to someother page and checks if you are logged in.. etc etc..

The way you are doing is not the proper way you have to follow the RFC
for HTTP1.1 for the same... the procedure used by you might work for
some sites but not for all the sites....

Cheers

http://www.ziksa.net
email: sa***@ziksa.net

Patrick wrote:
I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat

Jul 17 '05 #4

P: n/a
Ok, I am starting to understand what is going on now a little better.
I noticed that when I access

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

The server sends a cookie. And then when I access

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

I get the table of players and their respective points.

I used the following code

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();
System.out.println(uc.getHeaderField("Set-Cookie"));
}
To get the cookie, which was

CF_HA=2415676698; Domain=.dreamteamfc.com; expires=Tue, 14-Sep-04
22:25:46 GMT; Path=/

I think
CF_HA, is just a unique identifier, a variable which in incremented
by the server for each new client
Domain, is just the domain
expires, is just the expiry date
Path, hmm dunno

Now I hardcoded the cookie into the code, with a valid expiry date

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();

String cookie = "CF_HA=2415676698; Domain=.dreamteamfc.com;
expires=Tue, 14-Sep-04 22:25:46 GMT; Path=/";
uc.setRequestProperty("cookie",cookie);
int i = 0;

while ((i = uc.getInputStream().read()) != -1)
{ System.out.print((char) i);
}
}
Now, when I run this code I get the following error
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
at sun.net.http://www.protocol.http.HttpURLConn...tion.java:1133)
at Test.main(Test.java:46)
Is the following method

uc.setRequestProperty("cookie",cookie);

The correct way to send the cookie to the server?
Any pointers on what I should try next, I'm not quite sure where to
look.
pat

Chander <sa***@ziksa.net> wrote in message news:<aK*********************@news20.bellglobal.co m>...
As I menatined earlier read the rfc for http 1.1 when you are requesting
a page firsdt you get is the header you get information for size of the
data to follow, if server has responded as error, what type of error..
or the page is to be redirected, etc.. as per these information you are
to read the following lines or resend the request to the other page as
direction in the header..

Cheers

Jul 17 '05 #5

P: n/a
On 15 Sep 2004 07:38:22 -0700, go***********@yahoo.co.uk (Patrick)
wrote:
Ok, I am starting to understand what is going on now a little better.
I noticed that when I access

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

The server sends a cookie. And then when I access

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

I get the table of players and their respective points.

I used the following code

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();
System.out.println(uc.getHeaderField("Set-Cookie"));
}
To get the cookie, which was

CF_HA=2415676698; Domain=.dreamteamfc.com; expires=Tue, 14-Sep-04
22:25:46 GMT; Path=/

I think
CF_HA, is just a unique identifier, a variable which in incremented
by the server for each new client
Domain, is just the domain
expires, is just the expiry date
Path, hmm dunno

Now I hardcoded the cookie into the code, with a valid expiry date

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();

String cookie = "CF_HA=2415676698; Domain=.dreamteamfc.com;
expires=Tue, 14-Sep-04 22:25:46 GMT; Path=/";
uc.setRequestProperty("cookie",cookie);
int i = 0;

while ((i = uc.getInputStream().read()) != -1)
{ System.out.print((char) i);
}
}
Now, when I run this code I get the following error
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
at sun.net.http://www.protocol.http.HttpURLConn...tion.java:1133)
at Test.main(Test.java:46)
Is the following method

uc.setRequestProperty("cookie",cookie);

The correct way to send the cookie to the server?
Any pointers on what I should try next, I'm not quite sure where to
look.
pat

Chander <sa***@ziksa.net> wrote in message news:<aK*********************@news20.bellglobal.co m>...
As I menatined earlier read the rfc for http 1.1 when you are requesting
a page firsdt you get is the header you get information for size of the
data to follow, if server has responded as error, what type of error..
or the page is to be redirected, etc.. as per these information you are
to read the following lines or resend the request to the other page as
direction in the header..

Cheers

Well, I agree with Chander that you need to read the RFC on HTTP 1.1
(RFC 2068) which can be found at
http://www.ietf.org/rfc/rfc2068.txt?number=2068. I'd give you more
detailed advice, but I think a lot of it would be speculation (my
programmer friend is sitting next to me, so these are his words)

So here is the speculation. The issue that might be the cause of your
woes. In order for you to set properties in a URLConnection, the
properties MUST be set before you call getInputStream() or getHeader()
Jul 17 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.