473,326 Members | 2,110 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Download a webpage that contains javascript

I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat
Jul 17 '05 #1
5 4222
Have you checked its behavious whenyou try opening on Browser..? it
redirects you to someother page and checks if you are logged in.. etc etc..

The way you are doing is not the proper way you have to follow the RFC
for HTTP1.1 for the same... the procedure used by you might work for
some sites but not for all the sites....

Cheers

http://www.ziksa.net
email: sa***@ziksa.net

Patrick wrote:
I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat

Jul 17 '05 #2
I had a mistake in my previous post, it should have read:

If I navigate to
http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
And click on a link which calls the follow javascript function
"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
A new page pops up, the browser say its url is

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
Now with my code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

I can download the page at

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
But I cannot download the page at

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
That the javascript function

"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
In the page

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

calls.

When I try to download

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

With my code, all i get is

<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>


So how do i go about downloading the contents of the html page that is
in the window that is displayed when the javascript function above is
called?
regards,

pat

Chander <sa***@ziksa.net> wrote in message news:<7_*******************@news20.bellglobal.com> ...
Have you checked its behavious whenyou try opening on Browser..? it
redirects you to someother page and checks if you are logged in.. etc etc..

The way you are doing is not the proper way you have to follow the RFC
for HTTP1.1 for the same... the procedure used by you might work for
some sites but not for all the sites....

Cheers

http://www.ziksa.net
email: sa***@ziksa.net

Patrick wrote:
I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat

Jul 17 '05 #3
As I menatined earlier read the rfc for http 1.1 when you are requesting
a page firsdt you get is the header you get information for size of the
data to follow, if server has responded as error, what type of error..
or the page is to be redirected, etc.. as per these information you are
to read the following lines or resend the request to the other page as
direction in the header..

Cheers
Patrick wrote:
I had a mistake in my previous post, it should have read:

If I navigate to
http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
And click on a link which calls the follow javascript function
"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
A new page pops up, the browser say its url is

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
Now with my code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

I can download the page at

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH
But I cannot download the page at

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
That the javascript function

"javascript:dt_pop('PostPlayerList?catidx=1&title= GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
In the page

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

calls.

When I try to download

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

With my code, all i get is

<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>


So how do i go about downloading the contents of the html page that is
in the window that is displayed when the javascript function above is
called?
regards,

pat

Chander <sa***@ziksa.net> wrote in message news:<7_*******************@news20.bellglobal.com> ...
Have you checked its behavious whenyou try opening on Browser..? it
redirects you to someother page and checks if you are logged in.. etc etc..

The way you are doing is not the proper way you have to follow the RFC
for HTTP1.1 for the same... the procedure used by you might work for
some sites but not for all the sites....

Cheers

http://www.ziksa.net
email: sa***@ziksa.net

Patrick wrote:
I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/se...rList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
I'm guessing that when the web brower meets this code it executes the
javascript
location.replace("http://www.dreamteamfc.com")
And somehow loads the web page.
How do I go about downloading the actual web page that gets displayed
in a normal web browser.


Any help appeciated!
pat

Jul 17 '05 #4
Ok, I am starting to understand what is going on now a little better.
I noticed that when I access

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

The server sends a cookie. And then when I access

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

I get the table of players and their respective points.

I used the following code

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();
System.out.println(uc.getHeaderField("Set-Cookie"));
}
To get the cookie, which was

CF_HA=2415676698; Domain=.dreamteamfc.com; expires=Tue, 14-Sep-04
22:25:46 GMT; Path=/

I think
CF_HA, is just a unique identifier, a variable which in incremented
by the server for each new client
Domain, is just the domain
expires, is just the expiry date
Path, hmm dunno

Now I hardcoded the cookie into the code, with a valid expiry date

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();

String cookie = "CF_HA=2415676698; Domain=.dreamteamfc.com;
expires=Tue, 14-Sep-04 22:25:46 GMT; Path=/";
uc.setRequestProperty("cookie",cookie);
int i = 0;

while ((i = uc.getInputStream().read()) != -1)
{ System.out.print((char) i);
}
}
Now, when I run this code I get the following error
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
at sun.net.http://www.protocol.http.HttpURLConn...tion.java:1133)
at Test.main(Test.java:46)
Is the following method

uc.setRequestProperty("cookie",cookie);

The correct way to send the cookie to the server?
Any pointers on what I should try next, I'm not quite sure where to
look.
pat

Chander <sa***@ziksa.net> wrote in message news:<aK*********************@news20.bellglobal.co m>...
As I menatined earlier read the rfc for http 1.1 when you are requesting
a page firsdt you get is the header you get information for size of the
data to follow, if server has responded as error, what type of error..
or the page is to be redirected, etc.. as per these information you are
to read the following lines or resend the request to the other page as
direction in the header..

Cheers

Jul 17 '05 #5
On 15 Sep 2004 07:38:22 -0700, go***********@yahoo.co.uk (Patrick)
wrote:
Ok, I am starting to understand what is going on now a little better.
I noticed that when I access

http://www.dreamteamfc.com/dtfc04/se...nguage=ENGLISH

The server sends a cookie. And then when I access

http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167

I get the table of players and their respective points.

I used the following code

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();
System.out.println(uc.getHeaderField("Set-Cookie"));
}
To get the cookie, which was

CF_HA=2415676698; Domain=.dreamteamfc.com; expires=Tue, 14-Sep-04
22:25:46 GMT; Path=/

I think
CF_HA, is just a unique identifier, a variable which in incremented
by the server for each new client
Domain, is just the domain
expires, is just the expiry date
Path, hmm dunno

Now I hardcoded the cookie into the code, with a valid expiry date

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=1 67");
URLConnection uc = url.openConnection();

String cookie = "CF_HA=2415676698; Domain=.dreamteamfc.com;
expires=Tue, 14-Sep-04 22:25:46 GMT; Path=/";
uc.setRequestProperty("cookie",cookie);
int i = 0;

while ((i = uc.getInputStream().read()) != -1)
{ System.out.print((char) i);
}
}
Now, when I run this code I get the following error
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://www.dreamteamfc.com/dtfc04/se...ERS&gameid=167
at sun.net.http://www.protocol.http.HttpURLConn...tion.java:1133)
at Test.main(Test.java:46)
Is the following method

uc.setRequestProperty("cookie",cookie);

The correct way to send the cookie to the server?
Any pointers on what I should try next, I'm not quite sure where to
look.
pat

Chander <sa***@ziksa.net> wrote in message news:<aK*********************@news20.bellglobal.co m>...
As I menatined earlier read the rfc for http 1.1 when you are requesting
a page firsdt you get is the header you get information for size of the
data to follow, if server has responded as error, what type of error..
or the page is to be redirected, etc.. as per these information you are
to read the following lines or resend the request to the other page as
direction in the header..

Cheers

Well, I agree with Chander that you need to read the RFC on HTTP 1.1
(RFC 2068) which can be found at
http://www.ietf.org/rfc/rfc2068.txt?number=2068. I'd give you more
detailed advice, but I think a lot of it would be speculation (my
programmer friend is sitting next to me, so these are his words)

So here is the speculation. The issue that might be the cause of your
woes. In order for you to set properties in a URLConnection, the
properties MUST be set before you call getInputStream() or getHeader()
Jul 17 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Michael | last post by:
I am trying to allow a user to view a PDF in a new window. I currently have this working using the following: <a href="./pdf.do?parameter=01121980" target="top"><b>pdf</b></a> The problem...
2
by: Tom Youngquist | last post by:
I am trying to download a text file that my .NET page has just created based on entered parameters on the web page. Everything seems to work and the file is created. I am using the following code...
1
by: Alex | last post by:
Hi, Everyday, I download data from a webpage and manually input data into my MS Access database. I am thinking of automate the routine by a VB script. The webpage I am visiting will return a...
0
by: Ron Vecchi | last post by:
Hello, Through using this and many other Microsoft dotnet newsgroups I have been pointed towards many helpful open source projects and articles that have helped a great deal in my studies as...
13
by: Jitendra | last post by:
Hi! I m facing a problem with 'javascript:void(0)' Software Environ:- IE: 6.0.2600.0000 OS: Windows 2000 Professional with Service Pack 4 Problem:- I have a webpage with several links (<A>...
6
by: eureka | last post by:
Hi friends, I am developing a web application using Jsp and JS. I have a main Jsp page(Jsp1).Inside it I have an iframe having an Html- table which is created dynamically and contains all...
3
by: Gargoyle | last post by:
I am curious to know if any research has been conducted regarding the efficiency of having a single (large) .js file downloaded for a webpage compared to several smaller .js files. For example...
3
by: kyle | last post by:
Hi Guys, I am trying to mimic the behavior of download.com... where a page says "your download will start in 10 seconds" and then the download box pops up. Would anyone mind telling me how that...
1
by: Rakhi | last post by:
hello i want to alter the download settings of mozilla firefox browser using javascript of my application !! wat is happenin in my application is, on calling a method , it make a file ready...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.