liburl cant load webpage with Javascript

Uwe Mayer

Hi,

I want do use liburl to scan a webpage which is only accessible from within
my LAN environment. While mozilla manages to load the target URL properly
neither wget, nor liburl or liburl2 does.
I had a closer look at the html source and discovered a lot of Javascript,
including Cookies.

My suspicion is that the Javascript code needs to be executed for the page
to work properly. Also I don't know how liburl deals with Cookies, but
since they are handled by the Javascript in the source code they are
probably not considered at all.

In any case I get an IOError: connection refused, Error Code 111.

Does anyone know a way out of this?

Thanks for any hints,
Ciao
Uwe

Jul 18 '05 #1

Subscribe Reply

2316

Lorenzo Gatti

Uwe Mayer <me*****@hadiko .de> wrote in message news:<c8******* **@news2.rz.uni-karlsruhe.de>.. .

Hi,

I want do use liburl to scan a webpage which is only accessible from within
my LAN environment. While mozilla manages to load the target URL properly
neither wget, nor liburl or liburl2 does.
I had a closer look at the html source and discovered a lot of Javascript,
including Cookies.

My suspicion is that the Javascript code needs to be executed for the page
to work properly. Also I don't know how liburl deals with Cookies, but
since they are handled by the Javascript in the source code they are
probably not considered at all.

In any case I get an IOError: connection refused, Error Code 111.

Does anyone know a way out of this?

Thanks for any hints,
Ciao
Uwe

Mozilla is a web browser, and it implements cookies, DOM for HTML
pages, and a Javascript interpreter with objects representing browser
automation.
It's unlikely and inappropriate for low level HTTP implementations
like wget and liburl to have that kind of support for advanced web
features; maybe you can support cookies and Javascript in your
application.

In the specific case of "IOError: connection refused, Error Code 111",
however, the failure seems to happen at a lower protocol level: wrong
host names or port numbers, unavailable servers and maybe proxy
authentication requirements are the usual causes of refused
connections.

Lorenzo Gatti

Jul 18 '05 #2

John J. Lee

ga***@dsdata.it (Lorenzo Gatti) writes:

Uwe Mayer <me*****@hadiko .de> wrote in message news:<c8******* **@news2.rz.uni-karlsruhe.de>.. . [...]
I had a closer look at the html source and discovered a lot of Javascript,
including Cookies.

[...] Mozilla is a web browser, and it implements cookies, DOM for HTML
pages, and a Javascript interpreter with objects representing browser
automation.
It's unlikely and inappropriate for low level HTTP implementations
like wget and liburl to have that kind of support for advanced web [...]

JavaScript support is rare, but many libraries and tools support
cookies (including wget and my library, ClientCookie -- essentially a
drop-in replacement for urllib2). For JS, see my FAQ here (under
"Embedded script is messing up my web-scraping. What do I do?"):

http://wwwsearch.sourceforge.net/bits/GeneralFAQ.html

In the specific case of "IOError: connection refused, Error Code 111",
however, the failure seems to happen at a lower protocol level: wrong

[...]

Right.
John

Jul 18 '05 #3

Similar topics

3256

Cant get menu script to to put linked page in the same frame.

by: bayouprophet | last post by:

Cant get menu script to to put linked page in the same frame. I am new to Java and I am wondering what am I doing wrong? below are my java applet file, frame.html file, and my text file and one of my link file that should load next to the menu on the same page. And Thank You in advance. Here is my menu applet: <html>

Javascript

2667

Cant correctly load .js files when creating page dynamically in IE

by: Jake Lewis | last post by:

I have an HTML page that loads fine including the .js file <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <title>Untitled Document</title> <script language="JavaScript" type="text/JavaScript" src="ve3d.js" ></script> </head>

Javascript

1678

Authentication Q - Cant use basic or windows

by: arun.hallan | last post by:

Hi, I'm having problems with authentication. I have a set of users that are allowed to use a webpage. They are in domain A. My goal is to get the username of these users and then check them against my list. The webpage is hosted under a username in domain B. I cannot change any settings on the domains.

ASP.NET

1526

load external html vs 2005

by: Jeff Allan | last post by:

Hello, I am trying to load an external HTML page into a DIV tag with .NET 05 and I can't figure it out. Any suggestions? My Scenario: - Gridview populated with list of available files - I can get the external page to load in a new window just fine - I would like the detail to show on the same page, below the list. - I can't use the detail control and dynamically create the page as these

Visual Basic .NET

8196

load web page to variable

by: Tomek Toczyski | last post by:

I have an url of a webpage. I would like to load the content of the webpage to a variable. Sth like this: variable = loadwebpage( url ) How to do it? (I have just spent more than an hour googling for a recipe - without

Javascript

1906

Javascript HTML parser

by: pmarkov | last post by:

Hello, I need to build into my php application some javascript elements. My application check a given webpage for some information - lets suppose we have to check if there is a 'special link' on a 'special webpage'. I know how to parse this webpage using php and to check for the special link on it. The problem is that if I use a server side platform like php the 'special website' will know that I always do checking from the same IP address and...

Javascript

1455

database connectivity in javascript

by: Elavarasi | last post by:

Is it possible to have database connection in javascript.. i dont want to access the server database.. to access the local database i am asking... bcoz when i trying to open from asp(in localhost its working but in webpage its not working) .. i cant open the local db.. thats why i choose javascript.. will it work..?? regards ela..

Javascript

1934

Writing Cookies in Javascript.

by: sreemathy2000 | last post by:

I have a fully function webpage based on a Zipcode. I get the Zip code from the user once and write it the cookie and retrieve that everytime. I also have another webpage which also asks for the zip code. I have to synchronize these two. What i have done for this i accept the zip code also as querystrig parameter. So from 1 website when i enter the zip code, in the javascript i have image tag and i calll the otherwebsite and load it...

Javascript

9704

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

10562

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

10319

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...

Online Marketing

10070

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...

General

6845

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

5639

Windows Forms - .Net 8.0

by: adsilva | last post by:

A Windows Forms form does not have the event Unload, like VB6. What one acts like?

Visual Basic .NET

4282

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp

3803

How to add payments to a PHP MySQL app.

by: muto222 | last post by:

How can i add a mobile payment intergratation into php mysql website.

PHP

2978

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

General