Hi everyone,
I try to write a simple web crawler. It has to do the following:
1) Open an URL and retrieve a HTML file.
2) Extract news headlines from the HTML file
3) Put the headlines into a RSS file.
For example, I want to go to this site and extract the headlines: www.unstrung.com/section.asp?section_id=86
The problem is I do not know howto extract a headline from a HTML
file.
I mean HTML is not structured as XML, so I do not really know to solve
this problem. I notice that PHP has URL Functions to deal with HTML
file. For example, you have get_meta_tags () to extract meta tag
content attributes from a HTML file. But then, extract meta tag is
easy. With headlines, I don't really know where the headlines are on
a HTML file. Would anyone give me inputs on this?
This is not an impossible problem. If you look at Google News
( http://news.google.com/), they crawl the web and sort the headlines
on their site.
Thanks,
P. Ho 3 3882
On 3 Aug 2004 14:24:11 -0700, pe********@yahoo.com (Phong Ho) wrote: I try to write a simple web crawler. It has to do the following: 1) Open an URL and retrieve a HTML file. 2) Extract news headlines from the HTML file 3) Put the headlines into a RSS file.
For example, I want to go to this site and extract the headlines: www.unstrung.com/section.asp?section_id=86
The problem is I do not know howto extract a headline from a HTML file. I mean HTML is not structured as XML, so I do not really know to solve this problem. I notice that PHP has URL Functions to deal with HTML file. For example, you have get_meta_tags () to extract meta tag content attributes from a HTML file. But then, extract meta tag is easy. With headlines, I don't really know where the headlines are on a HTML file. Would anyone give me inputs on this?
This is not an impossible problem. If you look at Google News (http://news.google.com/), they crawl the web and sort the headlines on their site.
Whilst entirely possible in PHP, I'd use Perl for this, as there are many
extremely useful modules for this sort of thing. HTML::TableExtract,
HTML::Parser, WWW::Mechanize to name a few.
But make sure you're not infringing on their copyright by redistributing the
extracted headlines.
--
Andy Hassall / <an**@andyh.co.uk> / <http://www.andyh.co.uk>
<http://www.andyhsoftware.co.uk/space> Space: disk usage analysis tool
(v1.4.0 new 1st Aug 2004)
"Andy Hassall" wrote: On 3 Aug 2004 14:24:11 -0700, pe********@yahoo.com (Phong Ho) wrote:
I try to write a simple web crawler. It has to do the following: 1) Open an URL and retrieve a HTML file. 2) Extract news headlines from the HTML file 3) Put the headlines into a RSS file.
For example, I want to go to this site and extract the headlines: www.unstrung.com/section.asp?section_id=86
The problem is I do not know howto extract a headline from a HTML file. I mean HTML is not structured as XML, so I do not really know to solvethis problem. I notice that PHP has URL Functions to deal with
HTMLfile. For example, you have get_meta_tags () to extract meta tag content attributes from a HTML file. But then, extract meta tag
iseasy. With headlines, I don’t really know where the headlines are ona HTML file. Would anyone give me inputs on this?
This is not an impossible problem. If you look at Google News (http://news.google.com/), they crawl the web and sort the headlines on their site.
Whilst entirely possible in PHP, I’d use Perl for this, as there are many extremely useful modules for this sort of thing.
HTML::TableExtract, HTML::Parser, WWW::Mechanize to name a few.
But make sure you’re not infringing on their copyright by redistributing the extracted headlines.
Or do it the brute force way, using regular expressions. Just refer
to the manul for doing that. I don’t think php as well developed
modules for doing this (vs. Perl) but I could be wrong.
If you know regular expressions, this is an easy thing to do, by the
way.
-- http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-Extract-...ict136182.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=454893
I do not have experience with Perl but I would like to ask a question
concerning Perl. I will use web hosting from a web hosting company,they
support Perl(CGI-BIN) but I won't have the root account. Do you need a
root account to install a Perl Module?
Also, anyone know if there is a sample in PHP to extract the headlines
from a HTML file on the internet?
Thanks
steve wrote: "Andy Hassall" wrote: > On 3 Aug 2004 14:24:11 -0700, pe********@yahoo.com (Phong Ho) wrote: > > >I try to write a simple web crawler. It has to do the following: > >1) Open an URL and retrieve a HTML file. > >2) Extract news headlines from the HTML file > >3) Put the headlines into a RSS file. > > > >For example, I want to go to this site and extract the headlines: > >www.unstrung.com/section.asp?section_id=86 > > > >The problem is I do not know howto extract a headline from a HTML > >file. > >I mean HTML is not structured as XML, so I do not really know to > solve > >this problem. I notice that PHP has URL Functions to deal with HTML > >file. For example, you have get_meta_tags () to extract meta tag > >content attributes from a HTML file. But then, extract meta tag is > >easy. With headlines, I don’t really know where the headlines > are on > >a HTML file. Would anyone give me inputs on this? > > > >This is not an impossible problem. If you look at Google News > >(http://news.google.com/), they crawl the web and sort the headlines > >on their site. > > Whilst entirely possible in PHP, I’d use Perl for this, as > there are many > extremely useful modules for this sort of thing. HTML::TableExtract, > HTML::Parser, WWW::Mechanize to name a few. > > But make sure you’re not infringing on their copyright by > redistributing the > extracted headlines. >
Or do it the brute force way, using regular expressions. Just refer to the manul for doing that. I don’t think php as well developed modules for doing this (vs. Perl) but I could be wrong.
If you know regular expressions, this is an easy thing to do, by the way. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Jane Doe |
last post by:
Hi
I took a quick look in the archives, but didn't find an answer
to this one.
I'd like to display a list of HTML files in a directory, showing the
author's name between brackets after the...
|
by: Joe |
last post by:
I'm trying to extract part of html code from a tag to a tag code begins
with <span class="boldyellow"><B><U> and ends with
TD><TD> <img src="http://whatever/some.gif"> </TD></TR></TABLE>
I was...
|
by: Selen |
last post by:
I would like to be able to extract a BLOB from the database (SqlServer)
and pass it to a browser without writing it to a file. (The BLOB's
are word doc's, MS project doc's, and Excel spreadsheets....
|
by: Vjay77 |
last post by:
I posted this question, but I pressed 'post' and it disappeared. So
once again:
Problem:
I need to go to lets say www.site.com/page.html
Imagine that this html code is 6 mb long.
I need to...
|
by: trihanhcie |
last post by:
Hi,
I would like to extract the text in an HTML file
For the moment, I'm trying to get all text between <tdand </td>. I
used a regular expression because i don't know the "format between...
|
by: steveyjg |
last post by:
I want to extract the following data from a retrieved html file and store the information as strings.
'get the text of "title"
<h1 id="test_title">title</h1>
'get the contents of the value...
|
by: nkg1234567 |
last post by:
I'm trying to extract HTML from a website in the form of a string, and then I want to extract particular elements from the string using the substr function:
here is some sample code that I have thus...
|
by: veer |
last post by:
Hi
i am making a program in which i want to extract data from html file .
Actually there are two dates on html file i want to extract these dates but the main probleum is that these dates are...
|
by: rizwan6feb |
last post by:
I am trying to extract php code from a php file (php file also contains html, css and javascript code). I am using the following regex for this
<\?*?\?>
but this doesn't cater quotation marks...
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |