473,714 Members | 2,328 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Extract headlines from a HTML file.

Hi everyone,

I try to write a simple web crawler. It has to do the following:
1) Open an URL and retrieve a HTML file.
2) Extract news headlines from the HTML file
3) Put the headlines into a RSS file.

For example, I want to go to this site and extract the headlines:
www.unstrung.com/section.asp?section_id=86

The problem is I do not know howto extract a headline from a HTML
file.
I mean HTML is not structured as XML, so I do not really know to solve
this problem. I notice that PHP has URL Functions to deal with HTML
file. For example, you have get_meta_tags () to extract meta tag
content attributes from a HTML file. But then, extract meta tag is
easy. With headlines, I don't really know where the headlines are on
a HTML file. Would anyone give me inputs on this?

This is not an impossible problem. If you look at Google News
(http://news.google.com/), they crawl the web and sort the headlines
on their site.

Thanks,
P. Ho
Jul 17 '05 #1
3 3920
On 3 Aug 2004 14:24:11 -0700, pe********@yaho o.com (Phong Ho) wrote:
I try to write a simple web crawler. It has to do the following:
1) Open an URL and retrieve a HTML file.
2) Extract news headlines from the HTML file
3) Put the headlines into a RSS file.

For example, I want to go to this site and extract the headlines:
www.unstrung.com/section.asp?section_id=86

The problem is I do not know howto extract a headline from a HTML
file.
I mean HTML is not structured as XML, so I do not really know to solve
this problem. I notice that PHP has URL Functions to deal with HTML
file. For example, you have get_meta_tags () to extract meta tag
content attributes from a HTML file. But then, extract meta tag is
easy. With headlines, I don't really know where the headlines are on
a HTML file. Would anyone give me inputs on this?

This is not an impossible problem. If you look at Google News
(http://news.google.com/), they crawl the web and sort the headlines
on their site.


Whilst entirely possible in PHP, I'd use Perl for this, as there are many
extremely useful modules for this sort of thing. HTML::TableExtr act,
HTML::Parser, WWW::Mechanize to name a few.

But make sure you're not infringing on their copyright by redistributing the
extracted headlines.

--
Andy Hassall / <an**@andyh.co. uk> / <http://www.andyh.co.uk >
<http://www.andyhsoftwa re.co.uk/space> Space: disk usage analysis tool
(v1.4.0 new 1st Aug 2004)
Jul 17 '05 #2
"Andy Hassall" wrote:
On 3 Aug 2004 14:24:11 -0700, pe********@yaho o.com (Phong Ho) wrote:
I try to write a simple web crawler. It has to do the following:
1) Open an URL and retrieve a HTML file.
2) Extract news headlines from the HTML file
3) Put the headlines into a RSS file.

For example, I want to go to this site and extract the headlines:
www.unstrung.com/section.asp?section_id=86

The problem is I do not know howto extract a headline from a HTML
file.
I mean HTML is not structured as XML, so I do not really know to solve
this problem. I notice that PHP has URL Functions to deal with HTMLfile. For example, you have get_meta_tags () to extract meta tag
content attributes from a HTML file. But then, extract meta tag iseasy. With headlines, I donít really know where the headlines

are on
a HTML file. Would anyone give me inputs on this?

This is not an impossible problem. If you look at Google News
(http://news.google.com/), they crawl the web and sort the headlines
on their site.


Whilst entirely possible in PHP, Iíd use Perl for this, as
there are many
extremely useful modules for this sort of thing.

HTML::TableExtr act, HTML::Parser, WWW::Mechanize to name a few.

But make sure youíre not infringing on their copyright by
redistributing the
extracted headlines.


Or do it the brute force way, using regular expressions. Just refer
to the manul for doing that. I donít think php as well developed
modules for doing this (vs. Perl) but I could be wrong.

If you know regular expressions, this is an easy thing to do, by the
way.

--
http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-Extract-...ict136182.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=454893
Jul 17 '05 #3
I do not have experience with Perl but I would like to ask a question
concerning Perl. I will use web hosting from a web hosting company,they
support Perl(CGI-BIN) but I won't have the root account. Do you need a
root account to install a Perl Module?

Also, anyone know if there is a sample in PHP to extract the headlines
from a HTML file on the internet?

Thanks

steve wrote:
"Andy Hassall" wrote:
> On 3 Aug 2004 14:24:11 -0700, pe********@yaho o.com (Phong Ho) wrote:
>
> >I try to write a simple web crawler. It has to do the following:
> >1) Open an URL and retrieve a HTML file.
> >2) Extract news headlines from the HTML file
> >3) Put the headlines into a RSS file.
> >
> >For example, I want to go to this site and extract the headlines:
> >www.unstrung.com/section.asp?section_id=86
> >
> >The problem is I do not know howto extract a headline from a HTML
> >file.
> >I mean HTML is not structured as XML, so I do not really know to

> solve
> >this problem. I notice that PHP has URL Functions to deal with HTML > >file. For example, you have get_meta_tags () to extract meta tag
> >content attributes from a HTML file. But then, extract meta tag is > >easy. With headlines, I donít really know where the headlines

> are on
> >a HTML file. Would anyone give me inputs on this?
> >
> >This is not an impossible problem. If you look at Google News
> >(http://news.google.com/), they crawl the web and sort the headlines
> >on their site.

>
> Whilst entirely possible in PHP, Iíd use Perl for this, as
> there are many
> extremely useful modules for this sort of thing.

HTML::TableExtr act,
> HTML::Parser, WWW::Mechanize to name a few.
>
> But make sure youíre not infringing on their copyright by
> redistributing the
> extracted headlines.
>


Or do it the brute force way, using regular expressions. Just refer
to the manul for doing that. I donít think php as well developed
modules for doing this (vs. Perl) but I could be wrong.

If you know regular expressions, this is an easy thing to do, by the
way.


Jul 17 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
6917
by: Jane Doe | last post by:
Hi I took a quick look in the archives, but didn't find an answer to this one. I'd like to display a list of HTML files in a directory, showing the author's name between brackets after the file name. I can successfully extract the TITLE section, but no luck with the AUTHOR part. Any idea why?
3
6002
by: Joe | last post by:
I'm trying to extract part of html code from a tag to a tag code begins with <span class="boldyellow"><B><U> and ends with TD><TD> <img src="http://whatever/some.gif"> </TD></TR></TABLE> I was thinking of using a regular expression however I having hard time getting the desired string. I use htmlSource = urllib.urlopen("http://address/") s = htmlSource.read() htmlSource.close()
6
2223
by: Selen | last post by:
I would like to be able to extract a BLOB from the database (SqlServer) and pass it to a browser without writing it to a file. (The BLOB's are word doc's, MS project doc's, and Excel spreadsheets. How can I do this?
0
1619
by: Vjay77 | last post by:
I posted this question, but I pressed 'post' and it disappeared. So once again: Problem: I need to go to lets say www.site.com/page.html Imagine that this html code is 6 mb long. I need to extract information between bytes 5000 and 5020.
9
7305
by: trihanhcie | last post by:
Hi, I would like to extract the text in an HTML file For the moment, I'm trying to get all text between <tdand </td>. I used a regular expression because i don't know the "format between <tdand </td> It can be : <tdtext1 </td> or
1
3665
by: steveyjg | last post by:
I want to extract the following data from a retrieved html file and store the information as strings. 'get the text of "title" <h1 id="test_title">title</h1> 'get the contents of the value attribute <input name="test_code" type="text" value='<object </object>' > 'get the text of "category" or value of c <div class="smallText">
1
3660
by: nkg1234567 | last post by:
I'm trying to extract HTML from a website in the form of a string, and then I want to extract particular elements from the string using the substr function: here is some sample code that I have thus far: use HTTP::Request::Common; use LWP::UserAgent; use LWP::Simple; $ua = LWP::UserAgent->new;
1
3068
by: veer | last post by:
Hi i am making a program in which i want to extract data from html file . Actually there are two dates on html file i want to extract these dates but the main probleum is that these dates are different on each file. A word "AKTIVA" is always comes before these dates. i made this by seaching the activa word but after this i am not getting any idea how these dates can be accessed. i use one another method by transfering the whole data of...
3
4102
rizwan6feb
by: rizwan6feb | last post by:
I am trying to extract php code from a php file (php file also contains html, css and javascript code). I am using the following regex for this <\?*?\?> but this doesn't cater quotation marks (single and double quotes) and comments, i mean how can i skip php tags inside a string (and comments). Please have a look at the following code <?php include("db.php"); $name=$_REQUEST;
0
8802
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
8711
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9175
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9075
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9017
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
5948
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
1
3158
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2522
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2111
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.