473,405 Members | 2,261 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,405 software developers and data experts.

Help parsing continuous text from the html parsed page.

I've got this problem, where I need to extract an articles from many
different news sources (webpages).
I need to write some logic which would know how to extract the text
only and not a garbage around.

I am able to parse the text from html and all I get now is the text
from the site. But there is lot of short text, lot of advertising
text and garbage all around. In the middle is the article I need. I
was wondering on writing some program which would be smart enough to
figure out where article starts and ends... but apparently it's not
as easy as I thought. I could somehow depending on spaces and dots
make some fairly workable way of doing it. But I am not even sure how
I would go about that.

So when you look at this text in quotes, you'll see the article in the
middle. I will get these articles from many different sources so
wording will be always different, but it would generaly look
something like this:

The Globe and Mail
@import url('/cssv3/net5upcss.css');
if (navigator.userAgent.indexOf('MSIE 5') != -1) {
document.write(" ")
}

if((navigator.userAgent.indexOf('MSIE') != -1) &&
(navigator.userAgent.indexOf('Mac') != -1)) {
document.write(" ")
}
News
Investing
Technology
Vehicles
Careers
Home |
Business |
International |
Sports |
Travel |
Entertainment
d.body.style.visibility="hidden";
Search Site

Search Tips
showTextAd("USERNAME");



A strike at Bell Canada has been at least temporarily averted after the country's largest phone company made a new offer to 7,000
technicians that will be voted on over the next several weeks.
Joel Carr of the Communications Energy and Paperworkers Union of Canada, which represents the technicians, said the offer is
significant enough under CEP rules to warrant a vote by workers.
While he didn't discuss details, he said in general the offer is
“very much improved” from Bell's June offer,
which was rejected by 62 per cent of those that cast ballots.
The CEP filed notice of strike on Monday, meaning workers could have been on the pickets Thursday afternoon. Bell made what it called its
final offer late yesterday, and local union leaders from across
Ontario and Quebec meet tomorrow morning in Toronto to issue a
recommendation to workers. Pensions are the No. 1 issue.
“It represents the limit to which the company can go in meeting the priorities of the union and its members,” Ellen
Malcolmson, senior vice-president of operations at Bell, said in a
statement.
Voting, Mr. Carr said, will begin as soon as possible and take about two weeks.
Technicians represent about one sixth of Bell's work force of 42,500. The workers went on strike for five weeks in 1999, fighting
for job security, and the resulting contract expired last November.
There can be no strike while the workers vote on the new offer.
Bell is owned by BCE Inc. of Montreal, which has a majority stake in Bell Globemedia, owner of The Globe and Mail and CTV television.
Shares of BCE were up 25 cents or 1 per cent to $26.36 at midday on
the Toronto Stock Exchange.
While the Bell negotiations could reach a resolution, a strike at Aliant Telecom Inc. is nearly 13 weeks old. Aliant, majority owned by
Bell, is the dominant phone company in Atlantic Canada and 4,300 of
its workers — two-thirds of the work force — have
been on the picket lines since April 23.
Union leaders and the company met last week for the first time since the strike began, but no progress was made and no other meetings are
planned.











if (!mt_8734802b) {if (parent.mt_8734802b) {var mt_8734802b = parent.mt_8734802b;} else {var mt_8734802b=false;}} if (mt_8734802b) {
document.write('');
}

if (!mt_8734802c) {if (parent.mt_8734802c) {var mt_8734802c = parent.mt_8734802c;} else {var mt_8734802c=false;}} if (mt_8734802c) {
document.write('');
}



All I need is the way to figure out that article starts at from "A
strike at Bell Canada has been " and ends at "and no other meetings
are planned.", and that way it would be easy to extract important
text, but how to write this logic???
Uff...

Does anyone have any idea or can give me some help on how to write
some smart text extractor from the page?

Thanks a lot.

Martin Ho

Copy of this problem at:
http://www.groupsrv.com/dotnet/viewtopic.php?t=57362

Nov 20 '05 #1
2 1223
Wow, interesting. I would probably try to somehow count the spaces and if
there is no space every at least for 20 characters then get rid of them.
That way it could eventually somewhat work.
But this will need someone smarter then I to answer this question.
K.

Posted Via Usenet.com Premium Usenet Newsgroup Services
----------------------------------------------------------
** SPEED ** RETENTION ** COMPLETION ** ANONYMITY **
----------------------------------------------------------
http://www.usenet.com
Nov 20 '05 #2
Hi Martin,

Forget this one, this is not HTML this is JavaScript document Write which
uses some HTML, it is almost impossible to find a way to extract those text.

Large search companies have problems with finding things in this kind of
pages, so that you find a way when you are not extremely well in all aspects
of JavaScript and HTML is possible however not so sure.

I hope this helps anyway?

Cor
Nov 20 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

14
by: Viktor Rosenfeld | last post by:
Hi, I need to create a parser for a Python project, and I'd like to use process kinda like lex/yacc. I've looked at various parsing packages online, but didn't find anything useful for me: -...
6
by: egg | last post by:
Dear Sir/Madam, I'm about to start a project, the idea is to have a web interface to let user specify the URL of a target HTML document, based on that URL, retrieved and parsed the HTML document...
2
by: LoserInYourFaceEngineer | last post by:
Hello All: I'm having trouble with a recursive function. The function is supposed to identify nested folders in a hierarchical folder structure. The function "searchForFolders()" is...
16
by: Terry | last post by:
Hi, This is a newbie's question. I want to preload 4 images and only when all 4 images has been loaded into browser's cache, I want to start a slideshow() function. If images are not completed...
9
by: wardy | last post by:
I'm trying to undestand the impact of using content negotiation when rendering my Web pages to various different browsers as I would like to use the XHTML Strict DOCTYPE declaration. Reading the...
9
by: ankitdesai | last post by:
I would like to parse a couple of tables within an individual player's SHTML page. For example, I would like to get the "Actual Pitching Statistics" and the "Translated Pitching Statistics"...
8
by: Pratik Patel | last post by:
Hello, I used innerHTML to assign HTML content. but in my HTML page content have also some javascript function and it will run when page load. bu when HTML code assgin thru innerHTML then this...
4
by: poisonedapple | last post by:
Hi , I have posted this in one of my other thread but I am posting it again since it is infact on a different topic I got the table extracted and I have a huge document full of tables. From...
2
by: embz | last post by:
this post concerns three pages. 1. this page: http://www.katherine-designs.com/sendemail.php i get the following errors: a lot of it seems to deal with the PHP code i inserted to the page....
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.