Unstructured HTML extraction

[dayzman]

Hi,

I'm interested in a program that extracts the structure of unstructured
HTML documents. The program should be able to make good estimates about
different font styles used to represent headings, for example, some may
use <font size = 24> for headings and some may use <h1>, in the end,
both should output the same structure. The output can be in XML or
other formats. Manual driving should remain minimal. Does anyone know
of such program (preferably open-source)?

Cheers,
Michael

[Nick Kew]

In article <11**********************@z14g2000cwz.googlegroups .com>,
da*****@hotmail.com writes:

Hi,

I'm interested in a program that extracts the structure of unstructured
HTML documents. The program should be able to make good estimates about
different font styles used to represent headings, for example, some may
use <font size = 24> for headings and some may use <h1>, in the end,
Not quite sure what you mean, but does http://valet.webthing.com/access/
seem relevant to you?
both should output the same structure. The output can be in XML or
other formats.

Output is XML. What you see on the Web is transformed with XSLT to
HTML or RDF.

--
Nick Kew

[dayzman]

Hi,

Sorry for not making myself clear. What I'm trying to find is a program
that handles the inconsistent styles used on webpages to imply, e.g.,
headings -- where some people may use large fonts, some may use the
<h1> tag, and some may use block letters etc. The program needs to be
able to recognise the implication of the different styles used, and
"cleans up" the structure and outputs it in a readable format.
I hope it is more clear this time.

Cheers,
Michael

[dayzman]

Hi,

Sorry for not making myself clear. What I'm trying to find is a program
that handles the inconsistent styles used on webpages to imply, e.g.,
headings -- where some people may use large fonts, some may use the
<h1> tag, and some may use block letters etc. The program needs to be
able to recognise the implication of the different styles used, and
"cleans up" the structure and outputs it in a readable format.
I hope it is more clear this time.

Cheers,
Michael

[Nick Kew]

In article <11*********************@f14g2000cwb.googlegroups. com>,
da*****@hotmail.com writes:

Sorry for not making myself clear. What I'm trying to find is a program
that handles the inconsistent styles used on webpages to imply, e.g.,
headings -- where some people may use large fonts, some may use the
<h1> tag, and some may use block letters etc. The program needs to be

That's a hard AI problem. Incompetent authors abuse presentational
markup when they mean a heading, but someone might also legitimately
present something that isn't a heading as big and bold. That's why
tools like AccessValet highlight cases that look like a bogus
heading (e.g. <p><big>FOO</big></p>) for a human user to decide.

This topic has been done to death on appropriate newsgroups, as
google would have told you. Redirecting this.

P.S. Please fix your news client not to post everything twice.

--
Nick Kew