parse URL (href) from xhtml, xhtml -> text, for data

hawat.thufir

Given an xhtml file, how can I "export" the data to plain-text? That is,
I want:

google www.google.com
Whereas, if I copy and paste what the browser shows, I lose the URL and
end up with:

google
The idea is that I want to import the data to MySQL using the mysqlimport
command, but mysqlimport requires plain-text. The xhtml file in question:

[thufir@localhost Desktop]$ cat raw.xhtml -n
1 <?xml version="1.0" encoding="UTF-8"?><!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml"><head><meta
http-equiv="content-type" content="text/html; charset=utf-8" /><title
/><meta name="generator" content="StarOffice/OpenOffice.org XSLT
(http://xml.openoffice.org/sx2ml)" /><meta name="created"
content="2006-02-07T15:19:17" /><meta name="changed"
content="2006-02-07T15:36:55" /><base href="." /><style type="text/css">
3 @page { }
4 table { border-collapse:collapse; border-spacing:0;
empty-cells:show }
5 td, th { vertical-align:top; }
6 h1, h2, h3, h4, h5, h6 { clear:both }
7 ol, ul { padding:0; }
8 * { margin:0; }
9 *.ta1 { }
10 *.ce1 { font-family:Courier; color:#000000;
font-size:10pt; font-style:normal; text-shadow:none; font-weight:normal; }
11 *.ce2 { font-family:Courier; color:#000000; }
12 *.Default { font-family:'Bitstream Vera Sans'; }
13 *.Heading { font-family:'Bitstream Vera Sans';
text-align:center ! important; font-size:16pt; font-style:italic;
font-weight:bold; }
14 *.Heading1 { font-family:'Bitstream Vera Sans';
text-align:center ! important; font-size:16pt; font-style:italic;
font-weight:bold; }
15 *.Result { font-family:'Bitstream Vera Sans';
font-style:italic; font-weight:bold; text-decoration:underline; }
16 *.Result2 { font-family:'Bitstream Vera Sans';
font-style:italic; font-weight:bold; text-decoration:underline; }
17 *.co1 { width:0.8925in; }
18 *.ro1 { height:0.1756in; }
19 *.ro2 { height:0.1681in; }
20 </style></head><body dir="ltr"><table border="0"
cellspacing="0" cellpadding="0" class="ta1"><colgroup><col width="99"
/></colgroup><tr class="ro1"><td style="text-align:left;width:0.8925in; "
class="ce1"><p><a href="http://www.google.com/">google
</a>Â*Â*</p></td></tr><tr class="ro2"><td
style="text-align:left;width:0.8925in; " class="ce2" /></tr><tr
class="ro2"><td style="text-align:left;width:0.8925in; " class="ce2"
/></tr></table></body></html>[thufir@localhost Desktop]$ date
Tue Feb 7 15:52:34 EST 2006
[thufir@localhost Desktop]$

thanks,
Thufir

Feb 7 '06 #1

Subscribe Post Reply

2278

Joe Kesselman

First, you need to define what portions of the document are "data". It
sounds like what you want is just the links; is that correct?

If so, you need to search for <a> elements that have an href attribute,
pull out their content (which may be arbitrarily complex markup, please
remember -- rich text, images, etc. -- you need to define how much of
that you want to return and how you want it presented!), pull the value
of the href attribute, and report that pair of values.

Assuming that description of your problem is correct, you can do this by
writing a program that uses an XML parser and the SAX or DOM APIs, or
you can write an XSLT stylesheet such as the following. (WARNING: UNTESTED.)

<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="text" version="1.0" encoding="UTF-8" />

<xsl:template match="/">
<xsl:apply-templates select="//a[@href]"/>
</xsl:template>

<xsl:template match="a[@href]">
<xsl:value-of select="."/>
<xsl:text> </xsl:text>
<xsl:value-of select="@href"/>
<xsl:text>
</xsl:text>
</xsl:template>

</xsl:stylesheet>

Feb 8 '06 #2

hawat.thufir

Joe Kesselman wrote:

First, you need to define what portions of the document are "data". It
sounds like what you want is just the links; is that correct?
I like your approach: asking what is "data" in this case. Yes, I'm
after "just" the links for the example given. However, if I could get
the pair of values, "google" and "http://www.google.com" that'd be
stage two. For now, yes, I'd be happy with just the links.
If so, you need to search for <a> elements that have an href attribute,
pull out their content (which may be arbitrarily complex markup, please
remember -- rich text, images, etc. -- you need to define how much of
that you want to return and how you want it presented!), pull the value
of the href attribute, and report that pair of values.
Not sure I follow you there, I'm not after the actual google page,
simply the URL.
Assuming that description of your problem is correct, you can do this by
writing a program that uses an XML parser and the SAX or DOM APIs, or
you can write an XSLT stylesheet such as the following. (WARNING: UNTESTED.)

...

Right, thanks for writing a transform, I get the gist. That's actually
a big deal, I've read a tad about XSLT but it seemed arcane until just
now.

How do I get plain-text from the result, though? The result will be
XML, but not XHTML, which is a step in the right direction. PHP or
similar would be required to parse the XML resultant to get a
plain-text file with the link?
-Thufir

Feb 8 '06 #3

Martin Honnen

ha**********@gmail.com wrote:

Right, thanks for writing a transform, I get the gist. That's actually
a big deal, I've read a tad about XSLT but it seemed arcane until just
now.

How do I get plain-text from the result, though?

Joe's XSLT stylesheet has
<xsl:output method="text" version="1.0" encoding="UTF-8" />
so the output method is text and not XML.
XSLT can produce XML or HTML or text depending on the output method.
--

Martin Honnen
http://JavaScript.FAQTs.com/

Feb 8 '06 #4

Joe Kesselman

> How do I get plain-text from the result, though?

Note that the <xsl:output> statement in my example says to produce text
output. That says the output should be a free-form text stream rather
than XML. (Exactly how that differs from XML or HTML output modes is
described in the XSLT spec, if you want the details.)

I've said the text should be encoded as UTF-8; if you want the output in
a different encoding, that too can be specified via xsl:output. (Not all
processors support all encodings, admittedly.)
As I said, this is just one possible approach. You could hand-code a
solution almost as trivially, but I think I'm going to leave that as a
homework assignment for now. <smile/>

Feb 8 '06 #5

hawat.thufir

Joe Kesselman wrote:

How do I get plain-text from the result, though?
Note that the <xsl:output> statement in my example says to produce text
output.

Pardon, I didn't notice that until you mentioned it--thanks!

... I've said the text should be encoded as UTF-8; if you want the output in
a different encoding, that too can be specified via xsl:output. (Not all
processors support all encodings, admittedly.)
As I said, this is just one possible approach. You could hand-code a
solution almost as trivially, but I think I'm going to leave that as a
homework assignment for now. <smile/>

Doh!
Thanks for the help :)
I have to install a JRE and Saxon (?), then I'll give it a go.
-Thufir

Feb 8 '06 #6

hawat.thufir

Joe Kesselman wrote:

How do I get plain-text from the result, though?
Note that the <xsl:output> statement in my example says to produce text
output.

Pardon, I didn't notice that until you mentioned it--thanks!

... I've said the text should be encoded as UTF-8; if you want the output in
a different encoding, that too can be specified via xsl:output. (Not all
processors support all encodings, admittedly.)
As I said, this is just one possible approach. You could hand-code a
solution almost as trivially, but I think I'm going to leave that as a
homework assignment for now. <smile/>

Doh!
Thanks for the help :)
I have to install a JRE and Saxon (?), then I'll give it a go.
-Thufir

Feb 8 '06 #7

hawat.thufir

Joe Kesselman wrote:

How do I get plain-text from the result, though?
Note that the <xsl:output> statement in my example says to produce text
output.

Pardon, I didn't notice that until you mentioned it--thanks!

... I've said the text should be encoded as UTF-8; if you want the output in
a different encoding, that too can be specified via xsl:output. (Not all
processors support all encodings, admittedly.)
As I said, this is just one possible approach. You could hand-code a
solution almost as trivially, but I think I'm going to leave that as a
homework assignment for now. <smile/>

Doh!
Thanks for the help :)
I have to install a JRE and Saxon (?), then I'll give it a go.
-Thufir

Feb 8 '06 #8

Similar topics

cut strings and parse for images

by: Andreas Volz | last post by:

Hi, I used SGMLParser to parse all href's in a html file. Now I need to cut some strings. For example: http://www.example.com/dir/example.html Now I like to cut the string, so that only...

Python

MS XML DOM component can't parse reference

by: laughlin | last post by:

I'm using MS's XML DOM component to parse what I thought was XHTML. For some reason, the parser doesn't like the following reference that includes a couple of parameters following the "?" and...

.NET Framework

How to parse a XML doc with HTML tags within the texts

by: Francesco Moi | last post by:

Hi. I must parse this XML document: -------------- <doc> <item> <name>Jerry</name> <message>Hi<br>My name is Jerry</message> </item> </doc>

.NET Framework

XHTML: browser rendering parse tree instead. Why?

by: Neil Zanella | last post by:

Hello, When I save the following file with the .xhtml or .xml extension I get the XML parse tree and the following message instead of the actual document. This XML file does not appear to...

HTML / CSS

How to parse xmlDocument with XPath

by: John Barring | last post by:

Hi All, I am new to XPath stuff. I want parse XMLDocument with XPath and find out subset of information. If you look at following xml, for i.e how can i retrieve subset information such as...

.NET Framework

Regular Expression to Parse HTML

by: Charles Law | last post by:

Does anyone have a regex pattern to parse HTML from a stream? I have a well structured file, where each line is of the form <sometag someattribute='attr'>text</sometag> for example <SPAN...

Visual Basic .NET

Parse Error

by: Ben Allen | last post by:

Hi, Im currently getting the error: Parse error: parse error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING in...

PHP

parse xhtml with xpath???

by: Lore Leunoeg | last post by:

Hello Can I parse an XHTML Document with XPath? I tried the following expressions with the following XHTML Document with n o result: Removing the doctype and the namespace (xmlns) statemens...

.NET Framework

Use C++ to parse HTML

by: Bo Yang | last post by:

Hi, guys. I am now developing an application in which I need to fetch some html page, and then parsing it to get some intended content in it. Because HTML is not a standard XML format, so I am...

C / C++

Problem with location.href and frames

by: August Karlstrom | last post by:

Hi everyone, I have some problems loading a page into a frame from a different frame. In Firefox and Explorer the lower frame displays "Test..." but in some older version of Safari it is left...

Javascript

Is that possible of reading the .csv file in column wise and the column have different lengths ?

by: Sonnysonu | last post by:

This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

C / C++

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...

General

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...

Windows Server

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...

C / C++

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

Online Marketing

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

Windows Server

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

General

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

Microsoft Access / VBA

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...

C# / C Sharp