Convert ODP RDF to Static HTML Pages?

[DesignGuy]

I would like to download the RDF dump and generate static HTML pages (with
customizable headers and footers). I have only found one program called
iHierarchy that claims to do this ( http://simiax.com/ihier.html ) however
it is $199 and a demo to test is not available. Plus they're not answering
emails :(

Are there any other applications that will do this? Also, does any know or
care to guesstimate the size of the final static HTML output? I have a
dedicated server with about 60 GB of free space so hopefully there would be
room for the output.

I would need the program to maintain the exact ODP directory structure:

i.e. www.domain.com/ODP/Recreation/Outdoors/

[Me]

"DesignGuy" <do********@nowhere.com> wrote in message
news:vuW8d.316151$mD.193055@attbi_s02...

I would like to download the RDF dump and generate static HTML pages (with
customizable headers and footers). I have only found one program called
iHierarchy that claims to do this ( http://simiax.com/ihier.html ) however
it is $199 and a demo to test is not available. Plus they're not answering
emails :(

Are there any other applications that will do this? Also, does any know or
care to guesstimate the size of the final static HTML output? I have a
dedicated server with about 60 GB of free space so hopefully there would
be
room for the output.

I would need the program to maintain the exact ODP directory structure:

i.e. www.domain.com/ODP/Recreation/Outdoors/

Powerseek from http://www.focalmedia.net/ will do the job

Regards

Eddie

http://www.englandsportal.com

[John Doe]

On Wed, 06 Oct 2004 18:06:19 +0000, DesignGuy wrote:

I would like to download the RDF dump and generate static HTML pages
(with customizable headers and footers).

As long as you are complying with the proper attribution portion
of the "NOD" license that the ODP content is covered under, there are
quite a few dozen applications that can convert the RDF to HTML under
POSIX-compliant platforms (which excludes Microsoft Windows, of course).
I've been happily converting RDF to HTML for at least two years with some
perl glue.

[John Doe]

On Sun, 10 Oct 2004 15:06:51 -0400, John Doe wrote:

As long as you are complying with the proper attribution portion
of the "NOD" license that the ODP content is covered under

Oops, I should have clarified... if you're repurposing the content
from ODP, with your custom headers and footers, you have to include the
following attribution in every page:

<p><table border="0" bgcolor="#336600" cellpadding="3" cellspacing="0">
<tr>
<td>
<table width="100%" cellpadding="2" cellspacing="0" border="0">
<tr align="center">
<td><font face="sans-serif, Arial, Helvetica" size="2"
color="#FFFFFF">Help build the largest human-edited
directory on the web.</font></td></tr>
<tr bgcolor="#CCCCCC" align="center">
<td><font face="sans-serif, Arial, Helvetica" size="2"> <a
href="http://dmoz.org/cgi-bin/add.cgi?where=$cat">Submit
a Site</a> - <a href="http://dmoz.org/about.html"><b>Open Directory
Project</b></a> -
<a href="http://dmoz.org/cgi-bin/apply.cgi?where=$cat">Become
an Editor</a> </font>
</td></tr>
</table>
</td>
</tr>

This is documented here:

http://dmoz.org/become_an_editor/

[Philipp Lenssen]

DesignGuy wrote:

I would like to download the RDF dump and generate static HTML pages
(with customizable headers and footers). I have only found one
program called iHierarchy that claims to do this (
http://simiax.com/ihier.html ) however it is $199 and a demo to test
is not available. Plus they're not answering emails :(

Are there any other applications that will do this? Also, does any
know or care to guesstimate the size of the final static HTML output?
I have a dedicated server with about 60 GB of free space so hopefully
there would be room for the output.

I would need the program to maintain the exact ODP directory
structure:

i.e. www.domain.com/ODP/Recreation/Outdoors/

Just a Plan B suggestion here: screenscrape the content dynamically. No
RDF dump, no storage.

--
Google Blogoscoped
http://blog.outer-court.com

[John Doe]

On Mon, 18 Oct 2004 08:34:10 +0000, Philipp Lenssen wrote:

Just a Plan B suggestion here: screenscrape the content dynamically. No
RDF dump, no storage.

And risk getting your IP banned? Not a smart move. Google, Slashdot
and DMOZ all do this from time to time to keep abusers and abusive spiders
at bay.

One word: Don't.

[Philipp Lenssen]

John Doe wrote:

On Mon, 18 Oct 2004 08:34:10 +0000, Philipp Lenssen wrote:

Just a Plan B suggestion here: screenscrape the content
dynamically. No RDF dump, no storage.

And risk getting your IP banned? Not a smart move. Google, Slashdot
and DMOZ all do this from time to time to keep abusers and abusive
spiders at bay.

I'm using a mixture of Google API and screen-scraping and it worked
well so far (you can search, or browse):
http://findforward.com/?t=directory

After all it's called "open" directory, so I'm taking their word for it.

--
Google Blogoscoped
http://blog.outer-court.com

[Us]

"Philipp Lenssen" <in**@outer-court.com> wrote in message
news:2t*************@uni-berlin.de...

John Doe wrote:

On Mon, 18 Oct 2004 08:34:10 +0000, Philipp Lenssen wrote:

> Just a Plan B suggestion here: screenscrape the content
> dynamically. No RDF dump, no storage.

And risk getting your IP banned? Not a smart move. Google, Slashdot
and DMOZ all do this from time to time to keep abusers and abusive
spiders at bay.

I'm using a mixture of Google API and screen-scraping and it worked
well so far (you can search, or browse):
http://findforward.com/?t=directory

After all it's called "open" directory, so I'm taking their word for it.

http://dmoz.org/license.html

[Philipp Lenssen]

Us wrote:

"Philipp Lenssen" <in**@outer-court.com> wrote in message
news:2t*************@uni-berlin.de...

I'm using a mixture of Google API and screen-scraping and it worked
well so far (you can search, or browse):
http://findforward.com/?t=directory

http://dmoz.org/license.html

OK, where does it read that screen-scraping is not allowed? I'm
actually asking, because it sounds like the kind of legalese written
for lawyers, not webmasters (all upper-case?).

Let's face it, what Google and other search engines do all day is
screen-scraping. They extract information and present it on their site.
Like the Google cache. Oh wait, there's not even a "noarchive" thingie
in DMOZ.org.

--
Google Blogoscoped
http://blog.outer-court.com

[Us]

"Philipp Lenssen" <in**@outer-court.com> wrote in message
news:2t*************@uni-berlin.de...

Us wrote:

"Philipp Lenssen" <in**@outer-court.com> wrote in message
news:2t*************@uni-berlin.de...

>
> I'm using a mixture of Google API and screen-scraping and it worked
> well so far (you can search, or browse):
> http://findforward.com/?t=directory
> >

http://dmoz.org/license.html

OK, where does it read that screen-scraping is not allowed? I'm
actually asking, because it sounds like the kind of legalese written
for lawyers, not webmasters (all upper-case?).

Let's face it, what Google and other search engines do all day is
screen-scraping. They extract information and present it on their site.
Like the Google cache. Oh wait, there's not even a "noarchive" thingie
in DMOZ.org.

Your ignorance is matched by your poor eye sight. In fact only the last 2
paragraphs are in upper case and the terms and conditions should be clear
enough for anyone who is intending to use the data.

Incidently, Dmoz is a directory, unlike google that is a search engine and
does spider sites. Dmoz does not.

damnat quod non intelligunt

[Philipp Lenssen]

Us wrote:

"Philipp Lenssen" <in**@outer-court.com> wrote in message
news:2t*************@uni-berlin.de...

Us wrote:

>
http://dmoz.org/license.html
OK, where does it read that screen-scraping is not allowed? I'm
actually asking, because it sounds like the kind of legalese written
for lawyers, not webmasters (all upper-case?).

Your ignorance is matched by your poor eye sight. In fact only the
last 2 paragraphs are in upper case and the terms and conditions
should be clear enough for anyone who is intending to use the data.
So why don't you help me and point out the part where it reads I can't
do with FindForward.com what I'm doing now? Then I could adjust my site.

Incidently, Dmoz is a directory, unlike google that is a search
engine and does spider sites. Dmoz does not.

So? My point was that Dmoz lets itself be spidered by search engines
because it doesn't use the "noarchive" tag. FindForward.com is a search
engine (or meta engine, if you want). I'm adding functionality on top
of the Google API, and other web services.

[Follow-up to alt.internet.search-engines only]

[marathon]

On 19 Oct 2004 14:22:57 GMT, Philipp Lenssen in alt.internet.search-engines wrote:

Us wrote:

"Philipp Lenssen" <in**@outer-court.com> wrote in message
news:2t*************@uni-berlin.de...

> Us wrote:
> >> >
>> http://dmoz.org/license.html
>
> OK, where does it read that screen-scraping is not allowed? I'm
> actually asking, because it sounds like the kind of legalese written
> for lawyers, not webmasters (all upper-case?).

Your ignorance is matched by your poor eye sight. In fact only the
last 2 paragraphs are in upper case and the terms and conditions
should be clear enough for anyone who is intending to use the data. So why don't you help me and point out the part where it reads I can't
do with FindForward.com what I'm doing now? Then I could adjust my site.
Incidently, Dmoz is a directory, unlike google that is a search
engine and does spider sites. Dmoz does not.

So? My point was that Dmoz lets itself be spidered by search engines
because it doesn't use the "noarchive" tag. FindForward.com is a search
engine (or meta engine, if you want). I'm adding functionality on top
of the Google API, and other web services. [Follow-up to alt.internet.search-engines only]

Too bad, for some reason my newsreader isn't honouring this.

Very cute, you could have changed the followups at any time upthread,
but instead chose to do it now. Seems like someone is fearful of a
rebuttal...

--
marathon
"America is a stronger nation for the ACLU's uncompromising effort."
-- President John F. Kennedy

[Philipp Lenssen]

marathon wrote:

On 19 Oct 2004 14:22:57 GMT, Philipp Lenssen in
alt.internet.search-engines wrote:

So? My point was that Dmoz lets itself be spidered by search engines
because it doesn't use the "noarchive" tag. FindForward.com is a
search engine (or meta engine, if you want). I'm adding
functionality on top of the Google API, and other web services.

[Follow-up to alt.internet.search-engines only]

Too bad, for some reason my newsreader isn't honouring this.

Very cute, you could have changed the followups at any time upthread,
but instead chose to do it now. Seems like someone is fearful of a
rebuttal...

Actually, I typed lt.internet.search-engines so that caused some
confusion in my newsreader. In any case, when I restrict the follow-up,
then because I don't like crossposts. I happen to read along in both
newsgroups. If you want to stay on topic, then maybe you can give the
rebuttal you are talking about -- I'll be happy to hear it, as I'm here
to learn.

--
Google Blogoscoped
http://blog.outer-court.com

[marathon]

On 20 Oct 2004 07:43:22 GMT, Philipp Lenssen in
alt.internet.search-engines wrote:

marathon wrote:

On 19 Oct 2004 14:22:57 GMT, Philipp Lenssen in
alt.internet.search-engines wrote:

> So? My point was that Dmoz lets itself be spidered by search engines
> because it doesn't use the "noarchive" tag. FindForward.com is a
> search engine (or meta engine, if you want). I'm adding
> functionality on top of the Google API, and other web services.

> [Follow-up to alt.internet.search-engines only]

Too bad, for some reason my newsreader isn't honouring this.

Very cute, you could have changed the followups at any time upthread,
but instead chose to do it now. Seems like someone is fearful of a
rebuttal...

Actually, I typed lt.internet.search-engines so that caused some
confusion in my newsreader. In any case, when I restrict the follow-up,
then because I don't like crossposts. I happen to read along in both
newsgroups. If you want to stay on topic, then maybe you can give the
rebuttal you are talking about -- I'll be happy to hear it, as I'm here
to learn.

Nope not from me, I want to read the rebuttal as much as you appear to,
as I can always learn something new, to.

BTW cross posting isn't necessarily bad -- it's allowable when both are
in context for either/or newsgroups. In any decent newsreader,
cross posts can be marked read in the other groups. ;)

--
marathon
I came, I saw, I deleted all your files.

[Us]

"marathon" <[e-mail]> wrote in message
news:qn************@barnyard.sweetpig.dyndns.org.. .

On 20 Oct 2004 07:43:22 GMT, Philipp Lenssen in
alt.internet.search-engines wrote:

marathon wrote:

On 19 Oct 2004 14:22:57 GMT, Philipp Lenssen in
alt.internet.search-engines wrote: So? My point was that Dmoz lets itself be spidered by search engines
> because it doesn't use the "noarchive" tag. FindForward.com is a
> search engine (or meta engine, if you want). I'm adding
> functionality on top of the Google API, and other web services.

> [Follow-up to alt.internet.search-engines only]

Too bad, for some reason my newsreader isn't honouring this.

Very cute, you could have changed the followups at any time upthread,
but instead chose to do it now. Seems like someone is fearful of a
rebuttal...

Actually, I typed lt.internet.search-engines so that caused some
confusion in my newsreader. In any case, when I restrict the follow-up,
then because I don't like crossposts. I happen to read along in both
newsgroups. If you want to stay on topic, then maybe you can give the
rebuttal you are talking about -- I'll be happy to hear it, as I'm here
to learn.

Nope not from me, I want to read the rebuttal as much as you appear to,
as I can always learn something new, to.

BTW cross posting isn't necessarily bad -- it's allowable when both are
in context for either/or newsgroups. In any decent newsreader,
cross posts can be marked read in the other groups. ;)

--
marathon
I came, I saw, I deleted all your files.

"
2. Attribution Requirement. As a material condition of this Open Directory
License, you must provide the below applicable attribution statements on (1)
all copies of the Open Directory, in whole or in part, and derivative works
thereof which are either distributed (internally or otherwise) or published
(made available on the Internet and/or internally over any internal
network/intranet or otherwise), whether distributed or published
electronically, on hard copy media or by any other means, and (2) on any
program/web page from which you directly link to/access any information
contained within the Open Directory, in whole or in part, or any derivative
work thereof: "

Or in plain English you are welcome to use ODP data so long as you place the
correct attribution on each page ( as google does in it's directory"

You also mentioned the "Open" part. It is a common misconception that open
source software is free. It may be but it doesn't have to be.

Eddie

[Philipp Lenssen]

marathon wrote:

BTW cross posting isn't necessarily bad -- it's allowable when both
are in context for either/or newsgroups. In any decent newsreader,
cross posts can be marked read in the other groups. ;)

Hmm, that sounds very useful. I'm using XanaNews.

[Philipp Lenssen]

Us wrote:

Or in plain English you are welcome to use ODP data so long as you
place the correct attribution on each page ( as google does in it's
directory"

Just did, thanks.
<http://www.findforward.com>

[Colin Wilson]

"Philipp Lenssen" <in**@outer-court.com> wrote in message news:<2t*************@uni-berlin.de>...

marathon wrote:

BTW cross posting isn't necessarily bad -- it's allowable when both
are in context for either/or newsgroups. In any decent newsreader,
cross posts can be marked read in the other groups. ;)

Hmm, that sounds very useful. I'm using XanaNews.

BTW, You can turn this feature on/off in XanaNews. Tools/Options,
General tab. 'Automatic Crosspost Detection' tab.

--
Colin - author of XanaNews