How do search engines index multilingual content?

[Manfred Kooistra]

I am building a website with identical content in four different
languages. On a first visit, the search engine determines the language
of the content by the IP address of the visitor. What the user sees is
content in only one language at a time. He or she can then switch to
another language and set this as the preferred language, but again he
or she sees content in only this one other language.

The question now is: How do I get search engines to index ALL of the
content, in all languages?

Should I include the non-displayed content in DIVs with display set to
"none" (like we used to include complete websites in the noframes tag)?
Or do search engines ignore invisible DIVs?

Or can I somehow detect that a search engine is visiting and deliver a
page with the complete content of all four languages in it? Or would
that get me banned?

Or do I have to rely on the search engine following the local links to
the pages in the other languages? This might be a problem, because the
varying content is always displayed on the same page, so the URI stays
the same, and only one parameter changes, thus:
"content.php?la nguage=oneoffou rlanguages". In fact it might even be
impossible, because I do not want to transfer the language information
through the URL via GET, but want to send it through a form via POST.
So the URI is exactly the same for all languages (at least in the
version I am aiming at).

If you have solved this problem on your website or know how to go about
it, I'd be grateful for some help.

[Jukka K. Korpela]

Manfred Kooistra wrote:

I am building a website with identical content in four different
languages.
Fine. You have explicitly linked the different versions to each other,
right? With reasonable link texts like the name of the page in the other
language, or, as tolerable option, the name of the language in the
language itself, right? No flags, no dropdowns, mm'kay?
On a first visit, the search engine determines the language
of the content by the IP address of the visitor.
No it does not. The idea is absurd. There is no visitor (beyond the
search engine itself) when a search engine indexes your content.
Besides, using the IP address to determine the language of a person is
absurd, too.
What the user sees is
content in only one language at a time.
Fine, but he should have simple access (links) to the other versions, too.
He or she can then switch to
another language and set this as the preferred language,
Setting a preferred language would take place via a URL string or via
cookies. That would be extra bonus to some (many) users, but the solid
basis needs to be built first.
The question now is: How do I get search engines to index ALL of the
content, in all languages?
The way you make them find all of your pages in general: using links.
Should I include the non-displayed content in DIVs with display set to
"none" (like we used to include complete websites in the noframes tag)?
No, that would be absurd and destructive (especially when your style
sheet is not used).
Or do search engines ignore invisible DIVs?
They may, or they may not, or they may punish the page for suspected
keyword spamming or cloaking.
Or can I somehow detect that a search engine is visiting and deliver a
page with the complete content of all four languages in it?
Some indexing robots can be detected heuristically. But don't do it.
Or would that get me banned?
Hopefully yes.
Or do I have to rely on the search engine following the local links to
the pages in the other languages?
That's the general idea.
This might be a problem, because the
varying content is always displayed on the same page, so the URI stays
the same, and only one parameter changes, thus:
"content.php?la nguage=oneoffou rlanguages".
If you have reasons to suspect that this is a problem, then don't do
that. But I wouldn't be worried about search engines that ignore pages
with a simple query part of the form ?foo=bar - they probably exist, but
they are losers in the search engine competition.
In fact it might even be
impossible, because I do not want to transfer the language information
through the URL via GET, but want to send it through a form via POST.
Just don't do that. Simple, eh?
So the URI is exactly the same for all languages (at least in the
version I am aiming at).

That's a completely wrong idea. You might, however, use an _additional_
generic URL that is resolved to one of specific URLs, via language
negotiation at the HTTP level. See
http://www.cs.tut.fi/~jkorpela/multi/

[Philip Ronan]

Manfred Kooistra wrote:

I am building a website with identical content in four different
languages. On a first visit, the search engine determines the language
of the content by the IP address of the visitor.
I don't understand what you mean by that. The best way of specifying the
language of an HTML page is with a lang attribute in the HTML tag (e.g.,
<HTML lang="en">)
What the user sees is
content in only one language at a time. He or she can then switch to
another language and set this as the preferred language, but again he
or she sees content in only this one other language.

The question now is: How do I get search engines to index ALL of the
content, in all languages?
Use <LINK> elements to indicate where the oehr translations can be found.

<http://www.w3.org/TR/REC-html40/struct/links.html#edef-LINK>
Should I include the non-displayed content in DIVs with display set to
"none" (like we used to include complete websites in the noframes tag)?
Or do search engines ignore invisible DIVs?
They generally penalize you for doing that. This is a frequently abused
technique for stuffing keywords into web pages in order to rank higher in the
result listings.
Or can I somehow detect that a search engine is visiting and deliver a
page with the complete content of all four languages in it? Or would
that get me banned?
That's called "cloaking", and is also frowned upon by search engines.
Or do I have to rely on the search engine following the local links to
the pages in the other languages? This might be a problem, because the
varying content is always displayed on the same page, so the URI stays
the same, and only one parameter changes, thus:
"content.php?la nguage=oneoffou rlanguages".

So the URI is actually different. But I'm not sure this is the best way of
doing things. Apache servers have some very useful built-in features for this
sort of thing.

<http://www.google.com/search?q=apache %20content%20ne gotiation>

--
philronan [@] blueyonder [dot] co [dot] uk

[Jukka K. Korpela]

Philip Ronan wrote:

The best way of specifying the
language of an HTML page is with a lang attribute in the HTML tag (e.g.,
<HTML lang="en">)

The lang attribute is recommendable, but it has _no_ verified effect on
search engines.

The question now is: How do I get search engines to index ALL of the
content, in all languages?

Use <LINK> elements to indicate where the oehr translations can be found.

There is no evidence that shows that search engines utilize such <LINK>
elements. Surely normal links (<A> elements) are better, since they are
much more widely recognized by browsers _and_ search engines.

[Philip Ronan]

Jukka K. Korpela wrote:

Philip Ronan wrote:

The best way of specifying the
language of an HTML page is with a lang attribute in the HTML tag (e.g.,
<HTML lang="en">)
The lang attribute is recommendable, but it has _no_ verified effect on
search engines.

What do you mean by "verified"? Google mention that some pages provide
insufficient context for guessing the language of a web page. See
<http://www.google.co.u k/intl/en/help/faq_translation .html#link>, for
example:

Why don't all the results in translatable languages have the
"Translate" link?

We only offer the "Translate" link when we have a high degree of
confidence about the language of the selected page. Some pages may
contain multiple languages or insufficient text to provide a high
degree of certainty about the language in which they were written.
Are you suggesting Google don't use lang attributes to assist in this
process? Do you think they prefer to use a whole bunch of syntactic analysis
and word frequency tools to make this decision instead?

Use <LINK> elements to indicate where the oehr translations can be found.

There is no evidence that shows that search engines utilize such <LINK>
elements.

Really? What about <http://www.google.com/webmasters/bot.html#whatli nks>:
12. What kinds of links does Googlebot follow?

Googlebot follows HREF links and SRC links.

Are you saying that a <LINK> element with an href attribute is somehow *not*
an HREF link? Please explain.
Surely normal links (<A> elements) are better, since they are
much more widely recognized by browsers _and_ search engines.

That would be a useful addition, but not an absolute necessity as far as
search engines are concerned. Perhaps a pop-up menu would be neater than a
long list of A links.

--
philronan [@] blueyonder [dot] co [dot] uk

[Andreas Prilop]

On Mon, 30 Jan 2006, Philip Ronan wrote:

The lang attribute is recommendable, but it has _no_ verified effect on
search engines.
What do you mean by "verified"? Google mention that some pages provide
insufficient context for guessing the language of a web page. See

^^^^^^^^ !! <http://www.google.co.u k/intl/en/help/faq_translation .html#link>,
They don't mention the LANG attribute here.
Are you suggesting Google don't use lang attributes to assist in this
process? Do you think they prefer to use a whole bunch of syntactic analysis
and word frequency tools to make this decision instead?

I'm afraid, yes. Until recently, Google did the same to *guess*
the encoding (charset) of a page instead of reading the HTTP or META
charset parameter. Google Groups still ignore the charset parameter
of Usenet articles. Instead they use the group name and I-don't-know-
what-else to select an encoding for an article.

Example:
http://www.seekport.de/help/webmaster_tips.html#Sprache
mentions only a META tag. These simpletons don't know the LANG
attribute of HTML.

--
Netscape 3.04 does everything I need, and it's utterly reliable.
Why should I switch? Peter T. Daniels in <news:sci.lan g>

[Philip Ronan]

Andreas Prilop wrote:

On Mon, 30 Jan 2006, Philip Ronan wrote:

The lang attribute is recommendable, but it has _no_ verified effect on
search engines.
What do you mean by "verified"? Google mention that some pages provide
insufficient context for guessing the language of a web page. See

^^^^^^^^ !!

<http://www.google.co.u k/intl/en/help/faq_translation .html#link>,

They don't mention the LANG attribute here.

So read between the lines. It's quite obvious Google uses certain technqiues
to ascertain the language of a web page. Examining lang attributes (where
available) is an obvious method for achieving this with the least effort.

Here's something you can try: click the "Advanced search" link in Google, and
search for pages in English that contain the phrase "Der Spindoktor". Then
take a look at the lang attribute in the HTML tag of the top result.
Google Groups still ignore the charset parameter
of Usenet articles. Instead they use the group name and I-don't-know-
what-else to select an encoding for an article.

That's an inevitable problem caused by putting multiple articles (with
different charsets) in a single web page. Google Groups has plenty of other
problems, but this has nothing to do with lang attributes.

--
philronan [@] blueyonder [dot] co [dot] uk

[Alan J. Flavell]

On Mon, 30 Jan 2006, Philip Ronan wrote:

Andreas Prilop wrote:

Google Groups still ignore the charset parameter of Usenet
articles. Instead they use the group name and I-don't-know-
what-else to select an encoding for an article.

That's an inevitable problem caused by putting multiple articles
(with different charsets) in a single web page.

I can't agree.

Mozilla's Bugzilla made the same mistake, and some of the
charset-related bug reports are sheer incomprehensibl e as a
consequence - they contain a mish-mash of Chinese, Cyrillic and
whatever else, in their different encodings, served out as raw bytes.
But the mistake was made many years back...

At least their discussion shows that they have recognised their
mistake, and understand how to correct it - mapping the various
encodings into Unicode, and serving out the results accordingly -
probably in utf-8.

(This might cause problems for people who are discussing the finer
details of Han unification, but that can't be helped now.)

Google have already, in effect, implemented something like that for
indexing web content. Otherwise it wouldn't be possible to find texts
in koi8-r and Windows-1251 when searching with a utf-8-encoded query:
the kind of problems that Andreas was reporting some years back with
various search engines, which (to put it briefly) made a query in one
encoding, and only returned pages which used that same encoding.

They just need to apply the same principle to what their ggroups
thingy is serving out. Admittedly, ggroups have *other*, *serious*,
problems to attend to first, such as encouraging their users to follow
netiquette - to at least the extent needed to get them out of the
widespread killfiling that they've already earned. But I digress.

[Andreas Prilop]

On Mon, 30 Jan 2006, Philip Ronan wrote:

So read between the lines.
Never. I prefer to read the actual text. Internet Explorer reads
between the lines to guess the Content-Type, etc.
It's quite obvious Google uses certain technqiues
to ascertain the language of a web page.
But what are "certain techniques"?
Examining lang attributes (where
available) is an obvious method for achieving this with the least effort.
Yes. But many (including Google) seem to think "Warum einfach, wenn's
auch kompliziert geht?"
That's an inevitable problem caused by putting multiple articles (with
different charsets) in a single web page.
No, it isn't. All articles are converted to UTF-8 by Google and
presented to you in UTF-8.
Google Groups has plenty of other
problems, but this has nothing to do with lang attributes.

I never said that. I said:
Google's web search ignored and Google's Usenet archive still ignores
the charset parameter although this is the only valid method to
get the encoding of a web page or an article.

Therefore it is plausible that they also ignore the LANG attribute.
And we have no indication that Google does read the LANG attribute -
which is sad, btw.

[Andreas Prilop]

On Mon, 30 Jan 2006, Philip Ronan wrote:

Here's something you can try: click the "Advanced search" link in Google, and
search for pages in English that contain the phrase "Der Spindoktor". Then
take a look at the lang attribute in the HTML tag of the top result.

There is no such thing as "the top result".
http://www.google.com/search?q=%22De...=lang_en&hl=de
http://www.google.com/search?q=%22De...=lang_en&hl=en
http://www.google.com/search?q=%22De...=lang_en&hl=fr