Determine language of body of text?

[Mark B]

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[Mark B]

http://code.google.com/apis/ajaxlang...tation/#Detect

"Mark B" <no*****@none.comwrote in message
news:u$*************@TK2MSFTNGP06.phx.gbl...

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[MC]

Each language has a few words that are extremely common, such as "the", "a", "an", "of", "for" in English. You could look for the 5 or 10 most common words in each language, and see which language wins.

To find the most common words, use a word-frequency-table program to analyze some text samples.

[rossum]

On Thu, 23 Oct 2008 20:59:19 +1300, "Mark B" <no*****@none.comwrote:

>Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

As well as the other suggestions have a look at the characters used:
umlauts or the ss/beta character appear in German and acute or grave
accents in French.

Digram and trigram frequencies can also be a good indicator.

Essentially you are going to have to use some sort of statistical
method.

rossum

[JP]

There's some code here that writes the internet headers of an email to
a text file, which you could then parse for the language. For example,
some email headers include a "Content-Type" line which indicates the
character set used.

i.e.: Content-Type: text/plain; charset=US-ASCII

http://blogs.technet.com/kclemson/ar...n-outlook.aspx

HTH,
JP

On Oct 23, 3:59*am, "Mark B" <[e-mail]:

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[Dmitry Streblechenko]

Look at the MailItem.InternetCodepage property (corresponds to the
PR_INTERNET_CPID MAPI propety).

--
Dmitry Streblechenko (MVP)
http://www.dimastr.com/
OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <no*****@none.comwrote in message
news:u$*************@TK2MSFTNGP06.phx.gbl...

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[Mark B]

Do all incoming emails have this? Even if they are not originating from an
Outlook client?

(It's for an Outlook 2007 Add-in C# BTW).
"Dmitry Streblechenko" <dm****@dimastr.comwrote in message
news:%2****************@TK2MSFTNGP06.phx.gbl...

Look at the MailItem.InternetCodepage property (corresponds to the
PR_INTERNET_CPID MAPI propety).

--
Dmitry Streblechenko (MVP)
http://www.dimastr.com/
OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <no*****@none.comwrote in message
news:u$*************@TK2MSFTNGP06.phx.gbl...

>Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[Dmitry Streblechenko]

Most of them. But that really tells you more about the defaut code page of
the sender.

--
Dmitry Streblechenko (MVP)
http://www.dimastr.com/
OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <no*****@none.comwrote in message
news:%2***************@TK2MSFTNGP02.phx.gbl...

Do all incoming emails have this? Even if they are not originating from an
Outlook client?

(It's for an Outlook 2007 Add-in C# BTW).
"Dmitry Streblechenko" <dm****@dimastr.comwrote in message
news:%2****************@TK2MSFTNGP06.phx.gbl...

>Look at the MailItem.InternetCodepage property (corresponds to the
PR_INTERNET_CPID MAPI propety).

--
Dmitry Streblechenko (MVP)
http://www.dimastr.com/
OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <no*****@none.comwrote in message
news:u$*************@TK2MSFTNGP06.phx.gbl...

>>Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English.
I'm just trying to get the main language used.

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

Mark B wrote:

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English.
I'm just trying to get the main language used.

If you are willing to write some code, then you can detect
the language (but probably not the regional dialect).

* dictionary with common words
* special letters (forward and backward accents, umlauts etc.)
* distribution of letters
* distribution of pairs of letters

Arne