472,373 Members | 1,445 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,373 software developers and data experts.

tsearch2, ispell, utf-8 and german special characters

Hi!

Sorry to bother you, but I just don't know how to get tsearch2 configured correctly for my setup. I've got a 7.4.3 database-cluster initdb'ed with de_DE@euro as locale, the database is with Unicode encoding.

I made and installed contrib/tsearch2 after installing the dump/reload-patch http://www.sai.msu.su/~megera/postgr...e_7.4.patch.gz as advised by the docs. So far everything is looking good, I have generated a snowball stemmer dictionary and an ispell dictionary as described in the docs and created a new configuration 'default_german' as described.

This is working somehow:
SELECT to_tsvector('default_german',
'tsearch2 erlernen ist wie zur Schule zu gehen');
-> 'gehen':10 'schulen':8 'erlernen':3 'tsearch2':2

though I don't quite understand why "Schule" is converted to "schulen" and not the other way round, but so be it. My problem lies, as every so often, with the non-ascii-characters, namely german umlauts and the .

SELECT to_tsvector('default_german',
'ich mu tsearch2 begreifen ');

returns null. So does any phrase which contains or anything that's beyond ASCII.

Another thing is the ISpell functionality; the docs are quite vague on thispart when it comes to explaining which file(s) to use to create german.med.. In ISpell conventions, umlauts seem to be represented as A" a" O" o" U" u" and thus when doing

SELECT lexize('de_ispell', 'ther');
I receive NULL

whereas
SELECT lexize('de_ispell', 'A"ther');
gives me {"a\"ther"}
as result.

I downloaded igerman98-20030222.tar.bz2 from http://j3e.de/ispell/igerman98/dict/ which seems to be the recommended ISpell dictionary distribution forthe german language as noted on http://fmg-www.cs.ucla.edu/fmg-membe...l#German-dicts

Of course there are no german.0 or german.1 files in this distribution which would be the obvious counterparts to english.0 and english.1 mentioned inthe tsearch2-docs; there is however a file all.words built on installation, which seems to be the basis for building the hash-file later on. The first few lines of this file are

A"bte/N
A"btissin/F
a"chten/DIXY
A"chtens
A"chtung/P
a"chzen/DIXY
a"chzt/EGPX
A"cker/N

In order to get the .med-File I did sort -u -t/ +0f -1 +0 -T /usr/tmp -o german.med all.words

There is an option to generate another wordlist via make isowordlist - but this didn't resolve the umlaut-issue either, neither in the standard encoding provided in the package nor after conversion to UTF-8 (I tried both withand without a BOM).

Now has anybody actually managed to get a working configuration with tsearch2 and german language support in a unicode-database? What am I doing wrong? I just can't find any more hints in the docs, and there's a topic on the OpenFTS-Mailinglist with somewhat similar issues ( http://sourceforge.net/mailarchive/f...&forum_id=7671 ), but nothing which would actually help to resolve it.

Kind regards

Markus

Nov 23 '05 #1
0 1991

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

13
by: Nigel J. Andrews | last post by:
This will be a little vague, it was last night and I can't now do the test in that db (see below) so can't give the exact wording. I seem to remember a report a little while ago about tsearch v2...
1
by: psql-mail | last post by:
I have applied the recent tsearch2 patch and recompiled the tsearch2 module but I am still experiencing the same backend crashes as I previously described. Thanks for any help, Mat GDB...
1
by: Pavel Stehule | last post by:
Hello Can I use tsearch2 with aspell? I didn't find any info about it, and I don't know anything about difference between ispell and aspell. Thank You Pavel Stehule ...
9
by: Pavel Stehule | last post by:
Hello I try tsearch2 within czech environment. It is works fine, but I have two questions. 1. I have words "se", "ve" in my czech stop words. But I get this words in result. Why? Have I...
16
by: Ben | last post by:
I'm trying to make myself a dictionary for tsearch2 that converts numbers to their english word equivalents. This seems to be working great, except that I can't figure out how to make my lexize...
2
by: Fischer Ulrich | last post by:
Hi I have a problem with the restoring of a database which uses tsearch2. I made a backup as discribed in 'tsearch-v2-intro' on the tsearch2 page. Now i'm trying to restore it into a...
0
by: Ben | last post by:
I just made myself an ispell dictionary for tsearch2, thinking (very incorrectly, it turns out) that looking up a misspelled word with the ispell dictionary would return possible words that I...
3
by: Marcel Boscher | last post by:
Hello everybody, i tried to "J.U.S.T" install the FullTextSearchTool tsearch2 under the guidiance of : http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/...
2
by: Net Virtual Mailing Lists | last post by:
Hello, If I have a rule like this: CREATE OR REPLACE RULE sometable_update AS ON UPDATE TO table2 DO UPDATE cache SET updated_dt=NULL WHERE tablename='sometable'; CREATE OR REPLACE RULE...
1
by: Dawid Kuroczko | last post by:
I'm trying to use tsearch2 with database which is in 'UNICODE' encoding. It works fine for English text, but as I intend to search Polish texts I did: insert into pg_ts_cfg('default_polish',...
0
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge required to effectively administer and manage Oracle...
0
hi
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific technical details, Gmail likely implements measures...
1
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web server and have made sure to enable curl. I get a...
0
Oralloy
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the synthesis of my design into a bitstream, not the C++...
0
BLUEPANDA
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS starter kit that's not only easy to use but also...
0
by: Rahul1995seven | last post by:
Introduction: In the realm of programming languages, Python has emerged as a powerhouse. With its simplicity, versatility, and robustness, Python has gained popularity among beginners and experts...
2
by: Ricardo de Mila | last post by:
Dear people, good afternoon... I have a form in msAccess with lots of controls and a specific routine must be triggered if the mouse_down event happens in any control. Than I need to discover what...
1
by: ezappsrUS | last post by:
Hi, I wonder if someone knows where I am going wrong below. I have a continuous form and two labels where only one would be visible depending on the checkbox being checked or not. Below is the...
0
DizelArs
by: DizelArs | last post by:
Hi all) Faced with a problem, element.click() event doesn't work in Safari browser. Tried various tricks like emulating touch event through a function: let clickEvent = new Event('click', {...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.