By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
438,179 Members | 970 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 438,179 IT Pros & Developers. It's quick & easy.

unexpected query behavior with UTF text

P: n/a
I have recently come across an unusual behavior with Postgres 7.3.4 on a
Linux RH 9 system. My database has encoding set to "UNICODE", and the
table includes Japanese text. I'm trying to issue a query like this:

SELECT * FROM sales WHERE name='ja-text';

This query ignores all japanese characters in the comparison text. It
matches properly on ascii chars, but skips right over ja chars.

I tried using "LIKE" instead of "=", and this works fine! I would
expect "=" to do a character by character comparison, but it must not be.

I found a related issue on the mailing list, where locale setting was
causing something similar. However, my locale is set to "en_US.UTF-8",
which is the solution proposed to the other problem.

Is this a bug, or expected behavior? How can I correct this without
resorting to LIKE and escaping wildcard characters?

thanks!
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

Nov 12 '05 #1
Share this Question
Share on Google+
2 Replies


P: n/a
Indra Heckenbach <in***@macnica.com> writes:
I have recently come across an unusual behavior with Postgres 7.3.4 on a
Linux RH 9 system. My database has encoding set to "UNICODE", and the
table includes Japanese text. I'm trying to issue a query like this: SELECT * FROM sales WHERE name='ja-text'; This query ignores all japanese characters in the comparison text. It
matches properly on ascii chars, but skips right over ja chars.
Text = depends on strcoll(), which is locale-sensitive. It sure appears
that your locale is designed to ignore japanese characters :-(
I found a related issue on the mailing list, where locale setting was
causing something similar. However, my locale is set to "en_US.UTF-8",
which is the solution proposed to the other problem.


We have heard before that RH9's default locale setting is seriously
broken. This seems to be additional evidence for that opinion. I'd
recommend re-initdb'ing in locale C.

Also, you say "your locale", but how certain are you that that is the
database's locale, and not just the one in your own user environment?
It'd be a good idea to use pg_controldata to check the database settings.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05 #2

P: n/a
Hi,

Thanks for the response. I actually have tried to re initdb with
locale=C and I got the same results. If my locale is ignoring Ja chars,
how can I change that? I expected that UTF-8 would consider all
characters, being a universal encoding. A colleague also tried the
same test with a different server (locale also set to utf8).

Text = depends on strcoll(), which is locale-sensitive. It sure appears
that your locale is designed to ignore japanese characters :-(
I found a related issue on the mailing list, where locale setting was
causing something similar. However, my locale is set to "en_US.UTF-8",
which is the solution proposed to the other problem.
We have heard before that RH9's default locale setting is seriously
broken. This seems to be additional evidence for that opinion. I'd
recommend re-initdb'ing in locale C.

Also, you say "your locale", but how certain are you that that is the
database's locale, and not just the oe in your own user environment?
It'd be a good idea to use pg_controldata to check the database settings.

I tried to mannually initialize it, and my initdb.i18n exports
LANG="en_US.UTF-8". Is there any other way I can check?
thanks,
Indra
regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.