bug?!: lower() breaks unicode - PostgreSQL Database

Holger Klawitter

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi there,

I have a database with encoding UNICODE, when I run lower() on a column with
non ascii code the code for "sharp s" gets garbled. This happens with
postgres 7.3.[234] under Linux (Debian woody and SuSE 7.2). I have made sure
that LC_CTYPE=C for the server:

create table t ( a text, b text );
\encoding latin1 // my terminal is latin1
insert into t (a) values( 'Fuß' );
update t set b = lower(a);
select * from t;

ERROR: Could not convert UTF-8 to ISO8859-1

Apparantly the utf-8 special codes get lowercased, as the following selects
yield different results:

\encoding unicode // show me all
select a from t
select b from t

FuÃ // select a from t
fu // select b from t, should be "fuÃ"
The JDBC code breaks even more baldy.

java.lang.Array IndexOutOfBound sException: 9
at org.postgresql. core.Encoding.d ecodeUTF8(Encod ing.java:254)

- From the release docs of 7.4 it does not seems that this issue has not been
adressed. The database has been initialized to "de_DE@euro ", but this
shouldn't matter, should it?

Mit freundlichem Gruß / With kind regards
Holger Klawitter
- --
lists <at> klawitter <dot> de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/w0lx1Xdt0HKSwgY RAieWAJ9GHr/CAmh7mXYrM99LNz YimQa+qgCeIlKR
D0+YgVkdlbQtXbA Ed9T8/eE=
=NO8s
-----END PGP SIGNATURE-----
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postg resql.org so that your
message can get through to the mailing list cleanly

Nov 12 '05 #1

Subscribe Reply

1363

Similar topics

25959

convert Unicode to lower/uppercase?

by: Hallvard B Furuseth | last post by:

Has someone got a Python routine or module which converts Unicode strings to lowercase (or uppercase)? What I actually need to do is to compare a number of strings in a case-insensitive manner, so I assume it's simplest to convert to lower/upper first. Possibly all strings will be from the latin-1 character set, so I could convert to 8-bit latin-1, map to lowercase, and convert back, but that seems rather cumbersome.

Python

1722

xml encoding breaks

by: Phelim | last post by:

Hi. Im trying to form xml where the content is all unicode, but the xml seems to break at regular intervals, and will not see the content from that point on as unicode. It seems to add some extra characters onto the end of a node (which are usually garbage and a few characters long). After this point (which is always at the end of a particular node, the rest comes out as it would if it were not unicode.

.NET Framework

1887

function breaks when run too fast?

by: John Angel | last post by:

Hi, I have a pretty wierd situation. I have a javascript function that runs when I submit a page. The onsubmit code does quite a bit - it goes through a big textarea and replaces tabs with spaces, inserts/removes line breaks...it uses regular expressions to convert textarea content into somethign that can be sent to a mainframe application. Here's the problem: Sometimes special characters are pasted into the

Javascript

1719

Multi-Language upper(),lower()

by: Ara Anjargolian | last post by:

Just a quick question. Am I mistaken, or was multi-language support for upper(), lower() and friends recently added to 7.5? I could have sworn I saw mention of this on hackers, but, the TODO is not marked as done. Thanks, Ara Anjargolian

PostgreSQL Database

2108

problems with lower() and unicode-databases

by: peter pilsl | last post by:

postgres 7.4 on linux, glibc 2.2.4-6 I've a table containing unicode-data and the lower()-function does not work proper. While it lowers standard letters like A->a,B->b ... it fails on special letters like german umlauts (Ä , Ö ...) that are simply keeped untouched. Everything else (sorting etc.) is working fine and LC_COLLATE, LC_CTYPE and all the other locales were set proper to 'de_AT.UTF-8' (thats how my mandrake-systems calls the...

PostgreSQL Database

1731

lower/upper support and cyrillic

by: pginfo | last post by:

Hi , I tesetd the lower/upper support and cyrillic and it looks broken. My system: pg 7.4.3 freebsd 5.2.1. I created my database with 'UNICODE'.

PostgreSQL Database

4237

About upper() and lower to handle multibyte char

by: Weiping | last post by:

Hi, while upgrade to 8.0 (beta3) we got some problem: we have a database which encoding is UNICODE, when we do queries like: select upper('ÖÐÎÄ'); --select some multibyte character, then postgresql response: ERROR: invalid multibyte character for locale

PostgreSQL Database

26451

HTML UPPER case or lower case?

by: Eric Lindsay | last post by:

Should HTML 4.01 Strict markup be done in upper case or in lower case? I understand that HTML allows either upper or lower case. I also notice that XHTML apparently requires lower case. However I saw some mention that the HTML DOM uses upper case for markup elements. So, should I worry about what this means? I am inclined to go with lower case, for two reasons. Easier to change if I subsequently want to use XHTML.

HTML / CSS

7875

Problem with lower() for unicode strings in russian

by: Alexey Moskvin | last post by:

Hi! I have a set of strings (all letters are capitalized) at utf-8, russian language. I need to lower it, but my_string.lower(). Doesn't work. See sample script: # -*- coding: utf-8 -*- s1 = self.title s2 = self.title.lower() print s1 == s2

Python

9690

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

9550

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...

Windows Server

10501

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

10250

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

10032

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...

General

7574

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...

Microsoft Access / VBA

4149

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp

3764

How to add payments to a PHP MySQL app.

by: muto222 | last post by:

How can i add a mobile payment intergratation into php mysql website.

PHP

2944

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

General