By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,799 Members | 1,303 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,799 IT Pros & Developers. It's quick & easy.

Mixed UTF8 / Latin1 database

P: n/a
Hi,

I'm trying to restore a pg_dump-backed up database from one
server to another. The problem is that the db is "mixed encoded"
in UTF-8 and LATIN1... (weird but, yes it is ! It was ported once
from a hypersonic db... that screwed up something and now I'm
fighting with that...).

So, trying to restore that db into a UTF-8 encoded new one, gives
me errors ("invalid unicode character..."), but importing it
into a LATIN1 econcoded one, gives me weird characters (of course).

I'm wondering if anyone could have a script or something to help me
with this situation... :(

thanks.

--
Claudio Cicali
c.******@mclink.it
http://www.flexer.it
GPG Key Fingerprint = 2E12 64D5 E5F5 2883 0472 4CFF 3682 E786 555D 25CE

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 23 '05 #1
Share this Question
Share on Google+
2 Replies

P: n/a
> I'm wondering if anyone could have a script or something to help me
with this situation... :(


Knowing that Unicode is composed of plain ASCII characters,
you may perform a conversion test using PHP "recode" function.

You may test each record as follows:

$test = recode ("latin1..u8", $record);

If the $test value differs from the $record one, then it is a Latin1 string.

On the converse, it is Unicode UT-8. There is no garantee. Make a test on a
few values, I am not coding right now and only have access to email.

Cheers,
Jean-Michel
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 23 '05 #2

P: n/a
В Птн, 16.04.2004, в 16:26, Jean-Michel POURE пишет:
I'm wondering if anyone could have a script or something to help me
with this situation... :(
Knowing that Unicode is composed of plain ASCII characters,


It's not, Unicode is just a number to glyph mapping, and UTF-8, what you
probably mean, uses the eighth bit too, which cannot be said of ASCII -
ASCII is a 7 bit character set. you may perform a conversion test using PHP "recode" function.

You may test each record as follows:

$test = recode ("latin1..u8", $record);

If the $test value differs from the $record one, then it is a Latin1 string.


Unfortunately unless $record is all ASCII $test and $record will always
differ, because every byte stream is valid latin1 and can thus be
converted to UTF-8.

What you can do is to manually replace ä ü ö § and all other non-ASCII
byte values with their UTF-8 equivalent and then go through the data
manually to check it for correctness.

Another hint you can get at in your program is to try to check if your
input is valid UTF-8 and only convert from latin1 if it is not. You can
probably check for valid UTF-8 by seeing if the conversion from UTF-8 to
UTF-8 succeeds.

--
Markus Bertheau <tw*****@bluetwanger.de>
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ma*******@postgresql.org)

Nov 23 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.