470,636 Members | 1,507 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,636 developers. It's quick & easy.

Shift-JIS to UTF-8 conversion

Hello everybody,

I need to convert a Japanese Shift-JIS CSV file to Unicode UTF-8.

My machine is a Windows 98 english computer with Python 2.3.4

Any hints?.

Jul 19 '05 #1
4 7269
I think you do something like this (untested):

import codecs

def transcode(infile, outfile, incoding="shift-jis",
outcoding="utf-8"):
f = codecs.open(infile, "rb", incoding)
g = codecs.open(outfile, "wb", outcoding)

g.write(f.read())
# If the file is so large that it can't be read at once, do a loop which
# reads and writes smaller chunks
# while 1:
# block = f.read(4096000)
# if not block: break
# g.write(block)

f.close()
g.close()

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCjRzZJd01MZaTXX0RAg8YAJ4rQ8Fcpwi1AB2a/ZVdALGysct8jACfYdXm
in2aJ3xmdB0ncRZBWXmfMQs=
=bHjV
-----END PGP SIGNATURE-----

Jul 19 '05 #2
Hello,
I think the answer is basically correct but shift-jis is not a standard
part of
Python 2.3. You will either need to use Python 2.4 where the cjkcodes
are integrated or install them under Python 2.3. The link is
http://cjkpython.i18n.org/

You then also need:
import cjkcodecs.aliases

Richard

Jeff Epler wrote:
I think you do something like this (untested):

import codecs

def transcode(infile, outfile, incoding="shift-jis",
outcoding="utf-8"):
f = codecs.open(infile, "rb", incoding)
g = codecs.open(outfile, "wb", outcoding)

g.write(f.read())
# If the file is so large that it can't be read at once, do a loop which # reads and writes smaller chunks
# while 1:
# block = f.read(4096000)
# if not block: break
# g.write(block)

f.close()
g.close()

Jeff


Jul 19 '05 #3
PyTJ wrote:
I need to convert a Japanese Shift-JIS CSV file to Unicode UTF-8.

My machine is a Windows 98 english computer with Python 2.3.4

Any hints?.


First, you need to install codecs to support japanese encodings.
Python 2.3.* does not support SJIS by default.

I'll give you two options.

- Japanese Codecs
http://www.python.jp/Zope/download/JapaneseCodecs

http://ftp.python.jp/pub/JapaneseCod...in32-py2.3.exe

- CJKCodecs
http://cjkpython.i18n.org/
http://download.berlios.de/cjkpython...in32-py2.3.exe

If you only need Japanese support, Japanese Codecs might be handy.
On the other hand, CJKCodecs can handle much broader encodings.
Aside from that, starting from 2.4, Python ships with CJKCodecs,
so I'd recomment CJKCodecs without reservations.

-- george
Jul 19 '05 #4
On Fri, May 20, 2005 at 12:16:15AM -0700, rb*****@gmx.de wrote:
Hello, I think the answer is basically correct but shift-jis is not a
standard part of Python 2.3.


Ah, I was fooled --- I tested on Python 2.3, but my packager must have
included the codecs you went on to mention.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFCkcJCJd01MZaTXX0RAhNUAKCLbSsAAzxXe9UIjMXd5A N/wKcfbQCeI9j0
lpU5Zu0BgAdD2hTFvKB8kJs=
=Tof0
-----END PGP SIGNATURE-----

Jul 19 '05 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

388 posts views Thread by maniac | last post: by
43 posts views Thread by Mehta Shailendrakumar | last post: by
7 posts views Thread by Csaba Gabor | last post: by
9 posts views Thread by deepak | last post: by
4 posts views Thread by sandhya | last post: by
1 post views Thread by Korara | last post: by
???
reply views Thread by Stoney L | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.