Java sax UTF-8 parsing troubles -- PLEASE HELP...

[Aleksandar Matijaca]

Hi there,

I am in some need of help. I am trying to parse using the apache sax
parser
a file that has vaid UTF-8 characters - I keep end up getting a

sun.io.Malforme dInputException error.

This is my code:

infile = "<?xml version=\"1.0\"
encoding=\"UTF-8\"?><display_v alues><currency _display>\u00A5 Japanese
Yen</currency_displa y></display_values> ";

// the above is perfectly valid UNICODE symbol for Yen

XMLReader xr = new org.apache.xerc es.parsers.SAXP arser();

xr.setContentHa ndler(this);
xr.setErrorHand ler(this);

ByteArrayInputS tream bi = new
ByteArrayInputS tream(infile.ge tBytes());
Reader reader = new InputStreamRead er(bi,"UTF-8");
InputSource is = new InputSource(rea der);
is.setEncoding( "UTF-8");
xr.parse(is); // CRASHES RIGHT HERE...

this is the complete trace...

[8/29/04 22:38:40:756 GMT-05:00] 692c692c SystemErr R
sun.io.Malforme dInputException
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
java.lang.Throw able.<init>(Thr owable.java)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
sun.io.ByteToCh arUTF8.convert( ByteToCharUTF8. java)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
sun.nio.cs.Stre amDecoder$Conve rterSD.convertI nto(StreamDecod er.java)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
sun.nio.cs.Stre amDecoder$Conve rterSD.implRead (StreamDecoder. java)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
sun.nio.cs.Stre amDecoder.read( StreamDecoder.j ava)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
java.io.InputSt reamReader.read (InputStreamRea der.java)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.impl.XMLEnti tyScanner.load( Unknown Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.impl.XMLEnti tyScanner.scanQ Name(Unknown Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.impl.XMLNSDo cumentScannerIm pl.scanStartEle ment(Unknown
Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.impl.XMLDocu mentFragmentSca nnerImpl$Fragme ntContentDispat cher.dispatch(U nknown
Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.impl.XMLDocu mentFragmentSca nnerImpl.scanDo cument(Unknown
Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.parsers.XML1 1Configuration. parse(Unknown Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.parsers.DTDC onfiguration.pa rse(Unknown Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.parsers.XMLP arser.parse(Unk nown Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
org.apache.xerc es.parsers.Abst ractSAXParser.p arse(Unknown Source)
[8/29/04 22:38:40:776 GMT-05:00] 692c692c SystemErr R at
com.polyorb.tip ranavir.pdf.Con vertXML.cparse( ConvertXML.java )
What am I doing wrong here???

Thank you for any guideance...

Regards, Alex.

[Malcolm Dew-Jones]

Aleksandar Matijaca (al****@polyorb .com) wrote:
: Hi there,

: I am in some need of help. I am trying to parse using the apache sax
: parser
: a file that has vaid UTF-8 characters - I keep end up getting a

: sun.io.Malforme dInputException error.

: This is my code:

: infile = "<?xml version=\"1.0\"
: encoding=\"UTF-8\"?><display_v alues><currency _display>\u00A5 Japanese
: Yen</currency_displa y></display_values> ";

The string in java is not utf-8, it's utf-16, so if you pass the "raw
bytes" of the string to the parser then it isn't utf-8.

However, I haven't ever used the specific set of instructions you are
using, so I don't know for sure that is the problem.

[Martin Honnen]

Aleksandar Matijaca wrote:

I am in some need of help. I am trying to parse using the apache sax
parser
a file that has vaid UTF-8 characters - I keep end up getting a

sun.io.Malforme dInputException error.

This is my code:

infile = "<?xml version=\"1.0\"
encoding=\"UTF-8\"?><display_v alues><currency _display>\u00A5 Japanese
Yen</currency_displa y></display_values> ";

// the above is perfectly valid UNICODE symbol for Yen

XMLReader xr = new org.apache.xerc es.parsers.SAXP arser();

xr.setContentHa ndler(this);
xr.setErrorHand ler(this);

ByteArrayInputS tream bi = new
ByteArrayInputS tream(infile.ge tBytes());

I suspect the problem is here, getBytes using the platform's default
encoding (character set) while you want UTF-8 so try
infile.getBytes ("UTF8")
--

Martin Honnen
http://JavaScript.FAQTs.com/

[Aleksandar Matijaca]

MARTIN - THIS FIXED IT!!! It was the infile.getBytes ("UTF-8")
Martin and Malcolm, thank you very much for your suggestions.

All the best, Alex.
(Toronto)
Martin Honnen <ma*******@yaho o.de> wrote in message news:<41******* *************** @newsread4.arco r-online.net>...

Aleksandar Matijaca wrote:

I am in some need of help. I am trying to parse using the apache sax
parser
a file that has vaid UTF-8 characters - I keep end up getting a

sun.io.Malforme dInputException error.

This is my code:

infile = "<?xml version=\"1.0\"
encoding=\"UTF-8\"?><display_v alues><currency _display>\u00A5 Japanese
Yen</currency_displa y></display_values> ";

// the above is perfectly valid UNICODE symbol for Yen

XMLReader xr = new org.apache.xerc es.parsers.SAXP arser();

xr.setContentHa ndler(this);
xr.setErrorHand ler(this);

ByteArrayInputS tream bi = new
ByteArrayInputS tream(infile.ge tBytes());

I suspect the problem is here, getBytes using the platform's default
encoding (character set) while you want UTF-8 so try
infile.getBytes ("UTF8")

[Soren Kuula]

Aleksandar Matijaca wrote:

Hi there,
Hi, I can see you got your problem solved, but are you sure it is
_really_ doint what you want it to do (and are you aware what is
happening) ?

Assuming the type of your parameter infile is String:

Character encoding is the translation between character strings and byte
strings. I assume also that whatever made the String infile, it has
somehow managed to get the right chars out of the bytes in your file.

I think that this happens:
infile = "<?xml version=\"1.0\"
encoding=\"UTF-8\"?><display_v alues><currency _display>\u00A5 Japanese
Yen</currency_displa y></display_values> ";

// the above is perfectly valid UNICODE symbol for Yen

XMLReader xr = new org.apache.xerc es.parsers.SAXP arser();

xr.setContentHa ndler(this);
xr.setErrorHand ler(this);
ByteArrayInputS tream bi = new
ByteArrayInputS tream
1a) (as before): getBytes() returns PLATFORM_DEFAUL T-ENCODED byte string
representation of your String.(infile. getBytes());
or
1b) (after fix: )getBytes() returns UTF-ENCODED byte string
representation of your String.(infile. getBytes());
2) Your Reader then correctly DEcodes the byte stream into chars again Reader reader = new InputStreamRead er(bi,"UTF-8");
InputSource is = new InputSource(rea der); 3) the setEncoding statement should really have no effect; the
InputSource does not have the challenge of turning bytes into chars as
it already has a Reader (a source of chars, as opposed to a Stream
(source of bytes) so extract characters from. In other words, the
decoding work should have been done already) is.setEncoding( "UTF-8");
xr.parse(is); // CRASHES RIGHT HERE...

- because UTF_DECODE(SOME _OTHER_ENCODING _ENCODE(s)) is not necessarily
= s for some String s.

I think you read a file into a String (correctly decoded, maybe by
coincidence).
Then you encode that String (a String is just a sequence of chars) into
bytes and decode that back into chars again. No need for that !!

I suggest you let an InputStream read from your file, and use that
InputStream DIRECTLY as an argument to your InputSource. Reason : The
InputSource may be clever enough (I think it is) to UNDERSTAND the <?xml
encoding="blah" ... IN the XML file PROPER. Then it will automagically
use the proper decoding.

If that fails, you may try open a Reader on an InputStream in the file,
and then supply the encoding yourself (taking the risk that one day you
will prefer to write your XML files in some other encoding, and your
program will not work anymore).

Anyway encoding a String into bytes and then back to a source of chars
(a Reader) only adds confusion.

Soren

[Aleksandar Matijaca]

Yes, actualy, the string does have some UTF-8 characters which I am indeed
expecting. I am expecting a combination of Yen currency characters, British
pounds etc... This is an XML stream that needs to be parsed, modified, and
sent to FOP for PDF generation.

I have allways dealt with SAX parsing with plain Strings, and
that has allways worked, however, I realy did get stuck on this one...
Regards, Alex.

Soren Kuula <dongfang-remove_this@rem ove_this-bitplanet.net> wrote in message news:<3v******* *************@n ews000.worldonl ine.dk>...

Aleksandar Matijaca wrote:

Hi there,

Hi, I can see you got your problem solved, but are you sure it is
_really_ doint what you want it to do (and are you aware what is
happening) ?

Assuming the type of your parameter infile is String:

Character encoding is the translation between character strings and byte
strings. I assume also that whatever made the String infile, it has
somehow managed to get the right chars out of the bytes in your file.

I think that this happens:

infile = "<?xml version=\"1.0\"
encoding=\"UTF-8\"?><display_v alues><currency _display>\u00A5 Japanese
Yen</currency_displa y></display_values> ";

// the above is perfectly valid UNICODE symbol for Yen

XMLReader xr = new org.apache.xerc es.parsers.SAXP arser();

xr.setContentHa ndler(this);
xr.setErrorHand ler(this);
ByteArrayInputS tream bi = new
ByteArrayInputS tream

1a) (as before): getBytes() returns PLATFORM_DEFAUL T-ENCODED byte string
representation of your String.(infile. getBytes());
or
1b) (after fix: )getBytes() returns UTF-ENCODED byte string
representation of your String.(infile. getBytes());
2) Your Reader then correctly DEcodes the byte stream into chars again

Reader reader = new InputStreamRead er(bi,"UTF-8");
InputSource is = new InputSource(rea der);

3) the setEncoding statement should really have no effect; the
InputSource does not have the challenge of turning bytes into chars as
it already has a Reader (a source of chars, as opposed to a Stream
(source of bytes) so extract characters from. In other words, the
decoding work should have been done already)

is.setEncoding( "UTF-8");
xr.parse(is); // CRASHES RIGHT HERE...

- because UTF_DECODE(SOME _OTHER_ENCODING _ENCODE(s)) is not necessarily
= s for some String s.

I think you read a file into a String (correctly decoded, maybe by
coincidence).
Then you encode that String (a String is just a sequence of chars) into
bytes and decode that back into chars again. No need for that !!

I suggest you let an InputStream read from your file, and use that
InputStream DIRECTLY as an argument to your InputSource. Reason : The
InputSource may be clever enough (I think it is) to UNDERSTAND the <?xml
encoding="blah" ... IN the XML file PROPER. Then it will automagically
use the proper decoding.

If that fails, you may try open a Reader on an InputStream in the file,
and then supply the encoding yourself (taking the risk that one day you
will prefer to write your XML files in some other encoding, and your
program will not work anymore).

Anyway encoding a String into bytes and then back to a source of chars
(a Reader) only adds confusion.

Soren