Logging library unicode problem

[Victor Lin]

Hi,
I'm writting a application using python standard logging system. I
encounter some problem with unicode message passed to logging library.
I found that unicode message will be messed up by logging handler.

piese of StreamHandler:

try:
self.stream.wri te(fs % msg)
except UnicodeError:
self.stream.wri te(fs % msg.encode("UTF-8"))

It just write the message to stream. If there is some unicode error,
it would rewrite msg with utf8 encoding.

I write some code to try:

import sys
print u'¤¤¤å¦r´ú¸Õ'
print sys.stdout.enco ding
sys.stdout.writ e(u'¤¤¤å')

result of that program:

¤¤¤å¦r´ú¸Õ
cp950
Traceback (most recent call last):
File "update_stockpr ice.py", line 92, in <module>
sys.stdout.writ e(u'ä¸*æ?')
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
0-1: ordin
al not in range(128)

It show that....

1. print statement encode what it get with stream.encoding ?
2. stream.write don't do anything like encoding, just write it
(because it might be binary data?)

So the problem is : the StreamHandler of standard logging library use
stream.write to log message, if there is unicode error, unicode string
will be encode to utf8. This behavior mess my unicode up.

Here I modify the code of StreamHandler:

try:
print >self.stream, msg
#self.stream.wr ite(fs % msg)
except UnicodeError:
self.stream.wri te(fs % msg.encode("UTF-8"))

I replace stream.write with print statement, so that it will try to
use stream.encoding to encode msg. Now everything works fine.

My question is :
Could the behavior of StreamHandler be considered as a bug?
If it is, how to report this bug?
Is my solution correct?
Are there any side effect will caused by doing so?
If the code I write is fine, and solve that problem, how to report it
to Python's project?
I think this could be helpful for people who also encountered this
problem.

Thanks.
Victor Lin.

[Vinay Sajip]

On 13 Aug, 11:08, Victor Lin <Borns...@gmail .comwrote:

Hi,
I'm writting a application using python standardlogging system. I
encounter some problem with unicode message passed tologginglibrar y.
I found that unicode message will be messed up bylogginghandle r.

piese of StreamHandler:

try:
self.stream.wri te(fs % msg)
except UnicodeError:
self.stream.wri te(fs % msg.encode("UTF-8"))

It just write the message to stream. If there is some unicode error,
it would rewrite msg with utf8 encoding.

I write some code to try:

import sys
print u'¤¤¤å¦r´ú¸Õ'
print sys.stdout.enco ding
sys.stdout.writ e(u'¤¤¤å')

result of that program:

¤¤¤å¦r´ú¸Õ
cp950
Traceback (most recent call last):
File "update_stockpr ice.py", line 92, in <module>
sys.stdout.writ e(u'ä¸*æ?')
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
0-1: ordin
al not in range(128)

It show that....

1. print statement encode what it get with stream.encoding ?
2. stream.write don't do anything like encoding, just write it
(because it might be binary data?)

So the problem is : the StreamHandler of standardlogging library use
stream.write to log message, if there is unicode error, unicode string
will be encode to utf8. This behavior mess my unicode up.

Here I modify the code of StreamHandler:

try:
print >self.stream, msg
#self.stream.wr ite(fs % msg)
except UnicodeError:
self.stream.wri te(fs % msg.encode("UTF-8"))

I replace stream.write with print statement, so that it will try to
use stream.encoding to encode msg. Now everything works fine.

My question is :
Could the behavior of StreamHandler be considered as a bug?
If it is, how to report this bug?
Is my solution correct?
Are there any side effect will caused by doing so?
If the code I write is fine, and solve that problem, how to report it
to Python's project?
I think this could be helpful for people who also encountered this
problem.

Thanks.
Victor Lin.

Hi Victor,

Can you try modifying your patch to use the following logic instead of
the print statement?

if hasattr(self.st ream, 'encoding'):
self.stream.wri te(fs % msg.encode(self .stream.encodin g))
else:
self.stream.wri te(fs % msg)

Does this work in your scenario?

Regards,
Vinay Sajip