Logging library unicode problem

[Victor Lin]

Hi,
I'm writting a application using python standard logging system. I
encounter some problem with unicode message passed to logging library.
I found that unicode message will be messed up by logging handler.

piese of StreamHandler:

try:
self.stream.write(fs % msg)
except UnicodeError:
self.stream.write(fs % msg.encode("UTF-8"))

It just write the message to stream. If there is some unicode error,
it would rewrite msg with utf8 encoding.

I write some code to try:

import sys
print u'¤¤¤å¦r´ú¸Õ'
print sys.stdout.encoding
sys.stdout.write(u'¤¤¤å')

result of that program:

¤¤¤å¦r´ú¸Õ
cp950
Traceback (most recent call last):
File "update_stockprice.py", line 92, in <module>
sys.stdout.write(u'ä¸*æ?')
UnicodeEncodeError: 'ascii' codec can't encode characters in position
0-1: ordin
al not in range(128)

It show that....

1. print statement encode what it get with stream.encoding?
2. stream.write don't do anything like encoding, just write it
(because it might be binary data?)

So the problem is : the StreamHandler of standard logging library use
stream.write to log message, if there is unicode error, unicode string
will be encode to utf8. This behavior mess my unicode up.

Here I modify the code of StreamHandler:

try:
print >self.stream, msg
#self.stream.write(fs % msg)
except UnicodeError:
self.stream.write(fs % msg.encode("UTF-8"))

I replace stream.write with print statement, so that it will try to
use stream.encoding to encode msg. Now everything works fine.

My question is :
Could the behavior of StreamHandler be considered as a bug?
If it is, how to report this bug?
Is my solution correct?
Are there any side effect will caused by doing so?
If the code I write is fine, and solve that problem, how to report it
to Python's project?
I think this could be helpful for people who also encountered this
problem.

Thanks.
Victor Lin.

[Vinay Sajip]

On 13 Aug, 11:08, Victor Lin <[e-mail]:

Hi,
I'm writting a application using python standardloggingsystem. I
encounter some problem with unicode message passed tologginglibrary.
I found that unicode message will be messed up bylogginghandler.

piese of StreamHandler:

try:
self.stream.write(fs % msg)
except UnicodeError:
self.stream.write(fs % msg.encode("UTF-8"))

It just write the message to stream. If there is some unicode error,
it would rewrite msg with utf8 encoding.

I write some code to try:

import sys
print u'¤¤¤å¦r´ú¸Õ'
print sys.stdout.encoding
sys.stdout.write(u'¤¤¤å')

result of that program:

¤¤¤å¦r´ú¸Õ
cp950
Traceback (most recent call last):
File "update_stockprice.py", line 92, in <module>
sys.stdout.write(u'ä¸*æ?')
UnicodeEncodeError: 'ascii' codec can't encode characters in position
0-1: ordin
al not in range(128)

It show that....

1. print statement encode what it get with stream.encoding?
2. stream.write don't do anything like encoding, just write it
(because it might be binary data?)

So the problem is : the StreamHandler of standardlogginglibrary use
stream.write to log message, if there is unicode error, unicode string
will be encode to utf8. This behavior mess my unicode up.

Here I modify the code of StreamHandler:

try:
print >self.stream, msg
#self.stream.write(fs % msg)
except UnicodeError:
self.stream.write(fs % msg.encode("UTF-8"))

I replace stream.write with print statement, so that it will try to
use stream.encoding to encode msg. Now everything works fine.

My question is :
Could the behavior of StreamHandler be considered as a bug?
If it is, how to report this bug?
Is my solution correct?
Are there any side effect will caused by doing so?
If the code I write is fine, and solve that problem, how to report it
to Python's project?
I think this could be helpful for people who also encountered this
problem.

Thanks.
Victor Lin.

Hi Victor,

Can you try modifying your patch to use the following logic instead of
the print statement?

if hasattr(self.stream, 'encoding'):
self.stream.write(fs % msg.encode(self.stream.encoding))
else:
self.stream.write(fs % msg)

Does this work in your scenario?

Regards,
Vinay Sajip