Problems with email.Generator .Generator

[Chris Withers]

Hi All,

The following piece of code is giving me issues:

from email.Charset import Charset,QP
from email.MIMEText import MIMEText
charset = Charset('utf-8')
charset.body_en coding = QP
msg = MIMEText(
u'Some text with chars that need encoding: \xa3',
'plain',
)
msg.set_charset (charset)
print msg.as_string()

Under Python 2.4.2, this produces the following output, as I'd expect:

MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8"

Some text with chars that need encoding: =A3

However, under Python 2.4.3, I now get:

Traceback (most recent call last):
File "test_encoding. py", line 14, in ?
msg.as_string()
File "c:\python24\li b\email\Message .py", line 129,
in
as_string
g.flatten(self, unixfrom=unixfr om)
File "c:\python24\li b\email\Generat or.py", line 82,
in flatten
self._write(msg )
File "c:\python24\li b\email\Generat or.py", line 113,
in _write
self._dispatch( msg)
File "c:\python24\li b\email\Generat or.py", line 139,
in
_dispatch
meth(msg)
File "c:\python24\li b\email\Generat or.py", line 182,
in
_handle_text
self._fp.write( payload)
UnicodeEncodeEr ror: 'ascii' codec can't encode
character
u'\xa3' in position 41:
ordinal not in range(128)

This seems to be as a result of this change:

http://svn.python.org/view/python/br...37910&r2=42272

....which is referred to as part of a fix for this bug:

http://sourceforge.net/tracker/?func...70&atid=105470

Now, is this change to Generator.py in error or am I doing something wrong?

If the latter, how can I change my code such that it works as I'd expect?

cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

[Peter Otten]

Chris Withers wrote:

At worst, and most likely based on my past experience of (c)StringIO
being used to accumulate output, it won't make a jot of difference...

What past experience?

>>StringIO.Stri ngIO().write(un ichr(128))
cStringIO.Str ingIO().write(u nichr(128))

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\x80' in position
0: ordinal not in range(128)

Peter

[Steve Holden]

Chris Withers wrote:

Steve Holden wrote:

>>Is there a how-to for this anywhere? The email package's docs are
short on examples involving charsets, unicode and the like :-(

Well, it would seem like the easiest approach is to monkey-patch the
use of cStringIO to StringIO as recommended and see if that fixes your
problem. Wouldn't it?

No, not really, since at best that's a nasty (and I meant really nasty)
hack. I'm using the email package as part of a library that I'm building
which is to be used with various frameworks. Monkey patching modules is
about as bad as it gets in that situation...

At worst, and most likely based on my past experience of (c)StringIO
being used to accumulate output, it won't make a jot of difference...

Under those circumstances you probably know best ...

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Chris Withers]

Peter Otten wrote:

What past experience?

>>>StringIO.Str ingIO().write(u nichr(128))
cStringIO.St ringIO().write( unichr(128))

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\x80' in position
0: ordinal not in range(128)

OK, I stand corrected, although I suspect the bug is actually in
StringIO.String IO in that it doesn't barf on unicodes.

(Python 3000 and all that)

Which again leads us back to the email package: it used to do the right
thing from what I can see, and now it doesn't, and ends up trying to
write a unicode to a cStringIO, which (rightly, I guess) barfs...

Barry, Barry, where are you? ;-)

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

[Chris Withers]

Peter Otten wrote:

Chris Withers wrote:

>At worst, and most likely based on my past experience of (c)StringIO
being used to accumulate output, it won't make a jot of difference...

What past experience?

>>>StringIO.Str ingIO().write(u nichr(128))
cStringIO.St ringIO().write( unichr(128))

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\x80' in position
0: ordinal not in range(128)

Okay, more out of desperation than anything else, lets try this:

from email.Charset import Charset,QP
from email.MIMEText import MIMEText
from StringIO import StringIO
from email import Generator,Messa ge
Generator.Strin gIO = Message.StringI O = StringIO
charset = Charset('utf-8')
charset.body_en coding = QP
msg = MIMEText(u'Some text with chars that need encoding: \xa3','plain')
msg.set_charset (charset)
print repr(msg.as_str ing())
u'MIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\nContent-Type:
text/plain; charset="utf-8"\n\nSome text with chars that need encoding:
\xa3'

Yay! No unicode error, but also no use:

File "c:\python24\li b\smtplib.py", line 692, in sendmail
(code,resp) = self.data(msg)
File "c:\python24\li b\smtplib.py", line 489, in data
self.send(q)
File "c:\python24\li b\smtplib.py", line 316, in send
self.sock.senda ll(str)
File "<string>", line 1, in sendall
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xa3' in
position 297: ordinal not in range(128)

The other variant I've tried is:

from email.Charset import Charset,QP
from email.MIMEText import MIMEText
charset = Charset('utf-8')
charset.body_en coding = QP
msg = MIMEText('','pl ain',)
msg.set_charset (charset)
msg.set_payload (charset.body_e ncode(u'Some text with chars that need
encoding: \xa3'))
print msg.as_string()

Which is sort of okay:

MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"

Some text with chars that need encoding: =A3

....except it gets the transfer encoding wrong, which means Thunderbird
shows =A3 instead of the pound sign that it should :-(

....this is down to a pretty lame bit of code in Encoders.py which
basically checks for a unicode error *sigh*

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

[Chris Withers]

Chris Withers wrote:

...except it gets the transfer encoding wrong, which means Thunderbird
shows =A3 instead of the pound sign that it should :-(

...this is down to a pretty lame bit of code in Encoders.py which
basically checks for a unicode error *sigh*

OK, slight progress... here a new version that actually works:

from email.Charset import Charset,QP
from email.MIMEText import MIMEText
charset = Charset('utf-8')
charset.body_en coding = QP
msg = MIMEText('','pl ain',None)
msg.set_payload (u'Some text with chars that need encoding:\xa3', charset)
print msg.as_string()

MIME-Version: 1.0
Content-Type: text/plain; charset; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Some text with chars that need encoding:=A3

Okay, so this actually does the right thing... wahey!

....but hold your horses, if Charset isn't set to quoted printable, then
you end up with problems:

charset = Charset('utf-8')
msg = MIMEText('','pl ain',None)
msg.set_payload (u'Some text with chars that need encoding:\xa3', charset)

Traceback (most recent call last):
File "C:\test_encodi ng.py", line 5, in ?
msg.set_payload (u'Some text with chars that need
encoding:\xa3', charset)
File "c:\python24\li b\email\Message .py", line 218, in set_payload
self.set_charse t(charset)
File "c:\python24\li b\email\Message .py", line 260, in set_charset
self._payload = charset.body_en code(self._payl oad)
File "c:\python24\li b\email\Charset .py", line 366, in body_encode
return email.base64MIM E.body_encode(s )
File "c:\python24\li b\email\base64M IME.py", line 136, in encode
enc = b2a_base64(s[i:i + max_unencoded])
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xa3' in
position 40: ordinal not in range(128)

Now what?

*sigh*

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

[Chris Withers]

Chris Withers wrote:

print msg.as_string()

MIME-Version: 1.0
Content-Type: text/plain; charset; charset="utf-8"

^^^^^^^
Actually, even this isn't correct as you can see above...

charset = Charset('utf-8')
msg = MIMEText('','pl ain',None)
msg.set_payload (u'Some text with chars that need encoding:\xa3', charset)

Traceback (most recent call last):
File "C:\test_encodi ng.py", line 5, in ?
msg.set_payload (u'Some text with chars that need
encoding:\xa3', charset)
File "c:\python24\li b\email\Message .py", line 218, in set_payload
self.set_charse t(charset)
File "c:\python24\li b\email\Message .py", line 260, in set_charset
self._payload = charset.body_en code(self._payl oad)
File "c:\python24\li b\email\Charset .py", line 366, in body_encode
return email.base64MIM E.body_encode(s )
File "c:\python24\li b\email\base64M IME.py", line 136, in encode
enc = b2a_base64(s[i:i + max_unencoded])
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xa3' in
position 40: ordinal not in range(128)

....and I'm still left with this problem...

Has no-one ever successfully generated a correctly formatted email with
email.MIMEText where the message includes non-ascii characters?!

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

[Chris Withers]

Chris Withers wrote:

Has no-one ever successfully generated a correctly formatted email with
email.MIMEText where the message includes non-ascii characters?!

I'm guessing not ;-)

Well, I think I have a winner, but it required me to subclass MIMEText:

from email.Charset import Charset,QP
from email.MIMEText import MIMEText as OriginalMIMETex t
from email.MIMENonMu ltipart import MIMENonMultipar t

class MIMEText(Origin alMIMEText):

def __init__(self, _text, _subtype='plain ', _charset='us-ascii'):
if isinstance(_cha rset,Charset):
cs = _charset.input_ charset
else:
cs = _charset
if isinstance(_tex t,unicode):
_text = _text.encode(ch arset.input_cha rset)
MIMENonMultipar t.__init__(self , 'text', _subtype,
**{'charset': cs})
self.set_payloa d(_text, _charset)

charset = Charset('utf-8')
charset.body_en coding = QP
txt = u'Some text with chars that need encoding:\xa3'
msg = MIMEText(txt,'p lain',charset)
print msg.as_string()

Which gives:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

Some text with chars that need encoding:=C2=A3

It also works with non-QP charsets.

The reason the subclass is needed is because the
MIMNonMultipart .__init__ cannot handle a charset which isn't a simple
string. Since it's needed for that reason, it seems like the right place
to encode any incoming unicode.

So, by my count, there are two bugs:

1. email.MIMEText. MIMEText can't take a real Charset object to its
__init__ method.

2. email.Message.M essage.set_payl oad has no clue about unicode.

Does that sounds fair? If so, should I open SF issues for them?

cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

[Max M]

Chris Withers wrote:

Chris Withers wrote:

>print msg.as_string()

MIME-Version: 1.0
Content-Type: text/plain; charset; charset="utf-8"

^^^^^^^
Actually, even this isn't correct as you can see above...

>charset = Charset('utf-8')
msg = MIMEText('','pl ain',None)
msg.set_payloa d(u'Some text with chars that need encoding:\xa3', charset)

Has no-one ever successfully generated a correctly formatted email with
email.MIMEText where the message includes non-ascii characters?!

What is the problem with encoding the message as utf-8 before setting
the payload? That has always worked for me.
pl = u'Some text with chars that need encoding:\xa3'. encode('utf-8')
msg.set_payload (pl ,charset)

From the docs:

"""
The payload is either a string in the case of simple message objects or
a list of Message objects for MIME container documents (e.g. multipart/*
and message/rfc822)
"""

Message objects are always encoded strings. I don't remember seeing that
it should be possible to use a unicode string as a message.

The charset passed in set_payload(pl ,charset) is the charset the the
string *is* encoded in. Not the charset it *should* be encoded in.

--

hilsen/regards Max M, Denmark

http://www.mxm.dk/
IT's Mad Science

Phone: [phone]
Mobile: [phone]

[Peter Otten]

Chris Withers wrote:

Okay, more out of desperation than anything else, lets try this:

from email.Charset import Charset,QP
from email.MIMEText import MIMEText
from StringIO import StringIO
from email import Generator,Messa ge
Generator.Strin gIO = Message.StringI O = StringIO
charset = Charset('utf-8')
charset.body_en coding = QP
msg = MIMEText(u'Some text with chars that need encoding: \xa3','plain')
msg.set_charset (charset)
print repr(msg.as_str ing())
u'MIME-Version: 1.0\nContent-Transfer-Encoding: 8bit\nContent-Type:
text/plain; charset="utf-8"\n\nSome text with chars that need encoding:
\xa3'

Yay! No unicode error, but also no use:

File "c:\python24\li b\smtplib.py", line 692, in sendmail
(code,resp) = self.data(msg)
File "c:\python24\li b\smtplib.py", line 489, in data
self.send(q)
File "c:\python24\li b\smtplib.py", line 316, in send
self.sock.senda ll(str)
File "<string>", line 1, in sendall
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xa3' in
position 297: ordinal not in range(128)

Yes, it seemed to work with your original example, but of course you have to
encode unicode somehow before sending it through a wire. A severe case of
peephole debugging, sorry. I've looked into the email package source once
more, but I fear to understand the relevant parts you have to understand it
wholesale.

As Max suggested, your safest choice is probably passing in utf-8 instead of
unicode.

Peter

[Chris Withers]

Max M wrote:

From the docs:

"""
The payload is either a string in the case of simple message objects or
a list of Message objects for MIME container documents (e.g. multipart/*
and message/rfc822)
"""

Where'd you find that? I must have missed it in my digging :-S

Message objects are always encoded strings. I don't remember seeing that
it should be possible to use a unicode string as a message.

Yes, I guess I just find that surprising in today's "everything should
be unicode" world.

The charset passed in set_payload(pl ,charset) is the charset the the
string *is* encoded in. Not the charset it *should* be encoded in.

Indeed, although there's still the bug that while set_payload can accept
a Charset instance for its _charset parameter, the __init__ method for
MIMENonMultipar t cannot.

Incidentally, here's the class I finally ended up with:

from email.Charset import Charset
from email.MIMEText import MIMEText as OriginalMIMETex t
from email.MIMENonMu ltipart import MIMENonMultipar t

class MTText(Original MIMEText):

def __init__(self, _text, _subtype='plain ', _charset='us-ascii'):
if not isinstance(_cha rset,Charset):
_charset = Charset(_charse t)
if isinstance(_tex t,unicode):
_text = _text.encode(_c harset.input_ch arset)
MIMENonMultipar t.__init__(self , 'text', _subtype,
**{'charset': _charset.input_ charset})
self.set_payloa d(_text, _charset)

cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk