468,272 Members | 1,956 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,272 developers. It's quick & easy.

How to print first(national) char from unicode string encoded inutf-8?

Hi,

I have a problem with unicode string in Pylons templates(Mako). I will
print first char from my string encoded in UTF-8 and urllib.quote(),
for example string 'úukasz':

${urllib.unquote(c.user.firstName).encode('latin-1')[0:1]}

and I received this information:

<type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte
0xc5 in position 0: unexpected end of data

When I change from [0:1] to [0:2] everything is ok. I think it is
because of unicode and encoding utf-8(2 bytes).

How to resolve this problem?

Best regards
Sep 1 '08 #1
5 2145
On 1 Wrz, 15:10, "Marco Bizzarri" <marco.bizza...@gmail.comwrote:
2008/9/1 <sni...@gmail.com>:
Hi,
I have a problem with unicode string in Pylons templates(Mako). I will
print first char from my string encoded in UTF-8 and urllib.quote(),
for example string 'Łukasz':
${urllib.unquote(c.user.firstName).encode('latin-1')[0:1]}
and I received this information:
<type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte
0xc5 in position 0: unexpected end of data
When I change from [0:1] to [0:2] everything is ok. I think it is
because of unicode and encoding utf-8(2 bytes).
How to resolve this problem?
Best regards
--
http://mail.python.org/mailman/listinfo/python-list

First: you're talking about utf8 encoding, but you've written latin1
encoding. Even though I do not know Mako templates, there should be no
problem in your snippet of code, if encoding is latin1, at least for
what I can understand.

Do not assume utf8 is a two byte encoding; utf8 is a variable length
encoding. Indeed,

'a' encoded as utf8 is 'a' (one byte)

'├*' encode as utf8 is '\xc3\xa0' (two bytes).

Can you explain what you're trying to accomplish (rather than how
you're tryin to accomplish it) ?

Regards
Marco

--
Marco Bizzarrihttp://notenotturne.blogspot.com/http://iliveinpisa.blogspot.com/
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┼üukasz but ├ů┬üukasz
Sep 1 '08 #2
On Mon, Sep 1, 2008 at 3:25 PM, <sn****@gmail.comwrote:
>
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
--
http://mail.python.org/mailman/listinfo/python-list
That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.

Have you tried:

urlib.unquote(c.user.firstName)[0].encode('latin1') or

urlib.unquote(c.user.firstName)[0].encode('utf8')

I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.

Regards
Marco
--
Marco Bizzarri
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/
Sep 1 '08 #3

"Marco Bizzarri" <ma************@gmail.comwrote in message
news:ma*************************************@pytho n.org...
On Mon, Sep 1, 2008 at 3:25 PM, <sn****@gmail.comwrote:
>>
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
--
http://mail.python.org/mailman/listinfo/python-list

That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.

Have you tried:

urlib.unquote(c.user.firstName)[0].encode('latin1') or

urlib.unquote(c.user.firstName)[0].encode('utf8')

I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.
The OP stated that the original string was "encoded in UTF-8 and
urllib.quote()", so after urllib.unquote the string is in UTF-8 format.
This must be decoded into a Unicode string before removing the first
character:

urllib.unquote(c.user.firstName).decode('utf-8')[0]

The next problem is that the character in the OP's example string '┘' is not
present in the latin-1 encoding, but using utf-8 encoding demonstrates that
the full two-byte UTF-8 encoded character is collected:
>>import urllib
name = urllib.quote(u'┘ukasz'.encode('utf-8'))
name
'%C5%81ukasz'
>>urllib.unquote(name).decode('utf-8')[0].encode('utf-8')
'\xc5\x81'

-Mark

Sep 2 '08 #4
On 2 Wrz, 06:05, "Mark Tolonen" <M8R-yft...@mailinator.comwrote:
"Marco Bizzarri" <marco.bizza...@gmail.comwrote in message

news:ma*************************************@pytho n.org...
On Mon, Sep 1, 2008 at 3:25 PM, <sni...@gmail.comwrote:
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
--
http://mail.python.org/mailman/listinfo/python-list
That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.
Have you tried:
urlib.unquote(c.user.firstName)[0].encode('latin1') or
urlib.unquote(c.user.firstName)[0].encode('utf8')
I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.

The OP stated that the original string was "encoded in UTF-8 and
urllib.quote()", so after urllib.unquote the string is in UTF-8 format.
This must be decoded into a Unicode string before removing the first
character:

urllib.unquote(c.user.firstName).decode('utf-8')[0]

The next problem is that the character in the OP's example string '┘' is not
present in the latin-1 encoding, but using utf-8 encoding demonstrates that
the full two-byte UTF-8 encoded character is collected:
>>import urllib
>>name = urllib.quote(u'┘ukasz'.encode('utf-8'))
>>name
'%C5%81ukasz'
>>urllib.unquote(name).decode('utf-8')[0].encode('utf-8')
'\xc5\x81'

-Mark
@Mark, when I tried urllib.unquote(c.user.firstName).decode('utf-8')
[0].encode('utf-8'), I received this message:
> return render('/reports/create_report_step2.mako')
Module pylons.templating:344 in render
<< **cache_args)
return pylons.buffet.render(template_name=template,
fragment=fragment,
format=format, namespace=kargs,
**cache_args)
> format=format, namespace=kargs, **cache_args)
Module pylons.templating:229 in render
<< log.debug("Rendering template %s with engine %s",
full_path, engine_name)
return engine_config['engine'].render(namespace,
template=full_path,
**options)> **options)
Module mako.ext.turbogears:49 in render
<< info.update(self.extra_vars_func())

return template.render(**info)
> return template.render(**info)
Module mako.template:114 in render
<< declared by this template's internal rendering method are
also pulled from the given *args, **data
members. members."""
return runtime._render(self, self.callable_, args, data)

def render_unicode(self, *args, **data):> return
runtime._render(self, self.callable_, args, data)
Module mako.runtime:287 in _render
<< context = Context(buf, **data)
context._with_template = template
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
return context.pop_buffer().getvalue()>>
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
Module mako.runtime:304 in _render_context
<< # if main render method, call from the base of the
inheritance stack
(inherit, lclcontext) = _populate_self_namespace(context,
tmpl)
_exec_template(inherit, lclcontext, args=args,
kwargs=kwargs)
else:
# otherwise, call the actual rendering method specified>>
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
Module mako.runtime:337 in _exec_template
<< error_template.render_context(context,
error=error)
else:
callable_(context, *args, **kwargs)> callable_(context,
*args, **kwargs)
Module _reports_create_report_step2_mako:57 in render_body
<<
context.write(filters.decode.utf8(urllib.unquote(s tr(c.period.end))))
context.write(u' + ')

context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))

context.write(filters.decode.utf8(urllib.unquote(s tr(c.user.secondName)
[0:1])))
context.write(u'</h3>\r\n <input type="hidden"
name="works[]" value="')>>
context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))
Module encodings.utf_8:16 in decode
<<
def decode(input, errors='strict'):
return codecs.utf_8_decode(input, errors, True)

class IncrementalEncoder(codecs.IncrementalEncoder):> return
codecs.utf_8_decode(input, errors, True)
<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode
characters in position 0-1: ordinal not in range(128)
Sep 2 '08 #5
On 2 Wrz, 10:17, sni...@gmail.com wrote:
On 2 Wrz, 06:05, "Mark Tolonen" <M8R-yft...@mailinator.comwrote:
"Marco Bizzarri" <marco.bizza...@gmail.comwrote in message
news:ma*************************************@pytho n.org...
On Mon, Sep 1, 2008 at 3:25 PM, <sni...@gmail.comwrote:
>When I do ${urllib.unquote(c.user.firstName)} without encoding to
>latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
>--
>>http://mail.python.org/mailman/listinfo/python-list
That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.
Have you tried:
urlib.unquote(c.user.firstName)[0].encode('latin1') or
urlib.unquote(c.user.firstName)[0].encode('utf8')
I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.
The OP stated that the original string was "encoded in UTF-8 and
urllib.quote()", so after urllib.unquote the string is in UTF-8 format.
This must be decoded into a Unicode string before removing the first
character:
urllib.unquote(c.user.firstName).decode('utf-8')[0]
The next problem is that the character in the OP's example string '┘'is not
present in the latin-1 encoding, but using utf-8 encoding demonstrates that
the full two-byte UTF-8 encoded character is collected:
>>import urllib
>>name = urllib.quote(u'┘ukasz'.encode('utf-8'))
>>name
'%C5%81ukasz'
>>urllib.unquote(name).decode('utf-8')[0].encode('utf-8')
'\xc5\x81'
-Mark

@Mark, when I tried urllib.unquote(c.user.firstName).decode('utf-8')
[0].encode('utf-8'), I received this message:
return render('/reports/create_report_step2.mako')

Module pylons.templating:344 in render
<< **cache_args)
return pylons.buffet.render(template_name=template,
fragment=fragment,
format=format, namespace=kargs,
**cache_args)
> format=format, namespace=kargs, **cache_args)
Module pylons.templating:229 in render
<< log.debug("Rendering template %s with engine %s",
full_path, engine_name)
return engine_config['engine'].render(namespace,
template=full_path,
**options)> **options)
Module mako.ext.turbogears:49 in render
<< info.update(self.extra_vars_func())

return template.render(**info)
> return template.render(**info)
Module mako.template:114 in render
<< declared by this template's internal rendering method are
also pulled from the given *args, **data
members. members."""
return runtime._render(self, self.callable_, args, data)

def render_unicode(self, *args, **data):> return
runtime._render(self, self.callable_, args, data)
Module mako.runtime:287 in _render
<< context = Context(buf, **data)
context._with_template = template
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
return context.pop_buffer().getvalue()>>
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
Module mako.runtime:304 in _render_context
<< # if main render method, call from the base of the
inheritance stack
(inherit, lclcontext) = _populate_self_namespace(context,
tmpl)
_exec_template(inherit, lclcontext, args=args,
kwargs=kwargs)
else:
# otherwise, call the actual rendering method specified>>
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
Module mako.runtime:337 in _exec_template
<< error_template.render_context(context,
error=error)
else:
callable_(context, *args, **kwargs)> callable_(context,
*args, **kwargs)
Module _reports_create_report_step2_mako:57 in render_body
<<
context.write(filters.decode.utf8(urllib.unquote(s tr(c.period.end))))
context.write(u' + ')

context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))

context.write(filters.decode.utf8(urllib.unquote(s tr(c.user.secondName)
[0:1])))
context.write(u'</h3>\r\n <input type="hidden"
name="works[]" value="')>>
context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))
Module encodings.utf_8:16 in decode
<<
def decode(input, errors='strict'):
return codecs.utf_8_decode(input, errors, True)

class IncrementalEncoder(codecs.IncrementalEncoder):> return
codecs.utf_8_decode(input, errors, True)
<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode
characters in position 0-1: ordinal not in range(128)
ok, I resolved this problem $
{urllib.unquote(str(c.user.firstName)).decode('utf-8')[0]}

Could anyone explain me why this code works?
Sep 2 '08 #6

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

2 posts views Thread by Balaji | last post: by
2 posts views Thread by Timothy Babytch | last post: by
5 posts views Thread by Mel | last post: by
29 posts views Thread by JS | last post: by
4 posts views Thread by jceddy | last post: by
8 posts views Thread by lovecreatesbea... | last post: by
reply views Thread by zattat | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.