473,233 Members | 1,565 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,233 software developers and data experts.

How to print first(national) char from unicode string encoded inutf-8?

Hi,

I have a problem with unicode string in Pylons templates(Mako). I will
print first char from my string encoded in UTF-8 and urllib.quote(),
for example string 'úukasz':

${urllib.unquote(c.user.firstName).encode('latin-1')[0:1]}

and I received this information:

<type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte
0xc5 in position 0: unexpected end of data

When I change from [0:1] to [0:2] everything is ok. I think it is
because of unicode and encoding utf-8(2 bytes).

How to resolve this problem?

Best regards
Sep 1 '08 #1
5 2345
On 1 Wrz, 15:10, "Marco Bizzarri" <marco.bizza...@gmail.comwrote:
2008/9/1 <sni...@gmail.com>:
Hi,
I have a problem with unicode string in Pylons templates(Mako). I will
print first char from my string encoded in UTF-8 and urllib.quote(),
for example string 'Łukasz':
${urllib.unquote(c.user.firstName).encode('latin-1')[0:1]}
and I received this information:
<type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte
0xc5 in position 0: unexpected end of data
When I change from [0:1] to [0:2] everything is ok. I think it is
because of unicode and encoding utf-8(2 bytes).
How to resolve this problem?
Best regards
--
http://mail.python.org/mailman/listinfo/python-list

First: you're talking about utf8 encoding, but you've written latin1
encoding. Even though I do not know Mako templates, there should be no
problem in your snippet of code, if encoding is latin1, at least for
what I can understand.

Do not assume utf8 is a two byte encoding; utf8 is a variable length
encoding. Indeed,

'a' encoded as utf8 is 'a' (one byte)

'├*' encode as utf8 is '\xc3\xa0' (two bytes).

Can you explain what you're trying to accomplish (rather than how
you're tryin to accomplish it) ?

Regards
Marco

--
Marco Bizzarrihttp://notenotturne.blogspot.com/http://iliveinpisa.blogspot.com/
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┼üukasz but ├ů┬üukasz
Sep 1 '08 #2
On Mon, Sep 1, 2008 at 3:25 PM, <sn****@gmail.comwrote:
>
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
--
http://mail.python.org/mailman/listinfo/python-list
That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.

Have you tried:

urlib.unquote(c.user.firstName)[0].encode('latin1') or

urlib.unquote(c.user.firstName)[0].encode('utf8')

I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.

Regards
Marco
--
Marco Bizzarri
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/
Sep 1 '08 #3

"Marco Bizzarri" <ma************@gmail.comwrote in message
news:ma*************************************@pytho n.org...
On Mon, Sep 1, 2008 at 3:25 PM, <sn****@gmail.comwrote:
>>
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
--
http://mail.python.org/mailman/listinfo/python-list

That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.

Have you tried:

urlib.unquote(c.user.firstName)[0].encode('latin1') or

urlib.unquote(c.user.firstName)[0].encode('utf8')

I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.
The OP stated that the original string was "encoded in UTF-8 and
urllib.quote()", so after urllib.unquote the string is in UTF-8 format.
This must be decoded into a Unicode string before removing the first
character:

urllib.unquote(c.user.firstName).decode('utf-8')[0]

The next problem is that the character in the OP's example string '┘' is not
present in the latin-1 encoding, but using utf-8 encoding demonstrates that
the full two-byte UTF-8 encoded character is collected:
>>import urllib
name = urllib.quote(u'┘ukasz'.encode('utf-8'))
name
'%C5%81ukasz'
>>urllib.unquote(name).decode('utf-8')[0].encode('utf-8')
'\xc5\x81'

-Mark

Sep 2 '08 #4
On 2 Wrz, 06:05, "Mark Tolonen" <M8R-yft...@mailinator.comwrote:
"Marco Bizzarri" <marco.bizza...@gmail.comwrote in message

news:ma*************************************@pytho n.org...
On Mon, Sep 1, 2008 at 3:25 PM, <sni...@gmail.comwrote:
When I do ${urllib.unquote(c.user.firstName)} without encoding to
latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
--
http://mail.python.org/mailman/listinfo/python-list
That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.
Have you tried:
urlib.unquote(c.user.firstName)[0].encode('latin1') or
urlib.unquote(c.user.firstName)[0].encode('utf8')
I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.

The OP stated that the original string was "encoded in UTF-8 and
urllib.quote()", so after urllib.unquote the string is in UTF-8 format.
This must be decoded into a Unicode string before removing the first
character:

urllib.unquote(c.user.firstName).decode('utf-8')[0]

The next problem is that the character in the OP's example string '┘' is not
present in the latin-1 encoding, but using utf-8 encoding demonstrates that
the full two-byte UTF-8 encoded character is collected:
>>import urllib
>>name = urllib.quote(u'┘ukasz'.encode('utf-8'))
>>name
'%C5%81ukasz'
>>urllib.unquote(name).decode('utf-8')[0].encode('utf-8')
'\xc5\x81'

-Mark
@Mark, when I tried urllib.unquote(c.user.firstName).decode('utf-8')
[0].encode('utf-8'), I received this message:
> return render('/reports/create_report_step2.mako')
Module pylons.templating:344 in render
<< **cache_args)
return pylons.buffet.render(template_name=template,
fragment=fragment,
format=format, namespace=kargs,
**cache_args)
> format=format, namespace=kargs, **cache_args)
Module pylons.templating:229 in render
<< log.debug("Rendering template %s with engine %s",
full_path, engine_name)
return engine_config['engine'].render(namespace,
template=full_path,
**options)> **options)
Module mako.ext.turbogears:49 in render
<< info.update(self.extra_vars_func())

return template.render(**info)
> return template.render(**info)
Module mako.template:114 in render
<< declared by this template's internal rendering method are
also pulled from the given *args, **data
members. members."""
return runtime._render(self, self.callable_, args, data)

def render_unicode(self, *args, **data):> return
runtime._render(self, self.callable_, args, data)
Module mako.runtime:287 in _render
<< context = Context(buf, **data)
context._with_template = template
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
return context.pop_buffer().getvalue()>>
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
Module mako.runtime:304 in _render_context
<< # if main render method, call from the base of the
inheritance stack
(inherit, lclcontext) = _populate_self_namespace(context,
tmpl)
_exec_template(inherit, lclcontext, args=args,
kwargs=kwargs)
else:
# otherwise, call the actual rendering method specified>>
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
Module mako.runtime:337 in _exec_template
<< error_template.render_context(context,
error=error)
else:
callable_(context, *args, **kwargs)> callable_(context,
*args, **kwargs)
Module _reports_create_report_step2_mako:57 in render_body
<<
context.write(filters.decode.utf8(urllib.unquote(s tr(c.period.end))))
context.write(u' + ')

context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))

context.write(filters.decode.utf8(urllib.unquote(s tr(c.user.secondName)
[0:1])))
context.write(u'</h3>\r\n <input type="hidden"
name="works[]" value="')>>
context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))
Module encodings.utf_8:16 in decode
<<
def decode(input, errors='strict'):
return codecs.utf_8_decode(input, errors, True)

class IncrementalEncoder(codecs.IncrementalEncoder):> return
codecs.utf_8_decode(input, errors, True)
<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode
characters in position 0-1: ordinal not in range(128)
Sep 2 '08 #5
On 2 Wrz, 10:17, sni...@gmail.com wrote:
On 2 Wrz, 06:05, "Mark Tolonen" <M8R-yft...@mailinator.comwrote:
"Marco Bizzarri" <marco.bizza...@gmail.comwrote in message
news:ma*************************************@pytho n.org...
On Mon, Sep 1, 2008 at 3:25 PM, <sni...@gmail.comwrote:
>When I do ${urllib.unquote(c.user.firstName)} without encoding to
>latin-1 I got different chars than I will get: no ┘ukasz but ┼ ukasz
>--
>>http://mail.python.org/mailman/listinfo/python-list
That's crazy. "string".encode('latin1') gives you a latin1 encoded
string; latin1 is a single byte encoding, therefore taking the first
byte should be no problem.
Have you tried:
urlib.unquote(c.user.firstName)[0].encode('latin1') or
urlib.unquote(c.user.firstName)[0].encode('utf8')
I'm assuming here that the urlib.unquote(c.user.firstName) returns an
encodable string (which I'm absolutely not sure), but if it does, this
should take the first 'character'.
The OP stated that the original string was "encoded in UTF-8 and
urllib.quote()", so after urllib.unquote the string is in UTF-8 format.
This must be decoded into a Unicode string before removing the first
character:
urllib.unquote(c.user.firstName).decode('utf-8')[0]
The next problem is that the character in the OP's example string '┘'is not
present in the latin-1 encoding, but using utf-8 encoding demonstrates that
the full two-byte UTF-8 encoded character is collected:
>>import urllib
>>name = urllib.quote(u'┘ukasz'.encode('utf-8'))
>>name
'%C5%81ukasz'
>>urllib.unquote(name).decode('utf-8')[0].encode('utf-8')
'\xc5\x81'
-Mark

@Mark, when I tried urllib.unquote(c.user.firstName).decode('utf-8')
[0].encode('utf-8'), I received this message:
return render('/reports/create_report_step2.mako')

Module pylons.templating:344 in render
<< **cache_args)
return pylons.buffet.render(template_name=template,
fragment=fragment,
format=format, namespace=kargs,
**cache_args)
> format=format, namespace=kargs, **cache_args)
Module pylons.templating:229 in render
<< log.debug("Rendering template %s with engine %s",
full_path, engine_name)
return engine_config['engine'].render(namespace,
template=full_path,
**options)> **options)
Module mako.ext.turbogears:49 in render
<< info.update(self.extra_vars_func())

return template.render(**info)
> return template.render(**info)
Module mako.template:114 in render
<< declared by this template's internal rendering method are
also pulled from the given *args, **data
members. members."""
return runtime._render(self, self.callable_, args, data)

def render_unicode(self, *args, **data):> return
runtime._render(self, self.callable_, args, data)
Module mako.runtime:287 in _render
<< context = Context(buf, **data)
context._with_template = template
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
return context.pop_buffer().getvalue()>>
_render_context(template, callable_, context, *args,
**_kwargs_for_callable(callable_, data))
Module mako.runtime:304 in _render_context
<< # if main render method, call from the base of the
inheritance stack
(inherit, lclcontext) = _populate_self_namespace(context,
tmpl)
_exec_template(inherit, lclcontext, args=args,
kwargs=kwargs)
else:
# otherwise, call the actual rendering method specified>>
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
Module mako.runtime:337 in _exec_template
<< error_template.render_context(context,
error=error)
else:
callable_(context, *args, **kwargs)> callable_(context,
*args, **kwargs)
Module _reports_create_report_step2_mako:57 in render_body
<<
context.write(filters.decode.utf8(urllib.unquote(s tr(c.period.end))))
context.write(u' + ')

context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))

context.write(filters.decode.utf8(urllib.unquote(s tr(c.user.secondName)
[0:1])))
context.write(u'</h3>\r\n <input type="hidden"
name="works[]" value="')>>
context.write(filters.decode.utf8(urllib.unquote(c .user.firstName).decode('utf-8')
[0].encode('utf-8')))
Module encodings.utf_8:16 in decode
<<
def decode(input, errors='strict'):
return codecs.utf_8_decode(input, errors, True)

class IncrementalEncoder(codecs.IncrementalEncoder):> return
codecs.utf_8_decode(input, errors, True)
<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode
characters in position 0-1: ordinal not in range(128)
ok, I resolved this problem $
{urllib.unquote(str(c.user.firstName)).decode('utf-8')[0]}

Could anyone explain me why this code works?
Sep 2 '08 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Balaji | last post by:
Hello Eveybody, I have written a method which prints the prefix notation of any expression. here is the method... def PrintPrefix(expr): if expr.__class__==E: print expr.operator,...
2
by: Timothy Babytch | last post by:
Imagine you have some list that looks like ('unicode', 'not-acii', 'russian') and contains characters not from acsii. or list of dicts, or dict of dicts. how can I print it? not on by one, with...
5
by: Mel | last post by:
i want to have a block, like some articles i see, with the first char (very Fancy and huge) and have the rest of the block kind of wrap around this char (to the right of). something like below...
29
by: JS | last post by:
I would like to print char 'd': main(){ char g; g = 'a'; g = 'b'; g = 'c'; g = 'd';
4
by: jceddy | last post by:
Hey, I'm trying to write a file with unix-style newlines (ASCII character 10) from a c++ program on Windows...it seems that the most straightforward way to do that is just to print ( char )10, but...
3
by: applegreenss | last post by:
I am looking for a javascript function ( don't know what they call ths ) which will print a line of text, one letter at a time at a certain speed until the full string is printed. Can this be done...
2
by: lekshminair | last post by:
hello friends, can u help me. how to print first string in a line.(before first space) for example: String str={"Hello world java"} output Hello
8
by: lovecreatesbea... | last post by:
Thank you for your time. #include <stdio.h> int main(void) { int c; while ((c = getchar()) != EOF){
38
by: ssecorp | last post by:
char* reverse(char* str) { int length = strlen(str); char* acc; int i; for (i=0; i<=length-1; i++){ acc = str; } return acc; }
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, youĺll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.