473,326 Members | 2,113 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

2.3 encoding parsing bug

The documentation for encoding lines at

C:\Python23\Doc\Python-Docs-2.3.1\whatsnew\section-encodings.html

states:

"Encodings are declared by including a specially formatted comment in the
first or second line of the source file."

In fact, contrary to the implication, the Python 2.3 parser does not look
for lines of the form:

# -*- coding: <encoding> -*-

For example, Python improperly scans the following line for an encoding

#@+leo-ver=4-encoding=iso-8859-1.

and reports that iso-8859-1. (note trailing dot) is an invalid encoding!

The workaround for my app is to precede this line with the following line:

# -*- coding: iso-8859-1 -*-

This makes Python 2.3 happy.

To make myself perfectly clear: Python has absolutely no right to complain
about comment lines that do not have the form:

# -*- coding: <encoding> -*-

Edward
--------------------------------------------------------------------
Edward K. Ream email: ed*******@charter.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------
Jul 18 '05 #1
8 1417
"Edward K. Ream" <ed*******@charter.net> writes:
The documentation for encoding lines at

C:\Python23\Doc\Python-Docs-2.3.1\whatsnew\section-encodings.html

states:

"Encodings are declared by including a specially formatted comment in the
first or second line of the source file."

In fact, contrary to the implication, the Python 2.3 parser does not look
for lines of the form:

# -*- coding: <encoding> -*-

For example, Python improperly scans the following line for an encoding

#@+leo-ver=4-encoding=iso-8859-1.

and reports that iso-8859-1. (note trailing dot) is an invalid encoding!

The workaround for my app is to precede this line with the following line:

# -*- coding: iso-8859-1 -*-

This makes Python 2.3 happy.

To make myself perfectly clear: Python has absolutely no right to complain
about comment lines that do not have the form:

# -*- coding: <encoding> -*-


You should probably file a bug report for this.

Thomas
Jul 18 '05 #2
> "Edward K. Ream" <ed*******@charter.net> writes:
The documentation for encoding lines at

C:\Python23\Doc\Python-Docs-2.3.1\whatsnew\section-encodings.html

states:

"Encodings are declared by including a specially formatted comment in the
first or second line of the source file."

In fact, contrary to the implication, the Python 2.3 parser does not look
for lines of the form:

# -*- coding: <encoding> -*-

For example, Python improperly scans the following line for an encoding

#@+leo-ver=4-encoding=iso-8859-1.

and reports that iso-8859-1. (note trailing dot) is an invalid encoding!

The workaround for my app is to precede this line with the following line:

# -*- coding: iso-8859-1 -*-

This makes Python 2.3 happy.

To make myself perfectly clear: Python has absolutely no right to complain
about comment lines that do not have the form:

# -*- coding: <encoding> -*-

Thomas Heller <th*****@python.net> writes:

You should probably file a bug report for this.


Oops, I saw you did already.

Thomas
Jul 18 '05 #3
Edward K. Ream wrote:
To make myself perfectly clear: Python has absolutely no right to complain
about comment lines that do not have the form:

# -*- coding: <encoding> -*-


It does. Please see

http://www.python.org/doc/current/ref/encodings.html

This is the precise specification; Python looks for a certain regular
expression.

Regards,
Martin

Jul 18 '05 #4
> > To make myself perfectly clear: Python has absolutely no right to
complain
about comment lines that do not have the form:

# -*- coding: <encoding> -*-


It does. Please see

http://www.python.org/doc/current/ref/encodings.html

This is the precise specification; Python looks for a certain regular
expression.


Ah jeez :-)

The regular expression 'coding[=:]\s*([\w-_.]+)' matches so much more than
the "recommended" lines,

# -*- coding: <encoding> -*-

and

# vim:fileencoding=<encoding-name>

This is most annoying. It looks like Leo will have to change file formats
to accommodate this. I could hack a special case for .py files, I suppose,
but any such hack still amounts to a change in file format.

Is there any chance of modifying the re to reduce the possibility of
confusion and "false matches"? For example, matching only 'coding' and
'fileeencoding' (as words).

Thanks for your clarification of the situation. I suppose I'll have to look
more closely at PEP's in the future. These over-general encoding
declarations seem like a pretty low blow.

Edward

P.S. I just looked at pep 263:

[quote]
To define a source code encoding, a magic comment must
be placed into the source files either as first or second
line in the file:

#!/usr/bin/python
# -*- coding: <encoding name> -*-

More precise, the first or second line must match the regular
expression "coding[:=]\s*([\w-_]+)".
[end quote]

This was just a really bad idea, put forward in stealth, buried in an re.
Having a _restricted_ kind of special-purpose comment is one thing: having
a way-too-general kind of special-purpose comment is wrong, wrong, wrong.
It needlessly invalidates comments that _should_ have been none of Python's
business.

My guess is that I could have read this many times without having the
slightest hint of danger: the re bears almost no relation to the English
words. I'm gnashing my teeth.

EKR
--------------------------------------------------------------------
Edward K. Ream email: ed*******@charter.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------
Jul 18 '05 #5
"Edward K. Ream" <ed*******@charter.net> writes:
The workaround for my app is to precede this line with the following line:

# -*- coding: iso-8859-1 -*-

This makes Python 2.3 happy.


Presumably it would also work if you just included a pair of blank
lines (or perhaps to make it harder to accidentally remove, blank
comment lines), since Python is only going to check the first two
lines of the file.

It's still annoying, but at least you aren't then forced to bother
replicating a Python-matching line that actually contains encoding
information.

-- David
Jul 18 '05 #6
> Presumably it would also work if you just included a pair of blank
lines


Inserting two blanks lines to defeat the encoding convention would be very
bad programming style. Instead of blank lines I would insert:

# first line to defeat Python's encoding convention.
# second line to defeat Python's encoding convention.

Naturally, this is hardly an improvement over a line that explicitly
specifies the encoding.

Regardless of various workarounds, the essence of this situation is that pep
263 drastically changed the Python language. Indeed, it invalidated a file
format that I as an application designer should have had the right to
define, and _did_ have the right to define until Python 2.3 interfered. I
am not happy.

Edward
--------------------------------------------------------------------
Edward K. Ream email: ed*******@charter.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------
Jul 18 '05 #7
Edward K. Ream wrote:
Is there any chance of modifying the re to reduce the possibility of
confusion and "false matches"? For example, matching only 'coding' and
'fileeencoding' (as words).
Certainly. Propose a change to the specification, and suggest it to
python-dev. If the proposed change is acceptable, and somebodey
volunteers to provide an implementation, it will get implemented in
2.4. There is no chance of changing 2.3 in an incompatible way.
And there is, of course, no chance of changing the copies of
Python 2.3 that have already been installed.
Thanks for your clarification of the situation. I suppose I'll have to look
more closely at PEP's in the future. These over-general encoding
declarations seem like a pretty low blow.
I personally would have preferred a proper statement to declare the
encoding, such as

pragma encoding "iso-8859-1"

However, this approach was rejected as too intrusive, and a stealth
declaration in comments was considered more appropriate.
This was just a really bad idea, put forward in stealth, buried in an re.
Having a _restricted_ kind of special-purpose comment is one thing: having
a way-too-general kind of special-purpose comment is wrong, wrong, wrong.
It needlessly invalidates comments that _should_ have been none of Python's
business.
OTOH, LEO _should_ not have come up with its own syntax to specify an
encoding. Instead, LEO should have used established conventions, such
as

-*- coding: <codingname> -*-
My guess is that I could have read this many times without having the
slightest hint of danger: the re bears almost no relation to the English
words.


That is not true. The English language gives specific, recommended
examples. Users (i.e. python programmers) should use the recommended
syntax, instead of coming up with their own syntax that still matches
the regular expression.

The regular expression is introduced with the words "more precisely",
which always should make readers of formal specifications cautious.
In particular, this aspect is directed at developers of tools that
edit Python source, as this is the regular expression they need to
use to determine the encoding of the file. If LEO can read Python
files, this regular expression should have been used ever since
support for coding declarations was implemented.

Regards,
Martin

Jul 18 '05 #8
First, my apologies to python-dev et. al. for my irritable remarks re pep
263, http://www.python.org/peps/pep-0263.html and thanks to Michael Hudson
and Jeff Epler for their warm-hearted and generous responses to my
outbursts. It's so much easier to think now that there is no "vendetta"
going on :-)

This morning in the shower I realized that far from being "abused" by pep
263, Leo is, or will be, the beneficiary of pep 263. Indeed, having Python
recognize an encoding field in an #@+leo line is exactly what Leo's users
would want: it saves them from writing their own # -*- coding: <encoding
name> -*- line.

The reason Leo ran afoul of Python 2.3 is that Leo presently terminates the
encoding field with a period. Alas, periods may appear in encoding names.
Leo's convention is just wrong, so regardless of pep 263 Leo's file formats
will have to change in order to properly handle names such as
'japanese.sjis'.

The only remaining question in my mind is this: how likely is it for a user
to "innocently" match the regular expression "coding[:=]\s*([\w-_.]+)" by
mistake? I see now that Leo doesn't refute the assertion that it's not very
likely. Indeed, Leo's syntax _should_ have matched this re: the problem
arose not from any defect in pep 263 but from a very real bug in Leo.

In short, my opinion of pep 263 has undergone an almost 180 degree
turnaround. I like it, Leo's users will benefit from it, and it seems
unlikely that other people's existing code will suffer. Indeed, one would
typically expect an initial line containing "coding:" or "coding=" to be
followed by a valid Unicode encoding.

Two other thoughts:

1. The summaries of pep 263 such as
http://www.python.org/doc/2.3.3/what...encodings.html are not
accurate, that is, they do not affect what really happens. IMO, it would
make more sense to describe the re in English (as well as give the actual
re) and to give the rationale for making the re fairly general.

2. I wonder if it makes sense to do something besides throwing a syntax
error if the encoding isn't recognized. I suspect this topic has already
been discussed. Can anyone summarize it for me?

Many thanks to all who have responded, publicly or in private, to me on this
subject.

Edward
--------------------------------------------------------------------
Edward K. Ream email: ed*******@charter.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------
Jul 18 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: Mark | last post by:
I've run a few simple tests looking at how query string encoding/decoding gets handled in asp.net, and it seems like the situation is even messier than it was in asp... Can't say I think much of the...
15
by: Steven Bethard | last post by:
I'm having trouble using elementtree with an XML file that has some gbk-encoded text. (I can't read Chinese, so I'm taking their word for it that it's gbk-encoded.) I always have trouble with...
6
by: saumya.agarwal | last post by:
Hi, I am using libxml2 for xml parsing. When the client application sends data to libxml2 in UTF-8 format, it works fine. But, I have a scenarion in which the client application sends data to...
0
by: Zvi | last post by:
Hi All, I got trouble parsing xml returned by web service. XML data contains characters above 128, so ET.fromstring throws an error. Error is thrown from python's xmllib.py file, where it...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.