Undocumented Python 2.6 change: Py_None vs NULL when C implementationraises exception

[Heikki Toivonen]

I was debugging M2Crypto function written in C which changed behavior
between Python 2.6 and earlier Python versions. In an error condition
the function was supposed to raise exception type A, but with 2.6 it
raised type B, and further, there was no string value for the exception.

I tracked this down to the C code incorrectly returning Py_None when it
should have returned NULL. Changing the C code to return NULL made it
behave correctly in 2.6.

I don't know how common a mistake it is to return Py_None when NULL
should have been returned, but it might be worth a note in the list of
changes for 2.6 that this behavior changed, don't you think?

--
Heikki Toivonen

[Fredrik Lundh]

Heikki Toivonen wrote:

I was debugging M2Crypto function written in C which changed behavior
between Python 2.6 and earlier Python versions. In an error condition
the function was supposed to raise exception type A, but with 2.6 it
raised type B, and further, there was no string value for the exception.

I tracked this down to the C code incorrectly returning Py_None when it
should have returned NULL. Changing the C code to return NULL made it
behave correctly in 2.6.

I don't know how common a mistake it is to return Py_None when NULL
should have been returned, but it might be worth a note in the list of
changes for 2.6 that this behavior changed, don't you think?

the behaviour when setting the exception status without returning NULL
has never been well-defined; it pretty much depends on what kind of
error checking the code that runs later happens to use.

(this has been an entertaining source of obscure errors over the years;
code that uses PyErr_Clear may mask errors, and code that uses
PyErr_Occurred to check if something went wrong might raise the wrong
error, often at an unexpected location).

</F>

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Heikki Toivonen wrote:

I was debugging M2Crypto function written in C which changed behavior
between Python 2.6 and earlier Python versions. In an error condition
the function was supposed to raise exception type A, but with 2.6 it
raised type B, and further, there was no string value for the exception.

I tracked this down to the C code incorrectly returning Py_None when it
should have returned NULL. Changing the C code to return NULL made it
behave correctly in 2.6.

Can you please be specific what function you are talking about?

I don't know how common a mistake it is to return Py_None when NULL
should have been returned, but it might be worth a note in the list of
changes for 2.6 that this behavior changed, don't you think?

Perhaps. OTOH, perhaps the change is completely erroneous. In that case,
rather than documenting it, it should be reverted.

Unfortunately, as you keep the specific issue secret, none of this will
happen, as we have no clue what you are talking about.

I'm sure there are tons of silent changes, in this release, all past
releases, and all future releases, not only in Python, but in any
software.

Regards,
Martin

[Heikki Toivonen]

Martin v. Löwis wrote:

Heikki Toivonen wrote:

>I tracked this down to the C code incorrectly returning Py_None when it
should have returned NULL. Changing the C code to return NULL made it
behave correctly in 2.6.

Can you please be specific what function you are talking about?

Perhaps it wasn't clear that I was referring to the C code in an
extension, M2Crypto. I assumed that this affected all extension code
like this, which is why I didn't mention the actual lines. It appears
from Fredrik's comment that this might not be just a 2.6 issue, but that
this problem has cropped up in the past as well more or less randomly.

>I don't know how common a mistake it is to return Py_None when NULL
should have been returned, but it might be worth a note in the list of
changes for 2.6 that this behavior changed, don't you think?

Perhaps. OTOH, perhaps the change is completely erroneous. In that case,
rather than documenting it, it should be reverted.

Unfortunately, as you keep the specific issue secret, none of this will
happen, as we have no clue what you are talking about.

I don't know what change in Python caused the change in M2Crypto
behavior. I can only point you to the change I made in M2Crypto if you
are interested:
http://viewcvs.osafoundation.org/m2c...&r1=611&r2=531

If you revert that change and run the M2Crypto unit tests you will see
the single error in the tests.

I'm sure there are tons of silent changes, in this release, all past
releases, and all future releases, not only in Python, but in any
software.

Given that there is a long document showing the changes in each Python
release, I would hope all intended changes of significance would be
listed. Of course mistakes can happen, which was why I posted in the
first place.

I have no plans to track down the exact change in Python code that
caused this. There does not seem to be much point, since according to
Fredrik this seems to be an area that is practically undefined and the
M2Crypto code was clearly buggy.

--
Heikki Toivonen

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I have no plans to track down the exact change in Python code that

caused this. There does not seem to be much point, since according to
Fredrik this seems to be an area that is practically undefined and the
M2Crypto code was clearly buggy.

I see, and I agree with Fredrik's analysis. It might actually be that
there was *no* change in 2.6 causing this change in behavior, but just
a difference in data returned in the actual application. E.g. if a
PyInt_FromLong returns -1, the caller also needs to check for
PyErr_Occurred, which would then detect an earlier exception. So it
might be that if you have -1 in your data, you see the exception, but
if you have -2 instead, you won't see it.

Regards,
Martin