Python callbacks & PyGILState_Release()

[Randall Hopper]

What is the correct way to propagate exceptions from Python callbacks?

When I do this:

Python -> C++ -> Python Callback

(example attached) an exception raised in the callback doesn't make it back
across C++ to Python.

It appears that PyGILState_Release() at the bottom of the callback
wrapper is resetting the error state (as I can set a global based on
PyErr_Occurred() there, and can catch that up in the exception handler in
Python).

This obviously isn't correct. What should I be doing?

Thanks,

Randall
------------------------------------------------------------------------------
void callback_wrapper( void *user_data )
{
// Acquire interpreter lock
PyGILState_STATE gstate = PyGILState_Ensure();
...
// Call Python
pyresult = PyEval_CallObject( pyfunc, pyargs );
...
/*********** At this point, PyErr_Occurred() is true **************/
/****** But it's not true when we return through C++ to Python ******/

// Free interpreter lock
PyGILState_Release(gstate);
}

void registerCallback( PyObject *pyfunc )
{
...
// Ensure threads inited, so we can use Ensure/Release in callbacks
PyEval_InitThreads();

// Ref pyfunc
Py_INCREF( pyfunc );

// Call underlying C++ method, registering the above C++ callback
realRegisterCallback( callback_wrapper, pyfunc );
}

[Thomas Heller]

Randall Hopper <vi****@charter.net> writes:

What is the correct way to propagate exceptions from Python callbacks?

When I do this:

Python -> C++ -> Python Callback

(example attached) an exception raised in the callback doesn't make it back
across C++ to Python.

It appears that PyGILState_Release() at the bottom of the callback
wrapper is resetting the error state (as I can set a global based on
PyErr_Occurred() there, and can catch that up in the exception handler in
Python).

This obviously isn't correct. What should I be doing?

Thanks,

Randall
------------------------------------------------------------------------------
void callback_wrapper( void *user_data )
{
// Acquire interpreter lock
PyGILState_STATE gstate = PyGILState_Ensure();
...
// Call Python
pyresult = PyEval_CallObject( pyfunc, pyargs );
...
/*********** At this point, PyErr_Occurred() is true **************/
/****** But it's not true when we return through C++ to Python ******/
if (pyresult == NULL)
PyErr_Print();
// Free interpreter lock
PyGILState_Release(gstate);
}

PyErr_Print() will do the 'right' thing´s.

Thomas

[Randall Hopper]

Thomas Heller:
|> Python -> C++ -> Python Callback
|>
|> (example attached) an exception raised in the callback doesn't make it back
|> across C++ to Python.
....
|> void callback_wrapper( void *user_data )
|> {
|> // Acquire interpreter lock
|> PyGILState_STATE gstate = PyGILState_Ensure();
|> ...
|> // Call Python
|> pyresult = PyEval_CallObject( pyfunc, pyargs );
|> ...
|
| if (pyresult == NULL)
| PyErr_Print();
|
|> // Free interpreter lock
|> PyGILState_Release(gstate);
|> }
|
|PyErr_Print() will do the 'right' thing?s.

Thanks for the reply. However, this won't:

a) Stop the main Python script, and
b) Print the full stack trace (including Python and C++ SWIG wrapper)

Is there a clean way to save the full exception state in the callback
before the PyGILState_Release(), and restore it when we return across the
C++ wrapper?

If I knew what the proper "save" and "restore" exception state code bits
were, I could easily implement this with exception typemaps in SWIG.

Thanks,

Randall

P.S. Perhaps PyGILState_Release should take an argument instructing it to
exclude exception state when resetting the interpreter state back to its
original state.

[David E. Konerding DSD staff]

In article <ma**************************************@python.o rg>, Randall Hopper wrote:

Thomas Heller:
|> Python -> C++ -> Python Callback
|>
|> (example attached) an exception raised in the callback doesn't make it back
|> across C++ to Python.
...
|> void callback_wrapper( void *user_data )
|> {
|> // Acquire interpreter lock
|> PyGILState_STATE gstate = PyGILState_Ensure();
|> ...
|> // Call Python
|> pyresult = PyEval_CallObject( pyfunc, pyargs );
|> ...
|
| if (pyresult == NULL)
| PyErr_Print();
|
|> // Free interpreter lock
|> PyGILState_Release(gstate);
|> }
|
|PyErr_Print() will do the 'right' thing?s.

Thanks for the reply. However, this won't:

a) Stop the main Python script, and
b) Print the full stack trace (including Python and C++ SWIG wrapper)

Is there a clean way to save the full exception state in the callback
before the PyGILState_Release(), and restore it when we return across the
C++ wrapper?

If I knew what the proper "save" and "restore" exception state code bits
were, I could easily implement this with exception typemaps in SWIG.

Thanks,

Randall

P.S. Perhaps PyGILState_Release should take an argument instructing it to
exclude exception state when resetting the interpreter state back to its
original state.

Randall:

It's not the job of the PyGILState_* functions to manage exception details for you.

I always solved this problem a different way, by saving the exception in an instance variable
within the Python callback, and using a condition variable so that the main thread could abort on a callback's
failure. Notably, our callsbacks are always invoked from a C++ thread that the main Python interepreter didn't
create, and our main Python intepreter is blocked on a mutex most of the time.

I saved the exception state by retrieveing it from sys.exc_info(), which contains all the traceback object data as Python variable.
I think you can get the same info from your C++ callback wrapper, and use PyErr_Fetch and PyErr_Restore to save and restore the
exception sate.

Dave

[Randall Hopper]

David E. Konerding DSD staff:
|Randall Hopper wrote:
|> Is there a clean way to save the full exception state in the callback
|> before the PyGILState_Release(), and restore it when we return across the
|> C++ wrapper?
....
|I saved the exception state by retrieveing it from sys.exc_info(), which
|contains all the traceback object data as Python variable. I think you can
|get the same info from your C++ callback wrapper, and use PyErr_Fetch and
|PyErr_Restore to save and restore the exception sate.

Ok, thanks. I'll give this a shot!

Randy