Python callbacks & PyGILState_Rele ase()

[Randall Hopper]

What is the correct way to propagate exceptions from Python callbacks?

When I do this:

Python -> C++ -> Python Callback

(example attached) an exception raised in the callback doesn't make it back
across C++ to Python.

It appears that PyGILState_Rele ase() at the bottom of the callback
wrapper is resetting the error state (as I can set a global based on
PyErr_Occurred( ) there, and can catch that up in the exception handler in
Python).

This obviously isn't correct. What should I be doing?

Thanks,

Randall
------------------------------------------------------------------------------
void callback_wrappe r( void *user_data )
{
// Acquire interpreter lock
PyGILState_STAT E gstate = PyGILState_Ensu re();
...
// Call Python
pyresult = PyEval_CallObje ct( pyfunc, pyargs );
...
/*********** At this point, PyErr_Occurred( ) is true **************/
/****** But it's not true when we return through C++ to Python ******/

// Free interpreter lock
PyGILState_Rele ase(gstate);
}

void registerCallbac k( PyObject *pyfunc )
{
...
// Ensure threads inited, so we can use Ensure/Release in callbacks
PyEval_InitThre ads();

// Ref pyfunc
Py_INCREF( pyfunc );

// Call underlying C++ method, registering the above C++ callback
realRegisterCal lback( callback_wrappe r, pyfunc );
}

[Thomas Heller]

Randall Hopper <vi****@charter .net> writes:

What is the correct way to propagate exceptions from Python callbacks?

When I do this:

Python -> C++ -> Python Callback

(example attached) an exception raised in the callback doesn't make it back
across C++ to Python.

It appears that PyGILState_Rele ase() at the bottom of the callback
wrapper is resetting the error state (as I can set a global based on
PyErr_Occurred( ) there, and can catch that up in the exception handler in
Python).

This obviously isn't correct. What should I be doing?

Thanks,

Randall
------------------------------------------------------------------------------
void callback_wrappe r( void *user_data )
{
// Acquire interpreter lock
PyGILState_STAT E gstate = PyGILState_Ensu re();
...
// Call Python
pyresult = PyEval_CallObje ct( pyfunc, pyargs );
...
/*********** At this point, PyErr_Occurred( ) is true **************/
/****** But it's not true when we return through C++ to Python ******/
if (pyresult == NULL)
PyErr_Print();
// Free interpreter lock
PyGILState_Rele ase(gstate);
}

PyErr_Print() will do the 'right' thing´s.

Thomas

[Randall Hopper]

Thomas Heller:
|> Python -> C++ -> Python Callback
|>
|> (example attached) an exception raised in the callback doesn't make it back
|> across C++ to Python.
....
|> void callback_wrappe r( void *user_data )
|> {
|> // Acquire interpreter lock
|> PyGILState_STAT E gstate = PyGILState_Ensu re();
|> ...
|> // Call Python
|> pyresult = PyEval_CallObje ct( pyfunc, pyargs );
|> ...
|
| if (pyresult == NULL)
| PyErr_Print();
|
|> // Free interpreter lock
|> PyGILState_Rele ase(gstate);
|> }
|
|PyErr_Print() will do the 'right' thing?s.

Thanks for the reply. However, this won't:

a) Stop the main Python script, and
b) Print the full stack trace (including Python and C++ SWIG wrapper)

Is there a clean way to save the full exception state in the callback
before the PyGILState_Rele ase(), and restore it when we return across the
C++ wrapper?

If I knew what the proper "save" and "restore" exception state code bits
were, I could easily implement this with exception typemaps in SWIG.

Thanks,

Randall

P.S. Perhaps PyGILState_Rele ase should take an argument instructing it to
exclude exception state when resetting the interpreter state back to its
original state.

[David E. Konerding DSD staff]

In article <ma************ *************** ***********@pyt hon.org>, Randall Hopper wrote:

Thomas Heller:
|> Python -> C++ -> Python Callback
|>
|> (example attached) an exception raised in the callback doesn't make it back
|> across C++ to Python.
...
|> void callback_wrappe r( void *user_data )
|> {
|> // Acquire interpreter lock
|> PyGILState_STAT E gstate = PyGILState_Ensu re();
|> ...
|> // Call Python
|> pyresult = PyEval_CallObje ct( pyfunc, pyargs );
|> ...
|
| if (pyresult == NULL)
| PyErr_Print();
|
|> // Free interpreter lock
|> PyGILState_Rele ase(gstate);
|> }
|
|PyErr_Print() will do the 'right' thing?s.

Thanks for the reply. However, this won't:

a) Stop the main Python script, and
b) Print the full stack trace (including Python and C++ SWIG wrapper)

Is there a clean way to save the full exception state in the callback
before the PyGILState_Rele ase(), and restore it when we return across the
C++ wrapper?

If I knew what the proper "save" and "restore" exception state code bits
were, I could easily implement this with exception typemaps in SWIG.

Thanks,

Randall

P.S. Perhaps PyGILState_Rele ase should take an argument instructing it to
exclude exception state when resetting the interpreter state back to its
original state.

Randall:

It's not the job of the PyGILState_* functions to manage exception details for you.

I always solved this problem a different way, by saving the exception in an instance variable
within the Python callback, and using a condition variable so that the main thread could abort on a callback's
failure. Notably, our callsbacks are always invoked from a C++ thread that the main Python interepreter didn't
create, and our main Python intepreter is blocked on a mutex most of the time.

I saved the exception state by retrieveing it from sys.exc_info(), which contains all the traceback object data as Python variable.
I think you can get the same info from your C++ callback wrapper, and use PyErr_Fetch and PyErr_Restore to save and restore the
exception sate.

Dave

[Randall Hopper]

David E. Konerding DSD staff:
|Randall Hopper wrote:
|> Is there a clean way to save the full exception state in the callback
|> before the PyGILState_Rele ase(), and restore it when we return across the
|> C++ wrapper?
....
|I saved the exception state by retrieveing it from sys.exc_info(), which
|contains all the traceback object data as Python variable. I think you can
|get the same info from your C++ callback wrapper, and use PyErr_Fetch and
|PyErr_Restore to save and restore the exception sate.

Ok, thanks. I'll give this a shot!

Randy