It sounds like you're trying to do an SO(3) rotation, the exact parameterization of this depends on the basis of your matrix. In an orthonormal Eucledian basis, rotation in any arbitrary direction can be accomplished using 3 Euler angles; the parameterization of the determinant 1 matrix which does this can be found
here. To implement matrix algebra in C/C++, you might want to take a look at the
GNU Scientific Library (GSL) or the uBLAS library which is part of
boost.
Arnaud