I want to approximate the rotation in SO(3) (ie. 3D rotation) that minimizes some objective function.
I'm looking for a SO(3) rotation representation that lends itself to energy minimization. I'm writing a framework that performs minimization on shape parameters, where the goodness of the fit is given solely based on some arbitrary objective function which is a function of a rotation matrix.
One example solution would be to convert euler angles to matrix representation, minimize the objective function as a function of the angle parameters, and convert back to a matrix. This effectively to reparameterizing the objective function into some other rotation representation. However, one problem with euler angles is that the representation does not preserve uniformity and is prone gimbal locking problems. Thus I am afraid it may give biased results or create local minima.
I have heard it is possible to optimize based on quaternion or axis angle representations of 3D rotations, but I have been unable to find a good source or description.
(Added) More specifically. As suggested by Choward in the comments, a solution using quaternions would formaly be: Find some quaternion,$q^*$, that satisfies: $q^*=\arg \min f(R(q))$, where $R(q)$ is a rotation matrix as a function of the quaternion, and $f(\cdot)$ is some cost function being minimized as a function of the rotation matrix.
Edit2: Added clarification based on comment.