Having recently deployed my inverse kinematic engine into my ProxyMan character animation tool, I came across some unusual issues that weren't present during testing. I was getting these strange rotational results, where the IK would bend and position itself correctly, but some chain items were rotated around one of the axis incorrectly.
The following is a set of three renders that show a leg being moved by the IK solver, but the rotation around one of the axis flickers between three states.
If you look over the three images, you'll see that the leg chain has been bent to reach the cube goal. But if you look at the knee, which is the selected axis in all images above, you'll see that the rotation around the green axis band is not the same. It flickers between three states. You can even see this happening with the foot joint, which is at a 90-degree angle from where it should be in the last image. Ouch!
Here's what I know: the joints are moved to the correct position. They even rotate in a way that the rest of the chain meets it's goal. But they're always rotationally out of alignment on one axis. In the examples above, it's the green axis band, which is the Y-axis, or 'UP' axis.
At this stage, these are just my findings. I'm not sure what the fix is yet.