In light of finally receiving the first tranche of my optical motion capture (mocap) gear, I started to think about how I can connect things up together. To be able to track something, and then have that tracked data readily available in my 3D animation software for use. There are, of course, a few ways in which we could go about this. The mocap software itself can export into a number of file types. These include the usual suspects with FBX and BVH, but also include more rudimentary ways such as CSV.
Yep - you can export data into a spreadsheet! As odd at this may seem, there are legitimate reasons for doing this, such as being able to analyse the data later on for medical or scientific purposes.
For my needs however, I'm just after the pure tracked position and rotational data. And for that, I have the FBX and BVH exports, or I can connect to the software remotely through a network connection and pull the take data through a streaming service. To make the latter streaming option work, I need to program something myself.
Now, the other export options (FBX, BVH) are certainly things I'll use where I can. BVH in particular, because I intend on using it with my ProxyMan tool to animate characters (I hope..!!). However, upon testing the FBX exports, I found my animation package doesn't listen to the binary file types, and the ASCII versions do load but are static. That's to say, there's no animation data coming through.
I could write my own FBX importer I guess, but I'm hoping not have to do that. Seems a bit over-the-top for my needs (says me, programming a tool anyway...).
Instead, I'm going to try my own client streaming tools. I thought if I could make this work, it opens up a few options for me, particularly in the future for live streaming.
So, I chose to have a go at writing something myself. And I thought the best place for the initial idea is to sit in a tag plugin, that I could add to any object in a scene, and control the object's position and rotation using a live stream connection to the mocap system.
Thus far, I have managed to get a connection working. That was the first tick box. The next tick box was to be able to select a rigid body back inside the mocap recording, and grab it's position and rotational data for a particular frame. I was able to get both of these, but it did require some conversion. You see, the mocap data is in a particular 3D space; it's a right-handed coordinate system. However, my animation package is left-handed. So I have to convert the positional data over by rotating it around the Y axis 180 degrees, and then negating the Z axis. This however, doe snot work quite like this with the rotational data. I had troubles figuring this out, took me many hours. But I finally managed to solve this. The mocap data is in quaternions, and I couldn't just use the straight quaternion to pull a rotation matrix. I had to reverse both the Y and Z components, and the W component. Seems odd, but converting between 'handedness' it's not completely straight forward. That is, you may not be able to just mirror the X axis for example. You have to jump the handedness hurdle a little differently.
With those issues solved (* more on this below..), I was able to get a correct position and rotation interpretation of the rigid body frame data inside my animation software. Having made this possible, I added a bit more code to run the system with a live feed, and, I got this:
So, that's pretty cool. It's not a great example I know, but it does show the proof of concept. Now to find a way to record the data and use it as keyframes for animation.
My setup currently allows me to have the mocap software on a computer box and have it sit externally somewhere else. The idea being that I can track something (e.g., set the system up in the garage, and record there) and later pull the stream from any of the recorded takes back into my 'workstation' in another room. Things don't absolutely have to work this way, but it gives me options in the future.
And building the system into a tag like I have keeps things tidy and controllable. I'm working on a way to record things inside the animation package as well, currently it's jut a live feed but no actual data recording. That's the next box to tick. Beyond that, I need to start figuring out how I'm going to use all of these tools I've built in a production!
Exciting, lots more to come.
* Update
I've since found that there are also issues when your rotational values go beyond 360, or into negative territory. The system seems to flip them on each end, so -1 degree becomes 359 and 361 degrees becomes 1. This is problematic because if your rotations are hovering around either end of the scale, they constantly flip between 359 and 1. But in computer land, this means they're flipping also a full 360 degrees. I found this was happening when I was interpreting in-between keyframes. It thought the object was doing a full 360 turn, so it interpolated the values in between for a full rotation. I'm working on trying to find a fix for this!