Please note: I made this post originally a few years back, but I never finished it. I found it still in draft mode while looking through some older posts, so I'm posting it now for historic reasons, albeit with a few grammatical changes.
Some years ago I thought about building a tracking engine for The Dark Room. I thought it might be useful to composite some cheap effects onto my film images without having to use an external compositing program. But after spending some time on it while away on holiday, I couldn't get it to work and left it where it was.
Some years later, finding myself in a similar place, I began to refresh my memory on where I got to. I eventually decided to just start afresh. So I went searching and found a paper from the late 1990's with an algorithm in it. I tried to make a basic version using this algorithm. And to my pleasant surprise, I got it to work!
I've since kept plugging away at it to try and make it a more functional and usable engine. At present, it's a little broken because I'm converting things into new data containers. Once this change has been made however, it should set the foundations for further engines like a lens distortion corrector and a camera pose solver, to be made.
The beauty with these things is that if I can get the basics to work, I can customise the rest to work inside my other environments the way I want. It means my preview engine (that is, The Dark Room), potentially becomes much more powerful.
To briefly explain the image above, on the left side is a frame from an image sequence with a bunch of high contrast edges (black and white) to try and track. The sequence is just a 3D rendered image where the camera moves around the box. It doesn't matter that it comes from 3D, what matters is if we can track things in it. Tracking markers are green in the image on the left, with one coloured purple. On the right side is a zoomed section on a tracking marker from the left side. May be hard to see, but the purple marker (number 12) is what we're zoomed in on.
The 2D tracking might be useful on it's own, but what I really want is a 3D camera solver. But I can't make a 3D solver without a 2D tracking engine, and probably not without a lens distortion corrector either.
These would make a much more powerful previsualiser tool. I could do all my early post-production work between the camera and animation straight from inside my animation package. No external editor, compositor, or dedicated 3D tracking system required. This would be quite a feat on its own.