I've almost completed implementing an early version of the tracking interface, with tracking engine, into The Dark Room. This is the first of three main engines I need to make an interface for. But it wasn't all smooth sailing.
The tracking engine will allow the user to track features in an image; for example a mark on a wall, or patterns on the ground. The tracking markers can then be used to do things like add effects into the shot that follow the camera, or, work out the camera pose estimation to use in 3D animation rendering.
Let's begin with some of the things I had to do. I had to write some more graphics shaders for the viewport engine. One that would allow a simple 2D texture draw overlay, and another that would allow greyscale image viewing. In fact, I had to do some remodelling of how shaders work in the viewport draw engine, so that the interfaces could use them in their own draw routines. It took some time to reorganise the code to do this, but it's made a huge difference. There are all sorts of ways I could use the updated draw pipeline in the future. And I still think there are ways it could be improved.
I also spent some time connecting the tracking engine up to the real camera images. This might seem obvious, but I had only been testing with known image data up until now. But now I can do it on the real camera images. The image comparison below shows the first tracking results (look for the moving white and orange cross markers).
Moving the slider on the image above, you can see the markers track across the two frames fairly well. The marker locations aren't particularly good - trees like this don't make good tracking features! They just happened to be spots picked up by the auto-select function in the tracking engine. I'm not interested in auto-tracking itself, it was just a handy feature for the tracking engine to have to help quickly populate some features to test with. Even if they don't make for good tracking features!
The next step was to try and have it track features that are user-selected. In the camera footage, I was testing user features by placing a point near the top corner of one of the front windows. With a bit of re-coding, I managed to get the engine to recognise the custom feature placement. Below is an example of a user-selected feature being tracked across two frames. The feature may be hidden behind the drag bar to start with:
Initially, the tracking engine appeared to be quite slow. I suspected this was to do with two issues. One is the way I'm pulling the images and converting them to greyscale (the tracking engine works on greyscale image data). The second is probably to do with the resolution of the images. When testing on small images, it happily ran at real-time speeds. But larger resolution images took much longer. I updated the engine to work on smaller search window locations, and saw some good performance improvements in processing time. But the tracking itself started failing. This one was a real doozy. And it took more than a week worth of days to figure out.
To cut to the chase here, I made a silly mistake when copying memory from one data container to another. I was using the standard library memory copy function, which simply copies one block of memory into another. It does not do any sanity checks or converting of data; it quite literally copies one block of memory into another, at whatever size you tell it to. This was fine, I had the memory being copied from and into the right places. What I had wrong, was the size of the memory to copy. I told the function it was floating point data, when it actually was double precision. This meant it was only copying half of it! This explained all of the issues I was seeing, and why the tracking worked on some images, and not others. Because the markers on the larger resolutions just happened to be within the data that was copied. Whereas in smaller resolutions, there was a chance it was not. It was, literally, one word in one line of code that was the problem. This was a real nightmare to debug.
With this issue resolved, I think there might be a couple more ways I could make the engine work a little faster, but I'll work on these in time. For now, I just want things to work.
While implementing the tracking, I updated the engine so that tracking works in normalised image coordinate space. This means I can now track on any resolution image, without affecting the tracking outcome. This might be handy in future for a few reasons; firstly for speed, and secondly to help with failed tracks. If a track fails on a higher/lower resolution image, it may still track at another resolution with different levels of image detail.
At this point, I thought I'd try and track a feature across a longer sequence. And everything ground to a halt. I found another memory issue; this time something was taking up lots of memory, to the point where the program would eventually crash. It took me quite a while to track this issue down as well, but I believe it's something to do with the camera manufacturer's software. I've since contacted them about the issue, and am waiting a response.
In the meantime, I continued to work on the interface itself. The selected tracking window has been improved; I've now added buttons to track forward and backwards, frame by frame, and through all frames until failure. There's also a close button in the top right of the window. And the black window borders around the texture can now be used to resize the search area. The marker position can be moved manually inside the window. And the window itself can be moved around the viewport. The selected window now looks like this:
Another shader I wrote allowed me to draw lines for the tracking. You can see the green line in the image above. This shows the tracking result over a number of frames before and after the current frame. Handy for seeing where the marker has been, and is going. The green is also indicative of how well the marker has tracked; green is a good result, red would be poor. It looks pretty good in the image above. What's missing though is the marker itself. I need to add in a little window outline or circle - something that shows where the marker is in the current viewport frame. And, it needs to be selectable in the viewport.
This will probably involve having to make another shader or two. I have some ideas.
In addition to making the selected marker window more usable, I also fixed a colour space issue in the viewport. The viewport frame images look much better now, much crisper and cleaner looking colours. Here's an example screen capture with better looking colours:
I haven't been able to run a complete feature tracking test yet because of the memory issue crashing things. I'll post a video when this issue is resolved. Until then, I can only press on with building more interface functionality. Which is fine, as it needs to be done at some point anyway.