Reconstruction of non-uniform sampling motion field

Suppose we only have the motion information of a few points in the motion field, how to recover the whole true motion field? Is that even possible? The answer is yes, because we human can do it. It turn out that we are very good at giving an unique and stable solution to this inverse problem, we can always yield sound inference of motion only based on a few parts of an object.

To tackle this problem we need a kind of interpolation, or ‘fill-in’. This kind of inverse problem is classical, but sort of tricky in such a motion estimation setting because it is not uniformly sampled, and this sampling process is highly content dependant. So no fixed 'ideal' kernel will be find because it should vary from frame to frame. Another classical problem in motion estimation is aperture problem, my method addresses this problem with an ‘iron fist’: Fill in blank area (no motion information can be readily extracted) with ANY reliable motion with in or near by that area. A KNN-based method is used here. It is not very theoretically sound suggestion but a practically reasonable.

In the images showed above, blue and green color indicates the horizontal and vertical velocity respectively, and the brightness of the color means the speed (high or low). This algorithm’s complexity is relatively low so that it can run in real-time on my laptop at a QVGA resolution.

Further work: this problem can even fit in transductive online learning framework.