Looking at Optical Flow in real scenarios

In many scenarios knowing how an object moves between frames is important, for example if you are trying to separate moving objects from the background or to calculate the speed of a tennis ball. Generally within Computer Vision people default to the Lucas Kanade[1] Tracker or the Pyramid expansion, although this tracker is commonly used it is not generally considered to work particularly well in real world scenarios. To help understand a little better I have compared 3 different techniques over a selection of different video types varying in object and quality.

We compare:

  • Lucas-Kanade
  • Brox[2]
  • Sun[3]

To note in these experiments we use default settings provided by the implementations, with tweaking you will get better results. Also I am sure there are faster implementations than the matlab ones used. I also realise they are very hard test cases with low quality, lots of noise motion blur.

Vector Comparison

Frame 1 Frame 2 LucasKanade Brox Sun
horse-a horse-b pyrlk_flow horse_brox_flow horse_sun_flow
dance-a dance-b dance_pyrlk_flow dance_brox_flow dance_sun_flow
snow3-a snow3-b snowboard_pyrlk_flow snowboard_brox_flow snowboard_sun_flow



  Lucas-Kanade Brox Sun
Horse (1280x720) 105s 64s 619s
Dance(400x400) 26s 15s 134s
Snowboarder(720x576) 30s 28s 294s



Well as you can see they are inconclusive results, Sun’s approach I would say works better at too higher computational cost LK works well on the dance footage or at least in that specific example, less noisy frames of the dance footage I have found LK performed poorly on. Brox, seems generally not to cope very well in these cases. To conclude, optical flow is still a very very incomplete field, you have to test on your problem to find the best solution.


[1] Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. Proceedings of the 7th international joint … (Vol. 130, pp. 121–129). Retrieved from http://www.ri.cmu.edu/pub_files/pub3/lucas_bruce_d_1981_1/lucas_bruce_d_1981_1.ps.gz

[2] Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. Computer Vision-ECCV 2004, 4(May), 25–36. Retrieved from http://www.springerlink.com/index/87a4ckjqm92lp3j9.pdf

[3] Sun, D., Roth, S., & Black, M. J. (2010). Secrets of optical flow estimation and their principles. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2432–2439. doi:10.1109/CVPR.2010.5539939


This evaluation was done by taking openly available code for full references see the download file, that contains readme references in the relevant subfolders. If you have developed your own implementation of one of these methods or an alternative approach it would be great to expand with your results.

Saturday, August 18, 2012 10:32:41 PM Categories: Computer Vision Matlab Programming
Stuart James