DEPTH FROM OPTICAL FLOW

Tomas Hodan

Department of Information and Computing Sciences, Utrecht University

Abstract

This page is about an experimentation project performed by Tomas Hodan under supervision of dr. Robby T. Tan. The goal of the project is to estimate depth from image sequences through optical flow. Optical flow is the apparent motion of brightness patterns (objects, surfaces, edges, etc.) in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene. The basic task is to estimate optical flow of each pixel (represented by a flow vector) between two input images.

Below is an example of two slightly different input images from which we want to estimate an optical flow as similar as possible to the ground truth.

Input image #1

Input image #2 (slightly different than the first one)

For the visualization of optical flow is used color coding from Middlebury optical flow benchmark. Color of a pixel represents a vector starting in the middle of the coding circle and pointing to the corresponding color (the flow of the black pixels could not be determined). In this example, there is applied a threshold to flow vectors of all results to get similarly colored visualizations (flow in both x- and y-direction can be at most abs(5) pixels).

Optical flow ground truth of our example

Color coding of the optical flow

Temporary Results

Here are results of two dense methods (Horn-Schunck - in two variants, and Gunnar-Farneback) which estimate a flow vector for each pixel, and one sparse method (Lukas-Kanade) estimating flow vectors only for selected keypoints. Because for the depth estimation, dense data will be probably necessary, an interpolation of the sparse data is going to be implemented.

Horn-Schunck [1] (my implementation)

Horn-Schunck [1] (OpenCV's implementation)

Gunnar-Farneback (OpenCV's implementation)

Lucas-Kanade [2] (OpenCV's implementation)

Evaluation of Temporary Results

The much better results of the Lucas-Kanade method are caused by the fact, that only the sparse data were evaluated. A fair comparison will be possible after an interpolation of these sparse data.

Method	Endpoint Error		Angular Error
Method	Average	Std. dev.	Average	Std. dev.
Horn-Schunck [1] (my implementation)	0.198527	0.42984	0.10697	0.215792
Horn-Schunck [1] (OpenCV's implementation)	0.336752	0.559006	0.172015	0.264742
Gunnar-Farneback (OpenCV's implementation)	0.341871	0.56047	0.20237	0.31977
Lucas-Kanade [2] (OpenCV's implementation)	0.000247	0.006918	0.000164	0.004980

The angular error between a flow vector (u, v) and the ground-truth flow (uGT, vGT) is the angle in 3D space between (u, v, 1.0) and (uGT, vGT, 1.0)
The endpoint error is defined as sqrt((u−uGT)^2 +(v−vGT)^2)
A unit is one pixel.

References

Horn B.K.P. and Schunck B.G. Determining optical flow. Artificial Intelligence, 1981.
Lucas B.D. and Kanade T. An iterative image registration technique with an application to stereo vision. Proceedings of the 1981 DARPA Image Understanding Workshop, 1981.
Bruhn A., Weickert J., and Schnorr Ch. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision, 2005.
Sun D., Roth S., and Black M.J. Secrets of optical flow estimation and their principles. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
Bradski G. and Kaehler A. Learning OpenCV: Computer Vision with the OpenCV Library. O’Reilly Media, 2008.
Middlebury optical flow benchmark

Contact