Gradient space manipulation

Bart Liefers

Department of Information and Computing Sciences, Utrecht University

Abstract

This page is about an ongoing experimentation project performed by Bart Liefers under supervision of dr. Robby T. Tan. The main topic is the manipulation and reconstruction of images in the gradient domain. In this field, contrary to directly manipulating pixel values, the gradient of the image is edited. A possible application is to remove unwanted edges, for example by setting the local gradient to zero. After all gradient space manipulations are performed, the resulting gradient field has to be integrated to obtain a resulting image. Because the gradient field is generally non integrable, an approximation technique is used. The current approach is to use a Poisson solver, which yields a least squares approximation. This means that the squared difference between the original gradient field and the gradient of the resulting image is minimized. We used this technique to explore its use in image composition, object insertion and seamless stiching. We are currently investigating the applications of this technique in combination with image inpainting methods and shadow removal.



Final Report: PDF



Results:


Image composition:
Pasting a moon in the clear blue sky:
Combination of two images. Color levels may change after integration. Restored color levels using a different normalization method.




Smooth tiling:
In the next case the color difference between the left and right edge is propagated in the tiling. This is unwanted, but others can do better (here).


Object Insertion:
Inserting letters on a background (recreation of result from Howard Zhou & Jie Sun (here):
Inserting transparent object into scene:

Shadow removal using gradient-inpainting

In these images the shadows were removed by using an inpainting technique (Criminisi et al., 2004) that was modified such that the gradients around the shadow border were inpainted to obtain a shadow-free image. The region to be inpainted can be selected manually.


Left: original image.The yellow polygon represents the manually selected shadow-border region that needs inpainting.
Right: the result after gradient-domain inpainting inside this polygon and integration of the modified gradient field. Notice that the vertical structure of the wood and the white line on the road is maintained inside the border region by the algorithm.

Shadow detection

A physics-based model is used to create an image that is shadow-invariant (ie: different pixels, in sunlight and shadow, that represent the same surface will get the same value in the invariant image). Edge detection on this invariant image, compared with edge detection on the orignal image is used to automatically select shadow-border regions.




Top left: Input image
Top right: Edges of input image
Middle left: Invariant image
Middle right: Edges in invariant image
Bottom left: Shadow edge region bitmap.
Bottom: reconstruction created by first nullifying the gradient in the shadow edge region. Then integration, and finally inpainting (in the intensity domain) of the same shadow edge region. The result may be improved still, since the data-term in the inpainting algorithm was ignored.

References:


Last Update: March 2011