Research Interests:
|
My research focuses on computer vision, particularly in physics-based approaches: color constancy, shadow analysis, specular highlight removal, visibility enhancement in bad weather, multi-layered surfaces, etc. Since my arrival in Utrecht, I have worked in GATE project: 1. Automatic world generation based on real data and, 2. Detecting, interpreting and affecting user behavior. Currently, I involve in COMMIT project focusing on sensing emotion in video. |
Selected Results:
- Layered Surface Decomposition by Using the Spider Model [video | pdf ]
Many object surfaces are composed of layers of different physical substances, known as layered surfaces. These surfaces, such as patinas, water colors, and wall paintings, have more complex optical properties than diffuse surfaces. Although the characteristics of layered surfaces, like layer opacity, mixture of colors, and color gradations, are significant, they are usually ignored in the analysis of many methods in computer vision, causing inaccurate or even erroneous results. Therefore, the main goals of this paper are twofold: to solve problems of layered surfaces by focusing mainly on surfaces with two layers (i.e., top and bottom layers), and to introduce a decomposition method based on a novel representation of a nonlinear correlation in the color space that we call the "spider" model. (A joint work with T. Morimoto, R. Kawakami, and K. Ikeuchi. Published in CVPR 2010) [video]
- Visibility Enhancement in Bad Weather [webpage | pdf ]
Bad weather commonly degrades the visibility of outdoor scenes. A number of methods have been proposed to enhance the visibility. Unlike these methods, our goal is to develop a method that requires solely a single image taken from ordinary digital cameras, without knowing the geometrical information of the scene. The proposed method principally formalizes the image contrast and utilizes connections of pixels in term of Markov random fields. The experimental results on real images (for both color and gray images) show the effectiveness of the approach. The method is published in CVPR 2008. [webpage]
- Color Constancy Through Inverse-intensity Chromaticity Space: [ webpage | code | pdf ]
We introduce a single integrated method to estimate illumination colors from single-colored and multi-colored surfaces. Unlike existing dichromatic-based methods, the proposed method requires only rough highlight regions, without segmenting the colors inside them. In principle, it is based on "inverse-intensity chromaticity space", a novel two-dimensional space we introduce. (A joint work with K. Nishino, K. Ikeuchi. Published in CVPR 2003) [ webpage] [code] - Robust Highlight Removal: [
webpage | pdf ]
In the real world, the presence of highlights is inevitable, since there are many dielectric inhomogeneous objects which have both diffuse and specular reflections. To resolve this problem, we propose a method to separate the two reflection components. The method is principally based on the distribution of specular and diffuse points in a two-dimensional maximum chromaticity-intensity space. We found that, by utilizing the space and known illumination color, the problem of reflection component separation can be simplified into the problem of identifying diffuse maximum chromaticity. To be able to identify the diffuse maximum chromaticity correctly, an analysis of the noise is required, since most real images suffer from it. (A joint work with K. Nishino, K. Ikeuchi. Published in IEEE Trans. on PAMI 2005). [webpage]
- Highlight Removal for Complex Textured Surfaces: [
webpage | code | pdf ]
A number of methods have been proposed to remove highlights. However, to our knowledge, all methods that use a single input image require explicit color segmentation to deal with multicolored surfaces. Our proposed method resolve the segmentation problem. It is based solely on colors without requiring any geometrical information. One of the basic ideas is to iteratively compare the intensity logarithmic differentiation of an input image and its specular-free image. Specular-free image is an image that has exactly the same geometrical profile as the diffuse component of the input image, and that can be generated by shifting each pixel's intensity and maximum chromaticity non-linearly. (A joint work with K. Ikeuchi. Published in ICCV 2003 and IEEE PAMI 2005) [webpage] [code]
- Pose Recognition of Multiple Persons: [ pdf ]
The goal of this project is to segment, track, and estimate the poses of multiple persons in a scene captured with multiple cameras. Current development in the field mostly concentrated on the problems of a single person, which is considerably less problematic compared with those of multiple persons. Technically, one of the end results of our algorithm is full 3D skeletons representing the individuals in a scene. We expect that our algorithm could be practically useful for human-computer interactions, especially in gaming applications. (A joint work with X. Luo, P. van Beek, R. Veltkamp, W. Huerst, N. van de Aa).
- Consistent Surface Color for Texturing Large Objects in
Outdoor Scenes: [ pdf ]
Color appearance of an object is significantly influenced by the color of the illumination. When the illumination color changes, the color appearance of the object will change accordingly, causing its appearance to be inconsistent. To arrive at color constancy, we have developed a physics-based method of estimating and removing the illumination color. Our method is principally based on shadowed and non-shadowed regions. Previously researchers have discovered that shadowed regions are illuminated by sky light, while non-shadowed regions are illuminated by a combination of sky light and sunlight. Based on this difference of illumination, we estimate the illumination colors (both the sunlight and the sky light) and then remove them. To reliably estimate the illumination colors in outdoor scenes, we include the analysis of noise, since the presence of noise is inevitable in natural images. (A joint work with R. Kawakami, K. Ikeuchi. Published in ICCV 2005).
- Polarization-based Inverse Rendering: [ pdf ]
The goal of this project is to estimate the geometrical and photometrical properties of an object in order to realistically render the object'ss artificial images. To achieve the goal, my colleagues and I introduced a method to estimate geometrical, photometrical, and environmental information of a single viewed object in one integrated framework under fixed viewing position and fixed illumination direction. In our framework, photometrical information represents the texture and the surface roughness of an object, while geometrical and environmental information represent the 3D shape of an object and the illumination distribution, respectively. The proposed method estimates the 3D shape by computing the surface normal from polarization data, calculates the texture of the object from the diffuse only reflection component, determines the illumination directions from the position of the brightest intensity in the specular reflection component, and finally computes the surface roughness of the object by using the estimated illumination distribution. (A joint work with D. Miyazaki, K. Ikeuchi, K. Hara. Published in ICCV 2003).
- Specular-Free Images:
Highlights or specularities can cause many algorithms in computer vision to produce erroneous results. For complex textured surfaces, the existing methods that use a single input image cannot remove highlights correctly, since they require color segmentation. Thus, to resolve the problem we introduce 'specular-free image', an image that is free from the presence of highlights or specularities but has different color to the input image. This image can be generated straighforwardly by using a novel mechanism we call 'specular-to-diffuse mechanism' . The mechanism has two main advantages: first, it requires only a single colored image, and second, it operates based on a single pixel (local operation). The specular-free image, considering its properties, could be useful for various applications of computer vision that do not need actual surface colors but suffer from highlights, such as shape from shading, stereo, etc. Note that the specular-to-diffuse mechanism requires chromatic pixels and known illumination chromaticity, which the latter is obtainable using our color constancy algorithm [ webpage]