Emotion classification from facial images

Jan de Wit

Game and Media Technology, Utrecht University, Netherlands

Abstract

In this experimentation project report by a second year student of Game & Media Technology, an attempt was made to combine existing techniques to create a full pipeline for emotion classification from frontal face images. This includes face detection, feature detection and extraction and finally classification between seven different emotions. After detecting the face using Viola-Jones, Shape Models (snakes) are used to get a face model of which various measurements are derived. Finally, an attempt was made to extract texture data using Canny edge detection. Classification is currently based mostly on mouth data and is implemented using a multi-class version of AdaBoost. The final challenge is to combine the classifications from various sources (texture, shape model) into a final decision that is more robust than the individual classifications. The extended Cohn-Kanade (CK+) database is used to train the pipeline and test its performance. This technique could be used, among other fields, for lie detection, personalized advertising, security or entertainment. The results of classification based on mouth data alone indicate that the selection of mouth features as proposed in this paper should be useful for similar pro jects. The techniques treated in this paper are useful for anyone working with facial feature extraction or facial recognition, as well as related fields that desire a form of shape detection and model training.

Paper

Emotion classification from facial images [PDF] (version 2.0)
Jan de Wit

System Pipeline

Results

Aligning faces Facial feature point distribution Canny edge detector

Full pipeline program

Full pipeline program

The above screenshot shows the final pipeline, from input image (top left), shape model fitting (top right), scaling/alignment and Canny edge detection (bottom left) to final classification (bottom right).



References (selection)

  1. An introduction to active shape models
    T. Cootes [PDF]

  2. Algorithms for face and facial feature detection (MSc thesis)
    X. Luo (December 2007) [PDF]

  3. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression
    P. Lucey, J.F. Cohn, T. Kanade, J. Saragih and Z. Ambadar (CVPR, 2010) [PDF]

  4. Recognizing action units for facial expression analysis
    Y.L. Tian, T. Kanade and J.F. Cohn (IEEE transactions on pattern analysis and machine intelligence, February 2001) [PDF]



Links