Muharram Mansoorizadeh

Muharram Mansoorizadeh Tarbiat Modares University

Key research interests: Multimodal Emotion Recognition

Multimodal information fusion application to human emotion recognition from face and speech

Abstract -A multimedia content is composed of several streams that carry information in audio, video or textual channels. Classification and clustering multimedia contents require extraction and combination of information from these streams. The streams constituting a multimedia content are naturally different in terms of scale, dynamics and temporal patterns. These differences make combining the information sources using classic combination techniques difficult. We propose an asynchronous feature level fusion approach that creates a unified hybrid feature space out of the individual signal measurements. The target space can be used for clustering or classification of the multimedia content. As a representative application, we used the proposed approach to recognize basic affective states from speech prosody and facial expressions. Experimental results over two audiovisual emotion databases with 42 and 12 subjects revealed that the performance of the proposed system is significantly higher than the unimodal face based and speech based systems, as well as synchronous feature level and decision level fusion approaches. Keywords Multimodal feature extraction - Multimodal information fusion - Human computer interaction - Multimodal emotion recognition

Face and Facial Feature Tracking

Here are some initial results of facial feature tracking.

Speech Emotion Recognition: Comparison of Speech Segmentation Approaches

Recognition of emotional states carried in speech, is of a great interest in modern human computer interaction developments. To reliably detect the aroused emotion, a sufficiently long continuous speech segment is required. This research aims to analyze different segmentation approaches of speech signals. Berlin emotional speech database is used for data set generation. Time frame based and voiced segmentation approaches are applied and compared. The experimental results show that accurate emotion recognition is obtained when the speech segments are longer than a second or are composed of 10 to 12 voiced segments. Based on the findings of this research, voiced based segmentation generates more accurate results than the other methods

Facial Feature Localization(includes demo)

I've developed some MATLAB(R) scripts for facial feature localization. The included clip is a demo of scripts in action. I'm looking for guidelines to pack and publish the scripts so that it can be used by others.

The Database

The Database

Sample clip from the audio-visual emotion database

The subject acts like a man enjoying the blue sky.

Audio-Visual Emotion Database in Persian Language

I'm creating an audio-visual emotion database in Persian language. currently, recording phase is complete and audio/speech data is archived as digital streams. Clipping and annotation has just began.

Hybrid feature and decision level fusion of face and speech information for bimodal emotion recognition

A hybrid feature and decision level information fusion architecture is proposed for human emotion recognition from facial expression and speech prosody. An active buffer stores the most recent information extracted from face and speech. This buffer allows fusion of asynchronous information through keeping track of individual modality updates. The contents of the buffer will be fused at feature level; if their respective update times are close to each other. Based on the classifiers' reliability, a decision level fusion block combines results of the unimodal speech and face based systems and the feature level fusion based classifier. Experimental results on a database of 12 people show that the proposed fusion architecture performs better than unimodal classification, pure feature level fusion and decision level fusion.