Recognition of non-prototypical emotion from speech – The final frontier

After switching from acted to naturally occurring emotions and emotion-related states, from limited textual variation to spontaneous speech and reaching acceptable subject-independency it is time to face one of the last barriers prior to integration of emotion recognition from speech into real-life technology: non-prototypicality. In emotion processing, normally a pre-segmented subset of a full recording is taken consisting of somehow clear, i.e. more or less prototypical cases with respect to inter-labeller agreement. This is not only a clever move to push classification performance, it simply has grown out from the problem of class assignment in emotion processing: there is no simple and unequivocal ground truth. Using „realistic data?, however, not only means using spontaneous data, it means as well using all these data as in a dialog system, media retrieval or surveillance tasks. This second, sort of „quantitative? aspect, has still been neglected by and large. With this special session we aim to shed light on the performance loss to be expected, and find new paradigms to cope with this challenge as detection or spotting of emotion with potential garbage classes or decoding stages - potentially also including different quality measurement as ROC-curves with Equal Error Rates (EER), or the Area Under Curve (AUC).

A number of invited talks served as basis to initiate a panel discussion on dealing with non-prototypical emotion as the "final frontier" towards reliable real-life application of engines.

Contributions covered the field of emotion recognition from speech facing the following conditions:

• Natural emotion / emotion-related states / non-linguistic vocalisations
• Spontaneous speech
• Large datasets
• Non-prototypical emotions/states/vocalisations, i.e. dealing with any speech recorded and not only such with high inter-labeler-agreement, preferably also not pre-segmented.

We would like to thank the audience and our panelists (click on the names for the panel slides) Elisabeth André, Felix Burkhardt, Laurence Devillers, Shrikanth Narayanan, Jianhua Tao, and Björn Schuller, which each gave a talk on the topic and a short presentation answering two questions, we had asked them in advance, and following a short introduction by Anton Batliner:

• “What is the most promising approach that youcan imagine so far towards dealing with non-prototypical emotion?”
• “What are the most challenging problems we all have to face in this respect?”

Following, the paper presentations are listed with linked slides.

• Interpreting Ambiguous Emotional Expressions
  Emily Mower, Angeliki Metallinou, Chi-Chun Lee, Abe Kazemzadeh, Carlos Busso, Sungbok Lee, Shrikanth Narayanan
• Real-time vocal emotion recognition in artistic installations and interactive storytelling: Experiences and lessons learnt from CALLAS and IRIS
  Thurid Vogt, Elisabeth André, Johannes Wagner, Steve Gilroy, Fred Charles, Marc Cavazza
• A Multiple Perception Model on Emotional Speech
  Jianhua Tao, Ya Li, Shifeng Pan
• Emotion Detection in Dialog Systems: Applications, Strategies and Challenges
  Felix Burkhardt, Markus van Ballegooy, Klaus-Peter Engelbrecht, Tim Polzehl, Joachim Stegmann
• The Hinterland of Emotions: Facing the Open-Microphone Challenge
  Stefan Steidl, Björn Schuller, Anton Batliner, Dino Seppi

The target audience of the now open discussion are researchers working on analysis of emotion in a broad sense, and in particular individuals working on recognition of emotion from speech. Further researchers interested in application of analysis technology will have substantial interest in effects of non-prototypicality.