Guidelines for affective signal processing: From lab to life

Table of contents

1. Audience
2. Introduction
3. Issues of concern
4. Outcomes of / follow-ups for this session
5. Invited speakers' contributions
6. Presentations
7. References

Audience

All ACII2009 attendees who are applying physiological signals for emotion detection or are considering to do so. In addition, all who are interested in the constraints of Affective Signal Processing (ASP), both in lab and in the real world.

As ASP on a certain moment will affect everyone's life, we expect a broad interest to emerge for it. Hence, attendees from virtually all areas of science (e.g., engineering, design, and psychology) are invited to participate in this session.

Introduction

While a century ago emotions were considered to be too spiritual and human's health was explained in strictly physical and physiological terms, it is now generally acknowledged that emotions influence our long term physiological well-being, our physiological reactions/signals, as well as our cognitive processes. As a consequence, recently, Artificial Intelligence (AI) envisioned that emotions would both lead the path to true AI and enhance the communication between man and machine [2,3,8]. This notion resulted in a vast amount of research projects, all over the world; e.g., the EU project HUMAINE.

In these projects, it is often attempted to construct applications that measure and display the emotional state of their users, perhaps make them more aware of it. Another line of applications tries to adapt the application's functioning to the measured affective state. In both cases, it is vital that the emotional state is measured and interpreted in a correct way: the goal of ASP. However, this is by no means an solved case. Despite the tremendous efforts both by academic research and industry (e.g., [2,3,11]), the usage of ASP for real life everyday usage is still troubling [3,5].

Issues of concern

In practice, ASP is still suffering from some fundamental problems; for reviews, see: [1,4,9]. Some of the issues of concern are the following:

• Validity of ASP research; e.g., ground truth and ecological validity
• Generic features versus personalization?
• Lack of ASP databases and recording standards
• Limited robustness / noise-resistance
• Low reliability / classification performance
• Issues with real-time interpretation

The general purpose of this special session is to enhance the awareness of and discuss experiences with the methodological issues that so far have remained unsolved in the area of ASP

Outcomes of / follow-ups for this session

Possible outcomes of this session are: - Set up directives for ASP, both for lab and real world research. Including the early identification of possible problems.

• A discussion on the development of a database for ASP.
• A special issue with a journal with papers that fall in the scope of this special session.
• The initiation of a Special Interest Group (SIG) on ASP.

Invited speakers' contributions

Speakers from both academics and industry addressed the issues raised.

Presentations of the speakers and chairs: Egon L. van den Broek and Joyce H.D.M. Westerink, Elisabeth André, Jennifer A. Healey, Ben Mulder, Stijn de Waele.

Key-papers of the speakers where they address the same issue as in their presentation are provided in the list of references.

Biographies of the speakers and chair: Egon L. van den Broek, Joyce H.D.M. Westerink, Elisabeth André, Jennifer A. Healey, Ben Mulder, Stijn de Waele.

Egon L. van den Broek (Unversity of Twente, The Netherlands) and Joyce H.D.M. Westerink (Philips Research, The Netherlands):
Guidelines for Affective Signal Processing (ASP): From Lab to Life

Short Biography of Egon and Joyce.

Presentation of chairs.

This presentation presents the rationale behind ACII2009's special session: Guidelines for Affective Signal Processing (ASP): From lab to life. Although affect is embraced by both science and engineering, its recognition has not reached a satisfying level. Through a concise overview of ASP and the automatic classification of affect, we provide understanding for the problems encountered. Next, we identify guidelines for ASP: 1) four approaches to validation: content, criteria-related, construct, and ecological, 2) identification of users, 3) triangulation, and 4) signal processing issues. Each of these guidelines is briefly touched upon in this paper. A more exhaustive discussion on these guidelines, in perspective of the invited speakers' experience, will be provided through the session and its accompanying papers.

Prof. Dr. Elisabeth André (University of Augsburg, Germany):
Classification of Multimodal Emotional Signals [6]

Short biography of the author.

Presentation of the author.

Coping with differences in the expression of emotions is a challenging task not only for a machine, but also for humans. Since individualism in the expression of emotions may occur at various stages of the emotion generation process, human beings may react quite differently to the same stimulus. Consequently, it comes as no surprise that recognition rates reported for a user-dependent system are significantly higher than recognition rates for a user-independent system. Based on empirical data we obtained in our earlier work on the recognition of emotions from biosignals, speech and their combination, we discuss which consequences arise from individual user differences for automated recognition systems and outline how these systems could be adapted to particular user groups.

Dr. Jennifer A. Healey (Intel Corporation, USA):
Recording and processing of physiological signals from real life [5]

Short biography of the author.

Presentation of the author.

Recording and processing physiological signals from real life offers many challenges beyond those encountered in the laboratory [1,5]. Issues such as baselining and normalisation take on a time dependent meaning. It can not be expected that sensors will never move or that signal recording might never be interrupted, nor can it be expected that subjects will remain still, that ambient noise will be low or that unexpected events will not occur that cause jarring responses. Certain signals are affected differently by different kinds of interruptions and require different methods for processing. In this talk I will present recordings from galvanic skin response, blood volume pulse, electrocardiogram, electromyogram, respiration, accelerometer, and gyroscopic sensors and talk about various methods for analysing these signals and detecting and compensating for noise. Additionally I will discuss how to synchronise annotations and other ground truth with the signals for analysis.

Dr. ir. Ben (L.J.M.) Mulder (University of Groningen, The Netherlands):
Artifact-free real-time calculation of physiological parameters [7]

Short biography of the author.

Presentation of the author.

Using physiological signals in real-time has several consequences for data pre-processing and computation of derived measures as a function of time. Experience with artifact correction, certainly with heart rate data, has learned that subject-specific deviations are very difficult to detect and classify automatically without visual support. Many of these deviations have a physiological origin, which makes it even harder to decide whether or not to correct. From the other side, not correcting for such artifacts has large consequences for obtained (spectral) results. Another problem is related to the time resolution of spectral measures in relation to the reliability. In order to detect physiological state changes or physiological responses to task related events as fast as possible, time segments used should be short. However, this does not correspond with the knowledge that time duration should be large enough for reliable estimates. We will outline a procedure in which short-segment (spectral) parameters are computed as a function of time. These spectral profiles are used for further selection and averaging on the basis of task related events. Examples will be given from the cardiovascular domain (heart rate, blood pressure, respiration) while it will be shown that such an approach is suitable for other domains as well.

Dr. ir. Stijn de Waele (Philips Research, The Netherlands):
Experiences with adaptive models for biosignals in daily life [10]

Short biography of the author.

Presentation of the author.

We discuss the merits of adaptive statistical models for biosignals in a daily life context. Processing of this type of signals poses a number of challenges. First, it is clear that an adaptive model is needed to tailor for the differences in physiology between individuals, as well as adapt to someone?s current physiological state. Second, in a daily life setting we use unobtrusive measurement devices, which will lead to reduced signal quality compared to the laboratory setting. Third, low-power portable sensors allow for only limited data storage and data transmission. Two techniques to address these challenges are discussed in detail: the usage of the cumulative histogram and parametric models. We show applications to EEG, skin response (GSR) and heart rate (ECG, BVP) and we advise on how to obtain the most reliable results.

Presentations

1. Broek, E.L. van den & Westerink, J.H.D.M.: Guidelines for Affective Signal Processing (ASP): From Lab to Life.
2. André, E.: Towards User‐Independent Classification of Multimodal Emotional Signals.
3. Healey, J.A.: Affect Detection in the Real World:Recording and Processing Physiological Signals.
4. Mulder, L.J.M.: On artefact-free real-time computation of cardiovascular measures ? How to use available knowledge in on-line applications.
5. Waele, S. de: Experiences with adaptive statistical models for biosignals in daily life.

References

1. Broek, E.L. van den, Janssen, J.H., Westerink, J.H.D.M., and Healey, J.A. (2009). Prerequisites for Affective Signal Processing (ASP). In P. Encarnação and A. Veloso (Eds.), Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, p. 426-433. January 14-17, Portugal.
2. Broek, E.L. van den, Schut, M.H., Westerink, J.H.D.M., and Tuinenbreijer, K. (2009). Unobtrusive Sensing of Emotions (USE). Journal of Ambient Intelligence and Smart Environments, 1(3), 287-299.
3. Broek, E.L. van den and Westerink, J.H.D.M. (2009). Considerations for emotion-aware consumer products. Applied Ergonomics, 40(6), 1055-1064.
4. Fairclough, S. (2009). Fundamentals of physiological computing. Interacting with Computers, 21(1-2), 133-145.
5. Healey, J.A. & Picard, R.W. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2), 156-166.
6. Kim, J. and André, E. (2008). Emotion recognition based on physiological changes in listening music. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(12), 2067-2083.
7. Mulder, L.J.M., Dijksterhuis, C., Stuiver, A. and de Waard, D. (2009). Cardiovascular state changes during performance of a simulated ambulance dispatchers' task: Potential use for adaptive support. Applied Ergonomics, 40(6), 965-977.
8. Picard, R.W. (1997). Affective computing. Cambridge, USA: MIT Press.
9. Tractinsky, N. (2004). Tools over solutions? Comments on interacting with computers special issue on affective computing. Interacting with Computers, 16(4), 751-757.
10. Waele, S. de and Broersen, P.M.T. (2000). Order selection for the multirate Burg algorithm. Proceedings of the 10th European Signal Processing Conference, p. 1917-1922. Tampere, Finland.
11. Westerink, J.H.D.M., Ouwerkerk, M., Overbeek, T., Pasveer, W.F., and Ruyter, B. de (2008). Probing Experiences: From Academic Research to Commercial Propositions. Philips Research Book Series. Dordrecht, The Netherlands: Springer.