University of Twente

The Human Media Interaction (HMI) research group of the University of Twente consists of about 15 (including some parttime) staff members, a group of 12-15 temporary researchers (post-docs, junior researchers and Ph.D. students) and some administrative, technical and managerial support. At the University of Twente it is the main responsible group for research and teaching in human-computer interaction and its quality has been a factor in creating a special research kernel on HCI in the ICT research institute of the university (CTIT) and in establishing a Master's study Human Media Interaction at this university. Currently, every year, between twenty and thirty master's students finish their studies in the group. The main research activities are in the following areas:
• multi-modal and multi-party interaction
• embodied conversational agents; ambient intelligence
• affect in interaction; machine learning
• speech and language processing; multimedia retrieval and presentation
Four of our current projects that are very much related to HUMAINE  are on modeling multi-party interaction in smart environments, intelligent educational environments, virtual storytelling and design of emotion architectures. In addition there are research projects funded by external sources on multimodal interaction, dialogue systems and embodied conversational agents.

Prof. dr. A. Nijholt
Anton Nijholt coordinates the research of the Human Media Interaction research group in Twente. His main interest is the integration of research in the HMI group on multimodal interaction, virtual humans, multi-agent systems and virtual reality in several application-oriented projects. Three existing integration projects are INES (an educational environment that allows communication between a nursing student, a virtual tutor and a virtual patient), a virtual storytelling environment and research in the European AMI (Augmented Multi-party Interaction) project. One of the applications in this latter project is the design of a mixed reality meeting room where virtual humans (autonomous agents or agents representing humans) and other meeting participants can interact.

Dr. R. op den Akker
Rieks op den Akker works in multi-modal dialogue systems with a special focus on the verbal component and the way other aspects of behavior (including affective state and intentions) interact with the verbal way of expressing.
Research Interests.
• Emotion modeling: in tutoring systems and in agent-based virtual story telling.
• How do affect and emotion effect the way people interact with computers and how can we make computers so that they cope with these effects?
• Characterization of subjective, affective and factual contributions in dialogue systems.

Dr. D. Heylen
Dirk Heylen has been worked on generating facial expressions for embodied conversational agents and related issues. As a linguist and a computational linguist his primary interest is in applying insights from the study of human-human communication to human computer interaction and providing computation models of h2h communication. In working on embodied conversational agents, his interest has shifted from working on language as such towards nonverbal means of communication and particularly facial expressions. Trained as a linguist, originally, his first concern with generating appropriate expressions for agents is the question: what do real people do in naturalistic situations? Currently he is working on the analysis of multiparty interaction (meeting scenario's) analyzing the verbal and nonverbal means of communication and their relation with decision making (argumentation, agreement, etcetera) in the context of the European AMI project.
He is also working on intelligent tutoring systems that provide appropriate feedback to students, taking into account assumptions about the mental state of the students derived from observing their actions and nonverbal behavior.
Research Interests.
• Generation of appropriate conversational and emotional signals of embodied conversational agents, made sensitive to the state of the interlocutor (for instance gaze and smiles).
• Determining the interlocutor's state of mind by interpreting verbal and nonverbal cues.
• Reinterpreting classical speech act theory with insights from social psychology and cognitive emotion theories.
• Applying linguistic and psychological theories of human-human interaction to building virtual humans.

Dr. M. Poel
Mannes Poel has a background in pure mathematics and theoretical computer science. After working several years on verification of concurrent processes he shifted his interest in Neural Networks, Machine Learning and Artificial Intelligence in the context of Multi Agent Systems and Human Agent Interaction. Currently he supervises several Ph.D.’s who work on the subject of Machine Learning and Adaptivity. He has published several papers on Machine Learning and Human Media Interaction, some of them focusing on the role of emotions. He participates in the Bsik project ICIS and the European project  AMI. Mannes Poel is member of the program committee of the workshop “Affective Computing Entities” and several conferences on Artificial Intelligence.
Research Interests:
• Artificial Intelligence and Machine Learning.
• Machine Learning Techniques for Emotion Generation.
• Emotions and Behavior: architectures for emotional behavior.
• Machine Learning Techniques for Gesture and Emotion Recognition.
• Architectures for Emotional Embodied Virtual Agents.

Dr. Z. Ruttkay
Her research focuses around different aspects of modeling and generation of Embodied Conversational Agents (ECAs):
• ECA Evaluation.
• Individual, culturally different ECAs; adaptive ECAs.
• Facial animation; hand gesturing; synthesis of emotional speech and non-speech sounds.
• Dialogue patterns and refinements.
She is also interested in interactive, affective graphical art. Ruttkay developed a constraint-based facial animation framework to define and generate subtle (emotional and other) facial expressions. The EmotionDisc is another tool to generate a continuum of facial expressions based on the 6 basic emotional expressions. She co-operated in a series of dedicated experiments for testing the perceived effect of emotional and cognitive facial and vocal signals by ECAs. Currently she has been doing research on the manifestation of emotions and style in hand gestures. The GESTYLE language has been developed, opening the possibility to generate and evaluate ECAs with different gesturing habits. She is also working on a framework to refine and enrich dialogue schemes of ECAs, to be used, among others, to improve (the impression of) attention and adaptivity of ECAs.

Dr. M. Theune
Mariët Theune has a background in linguistics / computational linguistics. Her main interest is multimodal information presentation. The focus of her current research is on the combined generation of speech and appropriate gestures by embodied conversational agents (the ANGELICA project). This also involves the generation of natural prosody. Another, related, research interest is digital storytelling, which not only involves modeling the emotions of agents acting as characters in the story, but also appropriately expressing these emotions in natural, spoken language as generated by a ‘virtual storyteller’. Finally, she also works on multimodal presentation in the context of question answering (the IMOGEN project), where the challenge is to present the requested information in the most appropriate and user-friendly way.
Research interests:
• Language generation and prosody.
• Nonverbal communication by means of gestures.
• Digital storytelling.

Drs. R. Poppe
After receiving his MSc. in computer science at the University of Twente, Ronald Poppe is currently working towards a PhD title. Research interests include human body pose estimation, gesture recognition and multimodal interaction. There is a focus on the recognition of emotional affect from human body poses, especially stress.

Drs. K. Oinonen (until 01-01-2006)
Katri Oinonen has graduated with MSc degree on Artificial Intelligence from the University of Amsterdam. After working for the company Sentient Machine Research (SMR), she worked at the Department of Communication Science of the University of Amsterdam to study public response on an mobile multi-media museum guide (EU project SCALEX). For the last few years her main research interest has been interactive storytelling, computational story generation and emotional affective storytelling.