Personal tools
You are here: Home SIGs Speech SIG INTERSPEECH 2013 Computational Paralinguistics Challenge (ComParE)

INTERSPEECH 2013 Computational Paralinguistics Challenge (ComParE)

Homepage of the INTERSPEECH 2013 Computational Paralinguistics Challenge (ComParE)

 

Computational Paralinguistics Challenge (ComParE), Interspeech 2013

Social Signals, Conflict, Emotion, Autism



Last updated:

16 September 2013

Last addition:

Winners announced and closed.
Presentation Instructions: Please note that all accepted papers are to be presented as oral (talk) presentations and last 9+1 minutes. Please do not repeat description of the Challenge details, but focus on your methods and results.

Organisers:

Björn Schuller (TUM, Germany)
Stefan Steidl (FAU Erlangen-Nuremberg, Germany)
Anton Batliner (TUM, Germany)
Alessandro Vinciarelli (University of Glasgow, UK)
Klaus Scherer (Swiss Center for Affective Sciences, Switzerland)
Fabien Ringeval (University of Fribourg, Switzerland)
Mohamed Chetouani (Université Pierre et Marie Curie, France)

Sponsored by:

HUMAINE Association
Social Signal Processing Network
ASC-Inclusion



Officially started:

21 December 2012: Social Signals Sub-Challenge
1 January 2013: Emotion Sub-Challenge
8 January 2013: Autism Sub-Challenge
15 January 2013: Conflict Sub-Challenge



*FINISHED* Winners of the INTERSPEECH 2013 Computational Paralinguistics Challenge:
  • The Social Signals Sub-Challenge is awarded to:
    RAHUL GUPTA, KARTIK AUDHKHASI, SUNGBOK LEE, AND SHRIKANTH NARAYANAN
    Paralinguistic Event Detection From Speech Using Probabilistic Time-Series Smoothing and Masking
  • The Conflict Sub-Challenge Prize is awarded to:
    OKKO RÄSÄNEN AND JOUNI POHJALAINEN
    Random Subset Feature Selection in Automatic Recognition of Developmental Disorders, Affective States, and Level of Conflict from Speech
  • The Emotion Sub-Challenge Prize is awarded to:
    GÁBOR GOSZTOLYA, RÓBERT BUSA-FEKETE, AND LÁSZLÓ TÓTH
    Detecting Autism, Emotions and Social Signals Using AdaBoost
  • The Autism Sub-Challenge Prize is awarded to:
    MEYSAM ASGARI, ALIREZA BAYESTEHTASHK, AND IZHAK SHAFRAN
    Robust and Accurate Features for Detecting and Diagnosing Autism Spectrum Disorders

    The organisers congratulate the winners and thank all participants for their outstanding contributions - Overall the best result could be reached by fusion of these in the Sub-Challenges.

    If you are interested in using the corpora of the Challenge outside this event, please see here for new licenses and download.





Deadlines:

18 March 2013 Paper submission to INTERSPEECH 2013
24 May 2013 Final result upload
29 May 2013 Camera-ready paper


Download the Paper on the Challenge.




Call for Participation as PDF.

*Get started:*
License agreement for the dataset download (Social Signals Sub-Challenge)
License agreement for the dataset download (Emotion Sub-Challenge)
License agreement for the dataset download (Autism Sub-Challenge)
License agreement for the dataset download (Conflict Sub-Challenge)

*Read about it:* Paper on the Challenge to be cited.
*Participate:* Result submission is now open.
*FAQ:* Frequently asked questions.



The Challenge

After four consecutive Challenges at INTERSPEECH, there still exists a multiplicity of not yet covered, but highly relevant paralinguistic phenomena. In the last instalments, we focused on single speakers. With a new task, we now want to broaden to analysing discussion of multiple speakers in the Conflict Sub-Challenge. A further novelty is introduced by the Social Signals Sub-Challenge: For the first time, non-linguistic events have to be classified and localised – laughter and fillers. In the Emotion Sub-Challenge we are literally “going back to the roots”. However, by intention, we use acted material for the first time to fuel the ever on-going discussion on differences between naturalistic and acted material and hope to highlight the differences. Finally, the Autism Sub-Challenge picks up on Autism Spectrum Condition in children’s speech in this year. Apart from intelligent and socially competent future agents and robots, main applications are found in the medical domain and surveillance.

For these Challenge tasks, the SSPNET VOCALIZATION CORPUS (SVC), the SSPNET CONFLICT CORPUS (SC²), the GEMEP database, and the CHILD PATHOLOGICAL SPEECH DATABASE (CPSD) with high diversity of speakers of different and different languages covered are provided. The first – SVC - consists of 2,763 clips of 10 seconds extracted from 60 phone calls (120 speakers from Glasgow in total). The corpus contains more than 3k instances of laughter, back-channels, and fillers and will serve to evaluate features and algorithms for the determination and localisation of speakers’ social signals in speech. The second – SC² – contains 1,430 clips of 30 seconds extracted from the Canal9 Corpus – a collection of 45 Swiss political debates (in French) - including 138 subjects in total. The clips have been annotated in terms of conflict level by roughly 550 assessors recruited via Amazon Mechanical Turk. The third – GEMEP – contains 1.2k instances of emotional speech from 10 speakers in 18 categories. In the Challenge task, these will need to be classified into 12 categories and the two dimensions arousal and valence. Finally, CPSD provides speech as recorded in two university departments of child and adolescent psychiatry, located in Paris, France (Université Pierre et Marie Curie/Pitié-Salpêtière Hospital and Université René Descartes/Necker Hospital). It contains 9k instances of speech recordings from 99 children aged 6 to 18 years. 35 of these children show pervasive development disorders (PDD) either of Autism Spectrum Condition (ASC, 12 children), specific language impairment (SLI, 13 children) or PDD non-otherwise specified (PDD-NOS, 10 children) according to the DSM-IV criteria. A monolingual control group consists of 64 further children. The French speech includes sentence imitation - 2.8k instances - and storytelling - 6.2k instances, targeting grammatical and affective prosody, respectively. The corpora feature further rich annotation such as speaker meta-data, orthographic transcript, phonemic transcript, and segmentation. All four are given with distinct definitions of test, development, and training partitions, incorporating speaker independence as needed in most real-life settings. Benchmark results of the most popular approaches will be provided as in the years before.

In these respects, the INTERSPEECH 2013 Computational Paralinguistics Challenge (ComParE) shall help bridging the gap between excellent research on paralinguistic information in spoken language and low compatibility of results.

Four Sub-Challenges are addressed:

  • In the Social Signals Sub-Challenge, non-linguistic events – laughter and fillers – of a speaker have to be classified and localised based on acoustics.
  • In the Conflict Sub-Challenge, group discussions have to be automatically evaluated aiming at retrieving conflicts.
  • In the Emotion Sub-Challenge, the emotion of a speaker’s voice has to be determined by a suited learning algorithm and acoustic features.
  • In the Autism Sub-Challenge, the type of pathology of a speaker has to be determined by a suited classification algorithm and acoustic features.

The measures of competition will be Unweighted Accuracy and Area Under the receiver operating Curve (AUC). All Sub-Challenges allow contributors to find their own features with their own machine learning algorithm. However, a standard feature set will be provided per corpus that may be used. Participants will have to stick to the definition of training, development, and test sets. They may report on results obtained on the development set, but have only five trials (per Sub-Challenge) to upload their results on the test sets, whose labels are unknown to them. Each participation will be accompanied by a paper presenting the results that undergoes peer-review and has to be accepted for the conference in order to participate in the Challenge. In the case of the winners of a Sub-Challenge, the organisers preserve the right to re-evaluate the findings which may include that the participants make available according source code, scripts and executable including training routines for cross-check reproduction by the organisers. The organisers will not participate themselves in the Challenge. Participants are encouraged to compete in all Sub-Challenges.

Overall, contributions using the provided or equivalent data are sought for (but not limited to):

  • Participation in a Sub-Challenge
  • Contributions focussing on Computational Paralinguistics centred around the Challenge topics

The results of the Challenge will be presented at Interspeech 2013 in Lyon, France.
Prizes will be awarded to the Sub-Challenge winners.
If you are interested and planning to participate in the Speaker Trait Challenge, or if you want to be kept informed about the Challenge, please send the organisers an e-mail to indicate your interest.


To get started: Please obtain the License Agreement (SVC) or License Agreement (GEMEP) or License Agreement (CPSD) or License Agreement (SC²) to get a password and further instructions for the download of the datasets: Please fill it out for the dataset(s) you wish to obtain, and sign and fax, accordingly.
After downloading the data you can directly start your experiments with the train and development sets. Once you found your best method you should write your paper for the Special Event. At the same time you can compute your results per instance and Sub-Challenge task on the test set and upload them. We will then let you know your according performance result.

Paper on the Challenge: The introductive Paper on the Challenge provides extensive descriptions and baseline results. All participants will be asked to avoid repetitions of Challenge, data, or feature descriptions in their submissions, but include the following citation:

B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi, M. Mortillaro, H. Salamin, A. Polychroniou, F. Valente, S. Kim: “The Interspeech 2013 Computational Paralinguistics Challenge: Social Signals, Conflict, Emotion, Autism”, Proc. Interspeech 2013, ISCA, Lyon, France, 2013.

Result Submission: We've set up the following web site where you can upload and evaluate your predictions on the test set:

http://www5.cs.fau.de/ComParE/2013/

In the meantime, you (or the person who has signed the license agreements) should have received an e-mail with your log-in information and some important notes. Please read them carefully prior to uploading your arff files. If you have not received this information or if you encounter any other problems, please contact Stefan Steidl.

Paper Submission (all participants): Please be reminded that a paper submission is mandatory for the participation in the Challenge - however, paper contributions within the scope are also welcome if the authors do not intend to participate in the Challenge itself. In any case, please submit your paper until 18 March 2013 (and final results by 24 May 2013) using the standard style info and length limits, and submit to the regular submission system. However, you should choose only this Special Event (12.2 Computational Paralinguistics Challenge 2013). Please further remind that

  • The deadline for submission of the papers and results is the INTERSPEECH 2013 paper submission deadline: 18 March 2013. Remaining result upload trials can be saved for new Challenge results until 24 May 2013.
  • The papers will undergo the normal review process.
  • Papers shall not repeat the descriptions of database, labels, partitioning etc. of the SVC, SC², GEMEP and CPSD corpora but cite the introductive paper (cf. also above).

    B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi, M. Mortillaro, H. Salamin, A. Polychroniou, F. Valente, S. Kim: “The Interspeech 2013 Computational Paralinguistics Challenge: Social Signals, Conflict, Emotion, Autism”, Proc. Interspeech 2013, ISCA, Lyon, France, 2013.

  • Participants may contribute in all Sub-Challenges at a time.
  • A development set will allow for tests and results to be reported by the participants apart from their results on the official test set.
  • Papers may well report additional results on other databases.
  • An additional publication is planned that summarizes all results of the Challenge and results combination by ROVERING or ensemble techniques. However, this publication is assumed to be post INTERSPEECH 2013.


Frequently asked questions: you might find an answer to your questions here:
  • Q: Are there scripts to extract the new ComParE 2013 feature set with openSMILE available?
    A: There are, but they work with the upcoming new release of openSMILE that is at the moment only available for Linux. You can find the Linux binary and ComParE config files here. Thank you for using these only within the Challenge at the moment - the full release will follow soon.
  • Q: Why are there 1,098 frames per utterance in the feature files for the Social Signals Sub-Challenge, although the utterances are 11 seconds long and the frame shift is 10ms?
    A: The baseline setup uses a frame length of 25ms for MFCC calculation, as is standard in ASR. This means that for an utterance of 11 seconds length, the last full frame, namely the 1,098th frame, starts at 10.97 seconds.
  • Q: Do I have to include test set results for the paper submission?
    A: If you already have these, it will be informative for the reviewers. However, these only need to be included in the final camera ready version.
  • Q: Can we submit one paper to Interspeech describing participation in several Sub-Challenges?
    A: This is possible - obviously the paper should read well and contain all necessary details, as it needs to pass peer-review as condition for successful Challenge participation.
  • Q: Can we submit more than one paper to Interspeech describing participation in different Sub-Challenges?
    A: This is possible - you can submit, e.g., separate papers for the participation in different Sub-Challenges or for different approaches in the same Sub-Challenge. Each one will undergo peer-review independently.
  • Q: Can we have richer data of the observers/observations, so that we can also look at interobserver statistics?
    A: Items not available in the current download package will only be available after the Challenge. Then, however, this will be the case.-->
  • Q: My results are below the baseline - does it make sense to submit?
    A: Of course it does. We do not know whether the baseline will be surpassed and different experiences with the tasks on the same dataset will be of interest. Please remember that all submissions to the Challenge go through the normal reviewing process. Although it is very likely that the reviewers do know - and take into account - the baselines, the criteria are the usual, i.e., scientific quality; surpassing any baseline - be this the one given for this challenge, or another one known from the literature - is just one of the criteria. A paper reporting results above the baseline, but poorly written, runs high risks *not* to be accepted; in contrast, a paper which is well written, contributing to our knowledge, but with results below the baseline, has high chances to be accepted.
  • Q: When is the deadline for submitting the results?
    A: You will need to submit results by 24 May 2013 prior to camera ready paper submission to INTERSPEECH as a result on test needs to be included in your final paper version if you want to compete for the Sub-Challenge awards. All of the five result submissions per Sub-Challenge and participant can be saved for submission as late as 24 May 2013.
  • Q: How will the data be distributed to the participants? Are you sending the test data at the same time with training and development partitions?
    A: Yes, we do. However, labels are only given for training and development partitions. Further, we are not sending the data - you will need to download the data. Please first download, print, and sign the license agreements - one per data set once the agreements are available (cf. above) - and scan and mail or fax these to the addresses given on the agreements. You will then receive an email with download instructions.
  • Q: Can I use additional databases within the Challenge?
    A: You can - as long as these are well known and accesible to everybody.
  • Q: May I participate in several Sub-Challenges?
    A: Yes, of course. Every site may participate in every Sub-Challenge and the five upload trials are per Sub-Challenge. In how many Sub-Challenges one site participates is decided at the moment you submit your paper by the results you include. Please send us an additional mail to indicate which Sub-Challenges you participate in for us to know.
  • Q: Do I have to submit a paper in order to participate?
    A: Yes, the submission of a paper is mandatory. Please make sure to select the special event during the submission procedure. The prizes will be awarded during INTERSPEECH 2013 in Lyon.
  • Q: May I include results on other corpora in my paper?
    A: Yes, of course. As long as it fits the general focus these are of course very welcome.
  • Q: What are the other formalities with respect to the paper of mine?
    A: Please make sure to reference the official Paper on the Challenge and avoid repetition of the general data and Challenge description.
  • Thank you and welcome to the Challenge!



More Information will follow on a regular basis.



Document Actions
Powered by Plone

Portal usage statistics