Personal tools
You are here: Home SIGs and LIGs Speech SIG is14-compare

is14-compare

Homepage of the INTERSPEECH 2014 Computational Paralinguistics Challenge

 

Computational Paralinguistics Challenge (ComParE), Interspeech 2014

Cognitive & Physical Load



Last updated:

23 February 2014

Last addition:

Paper on the Challenge released.


Organisers:

Björn Schuller (Imperial College London, UK)
Stefan Steidl (FAU Erlangen-Nuremberg, Germany)
Anton Batliner (TUM, Germany)
Julien Epps (University of New South Wales / NICTA, Australia)

Sponsored by:

Association for the Advancement of Affective Computing (AAAC)
iHEARu
ASC-Inclusion



Officially started:

1 February 2014: Cognitive Load Sub-Challenge
4 February 2014: Physical Load Sub-Challenge



*FINISHED* Winners of the INTERSPEECH 2014 Computational Paralinguistics Challenge:
  • The Cognitive Load Sub-Challenge is awarded to:
    MAARTEN VAN SEGBROECK, RUCHIR TRAVADI, COLIN VAZ, JANGWON KIM, MATTHEW P. BLACK, ALEXANDROS POTAMIANOS, AND SHRIKANTH NARAYANAN
    Classification of Cognitive Load from Speech using an i-vector Framework
  • The Physical Load Sub-Challenge Prize is awarded to:
    HEYSEM KAYA, TUĞÇE ÖZKAPTAN, ALBERT ALI SALAH, SADIK FIKRET GÜRGEN
    Canonical Correlation Analysis and Local Fisher Discriminant Analysis based Multi-View Acoustic Feature Reduction for Physical Load Prediction

    The organisers congratulate the winners and thank all participants for their outstanding contributions - Overall the best result could be reached by fusion of these in the Sub-Challenges.

    If you are interested in using the corpora of the Challenge outside this event, please see here for new licenses and download.





Deadlines:

24 March 2014 Paper submission to INTERSPEECH 2014
16 June 2014 Final result upload
20 June 2014 Camera-ready paper




Baseline results are available:

Download the Paper on the Challenge.




Call for Participation as PDF.

*Get started:*
License Agreement (CLSE) for the dataset download (Cognitive Load Signals Sub-Challenge)
License Agreement (MBC) for the dataset download (Physical Load Sub-Challenge)


*Read about it:* Paper on the Challenge to be cited.
*FAQ:* Frequently asked questions.



The Challenge

The Interspeech 2014 Computational Paralinguistics ChallengE (ComParE) is an open Challenge dealing with states of speakers as manifested in their speech signal’s acoustic properties. There have so far been five consecutive Challenges at INTERSPEECH since 2009 (cf. the repository), but there still exists a multiplicity of not yet covered, but highly relevant paralinguistic phenomena. Thus, we introduce two new tasks by the Cognitive Load Sub-Challenge and the Physical Load Sub-Challenge.

For these tasks, the COGNITIVE-LOAD WITH SPEECH AND EGG database (CLSE), and the MUNICH BIOVOICE CORPUS (MBC) with high diversity of speakers and different languages covered (Australian English and German) are provided by the organisers. CLSE features Australian speakers recorded during different tasks: Story reading (neutral speech), Stroop test with time pressure with three cognitive load levels, reading span task across three working memory load levels, and Stroop test with dual task. Provided are the recordings and synchronised EGG (Laryngograph) signals (not for test set). The annotation features load level, time stamped mouse clicks, subjective ratings (often treated as a state of the art cognitive load indicator in psychology studies), and performance. Details are found in:

Tet Fei Yap: “Speech production under cognitive load: Effects and classification”, Diss., The University of New South Wales, 2012.

Next, MBC contains speech under physical exercising. Heart rate and skin conductance were measured by sensors. The data are described in:

Björn Schuller, Felix Friedmann, Florian Eyben: “Automatic Recognition of Physiological Parameters in the Human Voice: Heart Rate and Skin Conductance”, Proc. ICASSP, IEEE, Vancouver, Canada, 2013.

and

Björn Schuller, Felix Friedmann, Florian Eyben: “The Munich BioVoice Corpus: Effects of Physical Exercising, Heart Rate, and Skin Conductance on Human Speech Production”, Proc. LREC, ELRA, Reykjavik, Iceland, 2014 (to appear).

In these respects, the INTERSPEECH 2014 Computational Paralinguistics Challenge (ComParE) shall help bridging the gap between excellent research on paralinguistic information in spoken language and low compatibility of results.

Two Sub-Challenges are addressed:

  • In the Cognitive Load Sub-Challenge, the ternary level of cognitive load has to be classified based on acoustics.
  • In the Physical Load Sub-Challenge, the exercising state (running / resting) and by that heart rate state (high pulse / low pulse) have to be classified automatically.

The measure of competition will be Unweighted Accuracy. Transcription of the train and development sets will be known. Both Sub-Challenges allow contributors to find their own features with their own machine learning algorithm. However, a standard feature set will be provided per corpus that may be used. Participants will have to stick to the definition of training, development, and test sets. They may report on results obtained on the development set, but have only five trials to upload their results on the test sets, whose labels are unknown to them. Each participation has to be accompanied by a paper presenting the results that undergoes the normal Interspeech peer-review and has to be accepted for the conference in order to participate in the Challenge. The organisers preserve the right to re-evaluate the findings, but will not participate themselves in the Challenge.

We encourage both - contributions aiming at highest performance wrt. the baselines provided by the organisers, and contributions aiming at finding new and interesting insights wrt. these data. Overall, contributions using the provided or equivalent data are sought for (but not limited to):

  • Participation in a Sub-Challenge
  • Contributions focussing on Computational Paralinguistics centred around the Challenge topics

The results of the Challenge will be presented at Interspeech 2014 in Singapore, Singapore.
Prizes will be awarded to the Sub-Challenge winners.
If you are interested and planning to participate in the Computational Paralinguistics Challenge, or if you want to be kept informed about the Challenge, please send the organisers an e-mail to indicate your interest.




To get started: Please obtain the License Agreement (CLSE) or License Agreement (MBC) to get a password and further instructions for the download of the datasets: Please fill it out for the dataset(s) you wish to obtain, and sign and fax, accordingly.
After downloading the data you can directly start your experiments with the train and development sets. Once you found your best method you should write your paper for the Special Session. At the same time you can compute your results per instance and Sub-Challenge task on the test set and upload them: we will then let you know your performance result.

Paper on the Challenge: The introductive Paper on the Challenge provides extensive descriptions and baseline results. All participants will be asked to avoid repetitions of Challenge, data, or feature descriptions in their submissions, but include the following citation:

Björn Schuller, Stefan Steidl, Anton Batliner, Julien Epps, Florian Eyben, Fabien Ringeval, Erik Marchi, Yue Zhang: "The INTERSPEECH 2014 Computational Paralinguistics Challenge: Cognitive & Physical Load", Proceedings INTERSPEECH 2014, ISCA, Singapore, Singapore, 2014.

Result Submission: Opening soon...

Paper Submission (all participants): Please be reminded that a paper submission and at least one upload on the test set are mandatory for the participation in the Challenge - however, paper contributions within the scope are also welcome if the authors do not intend to participate in the Challenge itself. In any case, please submit your paper until 24 March 2014 (and final results by 10 June 2014) using the standard style info and length limits, and submit to the regular submission system. However, as topic you should choose only this Special Session (Computational Paralinguistics Challenge 2014). Please further remind that

  • The deadline for submission of the papers and results is the INTERSPEECH 2014 paper submission deadline: 24 March 2014. Remaining result upload trials can be saved for new Challenge results until 16 June 2014.
  • The papers will undergo the normal review process.
  • Papers shall not repeat the descriptions of database, labels, partitioning etc. of the CLSE and MBC corpora but cite the introductive paper (cf. also above).

    Björn Schuller, Stefan Steidl, Anton Batliner, Julien Epps, Florian Eyben, Fabien Ringeval, Erik Marchi, Yue Zhang: "The INTERSPEECH 2014 Computational Paralinguistics Challenge: Cognitive & Physical Load", Proceedings INTERSPEECH 2014, ISCA, Singapore, Singapore, 2014.

  • Participants may contribute in all Sub-Challenges at a time.
  • A development set will allow for tests and results to be reported by the participants apart from their results on the official test set.
  • Papers may well report additional results on other databases.
  • An additional publication is planned that summarizes all results of the Challenge and results combination by ROVERING or ensemble techniques. However, this publication is assumed to be post INTERSPEECH 2014.



Frequently asked questions: you might find an answer to your questions here:
  • Q: Are there scripts to extract the ComParE feature set with openSMILE available?
    A: Please see here.
  • Q: Do I have to include test set results for the paper submission?
    A: Yes. However, you can update these in the final camera ready version. In any case, at least one result submission has to be made before the paper deadline.
  • Q: Can we submit one paper to Interspeech describing participation in several Sub-Challenges?
    A: This is possible - obviously the paper should read well and contain all necessary details, as it needs to pass peer-review as condition for successful Challenge participation.
  • Q: Can we submit more than one paper to Interspeech describing participation in different Sub-Challenges?
    A: This is possible - you can submit, e.g., separate papers for the participation in different Sub-Challenges or for different approaches in the same Sub-Challenge. Each one will undergo peer-review independently.
  • Q: Can we have richer meta-data for the databases, so that we can also look at other things?
    A: Items not available in the current download package will only be available after the Challenge. Then, however, more info will partially given.
  • Q: My results are below the baseline - does it make sense to submit?
    A: Of course it does. We do not know whether the baseline will be surpassed and different experiences with the tasks on the same dataset will be of interest. Please remember that all submissions to the Challenge go through the normal reviewing process. Although it is very likely that the reviewers do know - and take into account - the baselines, the criteria are the usual, i.e., scientific quality; surpassing any baseline - be this the one given for this challenge, or another one known from the literature - is just one of the criteria. A paper reporting results above the baseline, but poorly written, runs high risks *not* to be accepted; in contrast, a paper which is well written, contributing to our knowledge, but with results below the baseline, has high chances to be accepted.
  • Q: When is the deadline for submitting the results?
    A: You will need to submit results by 16 June 2014 prior to camera ready paper submission to INTERSPEECH as a result on test needs to be included in your final paper version if you want to compete for the Sub-Challenge awards. All except one (which has to be used before the paper deadline on 24 March 2014) of the five result submissions per Sub-Challenge and participant can be saved for submission as late as 16 June 2014.
  • Q: How will the data be distributed to the participants? Are you sending the test data at the same time with training and development partitions?
    A: Yes, we do. However, labels are only given for training and development partitions. Further, we are not sending the data - you will need to download the data. Please first download, print, and sign the license agreements - one per data set once the agreements are available (cf. above) - and scan and mail or fax these to the addresses given on the agreements. You will then receive an email with download instructions.
  • Q: Can I use additional databases within the Challenge?
    A: You can - as long as these are well known and accesible to everybody.
  • Q: May I participate in several Sub-Challenges?
    A: Yes, of course. Every site may participate in every Sub-Challenge and the five upload trials are per Sub-Challenge. In how many Sub-Challenges one site participates is decided at the moment you submit your paper by the results you include. Please send us an additional mail to indicate which Sub-Challenges you participate in for us to know.
  • Q: Do I have to submit a paper in order to participate?
    A: Yes, the submission and acceptance of a paper is mandatory. Please make sure to select the special event during the submission procedure. The prizes will be awarded during INTERSPEECH 2014 in Singapore.
  • Q: May I include results on other corpora in my paper?
    A: Yes, of course. As long as it fits the general focus these are of course very welcome.
  • Q: What are the other formalities with respect to the paper of mine?
    A: Please make sure to reference the official Paper on the Challenge and avoid repetition of the general data and Challenge description.
  • Thank you and welcome to the Challenge!



More Information will follow on a regular basis.



Document Actions
Powered by Plone

Portal usage statistics