Personal tools
You are here: Home SIGs and LIGs Speech SIG is16-compare

is16-compare

Homepage of the Interspeech 2016 Computational Paralinguistics Challenge

 

Computational Paralinguistics Challenge (ComParE), Interspeech 2016

Deception, Sincerity & Native Language



Last updated:

12 October 2016

Last addition:

Paper on the Challenge released

Organisers:

Björn Schuller (University of Passau, Germany AND Imperial College London, UK)
Stefan Steidl (FAU Erlangen-Nuremberg, Germany)
Anton Batliner (TUM, Germany)
Julia Hirschberg (Columbia University, USA)
Judee K. Burgoon (University of Arizona, USA)
Eduardo Coutinho (University of Liverpool, UK)

Sponsored by:

Association for the Advancement of Affective Computing (AAAC)
audEERING UG
iHEARu
ETS



Officially started:

30 January 2016 (Sincerity Sub-Challenge)
6 February 2016 (Native Language Sub-Challenge)
19 February 2016 (Deception Sub-Challenge)



*FINISHED* Winners of the INTERSPEECH 2016 Computational Paralinguistics Challenge:

  • The Deception Sub-Challenge is awarded to:
    CLAUDE MONTACIÉ, MARIE-JOSÉ CARATY:
    Prosodic Cues and Answer Type Detection for the Deception Sub-Challenge
  • The Sincerity Sub-Challenge is awarded to:
    HEYSEM KAYA, ALEXEY A. KARPOV:
    Fusing Acoustic Feature Representations for Computational Paralinguistics Tasks
  • The Native Language Sub-Challenge is awarded to:
    ALBERTO ABAD, EUGÉNIO RIBEIRO, FÁBIO KEPLER, RAMON ASTUDILLO, ISABEL TRANCOSO:
    Exploiting Phone Log-likelihood Ratio Features for the Detection of the Native Language of Non-native English Speakers

The organisers congratulate the winners and thank all participants for their outstanding contributions - Overall the best result could be reached by fusion of these in the Sub-Challenges.

If you are interested in using the corpora of the Challenge outside this event, please see here for new licenses and download.





Deadlines:

23 March 2016 Paper submission to INTERSPEECH 2016
16 June 2016 Final result upload
24 June 2016 Camera-ready paper




Baseline results are available:

Download the Paper on the Challenge.




Call for Participation as PDF.

*Get started:*
License Agreement (Deception) for the dataset download (Deception Sub-Challenge)
License Agreement (Sincerity) for the dataset download (Sincerity Sub-Challenge)
License Agreement (Native Language) for the dataset download (Native Language Sub-Challenge)



*Read about it:* Paper on the Challenge to be cited.
*Participate:* Result submission is now open. Participants were contacted by email for the result submission procedure.
*FAQ:* Frequently asked questions.



The Challenge

The Interspeech 2016 Computational Paralinguistics ChallengE (ComParE) is an open Challenge dealing with states and traits of speakers as manifested in their speech signal’s acoustic properties. There have so far been seven consecutive Challenges at INTERSPEECH since 2009 (cf. the repository), but there still exists a multiplicity of not yet covered, but highly relevant paralinguistic phenomena. Thus, we introduce three new tasks by the Deception Sub-Challenge, the Sincerity Sub-Challenge, and the Native Language Sub-Challenge.

For the tasks, the data are provided by the organisers.

For the Deception Sub-Challenge Judee K. Burgoon and Aaron C. Elkins provide the DECEPTIVE SPEECH DATABASE (DSD). The audio is based on an experiment where university student participants were either imposters with false identities who stole an exam key from a university computer and then lied about the theft or were innocent and told the truth about their activities. All participants were interviewed by the AVATAR (Automated Virtual Agent for Truth-Assessment in Real-Time), which is an Embodied Conversational Agent system that provides a high degree of consistency in the interviews between subjects, something which is lacking from human interviews. Participants randomly assigned to the guilty/deception condition selected a false ID badge and were instructed to maintain that identity throughout the experiment. They were instructed to retrieve an exam key from a computer in the department’s main office. Participants in the innocent/truth condition maintained their own identity and retrieved a flyer from the same office. All participants were interviewed and asked an array of short-answer and open-ended questions including background questions that served as a truthful baseline, direct accusations regarding the theft, and Concealed Information Test (CIT) questions testing recognition of the stolen items. They were also asked some Cognitive Interview style and Behavioral Analysis Interview questions regarding motivations, what should happen to someone who steals, and so forth, the theory being that guilty people would deflect attention from themselves, be vague about other suspects, and suggest more leniency for the perpetrator. Ground truth for deception has been confirmed for guilty participants by (a) their assignment to condition, (b) retrieval of exam on a thumb drive, (c) maintenance of false ID during the interview, and (d) interview answers denying the theft. Confessors have been removed from the corpus. The full set of recordings includes more than 3 hours of speech.

For the Sincerity Sub-Challenge Julia Hirschberg and Alice E. Baird provide the SINCERITY SPEECH CORPUS (SSC). This dataset was created in the context of an art exhibition focused on the communication of sincerity, and particularly in the context of an apology. A number of individuals were asked to read six different sentences whose content is a form of apology. Further, each sentence was read in four different prosodic styles. The golden standard consists of the perceived sincerity as rated by a large group of annotators (at least 15 annotators per instance). The full set of recordings includes more than an hour of speech by several speakers.

For the Native Language Sub-Challenge Keelan Evanini and Educational Testing Service (ETS) provide the NATIVE LANGUAGE SPEECH CORPUS (NLSC). The full set of recordings includes more than 64 hours of speech from 5,132 non-native speakers of English drawn from 11 different L1 backgrounds (Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, and Turkish) with approximately 450 speakers per language representing a range of English speaking proficiencies. The spoken responses for each speaker are 45 seconds in duration responses and were provided in the context of the TOEFL iBT assessment of English for academic purposes.

The INTERSPEECH 2016 Computational Paralinguistics Challenge (ComParE) shall help bridging the gap between excellent research on paralinguistic information in spoken language and low compatibility of results.

Overall, three Sub-Challenges are thus addressed:

  • In the Deception Sub-Challenge, deceptive speech has to be identified.
  • In the Sincerity Sub-Challenge, perceived sincerity of speakers has to be determined for the first time.
  • In the Native Language Sub-Challenge, the native language of non-native English speakers from eleven countries has to be recognised.

All Sub-Challenges allow contributors to find their own features with their own machine learning algorithm. However, a standard feature set will be provided that may be used. Participants will have to stick to the definition of training, development, and test sets as given. They may report results obtained on the development sets, but have only five trials to upload their results on the test sets depending on the Sub-Challenges, whose labels are unknown to them. Each participation has to be accompanied by a paper presenting the results that undergoes the normal Interspeech peer-review and has to be accepted for the conference in order to participate in the Challenge. The organisers preserve the right to re-evaluate the findings, but will not participate themselves in the Challenge.

We encourage both - contributions aiming at highest performance w.r.t. the baselines provided by the organisers, and contributions aiming at finding new and interesting insights w.r.t. these data. Overall, contributions using the provided or equivalent data are sought for (but not limited to):

  • Participation in a Sub-Challenge
  • Contributions focussing on Computational Paralinguistics centred around the Challenge topics

The results of the Challenge will be presented at Interspeech 2016 in San Francisco, USA.
Prizes will be awarded to the Sub-Challenge winners.
If you are interested and planning to participate in the Computational Paralinguistics Challenge, or if you want to be kept informed about the Challenge, please send the organisers an e-mail to indicate your interest.




To get started: Please obtain the License Agreement (Sincerity Sub-Challenge) (Agreements for the other Sub-Challenges follow here once they opened) to get a password and further instructions for the download of the datasets: Please fill it out for the dataset(s) you wish to obtain, and print, sign, scan, and email accordingly. The agreement(s) has/have to be signed by a permanent staff member.
After downloading the data you can directly start your experiments with the data sets. Once you found your best method you should write your paper for the Special Session. At the same time you can compute your results per instance and Sub-Challenge task on the test set and upload them: we will then let you know your performance result.

Paper on the Challenge: The introductory Paper on the Challenge (the paper is now available) provides extensive descriptions and baseline results. All participants will be asked to avoid repetitions of Challenge, data, or feature descriptions in their submissions - of course, they have to describe shortly the essentials of the databases dealt with - but include the following citation:

Björn Schuller, Stefan Steidl, Anton Batliner, Julia Hirschberg, Judee K. Burgoon, Alice Baird, Aaron Elkins, Yue Zhang, Eduardo Coutinho, Keelan Evanini: "The INTERSPEECH 2016 Computational Paralinguistics Challenge: Deception, Sincerity & Native Language", Proceedings INTERSPEECH 2016, ISCA, San Francisco, USA, 2016.

Result Submission: Opened. Registered participants have been contacted by email.

Paper Submission (all participants): Please be reminded that a paper submission and at least one upload on the test set are mandatory for the participation in the Challenge. However, paper contributions within the scope are also welcome if the authors do not intend to participate in the Challenge itself. In any case, please submit your paper until 23 March 2016 (and final results by 16 June 2016) using the standard style info and respecting length limits, and submit to the regular submission system. However, as topic you should choose only this Special Session (Computational Paralinguistics Challenge 2016). Please note that

  • The deadline for submission of the papers and results is the INTERSPEECH 2016 paper submission deadline: 23 March 2016. Remaining result upload trials can be saved for new Challenge results until 16 June 2016.
  • The papers will undergo the normal review process.
  • Papers shall not repeat the descriptions of database, labels, partitioning etc. of the Sub-Challenge corpora but cite the introductive paper :

    Björn Schuller, Stefan Steidl, Anton Batliner, Julia Hirschberg, Judee K. Burgoon, Alice Baird, Aaron Elkins, Yue Zhang, Eduardo Coutinho, Keelan Evanini: "The INTERSPEECH 2016 Computational Paralinguistics Challenge: Deception, Sincerity & Native Language", Proceedings INTERSPEECH 2016, ISCA, San Francisco, USA, 2016.

  • Participants may contribute in all Sub-Challenges at a time.
  • A training and development partitioning will allow for tests and results to be reported by the participants apart from their results on the official test set.
  • Papers may well report additional results on other databases.
  • An additional publication is planned that summarises all results of the Challenge and results combination by ROVERING or ensemble techniques. However, this publication is assumed to be post INTERSPEECH 2016.



Frequently asked questions: you might find an answer to your questions here:
  • Q: Are there scripts to extract the ComParE feature set with openSMILE available?
    A: Please see here.
  • Q: Do I have to include test set results for the paper submission?
    A: Yes. However, you can update these in the final camera ready version. In any case, at least one result submission has to be made before the paper deadline.
  • Q: Can we submit one paper to Interspeech describing participation in several Sub-Challenges?
    A: This is possible - obviously the paper should read well and contain all necessary details, as it needs to pass peer-review as condition for successful Challenge participation.
  • Q: Can we submit more than one paper to Interspeech describing participation in different Sub-Challenges?
    A: This is possible - you can submit, e.g., separate papers for the participation in different Sub-Challenges or for different approaches in the same Sub-Challenge. Each one will undergo peer-review independently.
  • Q: Can we have richer meta-data for the databases, so that we can also look at other things?
    A: Items not available in the current download package will only be available after the Challenge. Then, however, more info will partially be given.
  • Q: My results are below the baseline - does it make sense to submit?
    A: Of course it does. We do not know whether the baseline will be surpassed and different experiences with the tasks on the same dataset will be of interest. Please remember that all submissions to the Challenge go through the normal reviewing process. Although it is very likely that the reviewers do know - and take into account - the baselines, the criteria are the usual, i.e., scientific quality; surpassing any baseline - be this the one given for this challenge, or another one known from the literature - is just one of the criteria. A paper reporting results above the baseline, but poorly written, runs high risks *not* to be accepted; in contrast, a paper which is well written, contributing to our knowledge, but with results below the baseline, has high chances to be accepted.
  • Q: When is the deadline for submitting the results?
    A: You will need to submit results by 16 June 2016 prior to camera ready paper submission to INTERSPEECH as a result on test needs to be included in your final paper version if you want to compete for the Sub-Challenge awards. All except one (which has to be used before the paper deadline on 23 March 2016) of the five result submissions per Sub-Challenge and participant can be saved for submission as late as 16 June 2016.
  • Q: How will the data be distributed to the participants? Are you sending the test data at the same time with training and development partitions?
    A: Yes, we do. However, labels are only given for training and development partitions. Further, we are not sending the data - you will need to download the data. Please first download, print, and sign the license agreements - one per data set once the agreements are available (cf. above) - and scan and mail or fax these to the addresses given on the agreements. You will then receive an email with download instructions.
  • Q: Can I use additional databases within the Challenge?
    A: You can - as long as these are well known and accessible to everybody and you clearly state the details in your paper.
  • Q: May I participate in several Sub-Challenges?
    A: Yes, of course. Every site may participate in every Sub-Challenge and the five upload trials are per Sub-Challenge. In how many Sub-Challenges one site participates is decided at the moment you submit your paper by the results you include. Please send us an additional e-mail to indicate which Sub-Challenges you participate in for us to know.
  • Q: Do I have to submit a paper in order to participate?
    A: Yes, the submission and acceptance of a paper is mandatory. Please make sure to select the special event during the submission procedure as first option. The prizes will be awarded during INTERSPEECH 2016 in San Francisco/USA.
  • Q: May I include results on other corpora in my paper?
    A: Yes, of course. As long as it fits the general focus these are of course very welcome.
  • Q: What are the other formalities with respect to the paper of mine?
    A: Please make sure to reference the official Paper on the Challenge (now available) and avoid repetition of the general data and Challenge description.
  • Thank you and welcome to the Challenge!



More Information will follow on a regular basis.



Document Actions
Powered by Plone

Portal usage statistics