Student mental state inference from unintentional body gestures using dynamic Bayesian networks

Applications that interact with humans would benefit from knowing the intentions or mental states of their users. However, mental state prediction is not only uncertain but also context dependent. In this paper, we present a dynamic Bayesian network model of the temporal evolution of students’ mental states and causal associations between mental states and body gestures in context. Our approach is to convert sensory descriptions of student gestures into semantic descriptions of their mental states in a classroom lecture situation. At model learning time, we use expectation maximization (EM) to estimate model parameters from partly labeled training data, and at run time, we use the junction tree algorithm to infer mental states from body gesture evidence. A maximum a posteriori classifier evaluated with leave-one-out cross validation on labeled data from 11 students obtains a generalization accuracy of 97.4% over cases where the student reported a definite mental state, and 83.2% when we include cases where the student reported no mental state. Experimental results demonstrate the validity of our approach. Future work will explore utilization of the model in real-time intelligent tutoring systems.