Toward Automated Classroom Observation Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate
Abstract : In this work we present a multi-modal machine learning-based system, which we call ACORN, to analyze videos of school classrooms for the Positive Climate (PC) and Negative Climate (NC) dimensions of the CLASS [1] observation protocol that is widely used in educational research. ACORN uses convolutional neural networks to analyze spectral audio features, the faces of teachers and students, and the pixels of each image frame, and then integrates this information over time using Temporal Convolutional Networks.The audiovisual ACORN’s PC and NC predictions have Pearson correlations of 0.55 and 0.63 with ground-truth scores provided by expert CLASS coders on the UVA Toddler dataset (cross-validation on n = 300 15-min video segments), and a purely auditory ACORN predicts PC and NC with correlations of 0.36 and 0.41 on the MET dataset (test set of n = 2000 videos segments). These numbers are similar to inter-coder reliability of human coders. Finally, using Graph Convolutional Networks we make early strides (AUC=0.70) toward predicting the specific moments (45-90sec clips) when the PC is particularly weak/strong. Our findings inform the design of automatic classroom observation and also more general video activity recognition and summary recognition systems.
? Systematic reviews on the use of machine learning within psychiatric and neuroscientific research emphasizes the need for a theory-driven machine learning approach, that is building on existing knowledge and hypothesis, to achieve relevant results and a deeper understanding.
? In other words, this is the method to get high quality ground truth data. Furthermore, the innovative explanation algorithm presented by Ribeiro et al. has the potential to accomplish further scientific understanding, as it gives both; insight into, and reasons for the predictions of the classifiers.
? Systematic reviews on the use of machine learning within psychiatric and neuroscientific research emphasizes the need for a theory-driven machine learning approach, that is building on existing knowledge and hypothesis, to achieve relevant results and a deeper understanding.
? In other words, this is the method to get high quality ground truth data. Furthermore, the innovative explanation algorithm presented by Ribeiro et al. has the potential to accomplish further scientific understanding, as it gives both; insight into, and reasons for the predictions of the classifiers.
• we present our proposed classification taxonomy for automatic MHMS works. graphically depicts the classification taxonomy which is divided into three main levels, namely: study type, study duration and sensing types.
• Subsequently, each level has different types of categories. For instance, the study type can be association, detection and forecasting. The study duration can be either short-term or long-term.
• In the same work, they proposed a state-change detection algorithm without explicitly recognizing the new state, i.e., detect when there is a change from a default state such that this could trigger a notification to visit a doctor for an exact diagnosis.
• Through stacking dilated convolution layers, the TCN can have a very large receptive field with relatively few layers and thus maintain computational efficiency.
• TCN are an alternative to LSTM and GRU networks and can retain high accuracy while reducing run-time costs. We thus tried using a TCN to predict CLASS scores from the audio feature. The TCN took took a 34- dim audio feature vector at each timestep and produces a single CLASS score estimate as output in the final timestep.
• The idea behind the attention is that we can identify the key participants present in each video frame using the self-attention weights. It also condenses the graph into a fixed-length representation.
• We found that both the attention mechanism and dropout were essential to obtain good performance with the GCN.
|