Decoupling Temporal Dynamics for Naturalistic Affect Recognition in a Two-Stage Regression Framework

Abstract

Automatic continuous affect recognition from multiple modalities is one of the most active research areas in affective computing. In addressing this regression problem, the advantages of a model, such as Support Vector Regression (SVR), or a model that can capture temporal dependencies within a predefined time window, such as Time Delay Neural Network (TDNN), Long Short-Term Memory (LSTM) or Kalman Filter (KF), have been frequently explored, but in an isolated way. The motivation is towards decoupling temporal information from its features at the semantic level, in order to exploit the slow-changing emotional property at decision level. This paper explores and proposes 2-stage regression framework where SVR, that has been regarded as the baseline approach on affective recognition task, is concatenated together with subsequent models. Extensive experiments have been carried out on a naturalistic emotion dataset, using eight modalities present in RECOLA database. The results shows the proposed framework can capture temporal information at the prediction level, and outperform state-of-theart approaches in continuous affective recognition.