Computational Modeling of Affective User Experience Using Multimodal Physiological and Behavioral Signals

초록

This work proposes a reproducible computational protocol for multimodal affective modeling that utilizes physiological signals. The goal of the protocol is to enable offline emotion recognition by integrating multiple bio signals using a unified deep learning framework. The proposed work consists of five steps: data collection, preprocessing, feature alignment, multimodal fusion, and evaluation. EEG, ECG, and GSR signals from publicly accessible AMIGOS data were used as the experimental baseline in this work. Bio signals were pre-processed and normalized to extract modality-specific features. Heterogeneous feature spaces were aligned across modalities using Deep Canonical Correlation Analysis, followed by a multimodal fusion network for classifying an affective state. The protocol has been evaluated with offline experiments and compared to conventional fusion and classification models using standard performance metrics such as accuracy, precision, recall, F1-score, and AUC. This study focuses on the development and validation of a computational framework for multimodal affective user experience modeling rather than the deployment of a real-time interactive system. With 92.1% accuracy for UX-affective state prediction and 94.2% F1-score for valence-arousal classification, the results consistently outperformed baseline models on emotional dimensions. These findings verified the effectiveness of the proposed multimodal fusion workflow for computational affective modeling by benchmarking physiological data. © 2026, Journal of Visualized Experiments Journal of Visualized Experiments.