Computational Modeling of Affective User Experience Using Multimodal Physiological and Behavioral Signals

Summary

This protocol describes a computational framework that models the affective user experience by integrating physiological and behavioral signals in a multimodal fashion, using techniques for correlation-based feature learning and multimodal fusion. This protocol proposes and tests a framework for multimodal affective modelling on the AMIGOS benchmark dataset.

Abstract

This work proposes a reproducible computational protocol for multimodal affective modeling that utilizes physiological signals. The goal of the protocol is to enable offline emotion recognition by integrating multiple bio signals using a unified deep learning framework. The proposed work consists of five steps: data collection, preprocessing, feature alignment, multimodal fusion, and evaluation. EEG, ECG, and GSR signals from publicly accessible AMIGOS data were used as the experimental baseline in this work. Bio signals were pre-processed and normalized to extract modality-specific features. Heterogeneous feature spaces were aligned across modalities using Deep Canonical Correlation Analysis, followed by a multimodal fusion network for classifying an affective state. The protocol has been evaluated with offline experiments and compared to conventional fusion and classification models using standard performance metrics such as accuracy, precision, recall, F1-score, and AUC. This study focuses on the development and validation of a computational framework for multimodal affective user experience modeling rather than the deployment of a real-time interactive system. With 92.1% accuracy for UX-affective state prediction and 94.2% F1-score for valence-arousal classification, the results consistently outperformed baseline models on emotional dimensions. These findings verified the effectiveness of the proposed multimodal fusion workflow for computational affective modeling by benchmarking physiological data.

Introduction

The complex interplay of thinking, feeling, and action shapes how people think and act. Affective computing is studying these relationships by leveraging cross-disciplinary knowledge from neuroscience, psychology, and artificial intelligence to build systems that are capable of analysing, understanding, and reacting to human emotion. This area has been increasingly applied to human–technology communication by incorporating expressive consciousness into responsive AI structures, making technology interact not just with intellectual but also with emotive conditions, resulting in more individualized and emotion-aware user knowledge. An emotion, a complex mental pro....

Protocol

The AMIGOS dataset used in this study is publicly available and was collected with prior institutional review board approval and informed consent, as reported in the original publication. This study involves only secondary analysis of the dataset, and no additional ethical approval was required.

The present method uses feature alignment and multimodal fusion approaches to handle multimodal physiological and behavioral data in order to describe perception–emotion correlations. This study proposes a computational model for affective user experience (UX) in interactive exhibitions, leveraging multimodal biophysical sensing and AI-based emotion....

Results

Assessment of the proposed system
To assess the proposed system, it performed experiments on the publicly available AMIGOS dataset, which provides synchronized measurements of EEG, ECG, GSR, video, and audio of 40 users exposed to emotionally engaging stimuli. For the purpose of this research, the authors used data from 33 participants (following preprocessing and removal of incomplete trials), resulting in 1,320 valid samples on the valence and arousal dimensions. The assessment emphasized emotion c.......

Discussion

Spatial, environmental, and physical interaction contexts, such as spatial layout, crowd density, or ambient environmental conditions, are explicitly not given in the AMIGOS dataset. Thus, such factors are also not directly modelled in the current experiments. The suggested computational framework for Affective User Experience (UX) modeling progresses much further than the base paper's foundational concepts that dealt with user, task-oriented child–robot interaction employing biophysical emotion detection. Gene.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Dataset	AMIGOS dataset	40 participants; EEG (128 Hz), ECG (1000 Hz), GSR (1000 Hz), facial video, self-reported valence/arousal labels	Multimodal ground truth data for affective state modeling
Physiological Sensors	EEG headset	Emotiv EPOC+ (14 channels, 128 Hz)	Capturing brain activity related to attention, arousal, and engagement
	ECG sensor	Biopac MP150 or equivalent (1000 Hz)	Heart rate variability and arousal
	GSR/EDA sensor	Shimmer GSR+ or equivalent (1000 Hz)	Skin conductance as measure of arousal
Behavioral Sensors	Eye-tracking device	Tobii Pro X2-60 or equivalent	Recording gaze fixation and saccades
	Facial expression recording	High-resolution video camera; analyzed with OpenFace (AUs, gaze vectors)	Extracting facial Action Units (AUs) and gaze cues
Environmental Inputs	Audio-visual recording setup	Microphone + Camera (synchronized with stimuli)	Capturing contextual stimuli during exhibition
Software / Toolkits	OpenFace	Open-source facial behavior analysis toolkit	Extracting Action Units (AUs), gaze direction
	MATLAB / Python (NumPy, SciPy, scikit-learn)	Signal preprocessing (resampling, z-score normalization, PSD computation)	Data preprocessing and feature extraction
	TensorFlowv2.13 / PyTorchv2.0	Deep learning framework for DCCA and MMFN	Model implementation and training
Algorithms / Models	Deep Canonical Correlation Analysis (DCCA)	Nonlinear feature alignment method	Learning correlated latent representations across modalities
	Multimodal Fusion Network (MMFN)	BiLSTM + Attention-based fusion layers	Hierarchical fusion of heterogeneous modalities for UX state classification
Evaluation Metrics	Accuracy, Precision, Recall, F1-Score, Cohen’s Kappa, AUC-ROC, Confusion Matrix	Implemented with scikit-learn / TensorFlow metrics	Model performance assessment
Computing Hardware	Workstation / GPU cluster	NVIDIA RTX 3080 (10GB) or equivalent, 32 GB RAM, Intel i9 processor	Model training and simulation