Dual-Branch Cross-Attention Network for Electroencephalography and Facial Multimodal Emotion Recognition During Painting Viewing

Summary

This protocol describes synchronized acquisition of electroencephalography and facial data in painting-viewing settings and a dual-branch cross-attention network for multimodal emotion recognition, including preprocessing, feature fusion, and classification of four emotional states.

Abstract

Emotion recognition during painting viewing remains difficult because aesthetic responses are dynamic, context-dependent, and only partly observable through outward behavior. Unimodal approaches based only on facial expressions or only on physiological signals, therefore, often provide incomplete representations of the viewer's experience. This article describes a reproducible method for constructing a dual-branch cross-attention network that integrates facial video with electroencephalography (EEG) data for multimodal emotion recognition in a painting exhibition context. The protocol begins with the establishment of a synchronized acquisition environment using a 64-channel EEG system and a high-resolution camera for concurrent recording during painting viewing. The subsequent procedure details signal preprocessing, including EEG filtering, segmentation, and differential entropy feature extraction, together with facial detection, alignment, and frame preparation for video analysis. The neural network contains two modality-specific branches. The EEG branch uses a convolutional neural network and a bidirectional long short-term memory network to model spatial-temporal neural features, whereas the facial branch uses a coordinate attention-enhanced lightweight convolutional network to extract visual expression features. A multi-head cross-attention module is then used to fuse the two streams by learning intermodal relevance. Under the experimental conditions reported here, the multimodal framework achieved higher classification accuracy than the unimodal comparison models in four-category emotion recognition. This method is most suitable for offline analysis in controlled acquisition settings and provides a practical framework for affective computing, empirical aesthetics, and exhibition-oriented human-computer interaction research.

Introduction

Emotion is a multidimensional psychological and physiological process shaped by external stimuli, internal appraisal, and ongoing cognitive regulation, and it therefore occupies a central position in human-computer interaction and cognitive science¹. In painting exhibition settings, emotion also mediates the relationship between visual artworks and viewer interpretation. For this reason, the identification of viewer emotional states has practical value for exhibition evaluation, spatial design, and empirical studies of aesthetic experience². At the same time, emotion measurement in semi-natural exhibition environments re....

Protocol

The study protocol was reviewed and approved by the Ethics Committee on Human Research Protection of Xingyi Minzu Normal University (Approval No. 2600AL0621). Written informed consent was obtained from all participants before their inclusion in the study and prior to data collection.

1. Participant recruitment and ethical compliance

1. Obtaining ethical approval
   1. Submit the complete experimental protocol, consent form, participant information sheet, and data-protection plan to the institutional review board (IRB) or ethics committee before starting the study.
   2. Obtain written approval and record....

Results

The performance of the proposed dual-branch cross-attention network was evaluated using the multimodal dataset collected from 327 participants during painting viewing. The primary outcome was four-class emotion classification performance for fear, happiness, calmness, and sadness. Additional analyses examined unimodal model behavior, multimodal convergence, sensitivity to the number of attention heads, comparative recognition performance, and the behavior of the facial and electroencephalography sub-networks. No separate.......

Discussion

This protocol addresses a practical problem in affective computing, namely, how to recognize emotional states in painting-viewing settings where visible expression and physiological response may not fully coincide at every moment. Under the experimental conditions reported here, the dual-branch framework achieved higher classification performance than the unimodal comparison models, which supports the value of combining electroencephalography and facial-expression information in exhibition-like environments. The findings.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Deep Learning Framework	PyTorch	v2.0 / https://pytorch.org	Used for model construction, training, and evaluation
EEG Acquisition System (64-channel)	Brain Products GmbH	actiCHamp Plus / v1.6	Used for high-resolution EEG signal acquisition during experiments
EEG Electrode Cap (10–20 system)	Brain Products GmbH	ActiCap / 64-channel	Standard international 10–20 electrode placement
High-Definition Camera	Sony Corporation	IMX327 Sensor	Used for facial expression video recording (≥1080p, 30 fps)