Visual Mapping of Multimodal Emotion Features for Intelligent Interaction Design

Summary

The work proposes an end-to-end multimodal emotion analysis framework that leverages transformer encoders, disentangled emotion representations, and graph-based cross-modal attention with visual mapping to improve emotion recognition accuracy across two datasets.

Abstract

By enabling machines to comprehend, interpret, and react to human affective states, visual mapping of multimodal emotion traits is crucial to intelligent interface design. Nevertheless, current multimodal emotion detection algorithms primarily focus on predictive performance, providing little insight into interpretability, interaction awareness, or feature-level cross-modal interactions. This work introduced a framework for the visual mapping of multimodal emotion aspects that combines interaction-oriented visualization, interpretable representation learning, and emotion prediction. Without using manually created features, transformer-based encoders were used to extract high-level emotion embeddings from textual, audio, and visual modalities. To improve robustness, emotion-specific and modality-specific information were separated using a disentangled representation learning technique. Additionally, a graph-based cross-modal attention network was created to use adaptive attention weighting to simulate subtle emotional relationships between modalities. A multi-view visual mapping module was created to depict temporal emotion trajectories, cross-modal attention distributions, and latent emotion embeddings in order to improve transparency. The Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset and the Chinese Multimodal Sentiment Analysis Dataset (CH-SIMS) were used to assess the suggested framework. According to experimental results, the suggested framework provided improved feature-level interpretability and interaction-aware visualization while consistently outperforming representative baseline techniques in terms of accuracy, F1-score, correlation, and mean absolute error. In addition, the visual mapping module facilitated qualitative interpretation of multimodal emotion representations, cross-modal interactions, and temporal emotion dynamics, supporting interaction-aware visualization and analysis.

Introduction

The ability to correctly identify and understand human sentiment has become crucial for improving interaction between humans and machines. In addition to being essential for improving human–computer interaction, emotion analysis has the potential to revolutionize a number of fields, including pre-diagnostic medical diagnosis¹, classroom behavioral analysis², patient psychological tracking³, fatigue identification in vehicles⁴, and intelligent management in smart home settings⁵. Recent developments in affective computing and deep learning⁶ have increased interest in creating real-time systems that can capture the intricacy of human emotion.

Many current methods primarily rely on unimodal data, such as textual, graphic, or auditory signs^7,8,9. These approaches repeatedly fail to detect refined signals such as sarcasm or context-specific nuances. Research has moved to multimodal emotion evaluation, which incorporates complementary data from multiple sources to overcome these restrictions^10,11,12. The advantages of merging multiple modalities have been demonstrated by baseline models such as early fusion-long short-term memory (EF-LSTM)13, light and FM deep neural networks (LF-DNN)¹⁴, tensor fusion networks (TFN), and low-rank multimodal fusion approaches. However, most of these approaches rely on offline feature generation and are not appropriate for dynamic, real-world situations.

In order to accommodate emotional states, multimodal emotion identification research and applications have grown within the past ten years¹⁵. One of the most challenging and significant areas of research nowadays is the continuous monitoring of emotional states using a variety of data sources¹⁶. The idea of multimodality emerged quite naturally with the emergence of such a varied knowledge system, and it has led to many advancements in the application of emotions in domains such as emotional computation, human-computer interaction (HCI), learning, video games, consumer service, medical care, user experience (UX) assessment, etc. However, it has not always been so simple to apply emotion recognition techniques in UX evaluation.

One of the primary reasons to focus on multimodal recognition rather than unimodal methods is the inherent disadvantages of relying on a single information source. Unimodal emotion detection systems can suffer from lower accuracy due to missing or unclear data, whereas MER schemes are more reliable and offer a more comprehensive, thoughtful understanding of expressive states by integrating multiple data sources¹⁷. For example, facial language alone may not be enough to precisely classify feelings, particularly when the gestures are complex or unclear. Though combining facial expressions with other forms of communication, such as language or physical cues, might increase the accuracy of feelings recognition.

Extensive research on emotion recognition has been conducted on several modalities. Early studies focused on identifying distinguishing features from different modalities, like graphic, acoustic, or text-based input. Though it is becoming increasingly evident that integrating various modalities can provide balanced emotional information, leading to more reliable and precise sentiment detection. This segment reviews key advances in single-modality and multimodal emotion analysis, focusing on baseline approaches, their shortcomings, and the rationale for our method.

The initial study on emotion recognition focused primarily on a single modality. In the visual area, groundbreaking research led to the categorization of prototypical facial movements²⁰. Subsequent advancements included techniques for capturing temporal dynamics, such as visual movement²¹ and hidden Markov models²², while facial responses were divided into action units using the Facial Action Coding System (FACS)²³. LSTMs and convolutional neural networks (CNNs) have been successfully used to extract meaningful features directly from raw audio signals; methodologies for audio-based emotion identification have advanced from conventional signal processing to deep learning²⁴. The extraction of contextual sentiment signals in text-based emotion analysis has been greatly enhanced by recurrent neural networks (RNNs) incorporating attention mechanisms and, more recently, by transformer-based models. Although their achievements, unimodal techniques are intrinsically incapable of accurately representing the complexities that people experience because they lack the supplementary information found in other modes.

Some attention mechanism-based models, such as the DialogXL model25, were proposed in 2021 to gather longer-term environmental data in the area of conversational sentiment identification. Kim et al.²⁶ introduced the EmoBERTa model in 2021 to increase the accuracy of ERC. This model uses speaker data to improve the features of speakers in conversations and their interactions with one another. In 2022, Li et al.²⁷ presented the CoG-BART method. The organization uses supervised contrastive learning to improve the model's ability to interpret relevant data. It should be noted that supervised learning requires a substantial amount of labeled data.

A typical example of such work is the Multimodal Interaction-Aware Representation (MIAR) framework²⁸, which uses feature interaction tokens and contrastive alignment to improve generalization across modalities. MIAR improves modality alignment and achieves strong performance across multiple datasets; however, it primarily focuses on predictive accuracy without addressing visual mapping. Similarly, the Cross-Attentional Audio-Visual Fusion model²⁹ effectively captures fine-grained audio-visual interactions for valence and arousal prediction but is limited to two modalities and lacks visualization capabilities. Temporal modeling approaches based on LSTM networks and autoencoder fusion successfully capture temporal dynamics of emotions, yet provide limited insight into modality interactions and the learned emotional representations. More recently, transformer-based methods such as the Transformer-based Multimodal Fusion Network (TMFN)³⁰ have demonstrated strong performance on CMU-MOSI and CMU-MOSEI through enhanced multimodal fusion and contrastive learning. Nevertheless, these approaches remain primarily performance-driven and offer limited support for explainability, feature-level interpretation, and interaction-oriented visualization.

To enhance the integration of textual, acoustic, and visual data, several multimodal emotion recognition techniques have been proposed. Although they were unable to capture intricate cross-modal interactions, early fusion techniques such as EF-LSTM and LF-DNN demonstrated the advantages of using multimodal information for sentiment and emotion analysis. Although the TFN improved performance on benchmark datasets like CMU-MOSI and CMU-MOSEI and introduced explicit modeling of inter-modal interactions, its limited interpretability and high computational complexity hindered its usefulness. To improve modality alignment and dynamic feature fusion, more modern techniques such as MIAR, cross-attentional fusion models, TMFN, and adaptive multimodal transformers have used transformer topologies and attention mechanisms. These methods primarily focused on predictive performance, offering little insight into feature-level emotional representations and cross-modal dependencies, despite achieving competitive results across common evaluation metrics such as accuracy, F1-score, and mean absolute error. Moreover, few studies have established visualization techniques that can disclose latent emotion embeddings, attention distributions, and temporal emotion dynamics, or they have integrated graph-based modeling of inter-modal interactions. The suggested approach incorporates multi-view visual mapping, graph-based cross-modal attention fusion, and disentangled representation learning to overcome these drawbacks and enhance multimodal emotion recognition performance.

Similarly, the Adaptive Multimodal Transformer³² proposes exchange-based fusion to adaptively mitigate noisy or missing modal information, thereby enhancing sentiment classification performance. Although this approach can effectively address discrepancies in the distribution of modal information, its design is task-specific to sentiment analysis and provides limited insight into the learned emotional representations. Another significant contribution is the Multimodal Fusion Model³³, which integrates CNNs, multiplicative LSTMs, and transformer-based attention for real-time multimodal emotion recognition. The model performs full fusion on video, audio, and text modalities and shows robust recognition performance. Nevertheless, like other existing models, it primarily focuses on predictive accuracy and lacks explicit features for visualization and interaction-oriented interpretability.

In conclusion, while present state-of-the-art multimodal emotion recognition approaches have achieved considerable success in capturing cross-modal interactions and enhancing prediction accuracy, most of them are centered on recognition-driven tasks. The areas of feature-level interpretability, visualizing emotional representations in the image space, and incorporating the proposed framework for intelligent interaction design have received little attention. It is with these challenges in mind that the proposed framework is introduced.

The majority of current methods focus primarily on improving predictive performance while offering little interpretability of learned emotional representations and cross-modal interactions, despite notable advancements in multimodal emotion recognition^28,29. Complex interactions among textual, acoustic, and visual modalities are frequently difficult for current fusion strategies to model, especially when modality discrepancies and information shortages occur ^28,31. While transformer-based designs and attention mechanisms have enhanced representation learning, their transparency and interpretability are often diminished by the lack of explicit separation of emotion-specific and modality-specific information^31,32,33. Additionally, only a small number of studies have investigated graph-based modeling of intermodal interactions or created visualization frameworks that can disclose temporal emotion dynamics, attention distributions, and emotion embeddings. These drawbacks demonstrate the necessity of an interpretable multimodal framework for intelligent human-machine interaction that combines interaction-oriented visual mapping with strong cross-modal learning.

This work presents an interpretable framework for visual mapping of multimodal emotion aspects in intelligent interface design. Without the need for manually created features, the system uses end-to-end deep learning to extract high-level emotional representations from textual, audio, and visual modalities. Cross-modal emotional interactions are modeled using a graph-based attention fusion mechanism that separates emotion-specific and modality-specific information, enabling disentangled representation learning and improving interpretability and robustness. Additionally, a multi-view visualization module is created to show temporal emotion dynamics, cross-modal attention patterns, and emotion embeddings. The efficacy and suitability of the suggested framework in intelligent human-machine interaction systems are evaluated through validation on benchmark multimodal emotion recognition datasets.

This work offers a unique framework for visual mapping of multimodal emotion aspects that goes beyond traditional prediction-oriented approaches to overcome the poor modeling of cross-modal interactions and the restricted interpretability in current multimodal emotion identification systems. Within a single architecture, the suggested approach combines transformer-based multimodal representation learning, disentangled emotion-aware feature modeling, graph-based cross-modal attention fusion, and interaction-oriented visualization. The framework clearly separates emotion-specific and modality-specific representations to improve interpretability, robustness, and cross-modal consistency, in contrast to current approaches that mainly concentrate on classification performance. Additionally, a graph-based attention mechanism is presented to enable adaptive modeling of inter-modal interactions by capturing fine-grained emotional interdependence among textual, audio, and visual modalities. A multi-view visual mapping module is created to increase transparency by revealing temporal emotion trajectories, cross-modal attention patterns, and latent emotion embeddings, which offer intuitive insights into the model's decision-making process. In addition to providing improved visualization and interpretability of multimodal emotional dynamics, experimental evaluation on the CH-SIMS and CMU-MOSEI benchmark datasets showed competitive performance in terms of accuracy, F1-score, mean absolute error, and correlation.

Protocol

The proposed work aims to improve intelligent interaction design by offering an interpretable framework for visual mapping of multimodal emotion features. The CH-SIMS and CMU-MOSEI databases, which are accessible to the public, were used in this work. Both datasets were acquired from their official sources and utilized in compliance with the usage guidelines, research standards, and licensing terms set forth by their respective creators. The datasets' multimodal content, including text, audio, and video data, was first gathered and annotated by the dataset providers in accordance with their authorized participant permissions and data collection protocols. There was no direct human interaction, no recruiting of new participants, and no gathering of personally identifiable information in this study. Every analysis was conducted using publicly available research datasets. Consequently, our secondary data analysis did not require any further ethical approval or Institutional Review Board (IRB) review. The study was carried out in compliance with appropriate institutional norms controlling the use of publicly available datasets, ethical research procedures, and applicable data-use policies.

A graph-based cross-modal attention mechanism was employed to learn emotion characterizations across modalities using emotion- or modality-specific characteristics to improve understanding. A multi-view visual mapping component is used to enhance real-time emotion-aware interactive design, and its efficiency is estimated for emotion recognition. The findings show that the proposed method improves both interpretability and efficiency for emotion-aware intelligent user interfaces in the real world. Figure 1 shows the architectural workflow of the proposed work. Figure 1 shows the complete workflow of the suggested visual mapping framework for multimodal emotion feature learning in the context of intelligent interaction systems. The workflow starts with three synchronized input modalities, namely text, audio, and video, which capture complementary emotional information from the linguistic, prosodic, and facial expression modalities, respectively. A modality-specific transformer encoder processes each modality to obtain high-level representations of the raw input data. The representations are then fed into a disentangled representation learning module, where the emotion-specific embeddings are disentangled from the modality-specific features to ensure consistency in the learned emotions across heterogeneous data sources. The emotion-specific embeddings from each modality are then aggregated by a graph-based cross-modal attention network, which models the inter-modal emotional dependencies and assigns dynamic importance weights to each modality based on the interaction context. Finally, the framework provides both emotion classification outputs and interaction insights, which facilitate transparent emotion-aware decision-making and adaptive intelligent interaction design.

Multimodal data acquisition

The CH-SIMS and CMU-MOSEI datasets are two existing public-domain multimodal datasets that have collected multimodal information, such as synchronized video, audio, and text. The CH-SIMS dataset comprises multimodal examinations of Chinese people, including 2,281 video segments, derived from multiple video segments from movies, television dramas, and variety shows. Studies on multimodal analysis of emotions using multimodal data are made more engaging and feasible by the addition of corresponding audio and text to these video segments. However, one of the largest multimodal datasets for the examination of opinion, feelings, and emotion is the CMU-MOSEI dataset, which was collected from over 23,000 video clips of phrases spoken by over 1,000 speakers across a variety of topics. The dataset was randomly divided into training (70%), validation (15%), and testing (15%) subsets while preserving class distribution.

Textual transcriptions, audio signals, and video frames are obtained and timestamp-aligned to match at a fine-grained temporal level. Frame-level integration ensures that the proposed representation learning and graph-based fusion mechanisms can jointly model and leverage emotional content arising from visual expressions, vocal tones, and linguistic content. Effective extraction of inter-modal emotional dynamics is enabled by this type of multimodal acquisition technique, which is essential for intelligent interaction design and comprehensible visual mapping.

Table 1 presents the key structures of the multimodal emotion datasets used in the proposed method. To obtain a wider range of related feelings, such as spoken words, voice intonations, and facial expressions, CH-SIMS and CMU-MOSEI integrate video, audio, and text modalities from these datasets. Furthermore, a limited number of data samples are available in CH-SIMS, enabling a thorough examination of modality interactions and emotion variation in China. On the other hand, the CMU-MOSEI dataset is more important for evaluating the durability of the representation learning and related visualization algorithms covered in this work, as it contains a large amount of data across various languages, subjects, and annotated emotion levels. Additionally, compared to earlier research, it reduces difficulties in selecting information when testing models.

Multimodal preprocessing and synchronization

The timestamp annotations from the CH-SIMS and CMU-MOSEI datasets were used to synchronize textual, auditory, and visual modalities, ensuring consistent multimodal representation learning. Each utterance functioned as the fundamental analysis unit and was linked to matching audio and video segments within the same temporal interval. Alignment was carried out at the utterance level. The transcript was divided into segments based on the start and end timestamps of each utterance. Transcript-audio-visual correspondence was established by extracting the corresponding video frames and audio signals that fell within the same time interval. While audio segments were processed using Wav2Vec 2.0 to produce acoustic representations, visual frames from the video segment were sampled at a predetermined rate and encoded using the Vision Transformer (ViT). The RoBERTa encoder was used to create textual embeddings from the utterance transcripts. Temporal synchronization was achieved by aligning all modality features to a single utterance boundary, since the three modalities produced feature sequences of varying lengths. A fixed-length format was used to normalize sequences of varying lengths. Longer sequences were shortened to the maximum permitted sequence length, while shorter sequences were zero-padded. During training, attention masks were used to separate padded elements from legitimate feature positions. Modality-specific embeddings were projected into a common latent space prior to fusion in order to overcome different sequence lengths across modalities. This guaranteed dimensional consistency and made it possible for the graph-based attention mechanism to facilitate successful cross-modal interaction learning. To preserve data quality and synchronization integrity, samples with corrupted files, missing annotations, or incomplete modality information were excluded from the studies.

Transformer-based feature extraction

A deep learning method is used to learn high-level emotion-aware feature representations from information across multiple modalities, such as vision, audio, and text, in an end-to-end manner after aligning the multimodal data and any required preprocessing. By using a transformer encoder to learn intrinsically discriminative feature representations from raw multimodal data, the proposed method eliminates the need for feature engineering. To facilitate consistency across the datasets used in the proposed approach, a fixed-size embedding is learned for each modality. For example, 768-dimensional feature embeddings are learned across each of the individual modalities, including image, audio, and text modalities, collectively forming a unified feature space used across the entire approach, particularly a feature extraction approach used across the proposed CH-SIMS dataset and the CMU-MOSEI dataset to learn high-level features across data of various sizes.

Visual modality: Vision transformer for emotion-aware frame embeddings

A Vision Transformer (ViT-Base) model was used to extract visual information from video frames in order to obtain emotion-relevant spatial representations. Before extracting features, each video segment's frames were scaled to (224 × 224) pixels and sampled at regular temporal intervals. Using a patch size of 16 × 16 pixels, the ViT-Base architecture produces a series of visual tokens that are processed by 12 transformer encoder layers. The retrieved frame-level representations were projected into a 768-dimensional embedding space using a pretrained ViT model as the visual backbone. The frame-level embeddings were aggregated using mean pooling to create the final visual representation for each segment. During training, image normalization and common augmentation methods, such as random cropping and horizontal flipping, were used to enhance generalization. The suggested multimodal fusion framework was then used to combine these visual embeddings with textual and auditory representations.

To maintain temporal dynamics in emotion, video samples from the CH-SIMS and CMU-MOSEI datasets will be uniformly sampled for the visual modality. Each video's frames, , can be split into N fixed-size sections, which are then flattened and inversely transferred to a latent embedding space with dimension . A series is created by adding positional embeddings and a learnable grouping token:

   (1)

where  stands for positional encoding and  is the patch embedding matrix.

Stacked transformer encoder layers, composed of feed-forward networks and multi-head self-attention, are used to process the embedded patch sequence. The transformation at layer l is described as:

   (2)

   (3)

To obtain the emotion-aware visual feature vector, the output equivalent to the class token is extracted as the feature vector . To address consistent visual emotion representations despite differences in language and content across datasets, frame-level embeddings from each video segment are combined to capture facial expressions and emotion evolution.

Audio modality using Wav2Vec 2.0 for prosody and semantic acoustic embeddings Contextualized speech embeddings were produced using a pretrained Wav2Vec 2.0 encoder as the acoustic backbone. A fixed-length acoustic feature vector for every phrase was obtained by aggregating the output hidden representations using mean pooling. The Wav2Vec 2.0 model was used to extract acoustic representations from audio signals to capture contextual and emotion-relevant speech characteristics. Before feature extraction, audio recordings were normalized and resampled to 16 kHz. To ensure synchronization between the textual and visual modalities, each utterance-level audio segment was isolated using the timestamp bounds provided by the dataset annotations. The pretrained acoustic encoder was adjusted during model training to enhance task-specific adaptation, enabling the derived representations to more accurately represent the emotional aspects of the speech signals. Graph-based cross-modal attention fusion and disentangled representation learning were then performed using the final audio embeddings.

In the audio modality, Wav2Vec-2.0 is used to model speech signals derived from initial speech information in the CH-SIMS and CMU-MOSEI datasets, directly extracting audio features related to emotion. A convolutional feature encoding exists for each input speech signal :

   (4)

where Z stands for low-level prosodic traits like melody, tone, and energy. A transformer-based context network is useful to the aforementioned structures, representing long-range time-based dependencies:

   (5)

Prosodic and semantic acoustic data, which are essential for expressing emotion, are included in these contextual acoustic representations. A specific-length audio embedding is created by performing a temporal pooling procedure:

   (6)

The design avoids the use of acoustic features such as MFCCs and achieves durability using English speech data from CMU-MOSEI and Chinese speech data from CH-SIMS by simply extracting representations from waveforms.

Text Modality using RoBERTa for contextual emotion embeddings

For the textual modality, the deep bidirectional transformer, RoBERTa, is applied to the speech transcripts from the two data sets to generate contextual semantics and affective meaning. RoBERTa-based transformer encoders were used to extract textual representations from transcript data to capture contextual semantic and emotional information. A pretrained RoBERTa model was used for the English-language CMU-MOSEI dataset, whilst a Chinese RoBERTa variant was used for the Chinese CH-SIMS dataset to better account for language-specific linguistic features. Text preprocessing included tokenization using the appropriate RoBERTa tokenizer for each language and text normalization. To ensure consistent batch processing, input sequences were converted to token embeddings and either padded or trimmed to a predetermined maximum sequence length. In accordance with the RoBERTa input format, special classification and separator tokens were introduced. A fixed-length textual embedding was obtained by aggregating the contextualized token representations produced by the transformer encoder using the final [CLS]-equivalent sentence representation. To adapt the learned representations to the emotion recognition task, the pretrained RoBERTa encoders were refined through multimodal training. The suggested multimodal fusion framework was then used to merge the textual embeddings with the audio and visual characteristics.

The token embeddings and positional encodings can be combined for each tokenized input, , to create the input representation in the deep bidirectional transformation technique known as

   (7)

This form is represented through the layers of the L transformer, which uses self-attention to discover the context dependencies among words:

  (8)

The contextual emotion embedding is obtained using the classification token's final hidden state:

   (9)

Sentiment polarity, intense emotions, and associated implications are examples of emotion-related linguistic characteristics that will be represented by this embedding. RoBERTa makes language-agnostic modeling easier. This enables the system to use the same embedding size for both English texts from the CMU-MOSEI dataset and Chinese texts from the CH-SIMS dataset.

Three modality-specific feature vectors of 768 dimensions, each visual f_v, audio f_a, and textual f_t modalities are extracted by the suggested deep feature representation learning step. In the intelligent interaction design, these learned representations create a unified multimodal feature space that serves as the foundation for feature-level visual mapping, graph-based cross-modal fusion, and disentangled representation learning.

Disentangled representation learning

After learning a deep feature representation, additional processing of the multimodal feature vectors from the visual, auditory, and text modalities can yield a disjointed emotion description. If the modality-specific feature embeddings of the auditory, visual, and text modalities used in the prior step are represented by f_v, f_a and f_t, accordingly, such that  and , the objective of this step is to factorize all of the modality features into two distinct latent depictions, which include the emotion depictions after taking into consideration the previous semantics of each of the different modalities.

This is accomplished by feeding each modality feature, f_m into two parallel projection networks: an emotion encoder  and a modality encoder , where the two encodings are produced.

  (10)

Where  the latent feature linked to the emotion is represented by  represents a latent feature specific to a modality. This allows emotion-specific embeddings to communicate with a space repeatedly across multiple inputs, including audio, visual, and text, while maintaining the unique structural data associated with each modality.

An effective separation is enforced by reconstructing with independence constraints. The reconstruction of the original modality feature is given by:

  (11)

where  is a decoder network. The reconstruction loss preserves the following data:

   (12)

Additionally, orthogonality, or decorrelation, between emotion and modality-specific depictions often limits information overlap:

   (13)

By achieving these objectives, the model might learn emotion representations that are reliable, understandable, and resistant to modality variability. Later graph-based cross-modal fusion and visual-mapping features, especially for intelligent interaction design, rely heavily on interpretable, structured emotion representations.

Emotion consistency across modalities

The addition of emotion consistency clearly enhances the effectiveness of the fusion among the modalities. This is because fusion strategies do not need to account for large gaps between the two representations, since the modality-specific emotion embeddings share a latent space. The two emotions' combined illustration is described as follows . Weighted fusion will be more stable when emotion-specific representations are consistent because variations between signals from various modalities will no longer have an impact on the result.

   (14)

By enforcing semantic alignment of emotional information, this goal minimizes discrepancies across modalities and ensures that emotion perception remains consistent regardless of the modality used to express it. Consequently, a cohesive and affective representation space is created by projecting emotional cues obtained from speech signals, facial expressions, and linguistic content.

Impact on cross-modal fusion

Cross-modal fusion becomes more effective when emotion consistency is introduced across modalities. Fusion mechanisms no longer have to make up for significant inter-modal representation gaps because the emotion-specific embeddings are aligned in a common latent space. The fused emotion representation is defined as follows:

  (15)

Where α_m represents the weight of attention given to modality m. The weighted fusion becomes more stable and comprehensible when emotion-specific representations are consistent, as the fusion result shows complementary emotional evidence rather than contradictory modality signals.

Mathematically, lowering  variance between various emotion-specific representation types in relation to  is equivalent to:

   (16)

where  represents the mean emotion expression. Reduced variance results in the input modalities being weighted according to their precise emotional significance rather than modality noise, ensuring improved network convergence and more accurate attention assessment.

The ultimate learning goal of disentangled emotion visualizations is to combine fragmentation, reconstruction, and emotion uniformity:

   (17)

which two hyperparameters, λ₁ and λ₂, control the relationship between cross-modal emotion reliability and disassociation. The suggested model acquires modality-invariant, interpretable, and fusible emotional representations to accomplish this, with an emphasis on providing a solid basis for later graph-based fusion and feature-level representation in intelligent interface design.

Graph-based cross-modal attention fusion

A graph-based cross-modal attention network was used to simulate fine-grained emotional relationships between modalities. To facilitate information flow across modalities, a fully connected multimodal graph was built, with each modality-specific embedding (text, audio, and visual) represented as a graph node. Modality relationships were used to establish the graph adjacency matrix, which was then improved via a learnable attention method. A shared latent space was created by concatenating and projecting the outputs from several attention heads. A unified multimodal emotion representation was then produced by aggregating node representations using attention-weighted pooling. The prediction and visualization modules for interaction-aware analysis and emotion recognition then received this fused representation. The graph-fusion module had a hidden representation dimension of 256 and two graph attention layers, each with eight attention heads. To ascertain the relative importance of nearby modalities, attention coefficients between connected nodes were calculated and standardized using the softmax function. Each graph-attention layer was followed by layer normalization, residual connections, and a dropout rate of 0.2 to increase training stability and reduce overfitting. After concatenating and projecting the outputs of several attention heads onto a common latent space, the node representations were combined into a single multimodal emotion embedding using attention-weighted pooling. After that, the fused representation was used for multi-view visual mapping and emotion prediction.

Diverse cross-modal interactions were captured using multi-head graph attention. The softmax function was used to normalize the attention coefficients between connected nodes for every node. To increase training stability and avoid overfitting, the graph fusion module's stacked graph-attention layers were followed by residual connections, layer normalization, and dropout regularization.

Let's, the terms  individually, represent the representations corresponding to the particular emotions gathered by the visual, audio, and text modalities; each component resides within the shared latent space . The models for the modality nodes use a notation for the graph , with the nodes of the collection  equivacorresponding to the three modality nodes themselves, to explicitly strengthen the emotional dependencies shared among the various modality representations.

After assigning an emotion-specific feature vector to each node in the graph, we have:

   (18)

Unlike traditional fusion methods, which use predetermined or fixed fusion weights, the proposed method learns adaptive edge weights based on their significance and interdependencies, both across channels and across emotional situations.

A Graph Attention Network (GAT) is utilized for cross-modal fusion. An attention coefficient is computed for each node i and its neighboring nodes, i.e., node j:

   (19)

The emotional effect of switching from mode j to modality i is measured by the normalized weights α_ij.

Next, each modality node's fused appearance is computed as a weighted grouping of its neighbors:

   (20)

where the activation function  is non-linear. Ultimately, the updated features of each node are combined to obtain a global fused emotion representation:

   (21)

It is a useful technique for interpretable emotion modeling and subsequent visual mapping because it captures complementary information from various modalities while preserving intricate inter-modal relationships.

Algorithm 1 (Supplementary File 1) provides the general scheme to carry out the graph-based cross-modal fusion. Initially, a graph is created with the emotion-specific characteristics linked to each of the visual, audio, and textual modalities. The attention scores for each graph's node pairs, which represent the combination of emotional dependency, are then calculated. The attention scores are then normalized to indicate the relative contribution of each modality to the specified emotional context, enabling learning of the attention weights. The general emotion embedding is produced in the last stage by aggregating the features associated with the graph's nodes. In this sense, the algorithm's general fusion of the multimodal features linked to the specified emotions shows promise for adaptability, interpretability, and contextual intelligence.

Figure 2 shows the structure and working of the proposed cross-modal attention network for multimodal sentiment fusion. The emotion-specific embeddings, derived independently for the text, auditory, and video modalities, are initially passed through modality-level attention mechanisms that learn the relative importance of linguistic, vocal, and facial information for a given emotional context. The attention-weighted representations of the modalities are then combined to form a unified emotion embedding, in which the attention matrix captures the inter-modal dependencies and interaction strengths. The attention matrix learned by the network dynamically modulates the contributions of the modalities, enabling it to focus on its strongest instructional modality while ignoring noisy, less informative ones. The fused emotion representation enables accurate emotion recognition and fine-grained analysis of emotional states such as neutral, embrace, and worry.

Visual mapping module

With the assistance of the visual mapping section, created specifically for this purpose, an intelligent interaction designer can convert the unified representation of the emotive state into a visual form that is understandable and readily consumable after the cross-modal fusion process, based on the graph.

To make it easier to grasp latent emotional representations, dimensionality-reduction techniques were used to visualize the learned multimodal emotion embeddings. After reducing feature dimensionality while maintaining most of the variance using Principal Component Analysis (PCA), the embeddings were projected into a two-dimensional space for display using t-distributed Stochastic Neighbor Embedding (t-SNE). A perplexity number of 30, a learning rate of 200, and 1,000 optimization iterations were used for t-SNE. Emotion-specific clusters and modality relationships were visualized using the obtained projections. The normalized cross-modal attention weights obtained by the graph-based attention network were used to create attention heatmaps. The relative contributions of textual, audio, and visual modalities during emotion prediction are depicted in these heatmaps. The learned attention coefficients were used as edge weights to create cross-modal interaction graphs, which allowed for the visual examination of inter-modal relationships and information flow inside the fusion framework. Emotion trajectory plots were created by monitoring the development of latent emotion embeddings across successive utterances to examine temporal emotional dynamics. These trajectories shed light on how emotional states and modality contributions evolve throughout time. Matplotlib and Scikit-learn are two Python packages used to create all the visuals. To evaluate interpretability, cluster formation, modality contributions, and temporal emotion dynamics, qualitative analyses of embedding visualizations, interaction graphs, and attention heatmaps were conducted.

Let the variable  represent the fused representation of the emotive state by the graph attention network function. In contrast to the output-level visualization of the emotive state plots, the module's main goal is to convert the emotive state representations and the corresponding weights of the attention mechanism into a comprehensible visual form.

Using the learned α_ji, an attention heatmap is created to visually examine each modality's contribution. The incoming attention weights are then added together to determine a composite importance score for each modality:

    (22)

where s_i represents modality I's relative emotional contribution. To help create an attention heatmap, the obtained values are normalized and mapped to a color map that makes it easy for the user to identify the prominent modality at various time steps or interactive states. Through various visual, auditory, or textual input modalities, such an interface helps the user understand how the system's attention mechanism operates.

The learned graph is specifically modeled as a "weighted interaction graph" to represent the cross-modal dependency of human emotions. The modality itself is represented by each node in the graph, and the strength of the interactions involving human emotions is represented by weights in the form of α_ji on the edges that connect them. The weighted graph above illustrates the intensity of human emotional interactions. The definition of i and j is:

   (23)

The graph visualization's line thickness or transparency is appropriately displayed thanks to these weights. This makes it easier to understand how the modalities interact. The designer can better understand how emotional indicators interact during the fusion process with the assistance of cross-modal interaction graphs.

The fused emotion embeddings  at various time steps are reduced to a lower-dimensional space by means of a dimensionality reduction algorithm such as PCA or t-SNE in order to display the evolution of temporal emotions:

   (24)

where the projection function is represented by . The emergence of 2D/3D emotion trajectories, which show emotion transitions, intensities, and stability in interactions, can be interpreted as an intuitive depiction of the evolution of emotional states over time. These aspects can be used for intelligent interaction, decision-making, and real-time monitoring.

Unlike conventional emotion systems that merely provide mappings to particular classes or scores, this system focuses on demonstrating how human emotions form within it. It reveals its decision-making process by offering visualizations of intermediate features, including weighing factors, interactions between models, and their corresponding paths. This allows the user to comprehend how each factor—visual, vocal, or linguistic affects it in producing its responses to human emotion.

Mapping emotions to comprehensible forms through visual representations can thus bridge the gap between emotion analysis using deep learning methods and their integration for intelligent interface design. The data gathered from both approaches can then be used to create more precise user interfaces. Thus, the suggested approach can give interaction designers vital information to create more precise user interfaces. To put it another way, emotion comprehension becomes a very active part of interaction design. Accuracy can only increase when emotion comprehension is actively incorporated into interaction design.

The visual mapping module makes explainability possible in a number of interconnected but different ways: feature-level explainability reveals underlying representations of emotion created by the DNN, enabling us to comprehend the semantics used to describe each feature related to emotion; modality-level explainability uses data from the interaction graphs and the visual attention heatmaps to describe how each modality visual, audio, or textual contributes to the decisions made to express emotions; and finally, the trajectories resulting from the emotion embedding allow us to understand how each visual and textual modality changes over time from a dynamic interaction perspective. Please refer to Algorithm 2 (Supplementary File 1).

The fused multimodal emotional features are mapped to visually significant representations for intelligent interaction design using the suggested visual mapping Algorithm 2. The graph-based multimodal attention weights are used to quantify differences across the input visual, audio, and textual elements at each time step. These differences are then mapped to visual representations to highlight the main sources influencing the emotions using heatmap visual elements. To provide an insightful view of the interaction between the input elements and convey the emotional evolution generated by the multimodal input elements, it then computes pairwise interaction strengths to create the multimodal interaction weights. In the meantime, a view of emotional evolution is created by mapping the fused emotional elements gathered over time into a low-dimensional space to produce a continuous emotional element evolution.

Emotion prediction and interaction feedback

The outcome of the fused emotion representation, represented as , is then utilized as a function for both interaction feedback and emotion prediction. Furthermore, emotion prediction is described as a multitask model comprising a regression task for continuous emotion intensity recognition and a classification task for discrete emotion recognition. The following softmax-based classification model is used for discrete emotion recognition:

   (25)

where  represents the expected emotion class probabilities and W_c and b_c are learnable constraints. The cross-entropy loss is utilized to enhance the classification objective:

  (26)

where y_k is the ground-truth tag, and K is the number of emotion classes. A regression branch is used to estimate emotion intensity in parallel, producing a prediction of various continuous affective attributes, including valence and arousal. Give it the following definition:

   (27)

where the expected emotion intensity score is indicated by . Mean-squared error loss is used to enhance the regression task:

   (28)

In general, the two tasks are combined in the prediction objective:

   (29)

where the objectives of regression and classification are balanced by λ. It suggests dynamic modification of the visualization module based on the identified emotion state to enable this type of interaction-aware feedback. The interaction context at a given time t is represented by c_t. The visualization module's response is provided by:

    (30)

where the visualization adaptation function is represented by . This makes it possible to adjust the modality heatmaps, interaction graphs, and emotion trajectories in real time based on anticipated changes and emotions. This indicates that the system provides substantial support for feedback in addition to performing well on emotion-state prediction.

Results

This work validates the proposed visual mapping framework for multimodal emotion features in terms of both interaction-aware interpretability and predictive performance, using two benchmark datasets, CH-SIMS and CMU-MOSEI. According to experimental results, the suggested approach consistently performs better than current multimodal emotion recognition techniques across a variety of metrics, including correlation, accuracy, F1-score, and mean absolute error (MAE). The disentangled emotion representation, the graph-based cross-modal fusion mechanism, and the end-to-end deep representation learning strategy all contribute to this improvement by enabling more stable and semantically aligned emotion modeling. The suggested approach provides richer feature-level insights through visual mapping, which goes beyond quantitative gains and makes it easier to understand modality contributions and emotional dynamics. Despite variations in language, cultural expression, and dataset size, the aforementioned performance gain is consistently observed across both datasets, demonstrating the suggested framework's strong generalization capacity.

The CH-SIMS and CMU-MOSEI public datasets for multimodal emotion are used in the proposed work. The CH-SIMS dataset comprises 2,281 video clips from films, TV dramas, and variety shows. Synchronized text, audio, and visual data are available in CH-SIMS. Both unimodal and multimodal sentiment labels are classified into positive, neutral, and negative categories. The large English multimodal dataset known as CMU-MOSEI contains about 23,453 annotated video clips featuring over 1,000 speakers discussing a wide range of subjects. This dataset contains both continuous emotion-intensity annotations, such as valence and arousal, and categorical emotion labels. By supporting a variety of linguistic, cultural, and emotional conditions, experimentation on these two datasets using the suggested framework strengthens the robustness and reliability. Table 2 shows the simulation environment details.

The main experimental and implementation settings used to assess the suggested framework are summed up in this simulation environment. To ensure uniform feature dimensionality across text-based, audio, and visual modalities, transformer-based encoders are applied uniformly. It employs a graph-based attention mechanism for cross-modal fusion, enabling adaptive learning of intermodal dependencies concerning emotional states.

The suggested framework extracts textual, auditory, and visual representations using transformer-based modality encoders. Fully connected layers with ReLU activation project the modality-specific embeddings into a shared latent space. Separate emotion-specific and modality-specific encoders, each with two fully connected layers and nonlinear activation and normalization, are then used to learn disentangled representations. The latent representations are projected into a 256-dimensional feature space, where information for each modality and emotion is learned separately. To maintain modality information and promote efficient disentanglement, reconstruction decoders are used. Before multimodal fusion and prediction, the graph-based cross-modal attention module models emotional dependencies between modalities using the latent representations. The network was trained using the Adam optimizer with an initial learning rate of 1 × 10⁻⁴ and a batch size of 32. Early stopping was employed based on validation loss to prevent overfitting. The total objective function comprised classification, reconstruction, and representation-consistency losses, each weighted by tunable hyperparameters. Model training was run for up to 100 epochs, and the best-performing model on the validation set was retained for testing.

The proposed framework was evaluated using classification metrics, including Accuracy, Precision, Recall, and F1-score, together with regression metrics such as Mean Absolute Error (MAE). Precision, Recall, and F1-score were computed using macro-averaging to account for class imbalance across emotion categories. For classification experiments, the predefined emotion/sentiment labels provided by the CH-SIMS and CMU-MOSEI datasets were used as ground-truth classes. For regression-based sentiment prediction, the datasets' continuous sentiment intensity scores served as the target variables. Confusion matrices were generated to analyze class-wise prediction performance and misclassification patterns with a confidence interval of 95%. To ensure robustness, each experiment was repeated 5 times using different random initializations, and the reported results correspond to the mean ±± standard deviation performance across all runs. Statistical significance was assessed using appropriate hypothesis-testing procedures, and confidence intervals were calculated where applicable. Training time was measured as the total elapsed time required for model convergence on the specified hardware platform.

A number of representative baseline models, including early fusion and late fusion, can be compared with the proposed method. TFN, LSTM-based methods, low-rank multimodal fusion models, adaptive multimodal transformers, and new cross-modal attention-based architectures. These baselines reflect cutting-edge techniques that primarily focus on improving emotion recognition accuracy through various fusion methods. The suggested framework's disentangled representation learning and graph-based fusion improve interpretability and interaction awareness in contrast to these methods, which primarily concentrate on output-level prediction. LSTM Fusion, TFN, Low-Rank Multimodal Fusion (LMF), Adaptive-Graph, and transformer-based techniques are examples of baseline models implemented using their official repositories or publicly accessible reference implementations. The authors' original hyperparameter settings were used when they were accessible. All retrained baselines were assessed using the same dataset splits, preprocessing techniques, optimization parameters, and evaluation criteria to provide a fair comparison. The comparison tables clearly identify the relevant references for data derived directly from earlier publications.

Accuracy

The accuracy of discrete emotion classification is measured for both CH-SIMS and CMU-MOSEI; however, the accuracy trend differs depending on the characteristics of the two datasets. Because CH-SIMS is a medium-sized dataset with an equal number of sentiment categories and carefully preprocessed video clips, its accuracy is high, suggesting that the model can capture subtle multimodal emotional information with little noise. However, accurately classifying emotions is challenging because CMU-MOSEI is a large-scale dataset with speakers from diverse backgrounds. Therefore, in addition to the model's ability to identify sentiment, the accuracy on the CMU-MOSEI dataset indicates its capacity to generalize and withstand a variety of interaction scenarios. The improvement in accuracy across the two datasets shows that the suggested approach generalizes well across datasets of different languages, sizes, and emotional complexity levels.

The classification accuracies of the suggested approach and other common baselines on the CH-SIMS and CMU-MOSEI datasets are shown in Table 3. As shown in Table 3, the proposed framework achieved the highest accuracy on both the CH-SIMS and CMU-MOSEI datasets, outperforming the strongest baseline (Adaptive-Graph) by approximately 3.3% and 3.1 percentage points, respectively. Lower standard deviation values also indicate greater stability across repeated experimental runs. Traditional LSTM-fusion approaches have lower accuracy because of their limited ability to capture complex cross-modal relationships. TFN and LMF improve the classification accuracy by modeling cross-modal relationships.

F1-Score

In emotion classification problems, the balance between recall and precision is measured using the F1-score. For multimodal emotion classification datasets, which are likely to have an uneven number of classes, it is therefore more suitable. In emotion classification tasks, the balance between recall and precision is measured using the F1-score. Since multimodal emotion classification datasets are expected to be unbalanced, the F1-score is more appropriate. The better the model is able to detect emotion classes, the higher the F1-score.

Figure 3 illustrates the evaluation of the F1-score of the suggested technique with the baselines on the CH-SIMS dataset. The LSTM-based baseline's ability to model complex cross-modal emotional relationships is revealed by its lowest F1-score. The F1-scores obtained by the assessed techniques on the CH-SIMS dataset are contrasted in Figure 3. As demonstrated, more sophisticated graph-based and transformer-based structures routinely outperform more conventional fusion-based methods. With an F1-score of 0.71, the LSTM model was the lowest, followed by TFN (0.74) and LMF (0.76). The F1-score was further raised to 0.79 using Adaptive-Graph, demonstrating the value of modeling inter-modal connections. With an F1-score of 0.82, the suggested framework outperformed Adaptive-Graph by 3.8%, LMF by 7.9%, TFN by 10.8%, and LSTM by 15.5%. These findings show that the proposed graph-based cross-modal attention mechanism and disentangled representation learning more effectively capture complementary emotional information across textual, audio, and visual modalities, leading to better classification performance.

The relative trends are consistent, even though all methods exhibit a slight decline in F1-score compared to CH-SIMS, as shown in Figure 4. The findings show a steady improvement in performance from traditional fusion methods to graph-based multimodal learning strategies. The models with the lowest F1-score were LSTM (0.69), TFN (0.72), and LMF (0.74). The Adaptive-Graph model's performance increased to 0.77, demonstrating how well cross-modal connections may be modeled. The suggested framework outperformed all other approaches, achieving the greatest F1-score of 0.80. The suggested disentangled emotion representation learning and graph-based cross-modal attention mechanism's contributions to capturing complementary emotional cues across textual, acoustic, and visual modalities are demonstrated by the improvement over the strongest baseline, Adaptive-Graph. These results show that the suggested framework for multimodal emotion recognition is reliable and efficient. The suggested approach demonstrates a significant improvement over both the graph-based approach and the conventional fusion method, further confirming the method's strong generalization capability. The improvement in the F1-score on the CMU-MOSEI dataset demonstrates that the proposed approach can achieve balanced emotion classification performance even in noisy, diverse environments.

Precision

In smart communication systems, precision is crucial because a false emotional signal can degrade the user experience. The ratio of accurately anticipated instances of a specific emotion to the total number of cases projected to be that emotion is known as precision. Because the disentangled emotion representations are less noisy and less prone to misclassification in the modality, the suggested model outperforms the baseline models on the CH-SIMS dataset in Figure 5.

The accuracy attained by several multimodal emotion recognition techniques on the CH-SIMS and CMU-MOSEI datasets is contrasted in Figure 5. As models move from traditional fusion-based techniques to more sophisticated graph-based architectures, precision steadily increases. The suggested framework attained the maximum precision of 0.82 on the CH-SIMS dataset, with precision rising from 0.71 for LSTM to 0.74, 0.76, and 0.79 for TFN, LMF, and Adaptive-Graph, respectively. On the CMU-MOSEI dataset, precision increased from 0.69 for LSTM to 0.72, 0.74, and 0.77 for TFN, LMF, and Adaptive-Graph, respectively. The suggested approach achieved the highest precision of 0.80. The suggested framework increased precision by roughly 3.8% on CH-SIMS and 3.9% on CMU-MOSEI when compared to the strongest baseline (Adaptive-Graph). These findings show that by capturing complementing emotional information across textual, acoustic, and visual modalities, the suggested disentangled representation learning and graph-based cross-modal attention mechanism successfully lowers false-positive predictions. The impact of the irrelevant modality is further mitigated by the graph-based cross-modal attention. The increased diversity of speakers and linguistic expressions on the CMU-MOSEI corpus causes precision to slightly decline for all models.

Recall

In emotion recognition tasks, recall, a measure of a model's capability to appropriately categorize emotional instances that belong to a specific class, is crucial because the lack of emotional information can negatively affect the quality of interaction. A higher recall value suggests that the model identifies fewer false negatives and more actual emotional expressions. Recall can be considered a measure of the efficiency with which emotional information is extracted from text, audio, and visual modalities in the context of multimodal emotion analysis.

The advantage of the proposed technique over the baseline approaches on the CH-SIMS and CMU-MOSEI datasets is shown in the recall comparison in Figure 6. As models evolve from conventional fusion-based methods to more sophisticated graph-based multimodal structures, results consistently show gains in recall. Recall on the CH-SIMS dataset increased from 0.70 for LSTM to 0.73, 0.75, and 0.78 for TFN, LMF, and Adaptive-Graph, respectively; the maximum recall of 0.82 was attained by the suggested framework. The suggested approach achieved the best recall of 0.80 on the CMU-MOSEI dataset, where recall rose from 0.68 for LSTM to 0.71, 0.73, and 0.76 for TFN, LMF, and Adaptive-Graph, respectively. The suggested framework produced relative improvements of roughly 5.1% on CH-SIMS and 5.3% on CMU-MOSEI when compared to the strongest baseline (Adaptive-Graph). These results show that by capturing complementary emotional cues across textual, acoustic, and visual modalities, the proposed disentangled representation learning and graph-based cross-modal attention mechanism successfully reduces false-negative predictions, thereby improving the identification of emotion classes across both datasets. Because traditional methods have limited capacity to model complex cross-modal relationships, they tend to have lower recall values. By modeling modality relationships more explicitly, TFN and LMF achieve moderate gains. By simulating the structured relationships among the modalities, the Adaptive-Graph method has further improved recall. For both datasets, the proposed method achieves the highest recall, indicating its better ability to detect emotional instances across different interaction scenarios. The improvement is more evident in the large-scale CMU-MOSEI dataset, demonstrating the robustness and generalization of the proposed framework.

Mean absolute error (MAE)

Mean Absolute Error (MAE) is used to assess the performance of continuous emotion-intensity prediction tasks, such as valence or arousal prediction. Unlike accuracy, MAE better reflects the proximity of the predicted emotional intensity to the actual emotional state. This is why MAE is more appropriate for datasets such as CH-SIMS and CMU-MOSEI, which have continuous affective labels. A lower MAE value indicates greater accuracy in emotion intensity prediction.

The MAE comparison between the proposed framework and baseline approaches on the CH-SIMS and CMU-MOSEI datasets is shown in Table 4. Traditional LSTM-based fusion methods tend to have higher MAE values due to their limited ability to model complex multimodal emotional dependencies. The TFN and LMF can reduce the MAE by modeling multimodal interactions, and the Adaptive-Graph approach further reduces it by learning modality graph relationships. The proposed framework achieves the smallest MAE across both datasets, indicating a more accurate estimation of emotion intensity. It is worth noting that the improvement is more significant on the CMU-MOSEI dataset, where the large-scale data variability and speaker conditions make the prediction task more challenging.

Confusion matrix for CH-SIMS dataset

The baseline methods of early fusion LSTM and TFN have significant confusion between the classes of neutral and positive emotions, according to the confusion matrices for the CH-SIMS dataset. This suggests that these methods are not very good at identifying subtle multimodal emotional expressions. On the other hand, the suggested method clearly exhibits strong diagonal dominance, with very few off-diagonal elements, thereby improving class separability. The suggested method's disentangled emotion learning mechanism ensures a consistent representation of emotions across modalities, and its graph fusion approach improves discrimination between closely related sentiment classes. Figure 7A–E shows the confusion matrix for the CH-SIMS dataset with various baseline methods.

Confusion matrix for the CMU-MOSEI dataset

The suggested framework uses modality-invariant emotion embeddings and adaptive attention weights to significantly reduce the misclassification rate, particularly for emotionally ambiguous inputs. This confirms that the suggested framework works well in a variety of large-scale multimodal interaction scenarios. Because of the dataset's size, speaker variation, and the continuous nature of sentiment labels, the CMU-MOSEI confusion matrices exhibit a higher overall misclassification rate. A considerable degree of confusion between various emotion categories is shown by the baseline methods. Figure 8A–E shows the confusion matrix for the CMU-MOSEI dataset with various baseline methods.

Training time per epoch

The computational trade-offs of advanced multimodal fusion techniques are highlighted by the comparison of training time per epoch. Because of the transformer encoders and graph attention mechanisms, the suggested model requires a little more training time than the straightforward fusion approaches, but this increase is not unfeasible. The proposed framework demonstrated strong performance on benchmark multimodal emotion analysis datasets and shows potential for integration into intelligent human–machine interaction systems. However, real-time deployment suitability was not evaluated in the current study and requires further investigation through analyses of latency, throughput, memory, and deployment. Notably, the improved convergence stability and superior predictive and interpretability performance make the additional computational cost worthwhile. Figure 9 shows the result of training time per epoch for the proposed method.

Ablation study

To determine the influence of each important element in the suggested framework, such as the visual mapping module, graph-based cross-modal fusion, and disentangled emotion representation, an ablation study was carried out. To comprehend the impact, each element was eliminated one at a time. According to the experimental results, the graph fusion strategy enhances cross-modal dependency modeling, and the contribution of disentangled emotion representation is most important for performance stability. The visual mapping module's auxiliary role is validated by the slight impact it has on performance when removed. The ablation study results under the same training and evaluation conditions are shown in Table 5. The greatest performance deterioration was observed after removing the graph-based cross-modal attention module, which reduced accuracy from 81.72% to 77.88% and the F1-score from 0.823 to 0.776. This underscores the significance of modeling intricate inter-modal interactions. The disentangled representation learning module's role in learning robust emotion-specific representations was demonstrated by the fact that removing it reduced accuracy by 2.78 percentage points and F1-score by 0.032. The visual mapping module's main advantage is interpretability rather than classification accuracy, as evidenced by the somewhat smaller decline in prediction performance upon its removal. The Basic Fusion configuration performed the worst, demonstrating that the best multimodal emotion recognition performance requires the combined influence of all suggested modules.

DATA AVAILABILITY:

The datasets used in this study are publicly available benchmark datasets. The CMU-MOSEI dataset is provided by the Carnegie Mellon University Multimodal SDK and described in Zadeh et al. (2018) (https://doi.org/10.18653/v1/P18-1208), with access available at https://multicomp.cs.cmu.edu/multimodal-language-analysis-in-the-wild-cmu-mosei-dataset-and-interpretable-dynamic-fusion-graph/. The CH-SIMS dataset is described in Yu et al. (2020) (https://doi.org/10.18653/v1/2020.acl-main.343) and is publicly accessible through resources provided by the authors, including https://github.com/thuiar/MMSA. All experiments were conducted using the official versions of these datasets. The raw extracted data, processed data files, preprocessing outputs, train/validation/test partitions, derived feature representations, and implementation details supporting the findings of this study are publicly available in the Zenodo repository at https://doi.org/10.5281/zenodo.21124921.

Figure 1: Schematic diagram of the proposed multimodal emotion feature visual mapping framework. Conceptual illustration of the overall architecture, showing multimodal feature extraction, disentangled representation learning, graph-based cross-modal attention fusion, and visual mapping components for interpretable multimodal emotion analysis.. Please click here to view a larger version of this figure.

Figure 2: Schematic workflow diagram of the proposed computational protocol.
Conceptual representation of the end-to-end processing pipeline, including dataset acquisition, preprocessing, modality-specific feature extraction, multimodal fusion, visualization, and emotion prediction stages Please click here to view a larger version of this figure.

Figure 3: Comparison of F1-score performance on the CH-SIMS dataset. Mean F1-score values obtained from five independent experimental runs (n = 5) using different random seed initializations. Error bars represent ±± one standard deviation (SD). Higher F1-score values indicate better emotion classification performance. Please click here to view a larger version of this figure.

Figure 4: Comparison of F1-score performance on the CMU-MOSEI dataset. Mean F1-score values obtained from five independent experimental runs (n = 5) using different random seed initializations. Error bars represent ±± one standard deviation (SD), and the corresponding mean ±± SD values are displayed above each data point. Higher F1-score values indicate better emotion classification performance. Please click here to view a larger version of this figure.

Figure 5: Precision comparison on the CH-SIMS and CMU-MOSEI datasets. Mean precision scores obtained by LSTM, TFN, LMF, Adaptive-Graph, and the proposed framework across five independent experimental runs (n = 5). Error bars represent ±± one standard deviation (SD) calculated from repeated runs with different random seed initializations. The proposed framework achieves the highest precision on both datasets, demonstrating improved discrimination of emotion classes through effective multimodal feature learning and graph-based cross-modal fusion. Please click here to view a larger version of this figure.

Figure 6: Recall comparison on the CH-SIMS and CMU-MOSEI datasets. Mean recall scores obtained by LSTM, TFN, LMF, Adaptive-Graph, and the proposed framework across five independent experimental runs (n = 5). Error bars indicate ±± one standard deviation (SD) derived from repeated experiments. The proposed framework consistently achieves the highest recall on both datasets, indicating superior capability in capturing multimodal emotional information and reducing false-negative predictions. Please click here to view a larger version of this figure.

Figure 7: Confusion matrix for dataset CH-SIMS with various baseline methods. Rows correspond to ground-truth emotion classes, and columns correspond to predicted classes. Cell values represent the percentage of samples normalized with respect to each true class. The confusion matrix was computed using the held-out CH-SIMS test partition. Higher diagonal percentages indicate better recognition accuracy for the corresponding emotion category. Confusion matrices of (A) Early fusion LSTM, (B) TFN, (C) LMF, (D) Adaptive Graph, and (E) Proposed method on the CH-SIMS dataset. Please click here to view a larger version of this figure.

Figure 8: Confusion matrix for the CMU-MOSEI dataset with various baseline methods. Rows represent actual emotion labels and columns represent predicted labels. Values are reported as class-normalized percentages on the CMU-MOSEI test set. The matrix illustrates both correctly classified samples (diagonal entries) and confusion among emotion categories (off-diagonal entries). Confusion Matrix on CMU-MOSEI (A) Early fusion LSTM, (B) TFN, (C) LMF, (D) Adaptive Graph, and (E) Proposed method on the CMU-MOSEI dataset. Please click here to view a larger version of this figure.

Figure 9. Comparison of training time per epoch on benchmark multimodal emotion datasets.
Mean training time per epoch obtained from five independent experimental runs (n = 5) for the CH-SIMS and CMU-MOSEI datasets under identical hardware and software settings. Error bars represent ±± one standard deviation (SD) calculated from repeated experiments using different random seed initializations. Lower values indicate higher computational efficiency, whereas stable training times across runs indicate improved reproducibility and training consistency. Please click here to view a larger version of this figure.

Dataset	Modalities	Number of Samples	Language	Emotion/Sentiment Labels
CH-SIMS34	Video, Audio, Text	2,281 video segments	Chinese	Multimodal and unimodal sentiment labels (negative, neutral, positive)
CMU-MOSEI35	Video, Audio, Text	~23,453 annotated clips	English	Sentiment and emotion intensity labels (continuous and categorical)

Table 1: Dataset description. Summary of datasets used within this study, modalities, number of samples, language, emotion/sentiment labels for CH-SIMS and CMU-MOSEI datasets.

Component	Specification
Programming Framework	Python with PyTorch
Visual Feature Extractor	Vision Transformer (ViT)
Audio Feature Extractor	Wav2Vec 2.0
Text Feature Extractor	RoBERTa
Fusion Strategy	Graph-based Cross-Modal Attention Network
Feature Dimension	768 per modality
Training Optimizer	Adam
Learning Rate	1.00E-04
Batch Size	16
Hardware	NVIDIA GPU (e.g., RTX series)
Evaluation Datasets	CH-SIMS, CMU-MOSEI
Latent Dimension	2.56E+02
Embedding Dimension	768
Activation Function	ReLU
Optimizer	Adam
Learning Rate	1.00E-04
Batch Size	32
Epochs	100
Early Stopping	Enabled
Loss Function	Classification + Reconstruction + Consistency

Table 2: Simulation environment. Simulation environment experimental setup and implementation details, such as model specifics, features, optimizer, and hardware used.

Method	CH-SIMS Accuracy (%)	CMU-MOSEI Accuracy (%)
LSTM (Early/Late Fusion)	72.84 ± 1.12	70.35 ± 1.26
TFN	74.91 ± 0.98	72.68 ± 1.10
LMF	76.23 ± 0.85	74.12 ± 0.94
Adaptive-Graph	78.46 ± 0.73	76.89 ± 0.81
Proposed Method	81.72 ± 0.61	79.94 ± 0.69

Table 3: Evaluating the suggested method's accuracy with baseline techniques on the CH-SIMS and CMU-MOSEI datasets. Results are reported as mean ±± standard deviation obtained from five independent experimental runs (n = 5) using different random seed initializations. All methods were evaluated using identical training, validation, and test partitions of their respective datasets to ensure a fair comparison.

Method	CH-SIMS MAE ↓	CMU-MOSEI MAE ↓
LSTM (Early/Late Fusion)	0.412 ± 0.014	0.465 ± 0.016
TFN	0.387 ± 0.012	0.439 ± 0.014
LMF	0.361 ± 0.010	0.412 ± 0.012
Adaptive-Graph	0.334 ± 0.009	0.379 ± 0.010
Proposed Method	0.298 ± 0.007	0.341 ± 0.008

Table 4: Result of mean absolute error. Results are reported as mean ±± standard deviation obtained from five independent experimental runs (n = 5) using different random seed initializations. All methods were evaluated using identical training, validation, and test partitions of their respective datasets to ensure a fair comparison. Comparison of MAE for continuous emotion intensity prediction using the proposed framework and baseline approaches on both datasets.

Model Variant	Accuracy (%)	F1-score
Full Model	81.72 ± 0.61	0.823 ± 0.008
Without Disentanglement	78.94 ± 0.74	0.791 ± 0.010
Without Graph Fusion	77.88 ± 0.81	0.776 ± 0.011
Without Visual Mapping	80.11 ± 0.66	0.807 ± 0.009
Basic Fusion	74.91 ± 0.98	0.742 ± 0.013

Table 5: Result of ablation study with baseline methods. Results are reported as mean ±± standard deviation obtained from five independent experimental runs (n = 5) using different random seed initializations. All methods were evaluated using identical training, validation, and test partitions of their respective datasets to ensure a fair comparison. Comparison of performance between model variants, indicating the effectiveness of learning from disentangled representations, graph-based fusion, and the visual mapping module.

Supplementary File 1: Algorithm 1 and Algorithm 2.Please click here to download this file.

Discussion

The experimental findings demonstrate that, across the CH-SIMS and CMU-MOSEI datasets, the proposed visual mapping framework for multimodal emotion characteristics outperforms current multimodal emotion recognition techniques. Compared with early and late fusion models, such as EF-LSTM and TFN, the proposed technique achieves higher accuracy and F1 Scores, demonstrating its robustness in handling diverse emotional information. The improvement in the method can be attributed to the disentangled emotion representation, which suppresses modality-specific noise and ensures emotion consistency across the visual, audio, and text modalities³⁶.

Recently, transformer-based multimodal models such as Adaptive Multimodal Transformers³⁷ and MIAR³⁸ have demonstrated impressive results in modeling cross-modal dependencies. Nonetheless, these models are primarily concerned with prediction and lack mechanisms to support feature-level interpretability. Conversely, the proposed system combines graph-based cross-modal attention with a visual mapping module, allowing for the transparent analysis of modality and emotional interactions. This is a crucial aspect currently missing from the literature, where emotional inference is treated as a black-box process.

The proposed framework demonstrated consistent performance across the CH-SIMS and CMU-MOSEI benchmark datasets. While the framework may have potential applications in intelligent human–machine interaction, healthcare monitoring, adaptive learning environments, and other emotion-aware systems, the current study evaluated the method only under controlled benchmark conditions. No real-world deployment, user-interface assessment, usability study, or human-subject evaluation was conducted. Therefore, the practical effectiveness of the framework in operational environments remains to be investigated. Future work will focus on deployment-oriented evaluations, user-centered studies, latency analysis, memory consumption assessment, and real-time interaction performance.

Moreover, the performance improvements across various culturally and linguistically diverse datasets demonstrate the validity of the proposed framework. Contrary to previous works validated on a single dataset or a specific interaction scenario, the proposed framework demonstrates robust performance across languages, speakers, and emotional expressions. Therefore, the proposed framework is very appropriate for practical intelligent interaction systems that demand accurate emotion recognition and interpretable decision-making.

The interpretability of the proposed framework was assessed through visualization-based analysis of the learned multimodal representations. Specifically, latent emotion embeddings, cross-modal attention maps, modality interaction graphs, and temporal emotion trajectories were examined to provide insights into the model's decision process and modality contributions. The objective of the visual mapping module is to enhance transparency by enabling observation of feature-level interactions and emotional dynamics. However, formal interpretability metrics, attention-faithfulness evaluations, and user studies were not included in the current work. Therefore, the reported interpretability benefits should be interpreted as visualization-based evidence rather than quantitatively validated explainability measures.

From a theoretical standpoint, this research makes the following contributions to multimodal affective computing: It proposes a systematic way of modeling emotions that clearly distinguishes between emotion-specific and modality-specific representations. By guaranteeing emotional consistency across modalities in a common latent space, the suggested framework thus advances the theory of representation learning in multimodal systems. The incorporation of graph-based attention mechanisms further formalizes the dependencies between the different modalities in expressing emotions.

From a practical perspective, the proposed framework is very beneficial for intelligent interaction design and emotion-aware user interfaces. The visual mapping module in the proposed framework allows system designers to understand the effect of various modalities on emotion prediction, which is very important for system designers. The proposed framework is very beneficial for applications such as affective conversational agents, adaptive learning systems, healthcare monitoring interfaces, and user experience evaluation platforms.

However, the proposed framework has some limitations. The use of graph attention mechanisms and transformer encoders adds complexity, which could be problematic for devices with limited capabilities. Furthermore, the current study ignores other physiological signals, such as EEG and eye-tracking, in favor of focusing on visual, audio, and text modalities. The computational efficiency of the suggested framework for real-time deployment was not specifically assessed, despite the fact that it achieved competitive prediction performance and improved visualization capability. Future research will examine deployment performance in real-world intelligent interaction contexts, inference delay, throughput, memory consumption, and model compression techniques. To further improve the accuracy of emotion modeling and interaction responsiveness, future research will focus on developing lightweight models, real-time optimization techniques, and incorporating additional modalities. Real-time deployment performance was not assessed, and the inclusion of transformer encoders and graph-based attention methods raises computing complexity. Only the CH-SIMS and CMU-MOSEI benchmark datasets were used to validate the methodology, which may introduce dataset-specific biases and restrict generalizability to other domains, languages, and user populations. Additionally, the current study does not include physiological signals like EEG or eye-tracking data; instead, it concentrates on visual, audio, and textual modalities. Although thorough implementation and simulation environment descriptions have enhanced reproducibility, source code and pretrained models are currently not made publicly available.

A novel visual mapping framework for multimodal emotion feature learning for intelligent interaction design is used in this work. The suggested method integrates end-to-end transformer-based representation learning, disentangled emotion modeling, and graph-based cross-modal attention to reach high predictive accuracy and interpretability. Experiments on the CH-SIMS and CMU-MOSEI datasets show that the proposed framework outperforms state-of-the-art multimodal emotion recognition models in terms of accuracy and F1-score. More importantly, the proposed visual mapping solution bridges the gap between emotion recognition and interaction strategy and offers a new direction for the design of interpretable, interaction-aware multimodal affective computing systems.

To facilitate interpretable, intelligent interaction design, this study introduced a novel visual mapping framework for learning multimodal emotion features. The suggested system successfully integrates emotion detection and interpretability within a single architecture by combining transformer-based multimodal feature extraction, disentangled emotion representation learning, graph-based cross-modal attention fusion, and visual mapping methods. In addition to offering clear visual representations of modality contributions, cross-modal interactions, and temporal emotion dynamics, the experimental evaluation on the CH-SIMS and CMU-MOSEI benchmark datasets showed improved performance over representative baseline methods across several metrics, including Accuracy, Precision, Recall, F1-score, and Mean Absolute Error (MAE). However, real-world deployment studies, user-centered evaluations, quantitative explainability assessments, and the integration of physiological modalities are not included in this study, which is restricted to evaluations on two benchmark datasets. Furthermore, environments with limited resources may face difficulties due to the computational cost of transformer encoders and graph-based attention processes. While future work will concentrate on more extensive validation across various datasets, integration of additional modalities, usability studies, reproducible resource release, and optimization for real-time deployment scenarios, the proposed framework overall shows the potential of interpretable multimodal emotion analysis for affective computing and intelligent interaction applications.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Adam	PyTorch Optimizer Library	https://pytorch-optimizers.readthedocs.io/en/latest/ (Learning Rate = 1 × 10-4)	Parameter optimization during model training
CH-SIMS	Original Dataset Repository	Latest Public Release	Benchmark dataset for Chinese multimodal sentiment/emotion analysis
CMU-MOSEI	CMU Multimodal SDK Repository	https://github.com/CMU-MultiComp-Lab/CMU-MultimodalSDK (Latest Public Release)	Benchmark dataset for multimodal sentiment and emotion recognition
Disentangled Emotion Encoder	Custom Implementation	https://pytorch-geometric.readthedocs.io/en/latest/(Dual Encoder Architecture)	Separation of emotion-specific and modality-specific latent representations
Graph Attention Network (GAT)	PyTorch Geometric	Multi-head GAT (https://pytorch-geometric.readthedocs.io/en/latest/)	Modeling cross-modal emotional relationships and adaptive feature fusion
Graph-Based Cross-Modal Attention Layer	Custom Implementation	Multi-Head Attention Fusion	Learning fine-grained emotional dependencies among modalities
Matplotlib	Matplotlib Project	3.7+	Generation of plots and visual analytics
NetworkX	NetworkX Project	3.0+	Construction and visualization of cross-modal interaction graphs
NVIDIA GPU	NVIDIA Corporation	CUDA-enabled GPU	Acceleration of transformer and graph neural network training
Principal Component Analysis (PCA)	Scikit-learn	sklearn.decomposition.PCA	Initial reduction of high-dimensional emotion embeddings
Python	Python Software Foundation	3.8+	Core programming language for implementing the multimodal emotion analysis framework
PyTorch	PyTorch Foundation	2.0+	Development, training, and optimization of deep neural networks
RoBERTa	Hugging Face Transformers	https://huggingface.co/docs/transformers/index (roberta-base, 768-dim embeddings)	Extraction of contextual linguistic and semantic emotion representations
Seaborn	Seaborn Project	0.12+	Generation of attention heatmaps and statistical visualizations
t-distributed Stochastic Neighbor Embedding (t-SNE)	Scikit-learn	scikit-learn.org (Perplexity = 30, Iterations = 1000)	Visualization of latent emotion clusters and trajectories
Ubuntu Linux	Canonical Ubuntu	20.04 LTS or later	Experimental execution environment
Vision Transformer (ViT-Base)	Google Research Vision Transformer	ViT-Base/16, 768-dim embeddings	Extraction of emotion-aware visual representations from video frames
Wav2Vec 2.0	Meta AI Research	Base Model, 768-dim embeddings	Extraction of contextual acoustic and speech emotion features