Development of Interactive Artificial Intelligence Tools for Personalized Somatosensory and Rhythm Evaluation in Intelligent Music Education Platforms

Summary

This study presents a reproducible somatosensory music-learning protocol combining residual LSTM recognition with TRPO for adaptive difficulty. It covers preprocessing, FFT features, training, personalization, and evaluation. On a public dataset, the hybrid model reached Acc 95.0 / P 93.5 / R 94.6 / F1 94.2 over three subject-disjoint folds.

Abstract

Traditional music education often lacks interactivity and real-time adaptability, especially in remote settings. This study introduces a personalized somatosensory framework, TRPO-ResLSTM, for music education platforms. The system captures movement, rhythm, and response time, preprocesses data with Wiener filtering and Z-score normalization, and extracts features via FFT. Gesture recognition is performed by DeepRes-LSTM, while adaptive difficulty is regulated by TRPO reinforcement learning. Incremental learning ensures personalization across sessions. Experiments on a publicly available, anonymized gesture-rhythm dataset (n = 2,730 samples; training/validation/test split 70/15/15) show superior performance over multimodal baselines, achieving 95% accuracy, 93.5% precision, 94.6% recall, and 94.2% F1-score. Ablation studies confirm the individual contributions of TRPO and Res-LSTM. The innovation of this protocol lies in integrating reinforcement learning with residual temporal modeling for adaptive gesture recognition, enabling stable yet personalized learning. This work demonstrates that adaptive, gesture-responsive tools can enhance engagement, personalization, and progressive skill development in intelligent music education. Limitations include reliance on a single dataset and the need for real-learner validation, which define directions for future work.

Introduction

Recent advances in artificial intelligence (AI) and somatosensory technology are reshaping music education by enabling learners to interact with music through body movements, where gestures are translated into notes, rhythms, or controls for virtual instruments^1,2. These interactive features enhance engagement, retention, and creativity compared to traditional classroom instruction, and somatosensory tools allow students to practice rhythm, coordination, and expression through body percussion, conducting gestures, and ensemble simulations³. Combined with AI-driven adaptive pathways, learners receive individualized content, real-time feedback, and progressive skill development that improve motivation and outcomes^4,5.

Despite these developments, existing platforms often rely on limited modalities, lack continuity of personalization, or fail to adapt to diverse cultural and physical learning styles^6,7. Traditional approaches also fall short in delivering real-time, data-driven adjustments that reflect the learner's evolving capabilities. For example, motion capture and wearable devices can generate rich datasets but are often underutilized in adaptive instruction^8,9. Furthermore, while music libraries and learning management systems have expanded accessibility, they rarely provide dynamic personalization across sessions, which is critical in multicultural and heterogeneous learning contexts¹⁰.

To address these gaps, this study proposes a novel Trust Region Policy Optimized Deep Residual Long Short-Term Memory (TRPO-ResLSTM) framework for music education platforms¹¹. The system integrates advanced preprocessing methods, including Wiener filtering and Z-score normalization, with Fast Fourier Transform for frequency-domain feature extraction. Res-LSTM provides robust recognition of gestures and temporal sequences, while TRPO reinforcement learning dynamically adjusts task difficulty in response to learner performance. Incremental learning further strengthens personalization by updating models across sessions.

Experiments were conducted on the Kaggle music gesture and rhythm dataset comprising 2,730 samples, split into training, validation, and testing subsets. Results show that the proposed method consistently outperforms baseline multimodal architectures, achieving accuracy, precision, recall, and F1 values in the range of 93%-95%. Ablation analyses confirm the effectiveness of both TRPO and Res-LSTM components. By enhancing rhythm accuracy, user engagement, and policy stability in real-time, the framework provides a practical solution for improving music education efficiency in resource-constrained and remote learning environments. Related work on AI-driven music education has highlighted the potential of somatosensory engagement, adaptive learning personalization, and even applications in music therapy and automated composition^12,13. This study builds on these findings by offering a reproducible protocol that combines reinforcement learning with deep temporal modeling to advance the field of intelligent music education.

Protocol

This study analyzed anonymized, publicly available data and did not involve human subjects or animals. Therefore, additional ethical approval was not required.

1. Overview

This protocol describes a reproducible framework for somatosensory music education based on deep residual LSTM recognition and Trust Region Policy Optimization (TRPO) for adaptive difficulty control. It includes dataset preparation, preprocessing, frequency-domain feature extraction, model architecture, training, personalization, and evaluation. Figure 1 illustrates the end-to-end workflow¹⁴.

2. Dataset

A publicly available, anonymized music gesture and rhythm dataset was used that records body responses to auditory cues such as tempo and beat intensity. The dataset provides multimodal time-series data suitable for studying rhythm execution and learning behaviors. Each record contains movement patterns, timing, and rhythm accuracy measures, response feedback, and task completion time. As the dataset was fully anonymized and publicly accessible, no additional institutional review board approval was required¹⁵. For experiments, data were partitioned by subject into training, validation, and test subsets to avoid identity leakage; detailed statistics and split ratios are reported in Table 1. Three subject-disjoint folds were used, fixed random seeds across all runs, and identical preprocessing for all model variants to ensure comparability¹⁶.

The dataset yielded a balanced distribution of rhythmic gestures, with training, validation, and test partitions showing comparable variance in timing and motion features. Descriptive statistics (median ± IQR) for tempo deviation and motion amplitude were comparable across splits, indicating minimal covariate shift.

3. Data preprocessing

1. Z-score normalization
   Raw data were standardized using Z-score normalization. For channel c and time t:
       (1)
   We computed μ_c and σ_c on the training set only and applied them to validation and test sets to avoid leakage¹⁷.
   After normalization, all channels showed near-zero mean and unit variance, ensuring comparability across participants. Batch-wise diagnostics confirmed no drift between folds.
2. Wiener filtering
   To suppress noise, we applied a Wiener filter in the frequency domain:
        (2)
   where Y(k) is the observed spectrum, \hat{X}(k) is the denoised estimate, and S_xx(k), S_nn(k) denote signal/noise power spectral densities. We used a window length and overlap consistent with downstream FFT to maintain phase coherence¹⁸.
   Wiener filtering reduced high-frequency noise by ~30% while retaining dominant rhythmic components. Signal-to-noise ratio improved without attenuating beat-locked peaks.
3. Feature extraction (FFT)
   Short-time FFT features were used on overlapping windows:
       (3)
   Extracted descriptors included dominant frequency, spectral flux, and band-energy ratios. Tempo-locked peak prominence and inter-peak variability were also computed to capture micro-timing stability¹⁹.
   FFT revealed clear spectral peaks aligned with musical tempo (2-3 Hz), confirming rhythmic structure in the dataset. Peak-to-noise ratios exceeded 6-8 dB on average in correctly executed trials.

4. Model: TRPO-ResLSTM

1. Residual LSTM (ResLSTM)
   Temporal patterns were modeled using stacked LSTMs with residual shortcuts:
   (4)
   where P is identity or a projection to match dimensions. Residual connections mitigate vanishing gradients and enable deeper temporal stacks while preserving training stability²⁰.
   Residual connections improved gradient flow and classification accuracy compared to plain LSTM. Ablations showed +0.7-1.1 pp accuracy over non-residual stacks at comparable parameter counts.
2. Trust Region Policy Optimization (TRPO)
   TRPO controlled task difficulty dynamically. The personalized reward was:
      (5)
   with success score s_t, tempo deviation Δtempot, gesture dissimilarity d_t (e.g., DTW distance or classification loss), and a switch indicator u_t (penalizing frequent difficulty changes). We optimized a trust-region objective with a KL constraint:
       (6)
   1. Reinforcement-learning setup and notation
      Adaptive difficulty was formulated as a finite-horizon MDP in which the state s_t aggregates windowed somatosensory features (accelerometer channels, hand-joint positions, rhythm descriptors after normalization, Wiener filtering, and FFT), and the action a_t is a discrete difficulty level controlling tempo tolerance and gesture strictness. The reward r_t balances task success, timing deviation, and engagement, with a small penalty on excessive difficulty switching to discourage oscillation. Policy updates follow TRPO with a KL-divergence constraint for conservative steps. In Eqs. (5-6), g(y,x) denotes the task-specific loss gradient, W(ζ) is an L₂ regularizer over parameters ζ, π_θ is the stochastic policy with parameters θ, D_KL defines the trust region, γ is the discount factor, and δ is the trust-region radius. Hyperparameters α,β,γ,δ were selected via grid search on the validation split (ranges in Table 2) to balance stability and responsiveness; early stopping was triggered when the average KL reached 0.9²¹.
   2. Rationale versus alternatives
      TRPO's KL-constrained on-policy updates were preferred for small, session-level batches and non-stationary learner behavior; PPO/SAC remain promising and will be benchmarked in future work²².
      TRPO achieved stable learning and smoother difficulty adjustment than baseline controllers, with consistent convergence. Learning curves showed monotonic improvement and earlier KL stabilization for TRPO-ResLSTM than for single-component baselines.
3. Personalization and session updates
   Incremental updates refined both ResLSTM and TRPO models after each user session with small learning rates and rehearsal buffers. We used a small rehearsal buffer of recent trials per learner to prevent drift, and limited per-session updates to a fixed budget to preserve stability. Personalization effectiveness was measured as the relative gain in F1 between a learner's first and last session within a fixed horizon²³.
   Session-to-session personalization increased user-specific accuracy by 2%-3% without catastrophic forgetting. Gains were largest for learners with mid-range baseline accuracy, suggesting headroom for adaptive scaffolding.
4. Algorithm and implementation
   The complete pseudo-code ("Algorithm 1: TRPO-ResLSTM") and a reference Python 3.10.1 implementation are provided as. All figures and tables include measurement definitions, error bars, and sample sizes. We report mean ± SD over three subject-disjoint folds and assess inter-model differences with repeated-measures ANOVA or Friedman tests as appropriate, with multiplicity-adjusted post-hoc comparisons (α = 0.05). To ensure reproducibility, we list package versions and GPU/CPU specs in the Table of Materials and include a README with environment and seed configuration²⁴.
   The protocol consistently reproduced improvements over multimodal baselines, validating its reproducibility. Independent re-runs across seeds yielded <0.5 pp variance in accuracy for the integrated model.

Results

Experimental Setup
The TRPO-ResLSTM framework was implemented in Python 3.10.1 with GPU acceleration. The computing environment, motion-sensing hardware, and Python libraries are listed in the Table of Materials. Data were split into subject-disjoint training/validation/test partitions as shown in Table 1 (70/15/15). Key hyperparameters are summarized in Table 2. Three models were evaluated: baseline TRPO, baseline ResLSTM, and the integrated TRPO-ResLSTM. This setup enabled consistent assessment of gesture recognition, adaptive difficulty control, and personalization in music-learning scenarios.

Confusion Matrix
Classification performance was first examined using a confusion matrix on the held-out test set (Figure 2). Rows denote true labels and columns denote predictions; values are row-normalized. Strong diagonal dominance indicates robust recognition across gesture categories, with only minor confusions among closely related rhythmic patterns.

Feature importance analysis
To understand model behavior the importance of permutation-based features was estimated on validation folds (Figure 3). Accelerometer-derived motion signals and hand-joint positions were most informative, followed by temporal rhythm features. These results support somatosensory, gesture-driven interaction as a core mechanism for personalized feedback in music education.

Task completion time distribution
System responsiveness was assessed via the distribution of task completion times (Figure 4). The histogram shows a unimodal profile with stable dispersion across learners, indicating that the protocol maintains consistent task difficulty and throughput under TRPO-based adaptation. For interpretability, we report the median and interquartile range in the legend.

Performance Metrics
On the test set, TRPO-ResLSTM achieved an accuracy of 95.0%, precision 93.5%, recall 94.6%, and F1-score 94.2% (Figure 5). Further, report education-oriented outcomes were reported: rhythm-synchronization accuracy 92.6%, adaptive-response 90.8%, personalization effectiveness 91.5%, and retention rate 85.3%.

Definitions: rhythm-synchronization accuracy quantifies temporal alignment with the target beat; adaptive-response measures the proportion of trials in which the difficulty is correctly re-estimated following a performance change; personalization effectiveness is the relative improvement in recognition F1 after repeated sessions for the same learner; retention rate is the proportion of learners returning for a subsequent session within a predefined interval.

All values are reported as mean ± SD over 3 subject-disjoint folds. Inter-model differences were assessed with repeated-measures ANOVA (or Friedman tests when normality was violated) followed by multiplicity-adjusted post-hoc comparisons; p < 0.05 was considered significant.

Comparative and ablation analysis
Ablation results (Table 3, Figure 6) show complementary benefits from combining trust-region reinforcement learning with residual temporal modeling. TRPO provided stable policy updates but limited representational power for fine-grained gestures; ResLSTM captured temporal dynamics yet lacked on-the-fly adaptation. The integrated TRPO-ResLSTM delivered the best balance, reaching 95.0% accuracy and the highest precision/recall/F1 across folds while preserving real-time adaptability and personalization. Theoretical distinctions from alternative RL methods (e.g., PPO, SAC) were discussed in the Discussion; a full empirical benchmarking lies beyond the scope of this dataset and is identified as future work.

DATA AVAILABILITY:
The data used in the study are available from the authors upon reasonable requests.

Figure 1: System workflow for adaptive, personalized music learning. Pipeline of data collection, preprocessing (Z-score normalization; Wiener filtering), FFT-based feature extraction, gesture recognition with ResLSTM, adaptive difficulty control with TRPO, and incremental personalization. Arrows indicate data flow between modules. Please click here to view a larger version of this figure.

Figure 2: Confusion matrix of gesture recognition on the test set. Row-normalized confusion matrix (n = 409 samples). Rows: true labels; columns: predictions. Colorbar indicates proportion (0-1). Diagonal entries reflect per-class recall; off-diagonals highlight confusions among rhythmically similar categories. Please click here to view a larger version of this figure.

Figure 3: Permutation-based feature importance across sensor inputs. Relative importance (0-1) for accelerometer signals, hand-joint positions, and rhythm features, computed via permutation on validation folds. Bars show mean ± SD over 3 folds. Please click here to view a larger version of this figure.

Figure 4: Distribution of task completion times. Histogram of task durations (seconds) across learners (n = 409 tasks). The kernel-density overlay summarizes central tendency and spread. Reported in the panel are median and IQR; bin width and KDE bandwidth follow standard automatic rules. Please click here to view a larger version of this figure.

Figure 5: Model performance and education-oriented outcomes. Bars show mean ± SD over 3 folds. Reported metrics: gesture-recognition accuracy, precision, recall, F1; rhythm-synchronization accuracy; adaptive-response; personalization effectiveness; retention rate. See Results for formal definitions and statistical testing. Please click here to view a larger version of this figure.

Figure 6: Comparative performance of TRPO, ResLSTM, and TRPO-ResLSTM. Accuracy, precision, recall, and F1 for the three models (mean ± SD, 3 folds). Significance marks denote post-hoc comparisons after repeated-measures ANOVA/Friedman tests (adjusted p < 0.05). Please click here to view a larger version of this figure.

Table 1: Dataset partitioning (subject-disjoint). Counts and proportions (%) for training/validation/test splits; no subject appears in more than one split. Please click here to download this Table.

Table 2: Hyperparameters for TRPO-ResLSTM. Batch size, learning rate, LSTM hidden units, dropout, discount factor (γ), trust-region radius (δ), optimization settings, and early-stopping criteria. Please click here to download this Table.

Table 3: Comparative results across models. Test-set accuracy, precision, recall, and F1 (mean ± SD, 3 folds). The integrated TRPO-ResLSTM outperforms both baselines, as shown in Figure 6. Please click here to download this Table.

Discussion

This study proposes a hybrid protocol, TRPO-ResLSTM, that integrates reinforcement learning and residual temporal modeling for gesture-based music education. By combining the stability of Trust Region Policy Optimization (TRPO) with the sequence-learning capacity of residual LSTMs, the framework delivers real-time gesture recognition together with adaptive difficulty control, enabling personalized feedback and progressive skill acquisition²⁵. To ensure reproducibility, subject-disjoint folds, fixed seeds, and identical preprocessing across models were used, and the mean ± SD was reported over three folds with significance testing aligned to the Results section.

The contribution of this work lies in advancing personalized and interactive learning environments. Unlike systems that rely on static instruction or unimodal recognition, TRPO-ResLSTM dynamically adapts to learners' rhythmic and movement patterns, which supports engagement and retention in practice-oriented session²⁶. The education-oriented indicators we report (rhythm synchronization, adaptive response, personalization effectiveness, retention) operationalize learner-centric outcomes rather than only ML metrics, and they map to recent calls for multimodal analytics in authentic learning setting²⁷.

Prioritization of TRPO was required because KL-constrained on-policy updates offer stable adaptation under small, session-level batches and non-stationary learner behavior that is typical of classroom and remote use. This conservative update rule reduces the risk of policy oscillation when difficulty is adjusted between consecutive actions. Alternative algorithms may also be viable: PPO approximates trust-region updates via clipping and can be sample-efficient, while SAC leverages off-policy maximum-entropy updates and replay for robust exploration^28,29. A comprehensive empirical benchmark across multiple datasets is beyond the scope of the present study and is planned as future work; here, on theoretical rationale and ablations were focused within the task constraints.

Despite these achievements, limitations exist. TRPO incurs non-trivial compute and memory costs, which may constrain deployment on mobile or low-resource classroom devices³⁰. This motivates lightweight policies, model compression, and hardware-aware training for on-device or edge execution. Cultural variability in gesture and rhythm practices may also limit generality without diverse, cross-cultural data. Ethical considerations arise because continuous motion monitoring implicates consent, minimization, and governance of learner data³¹. Future iterations should integrate privacy-by-design safeguards, including local feature extraction, short-horizon retention, and clear consent flows for educational contexts.

Alternative methodological routes remain valuable. Classical sequence models such as hidden Markov models can support interpretable pipelines and small-data regimes³². Self-supervised representation learning on unlabeled motion streams may further reduce annotation burdens and improve transfer. Hybrid controllers that couple rule-based safety checks with a learned policy may also be appropriate when classroom constraints require tight guardrails.

The potential applications of TRPO-ResLSTM extend beyond formal education. The framework could support music therapy, cognitive rehabilitation, and interactive art, where adaptive feedback to human motion is critical³³. In rehabilitation settings, movement-contingent feedback can scaffold motor relearning while preserving engagement, which aligns with emerging practice in technology-supported therapy. For creative practice and performance, the same pipeline can personalize real-time responses to embodied musical gestures.

Looking ahead, multimodal integration, voice, facial expression, and affect can provide more comprehensive feedback in music learning³⁴. Incorporating affect estimates would allow the system to adjust instruction based on moment-to-moment motivation or frustration. Broader inclusion of instruments and musical styles from diverse cultural contexts will enhance accessibility and global relevance³⁵. Finally, control in-the-wild studies that track educational outcomes over weeks rather than sessions, along with a formal benchmark comparing TRPO, PPO, and SAC under identical protocols.

Conclusion
This work demonstrates that music education platforms can be strengthened by adaptive, gesture-responsive AI methods. Traditional approaches often lack interactivity, personalization, and real-time flexibility, especially in remote or platform-based settings³⁶. By leveraging somatosensory input, standard preprocessing (Z-score normalization and Wiener filtering), and FFT-based features, the proposed TRPO-ResLSTM pipeline provides robust real-time gesture recognition and adaptive instruction.

In experimental validation on a public gesture-rhythm dataset, the hybrid model achieved 95.0% accuracy, 93.5% precision, 94.6% recall, and a 94.2% F1-score, and improved education-oriented indicators such as rhythm synchronization, adaptive response, personalization effectiveness, and retention. These outcomes indicate not only predictive strength but also learner-centric value, consistent with evidence that AI-enabled personalization can enhance responsiveness and engagement when deployed responsibly³⁷. The subject-disjoint evaluation and fold-averaged reporting provide a reproducible baseline for future extensions.

In summary, integrating conservative RL updates for policy control with residual temporal modeling for recognition offers a scalable foundation for individualized, interactive music learning. The same principles support broader applications across education, therapy, and rehabilitation³⁸. Future work will extend to multimodal affect-aware feedback, cross-cultural repertoires, and lightweight deployments suitable for classrooms and clinics.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Accelerometer sensor data	Kaggle (Public domain)		Multimodal input signals (motion patterns, timing features) included in dataset
GPU workstation	NVIDIA Corporation, USA		Training hardware: NVIDIA RTX 3080 (10 GB), 32 GB RAM, Ubuntu 20.04
Hand - joint position data	Kaggle (Public domain)		Somatosensory input for gesture recognition
Matplotlib (v3.7)	https://matplotlib.org		Visualization library for plotting figures and performance metrics
NumPy (v1.23)	https://numpy.org		Numerical computation library for array operations
Public music gesture and rhythm dataset	Kaggle (Public domain)		Anonymized dataset of 2,730 samples recording body responses to tempo and beat; used for training/validation/testing (70/15/15)
Python 3.10.1	Python Software Foundation, https://www.python.org		Programming environment for model implementation and analysis
PyTorch (v1.13)	https://pytorch.org		Deep learning framework for implementing ResLSTM and TRPO modules
scikit - learn (v1.2)	https://scikit-learn.org		Machine learning utilities for preprocessing and evaluation
SciPy (v1.10)	https://scipy.org		Scientific computing library (used for Wiener filtering)