Review Article
This review examines how multimodal architectures that combine auditory, articulatory, and visual information, along with transformer and TCN models, can improve phoneme identification in dysarthric speech. It emphasizes their potential to improve speech intelligibility assessment and personalized therapy beyond the capabilities of typical ASR systems.
Due to articulation issues and phoneme variability, speech disorders such as dysarthria pose significant challenges in clinical rehabilitation. Traditional automatic speech recognition (ASR) systems, which are typically trained on normative datasets, often fail to accurately decode dysarthric speech. Recent breakthroughs in artificial intelligence and deep learning have allowed the integration of multimodal architectures, such as merging auditory, articulatory, and visual information to record complex speech patterns, making it increasingly possible. In this review, we have explored the convergence of transformers and temporal convolutional networks (TCNs) within multimodal frameworks to address phoneme labeling imprecision in dysarthric speech. Here, we discuss how transformer-based contextual modeling and TCN-driven temporal precision can enhance phoneme boundary detection, classification, and rehabilitation feedback. The review also discusses the potential of such hybrid systems in individualized speech therapy, interpretability issues, and clinical applications. Multimodal architectures are a translational approach to enhancing therapeutic monitoring, communication aids, and speech intelligibility assessment for people with motor speech disorders by bridging clinical medicine and medical informatics.
Nervous system impairment can often lead to drastic and abrupt health consequences, which include musculoskeletal disorders, neurovascular disorders, and others like neurogenic communication disorders. Neurogenic communication disorders include disorders like dysarthria and apraxia, which can be defined as slurred speech due to muscle weakness and difficulty planning speech movements, respectively1. Speech is made up of five subsystems: breathing, phonation, resonance, articulation, and prosody, all of which must work together synergistically and seamlessly to communicate effectively2. Dysarthria, caused by dysfunction in these subsystems, reduces audibility, naturalness, intelligibility, and communication efficiency, which, in turn, significantly impacts patients and their families’ lives. Dysarthria can be caused by a variety of neurological illnesses and underlying pathologies, including dysfunctions in the cerebral cortex, basal ganglia, cerebellum, basal nuclei, cranial nerves, or peripheral nerves. Other factors include primary motor problems of the tongue, larynx, and throat3.
Dysarthria is an umbrella term for motor speech disorders caused by a change in neuromuscular control that affects breathing, phonation, resonance, speech production, vocalization, and rhythm. Thus, dysarthria is a speech disorder that is often associated with neuromuscular impairments and injuries, where the muscles that are used for speech are either weak, slow, or paralyzed. Specific types of muscles that are involved in speech include those of the tongue, throat, face, lips, vocal cords, jaws, and muscles of the upper respiratory tract. The damage to these muscles may take various forms, including injuries to the motor neurons that innervate them or to their vasculature, depriving them of oxygen. Typical neurological damages that impair the speech conduction include upper motor neuron damage that can lead to the stiffness or spasticity of speech muscles, lower motor neuron injuries that can lead to the muscle weakness or paralysis as a result of interrupted nerve signals, cerebellar injuries which often result in lack of coordination among the muscles (ataxia) or the basal ganglionic damage that impairs coordinated movements and may result in involuntary movements (hyperkinetic) or rigidity of the muscles (hypokinetic movements)5.
Dysarthria is a common symptom of various neurological disorders and is frequently linked to progressive neurological diseases. It significantly impacts the patient and their families, as communication plays a crucial role in expressing one's personality and maintaining social connections. The disorder is characterized by decreased speech intelligibility. The spoken language content stays intact, allowing the patient to write and interpret both spoken and written language; on the other hand, anarthria is the most severe kind, resulting in full loss of motor speech output6. There are six primary classifications of dysarthria: flaccid dysarthria, which is linked to lower motor neuron dysfunction; spastic dysarthria, which arises from damage to upper motor neurons connected to the motor regions of the cerebral cortex; ataxic dysarthria, primarily resulting from cerebellar issues; and hyperkinetic dysarthria and hypokinetic dysarthria, both associated with disorders of the extrapyramidal system. The sixth type is typically referred to as mixed dysarthria and is often associated with damage in multiple areas, leading to speech characteristics that exhibit traits from at least two categories. The speech disturbances associated with these six principal types of dysarthria are unique and can aid in the diagnosis and treatment of dysarthria5,6.
The etiological factors contributing to challenges with articulation encompass infections (such as Creutzfeldt–Jakob disease and acquired immunodeficiency syndrome), vascular issues (both ischemic and haemorrhagic strokes), tumors (brain neoplasms), demyelinating illnesses (including multiple sclerosis and Guillain–Barré syndrome), degenerative disorders (like Parkinson's disease and Huntington's disease), injuries (traumatic brain injuries and cerebral palsy), toxic exposures (to heavy metals, alcohol, and drugs), and genetic conditions (for example, SANDO)7. Additionally, non-neurological factors, such as cleft lip or palate, may also cause articulation issues, but these do not fall under the category of dysarthria8.
The importance of dysarthria in clinical settings lies in its significant impact on individuals' quality of life. Because communication is essential for social, educational, and professional engagement, those who suffer from it often face feelings of isolation and diminished well-being6. Accurately diagnosing the various types of dysarthria, for example, flaccid, spastic, ataxic, hyperkinetic, hypokinetic, and mixed, is essential for identifying the underlying neurological issues and evaluating rehabilitation approaches. Speech-language pathologists and clinicians often depend on analyzing the characteristics and severity of dysarthria to customize therapy, establish achievable communication objectives, and assess the necessity for augmentative and alternative communication methods9. For those with speech difficulties, automatic speech recognition (ASR) systems have been found to improve accessibility and social interaction. Nevertheless, insufficient acoustic models have hindered the broader success of ASR in addressing disordered speech. Thus, this paper delves into phoneme problems in dysarthric speech, existing ASR systems and their limits, creative multimodal techniques, transformer-based contextual modeling, and TCNs for temporal and sequential refinement.
Even with substantial progress in ASR technology, the top ASR systems still have numerous shortcomings for individuals with speech impairments. These shortcomings may partially stem from the difficulties in obtaining a diverse and representative training dataset of disordered speech. One challenge in disordered speech ASR is likely linked to the wide range of abnormal speech characteristics that vary from person to person, and the inadequate representation of these variations in training datasets10. Despite this, research on ASR related to dysarthria has often overlooked this diversity.
ASR systems are divided into three categories: speaker-independent (SI), speaker-dependent (SD), and speaker-adaptive (SA). SI systems perform well with healthy speakers but struggle with degraded speech and perform better when training data includes dysarthric speakers. In contrast, SD systems focus on individual users, attaining excellent accuracy, whereas SA systems adapt to user speech over time and outperform both SI and SD models. However, both SA and SD systems require training data, which presents yet another barrier for those with neurodegenerative disorders11. Despite significant progress, existing ASR models are still unsuitable for disabled speakers, owing to a lack of different training datasets. To fill this gap, thorough data-gathering methodologies that capture various aspects of dysarthria must be developed, thereby improving ASR performance for difficult-to-recognize speakers.
To overcome these limitations, multimodal strategies that blend acoustic, articulatory, and visual signals can effectively represent speech production and refinement in dysarthric symptoms in a comprehensive manner. For example, data on articulatory motion, such as tongue imaging, is obtained using ultrasonography and electromagnetic articulography. This data is often combined with visual lip movements, and can compensate for impaired acoustic information, enhancing the ability to distinguish and differentiate phonemes12. This redundancy found in multimodal systems is consistent with clinical evaluation methods, strengthening the capabilities of therapists to observe orofacial movements alongside auditory speech. Conversely, deep learning frameworks, particularly transformers and TCNs, offer superior modeling of temporal dependencies and variations within speech sequences. These models function by allowing transformers to capture overarching contextual relationships, enabling phoneme identification even when the acoustic signals are diminished, obscured, or reduced13. TCNs contribute further by providing stable receptive fields and temporal smoothing, which enhances the accuracy of phoneme boundary detection. When integrated within multimodal fusion frameworks, both of these approaches greatly enhance the accuracy of phoneme labeling in dysarthric speech and facilitate more precise, feedback-oriented clinical rehabilitation. Therefore, these multimodal deep learning architectures are directly aligned with real-time clinical objectives, such as enhancing the measurement of speech intelligibility, monitoring the rehabilitation journey, and providing technology-assisted therapies customized to the specific motor speech impairment profiles of individuals14.
While these multimodal deep learning architectures hold great potential for resolving phoneme labeling imprecision, their effective transfer from experimental frameworks to dependable diagnostic tools requires a uniform operational structure. To address this, the following study describes a comprehensive computational workflow encompassing multimodal data collection, feature extraction, and model training. The goal is to ensure that these complex systems are reproducible and verifiable across a wide range of clinical situations. Establishing such a transparent methodological pipeline is critical for speech-language pathologists and clinical engineers to validate algorithms across a wide range of patient characteristics and impairment levels. Finally, this systematic methodology serves as a precursor to clinical implementation, enabling the integration of advanced speech models into regulatory evaluations, electronic health systems, and long-term treatment-monitoring platforms.
Access restricted. Please log in or start a trial to view this content.
Overview of multimodal data acquisition in dysarthric speech
Multimodal data collection is necessary to thoroughly examine speech motor control and to develop efficient diagnostic and rehabilitation techniques, given the complexity and diversity of dysarthric symptoms.
Acoustic recording setup
The acoustic channel remains central. Recordings are best conducted in an acoustically treated environment to mitigate reverberation and ambient noise. Typical microphones are high-quality condenser and are either cardioid head-mounted or table-top units, positioned consistently to control signal level and avoid variation across sessions. Sampling rates of 16 kHz or 44.1 kHz are very common, where 16 kHz suffices for intelligibility and prosody analysis, while 44.1 kHz gives greater fidelity, which is useful for fine-grained acoustic/phonatory research. Multichannel audio interfaces allow synchronous capture of multiple streams and must maintain consistent gain settings and headroom to avoid clipping or floor noise. For reproducibility, establishing calibration and documenting microphone gain, ambient noise floor, and drift are important. In dysarthric speech corpora, audio quality is particularly important because acoustic signal defects may be subtle and can be easily masked by poor recording conditions15.
Optional articulatory / lip-tracking / EMG / ultrasound modalities
To go beyond the acoustic waveform, additional modalities are often required to capture articulatory kinematics and physiological muscle activity. Thus, lip-tracking or facial motion capture allows quantification of lip/jaw opening, movement, speed, and symmetry. Electromagnetic articulography (EMA) tracks tongue, jaw, and lip sensors in three dimensions and provides temporal/positional resolution of articulators while surface electromyography (sEMG) records muscle activity to explore neuromuscular activation deficits underlying dysarthria. Ultrasound tongue imaging (UTI) provides real-time imaging of the tongue surface during articulation, useful in subjects who cannot tolerate EMA. Multimodal systems show that the identification of speech motor deficits is improved by concurrent acoustic, supplemented with articulatory/visual capture16.
Ethical considerations and patient consent
Working with dysarthric speakers frequently involves vulnerable populations. Researchers must secure institutional review board (IRB) or ethics committee approval, and ensure informed consent that explains recording modalities (audio, video, physiological sensors), storage, anonymization, future use, and withdrawal rights. Consent forms should explicitly address video capture (lip/facial tracking) and, optionally, sensitive modalities like EMG. Data privacy must be managed, particularly when video or facial images are stored. It is necessary to assess the participant's level of comfort, exhaustion, movement limitations, and potential risk. Sessions should include rest periods, and participants should be told they can stop at any time without facing consequences. The scarcity and diversity of participants in dysarthric speech corpus research further emphasize the importance of ethical rigor17.
Data pre-processing steps (segmentation, noise reduction, normalization)
The MAV-HuBERT system aligns acoustic and visual data temporally at the frame level, extracting 26-dimensional log-filter bank energies from the original waveform and synchronizing them with 25 Hz visual frames collected by Dlib-based face identification. The consecutive audio frames are typically combined to stabilize short-term fluctuations, and the fused visual regions are spatially normalized prior to multimodal feature fusion. These pre-processing activities provide continuous temporal consistency and reduce variability across audio and visual modalities, resulting in higher downstream identification accuracy15,18.
Feature engineering and computational workflow
Multimodal deep learning models require synchronized representations of acoustic, articulatory, and visual information to accurately label phonemes in dysarthric speech. This section provides an outline of the feature representation techniques used in multimodal speech analysis, followed by a step-by-step computational workflow for extracting and aligning these features prior to model training.
Feature extraction and representation
In multimodal speech analysis, raw audio and motion signals must be converted into meaningful representations that can be processed by deep learning models. One of the most widely used representations is the spectrogram, which shows how signal energy varies over time and across frequencies. Spectrogram-based representations are useful for capturing characteristics such as slurring, breathiness, and irregular timing, which are commonly observed in dysarthric speech19.
Another commonly used feature is the Mel-Frequency Cepstral Coefficients (MFCCs), which represent the spectral properties of speech in a way that approximates human auditory perception. MFCC features are widely used in speech recognition because they provide compact, noise-robust representations that remain stable even when speech is distorted20. In addition to acoustic features, multimodal approaches incorporate articulatory information, such as movement trajectories of the tongue, lips, and jaw. These signals can be obtained using electromagnetic articulography (EMA), ultrasound tongue imaging (UTI), or optical motion tracking. Articulatory features provide direct information about speech motor control and help compensate for degraded acoustic signals21.
Visual information is also useful for analyzing dysarthric speech. Facial landmarks extracted from video recordings, including lip contour, jaw displacement, and mouth opening, provide additional cues about articulation. These visual features improve phoneme discrimination, particularly when the acoustic signal is unclear. For supervised learning, all feature streams must be aligned with phoneme-level transcriptions, ensuring that each time frame corresponds to the correct speech unit. Accurate alignment is essential for training phoneme labeling models, especially in dysarthric speech, where phoneme boundaries are often blurred22.
Computational workflow
This section demonstrates how every step is required to reproduce the multimodal phoneme-labeling workflow, from feature extraction to model evaluation (Figure 1).
Acoustic feature extraction through spectrograms and MFCCs
First, the short-time Fourier transform (STFT) is used to transform the raw speech signal into a time-frequency representation. To create a spectrogram, the signal is split up into brief overlapping frames, and each frame is subjected to the Fourier transform.
Mel-Frequency Cepstral Coefficients (MFCCs) are used to extract perceptually weighted acoustic features from the spectrogram. For speech recognition and dysarthria analysis, MFCCs offer compact and noise-robust representations23,24. Due to their robustness and effectiveness, MFCCs are commonly used in applications related to speech and speaker recognition. Thus, due to their usefulness, MFCCs are then extracted to calculate and approximate the human auditory perception levels and provide compressed, noise-robust acoustic features, thereafter25.
Acquisition of articulatory trajectories
Articulatory motion data are obtained to capture the physical movement of speech organs. Electromagnetic articulography (EMA) uses sensors attached to the tongue, lips, and jaw to track articulatory motion in three dimensions. Alternatively, ultrasound tongue imaging (UTI) can be used to estimate tongue movement non-invasively. The recorded trajectories are filtered, normalized, and down-sampled to match the frame rate of acoustic features to ensure temporal consistency26. Converting speech into ultrasound tongue imaging (UTI) sequences is a technique for estimating ultrasound tongue trajectories from speech data by mapping auditory characteristics to physiological tongue motions. This methodology provides a non-invasive alternative to direct articulatory data collecting techniques, which are necessary for monitoring and treating speech and vocal tract abnormalities, while avoiding the need for specific equipment and clinical experience inherent in direct measurement approaches27,28. Based on the availability of articulatory features, like movement trajectories of tongue, lips, and jaws, which are recorded using EMA or ultrasonography (UTI), the signals are filtered and down-sampled to match the rate of acoustic frames.
Detection of visual landmarks
For the inputs of speech and videos, facial key inputs (Lip corners, motion of jawlines) need to be extracted through tools such as Mediapipe or OpenFace, and then these are normalized to remove head-pose effects. MediaPipe employs a deep learning framework to estimate facial and body poses, providing 33 landmarks for general pose recognition, with 11 specifically for the face. Its effectiveness can vary, particularly in cases of occlusions. The MediaPipe FaceMesh model improves reliability by using 468 3D landmarks of the face along with a geometric method for head pose estimation29. OpenFace 2.0 functions as a toolbox for analyzing facial behaviors, enabling the detection of facial landmarks, head pose estimation, eye gaze tracking, and recognition of facial action units. It supports integration with various programming languages and can process live video feeds or still images. Outputs, such as facial landmarks and gaze vectors, can be exported in CSV format. OpenFace 2.0 uses the Convolutional Experts Constrained Local Model (CE-CLM), which includes a point distribution model to account for variations in landmark shapes and patch experts for differences in local appearances29,30.
Phoneme transcription alignments
The next step is to temporarily align all of the featured streams (acoustic, articulatory, and visual) to phoneme-level annotations using forced alignment tools such as the Montreal Forced Aligner (MFA), ensuring synchronized multimodal inputs for computational model training. Forced alignment (FA) is the technique of aligning transcripts with audio signals in order to determine the temporal boundaries of speech units. This allows for more accurate audio processing and advances linguistic study. It is increasingly employed in phonetic studies and has proven to be substantially faster than manual alignment. Although generally associated with automatic speech recognition (ASR) systems, FA is a broader task within automatic speech processing that includes spoken language analysis and transcription22. The Montreal forced aligner (MFA) is a premier FA toolkit that uses HMM-GMM algorithms to achieve effective alignment. Despite advances in ASR technology, traditional approaches such as HMM-GMM are still popular for FA tasks. The MFA uses MFCC characteristics and a four-stage training method to create acoustic models with accurate phonetic alignment31.
Components of the architecture of the models
Multimodal deep learning models for dysarthric speech recognition consist of several key components that work together to capture contextual, temporal, and multimodal information. The main components include a contextual encoder, a temporal refinement module, and a multimodal fusion mechanism. Each component plays a specific role in improving phoneme labeling accuracy in disordered speech.
Transformer-based contextual encoder
The transformer encoder is used to model long-range dependencies in speech sequences. Dysarthric speech often contains irregular timing and unclear phoneme boundaries, making it difficult for conventional models to capture contextual information. Transformers use multi-head self-attention to analyze relationships between different time frames in the input sequence. This allows the model to consider both past and future speech frames when predicting phonemes. As a result, the encoder can better handle the articulation and pronunciation variations common in dysarthria. In the multimodal architecture, the transformer receives synchronized acoustic, articulatory, and visual features and produces context-aware representations that are passed to the next stage of the model32,33.
TCN-based temporal sequence refinement
After contextual encoding, the feature sequence is processed by a Temporal Convolutional Network (TCN) to improve temporal precision. Dysarthric speech often contains unstable timing, pauses, and elongated phonemes, which require additional refinement beyond contextual modeling. TCNs use dilated causal convolutions to capture long temporal dependencies while maintaining computational efficiency. This structure allows the model to smooth noisy predictions and maintain consistent phoneme boundaries across time. In the architecture, the TCN receives the output of the transformer encoder and refines the sequence to produce stable and temporally consistent feature representations33,34,35.
Multimodal fusion strategy
To increase diagnostic precision in dysarthria evaluation, multimodal fusion incorporates information from numerous sources such as voice, text, video, and physiological sensors. The major techniques consist of three distinct steps: early fusion, late fusion, and intermediate fusion36. Early fusion combines data from many modalities at a fundamental level, allowing models to understand complex relationships from the start. Its use is limited since it requires synchronizing several sampling rates and data types. Late fusion processes each modality using independent models before combining the results for final evaluations. This technique is flexible and effective when data from one modality is missing. Furthermore, intermediate fusion integrates modalities at a mid-level, often using cross-attention techniques to detect dependencies. This method has been particularly useful in clinical contexts, improving results by combining language references with acoustic data. In summary, intermediate fusion with cross-attention produces superior results in automatic dysarthria assessment than methods based on a single modality36,37.
Best practices for training and evaluating dysarthria models:
Developing sinewy models for dysarthria assessment and recognition requires careful attention for partitioning the dataset, evaluation of the metrics, and interpretability. Recent research has highlighted standardized approaches and innovative solutions to address these unique challenges of dysarthric speech, which include data scarcity, variability, and clinical relevance38,39.
Dataset partitioning strategies
To ensure that training, validation, and test datasets do not exchange speaker information, preventing data leaking and overfitting, stratified group K-Fold cross-validation is now commonly employed38,39. This approach enables models to maintain balanced distributions and reliable performance evaluations, even when working with limited or diverse datasets. Since partitioning by speaker is essential for preventing bias, common splits consist of train/test ratios of 75:25 or 85:15, along with further division for validation40. To handle limited dysarthric data, popular data augmentation approaches such as synthetic data generation, tempo perturbation, and transfer learning from healthy speech are applied41,42.
Evaluation metrics
Model interpretability considerations
Thus, a thorough dysarthria model pipeline includes stratified, speaker-independent data splits, multifaceted evaluation (PER, intelligibility, clinical input), and interpretability through attention mechanisms. These approaches ensure that model systems for assessing and recognizing dysarthria are robust, clinically relevant, and transparent.
Representative outcomes
Several studies have reported improvements in phoneme boundary precision when multimodal features are used. Dysarthric speech often contains blurred or prolonged articulatory transitions, making it difficult to determine the exact boundary between phonemes. Published models that combine acoustic, articulatory, and visual features have shown better agreement with reference annotations than unimodal systems. These improvements are often visualized using alignment curves, where multimodal models demonstrate reduced timing errors, especially during consonant–vowel transitions46. Literature reports also describe improvements in phoneme classification accuracy. Multimodal architectures have been shown to reduce substitution and deletion errors, particularly for fricatives and vowels that are frequently distorted in dysarthria. Confusion matrices reported in previous studies indicate that combining visual and articulatory information helps the model distinguish between similar phonemes more reliably. The increase in phoneme boundary precision is one prominent example. The articulatory transitions and changes in dysarthric speech are often elongated or blurred, which makes it difficult to interpret where one phoneme ends and another begins. To more precisely identify phoneme onset and offset, the multimodal system makes use of shared data from visual lip movements, articulatory trajectories, and audio. Compared to those produced by unimodal acoustic models, the predicted phoneme boundaries visualized match true annotations more closely. Alignment curves are used to visualize these improvements, where the multimodal model shows fewer errors in timing, particularly for transitions between vowels and consonants (VC and CV). In the clinical setting, this can result in improved segmentation maps, which further help speech therapists and clinicians to identify particular issues with motor control, such as insufficient lip rounding or delayed jaw closure47.
Additionally, the model can produce differential classification outputs that could be used to identify the most likely spoken phoneme sequence. The multimodal system exhibits a decline in phoneme substitution and deletion errors when compared to traditional and baseline models. For example, the incorporation of visual shape of lips and articulator position cues improves the classification accuracy of fricatives, such as /s/ and /ʃ/, which are frequently found distorted and disoriented in dysarthria. Confusion matrices can also be used to demonstrate such improvements, where improved phoneme discrimination and bifurcation are indicated by decreased cross-category confusion48.
Speech intelligibility measurement before and after model-supported correction can be compared using a clinical relevance chart. Metrics like stability and timing of syllables, estimation of vowel space area, precision rating of consonants, and overall intelligibility score can be included in this chart. In general, the output that has already been corrected exhibits improved prosody, rhythm, and articulation boundaries. For instance, after correction, the representation of vowel space usually enlarges, indicating enhanced vowel acoustic distinctiveness, a crucial therapeutic goal in many dysarthria treatments39.
Importantly, these model outputs are intended to support clinical decision-making by enhancing rather than replacing clinician judgment. The system can emphasize specific phonemes or articulatory transitions that significantly deviate from anticipated or theoretically hypothesized patterns. With this information, therapists can tailor their therapy goals to include: (a) improve tongue elevation consistency for alveolar consonants (e.g., /t/, /d/), (b) concentrate on lip protrusion for rounded vowels (e.g., /u/), (c) improve onset timing for voiced stop consonants.
Published work emphasizes that these outputs should complement clinical interpretation, not replace physician judgment. Model visualizations, such as attention maps and phoneme alignment plots, can assist therapists in identifying specific articulatory issues, such as insufficient tongue elevation, decreased lip rounding, or delayed voice onset. Overall, data from several studies show that multimodal deep learning architectures increase technical performance indicators while also providing interpretable outputs that can help with therapy planning and rehabilitation progress tracking.
Clinical integration and rehabilitation workflow
The implementation of digital technologies and advanced speech models in dysarthria rehabilitation is transforming patient results, the delivery of therapy, and clinical practices. This section addresses the framework of feedback-oriented articulatory training, the evolving roles of speech therapists and patient monitoring, the integration of model outputs into therapeutic tools, and significant usability concerns.
How do these model outputs feed into speech therapy tools?
Modern AI and deep learning models, which include ASR and severity classifiers, could generate detailed, objective data on speech intelligibility, articulation errors, and severity levels48. These outputs, these days are increasingly embedded in digital therapy platforms and mobile applications, which provide real-time, personalized feedback to patients and therapists49. For instance, tablet-based applications and cloud-based tele-monitoring systems use model-generated metrics to visualize progress, highlight specific articulatory deficits, and adapt exercise difficulty. These systems allow objective tracking of therapy progress, enable personalized exercise selection, and support remote monitoring through digital platforms50,51,52.
Feedback-based articulatory training modules
Feedback-based training modules are central to digital dysarthria rehabilitation. Previous studies have reported that digital rehabilitation modules often include biofeedback, automated error detection, and adaptive exercise design. Biofeedback in speech therapy utilizes visual and auditory cues, such as waveform displays and visualizations of articulator movements, to provide real-time guidance to patients. Additionally, automated error detection is facilitated through AI models that identify phonological or articulatory errors, offering immediate corrective feedback to enhance learning. The training process is hierarchical and adaptive, meaning that it progresses from simpler to more complex tasks, which specifically target speech sounds, syllables, or words. These exercises are dynamically adjusted based on the patient's performance, ensuring personalized learning experiences. Furthermore, the incorporation of gamification elements and virtual reality (VR) into some platforms serves to increase motivation and compliance among patients, making the therapeutic process more engaging. These modules thus support intensive, repetitive practice, which is the key for motor learning and speech improvement while allowing for independent or therapist-guided use51,53,54.
Role of speech therapists and patient monitoring
Speech-language therapists (SLTs) remain central to the rehabilitation process, even as digital tools have become more prevalent in the last decade. Assessment and goal setting involve interpreting model outputs to select appropriate therapy targets and customizing differential treatment plans tailored to individual patient needs. Supervision and feedback are vital for monitoring patient progress; experts provide feedback on the rectification of speech and make necessary adjustments to therapy as required. Additionally, patient education and support are emphasized, teaching patients to effectively utilize digital tools in their rehabilitation process. Multidisciplinary collaboration is crucial, as professionals from various disciplines work together to address the broader needs of rehabilitation, ensuring a comprehensive approach to patient care. Patient monitoring is enhanced by digital platforms, which allow therapists and clinicians to track patient compliance, recovery, performance, and outcomes remotely, facilitating timely interventions and ongoing support51,52.
Usability considerations: patient comfort and repeatability
For digital rehabilitation technologies to be successfully implemented, usability is crucial, with a focus on user-friendly interfaces that meet a variety of patient needs. Flexible therapy methods are made possible by mobile platforms, which improve comfort and accessibility. Important features like adaptable modules and automated feedback encourage consistent, autonomous participation, which is essential for motor learning. Pilot tests show high acceptance and satisfaction, but there are still technical issues. Continuous feedback from patients and therapists helps enhance these tools to match real-world requirements. With an emphasis on developing meaningful therapeutic experiences, the use of AI and feedback-driven training in speech therapy is improving efficacy, accessibility, and personalization in dysarthria rehabilitation48,51,52,53,54.
Access restricted. Please log in or start a trial to view this content.
This review describes the increasing application of multimodal deep learning architectures to phoneme labeling and speech restoration in dysarthria․ These architectures combine acoustic‚ articulatory and visual analysis to overcome the limitations of audio-based speech recognition in pathological speech characterized by low intelligibility and non-standard articulatory configurations․ Electromagnetic articulography‚ ultrasound tongue imaging‚ and facial landmarking are more holistic models of speech production that may a...
Access restricted. Please log in or start a trial to view this content.
The authors declare no conflict of interest.
The authors acknowledge the Nanjing University of Chinese Medicine Affiliated Hospital of Nanjing, University of Chinese Medicine, for providing the necessary fellowships and funding (Jiangsu Province Postgraduate Scientific Research and Innovation Program KYCX25_2282).
Access restricted. Please log in or start a trial to view this content.
Access restricted. Please log in or start a trial to view this content.
Request permission to reuse the text or figures of this JoVE article
Request Permission