A Structured Rehabilitation Protocol for Improved Multifunctional Prosthetic Control: A Case Study

Summary

As prosthetic development moves towards the goal of natural control, harnessing amputees’ inherent ability to learn new motor skills may enable proficiency. This manuscript describes a structured rehabilitation protocol, which includes imitation, repetition, and reinforcement learning strategies, for improved multifunctional prosthetic control.

Abstract

Advances in robotic systems have resulted in prostheses for the upper limb that can produce multifunctional movements. However, these sophisticated systems require upper limb amputees to learn complex control schemes. Humans have the ability to learn new movements through imitation and other learning strategies. This protocol describes a structured rehabilitation method, which includes imitation, repetition, and reinforcement learning, and aims to assess if this method can improve multifunctional prosthetic control. A left below elbow amputee, with 4 years of experience in prosthetic use, took part in this case study. The prosthesis used was a Michelangelo hand with wrist rotation, and the added features of wrist flexion and extension, which allowed more combinations of hand movements. The participant’s Southampton Hand Assessment Procedure score improved from 58 to 71 following structured training. This suggests that a structured training protocol of imitation, repetition and reinforcement may have a role in learning to control a new prosthetic hand. A larger clinical study is however required to support these findings.

Introduction

Replacing hand function in amputees is a difficult endeavor. Coordinating highly skilled hand movements is not an innate ability, and takes humans years of learning to develop.^1–5 After the traumatic loss of a hand, replicating this capability by prosthetic means is not a trivial task and may require a period of sustained learning.

Prosthetic design and interfacing methods for their control are subject to rapid technological innovations, with the goal of multifunctional control in a natural manner.⁶ The complexity of these control systems increases substantially to provide more functions for amputees. To ensure accurate control of these systems, and to reduce abandonment of new technologies, adequate training needs to be established. This is likely to be more successful if it is based on the amputees’ inherent learning strategies.

Vision can play an important role during learning of hand movements. Behavioral studies have shown that by observing the actions of others⁷ or using visual cues⁸, able-bodied individuals learn and coordinate new movements. Through a process of observation, understanding and execution of an observed action, individuals are able to imitate the actions of others. Specific cortical networks, which may include a mirror-neuron system (MNS), are believed to underlie this capability, and may have a role in controlling prosthetic limbs.^9–11

The role of imitation might not just be limited to executing actions that have already been seen, but together with the MNS, allow the execution of movements that have not yet been observed but extrapolated from the observer’s motor repetoire.¹² Indeed, imitation may not necessarily be an innate ability, but an accruement of motor skills over time that lead to experienced and sophisticated actions.¹³ The strength of observing actions, over just simply imagining them, has been shown to improve learning new tasks.¹⁴ Thus, imitation may be a pragmatic approach to training amputees, as evidence suggests it a goal directed process¹⁵, with the target in the rehabilitation setting of enabling useful prosthetic hand function.

Rehabilitation studies have separately shown that visual cues, such as virtual simulations of a prosthetic hand, encourage amputees during rehabilitation training.¹⁶ In addition, the use of repetition when conducted in a blocked paradigm has been shown to enable rapid learning of upper-limb prosthetic control.¹⁷ While virtual simulations have been proven to be equally effective as real control of prosthetic hands in enabling abled-body users to control myoelectric devices,¹⁸ their effect on amputees using standardized outcome measures is not clear. Finally, where protocols for upper limb amputation training exist, the role of imitation in learning of prosthetic control is not explicitly discussed.^19,20

This study aims at understanding if the use of imitation, in combination with repetition and reinforcement, has a positive impact on the learning of multifunctional prosthetic control as part of a structured training program.

Presented herein is a case report of a transradial amputee who was trained to use a multifunctional prosthetic hand. The participant had previously become accustomed to operating traditional myoelectric prostheses. Using visual cues, both in the form of imitation of a healthy demonstrator and as simple computer visual feedback, the amputee quickly improved handling of his new device.

Protocol

This study was carried out in accordance with the Declaration of Helsinki, as approved by the local research ethics committee. The study was explained in full detail to the participant prior to commencement, allowing the participant the time to weigh up the decision to voluntarily take part in the study and confirm his participation by informed, written consent.

Note: One man, aged 27 years, took part in the study. The participant had normal vision, was a left below-elbow amputee, and was an experienced user (4 years total prosthesis use). Prior to commencing this study the prosthesis he used on a daily basis was a 4 channel myoelectric prosthetic hand with wrist rotation for 12-15 hr per day for 15 months. The participant’s right hand had previously been surgically reconstructed, but had no other physical or neurological impairment.

1. Study Design

1. Split the study over two sessions: naïve use, and use following structured training.
   Note: This is to allow intra-subject comparison before and after training respectively.
2. Ensure that these two sessions are at least three months apart, so as to be treated as independent from each other.
3. At the beginning of both sessions, fit a customized socket and prosthesis to the participant. Ensure that the prosthetic hardware and control algorithms match those detailed in the Materials section of this protocol. Ensure that the participant is not able to use the customized prosthesis in the intervening time between sessions.
4. Train the patient according the steps outlined in the Naïve Session and Structured Training Session sections of this protocol. At the beginning of each of these sessions, calibrate the prosthetic hardware. Use the collected calibration data for real-time prosthetic control.
5. Once the Naïve Session and Structured Training Session are complete, assess the participant’s performance using the Southampton Hand Assessment Procedure (SHAP) outcome measure.²³ Compare the SHAP scores to a baseline measure using the participant’s standard prosthesis (obtained before either training sessions).

2. Materials

1. Fit the participant with a custom-built socket. Attach a commercially available prosthesis according to manufacturer’s instructions. Equip the prosthetic hand with prototype components that allow actuated wrist flexion, extension and rotation. This enables the participant to control the hand with 3.5 degrees of freedom (DoFs) (Table 1).
   Note: In this experiment a Michaelangelo Hand (See Materials List) was used. Other terminal devices capable of wrist rotation, flexion and extension, together with standard grip functions would also be appropriate.
2. Record EMG signals using eight equidistantly placed raw signal electrodes around the stump, and an on-board decoding system at the sampling rate of 1,000 Hz and digitized with 10-bit depth. Perform the initial filtering and amplification within the electrodes themselves according to the vendor’s specifications. Use a personal computer (PC) to conduct the main processing, which communicates with acquisition hardware and controls the prosthesis via a wireless connection.
   Note: In this study the surface EMG electrodes and on-board decoding system (AxonBus) used were from Otto Bock. Other manufacturers of similar devices would also be appropriate. The wireless connection was via Bluetooth, and likewise other modalities could be applied.

3. Control Algorithm

1. Use a control algorithm that provides simultaneous and proportional prosthetic control across multiple DoFs.²¹ The algorithm used in this study was a two-stage decision making paradigm, so that context-dependent movement estimation was possible.
2. Upon system training, which contains all the controllable single DoF movements, record an incoming electromyogram (EMG).
3. In the first stage, assess intrinsic dimensional information of the intended motion based on the Mahalanobis distance of the newly calculated EMG feature vector from the training data. Make a decision as to whether the user’s intention was to perform a fine 1-DoF or a coarser simultaneous 2-DoF motion.
   Note: Mahalanobis distance of a feature vector x to class i with the class mean vector µ_i and covariance matrix Σ_i is calculated as:
   
   As described in Amsuess et al, the newly calculated feature vector is mapped to the high dimensional space and the Mahalanobis distance of the transformed point to any of the trained class points is taken as the measure for novelty.²¹ An empirically determined threshold to that distance gives the decision for novelty (2-DOF) or not (1-DOF).
4. In the second stage, based on the previous decision, use one of two parallel estimators — one dealing with the sequential movements (SEQ-E) and the other handling simultaneous motions (SIM-E) — to provide the control signals for the prosthesis.
   Note: SEQ-E is in essence a proportional estimator (i.e., the strength of muscular contractions) based on common spatial patterns (CSP)²¹, while SIM-E is a linear regressor, which simultaneously steers 2 DoFs of the wrist.

4. Software Framework

Note: The software framework used in this study allowed handling of the communication between the prosthetic hardware and the embedded control algorithm. It also offered visually supportive training tools needed for maximizing participant training.

1. Display the root mean square (RMS) of the EMG signals collected from the 8 equidistantly placed electrodes in a form of a polar plot of EMG amplitude as a function of electrode location. This visual feedback enables easy monitoring of the spatial distribution of the EMG in the transverse plane of the forearm. Using such a setup, each of the user’s motions can thereby elicit a distinct pattern²² in the polar plot, which can then be saved and used to train for the repeatability of the specific gesture.
   Note: The framework enables the collection of EMG data in the standard pattern recognition manner.²³ For each of the EMG channels the RMS over 40 msec is calculated as
   
   resulting in observations for every ensemble window.
2. For initial calibration collect the maximal long-term voluntary contraction (MLVC) values for each intended motion. Prompt the participant by using the demonstrator’s hand to perform the desired motion while giving vocal and visual instructions for 5 sec.
3. After calibration, present the participant with a set of trapezoidal cues. These force profiles contain plateaus set at 30%, 60% and 90% of the calibrated maximum.
4. Within each trial, instruct the participant to steer the red pointer along the cue by modulating the force level of the prompted motion (Figure 1). The vertical position of the pointer corresponds to the summed RMS values across all eight channels. Set the duration of the trial to 5 sec with plateau interval corresponding to the middle 3 sec.

5. Naïve Session

Note: During the naïve training session, the participant had no prior experience of the prosthetic control scheme used in this study.

1. Do not give the participant any formal clinical training, but only instruct that 8 actions of the residual limb, of which one is a resting state, will allow control of a visual target on a computer screen. These tasks are similar to those used in classic pattern recognition approaches for prosthetic control²³, and for those methods the participant in this study had approximately 60 hr of previous experience.
2. Display the required movements on screen in terms of text and a static picture while following a visual cue (Figure 1).
3. Show the participant his EMG activation patterns, which correspond to eight specific and unique polar plots (Figure 2).
4. Use audible instructions to encourage the participant to follow the visual cue. These audible instructions must be identical if used in the structured training session.
5. Repeat the tasks three times with different arm positions (relaxed, reaching in front, reaching across) to enhance system training. Keep in mind that there are 8 different actions and three force levels, once all arm positions are covered, system training input sums up to the total of 72 individual samples.
6. Once complete, allow the participant the opportunity to practice real-time control prior to completing the SHAP outcome assessment.
7. Ensure the participant does not have access to the customized prosthesis and control algorithms beyond the end of the naïve session.

6. Structured Training Session

1. Three months after the naïve session, perform a structured training session.
2. Structure the session in the following ordered steps (Figure 3):
   1. For imitation, instruct the participant to directly imitate the desired eight actions (Table 1) performed by the demonstrator in real-time. Execute each action for 3 sec.
   2. For repetition, ask the participant to repeat the action that has been imitated 10 times, so each action is performed for 30 sec.
   3. For reinforcement & computer system training, ask the participant to now engage with the computer’s visual feedback, that is exactly the same setup as the Naïve session. Ensure that there is no difference between these two sections.
   4. For prosthetic control, ask the participant to practice real-time control of the customized prosthesis before completing the outcome assessment.
3. During imitation, seat the participant at a 45° angle from the demonstrator and provide with a full and unobstructed view of the demonstrator’s hand matching the affected side of the participant during all actions (Figure 4). No visual cues from a computer screen should be available to the participant at this time.
4. For repetition, during the participant’s actions, have the demonstrator observe the corresponding EMG activity as represented by the polar plots of each movement (Figure 4). Once the demonstrator has determined that the participant can produce unique and repeatable EMG activation patterns for each movement, ask the participant to repeat the actions for 30 sec without any visual cues.
   Note: There are total of 8 unique actions — seven of them (wrist pronation/supination, wrist flexion/extension, hand open, key grip and fine pinch) requiring muscle activation, and the eighth being no action which represents a steady relaxed state.
5. After reinforcement & computer system training, present the participant with visual feedback of his eight actions, exactly as was seen in the naïve session, which correspond to the eight unique and specific polar plots on the computer screen (Figure 3). To tune performance, ask the participant to perform the actions while viewing the real-time polar plots with recorded motion overlays to reinforce learning, typically between 2-4 attempts for each movement. Once confident the participant can then complete the exact same tasks that were performed in the naïve session.

7. Prosthetic Control

1. Use the training data sets from each session to calibrate and adjust the prosthesis for real time control.
2. Initially, only allow the participant to control the prostheses by sequential proportional control, i.e., one movement at a time, with the speed of the device proportional to the levels of muscular contractions.
3. Once each of the eight actions are performed in a repeatable and reliable manner, switch the control scheme to proportional and simultaneous control, allowing more than one movement of the wrist at a time.
4. Have the participant practice simple tasks, such as picking up a bottle and laying it on its side (2 attempts is sufficient). Allow a period of rest before the outcome assessment is performed. In the case of this study, 2 hr of rest for the naïve session, and 24 hr of rest for the structured session.

8. Outcome Measurement

1. Evaluate global upper extremity function both in the naïve and structured training sessions using the SHAP, which monitors hand and upper extremity function closely related to activities of daily living (ADLs). The tasks performed in the SHAP include manipulating light and heavy objects, as well as tasks of ADL such as cutting an object with a knife or undoing buttons. The SHAP has been validated for assessment of pathological and prosthetic hand function.²⁴
   Note: This measurement was chosen as the participant in this study had been routinely followed up with this outcome measure by his clinical team.

Representative Results

The baseline SHAP performance of the participant with his daily prosthesis was 81 when measured by the clinical staff 8 months prior to testing. A SHAP score of 100 represents able-bodied hand function.²⁴ The participant scored an overall SHAP score of 58 during the naïve session with the more advanced prosthesis control system. However, 3 months later and with no further interaction with the new system, aside from the structured training, the participant achieved a SHAP score of 71 with the same advanced system (Table 2).

When the overall SHAP score was broken down into functional profile assessment, it was observed that the participant had performed well in all functional groups (spherical, power, tip, lateral and extension grasp), except for tripod grasp. However, the largest observed improvement was during extension, a function that the new control scheme and prosthesis provided while his traditional prosthesis did not (Figure 5). This may have also contributed to the improvement in spherical grasp, which was better after the structured training session than the baseline or the naïve session. In addition complex ADL movements, which involved combined movements of the wrist and hand, such as jug and carton pouring were executed best after the structured training session using the advanced prosthetic system.

Figure 1. Example of visual cues used for participant reinforcement and system training. The blue target profile represents the desired level of EMG contraction produced during a certain movement. The red tracking line represents the participant’s efforts. Please click here to view a larger version of this figure.

Figure 2. The profiles for the active movements, referred to as polar-plots, of the individual movements produced by the participant during the imitation task. These were reinforced during system training and eventually used to control the prosthetic hand. Please note, that rest or no-movement is considered a unique action, and as such does not produce an overlay. Please click here to view a larger version of this figure.

Figure 3. This schematic represents the structured training session. The participant first observed and imitated the actions of demonstrator. Before viewing his performance as graphs on a computer screen, he repeated the learned movements with no visual feedback. The learned movements were reinforced by matching muscular contractions to recorded EMG patterns, and then used to train the system’s control algorithms, which enabled multifunctional prosthetic control.

Figure 4. Experimental setup during the structured training session. The participant had a full and unobstructed view of the demonstrator’s left hand during imitation. During the repetition phase, the demonstrator would give audible instruction to ensure the participant’s movements matched the contractions produced during the imitation phase. Finally, during system training, the movements were reinforced using visual cues that were displayed on the computer screen to both the participant and demonstrator.

Figure 5. The breakdown of overall SHAP scores between, the baseline (BL) the naïve session (NS) and the structured training session (STS). Please click here to view a larger version of this figure.

Prosthetic Function	Phantom Limb Motion
Pronation	Wrist rotation inwards with fully relaxed fingers
Supination	Wrist rotation outwards with fully relaxed fingers
Flexion	Ulnar deviation
Extension	Wrist extension
Palmar grip	Thumb adduction slightly crossing posteriorly towards the back of the hand
Fine pinch	Opposition of thumb to the first three fingers, slight extension of the little finger
Hand open	Opening of the hand with focus on the extension of the middle three digits
No movement	Full relaxation of the hand and wrist

Table 1. Desired prosthetic functions mapped to the phantom limb motions, which the participant was capable of visualizing and executing with his remaining anatomy.

Abstract Objects
	BL	NS	STS		BL	NS	STS
Light Sphere	2.46	2.66	2.5	Heavy Sphere	3.25	4.78	2.1
Light Tripod	2.35	3.56	2.78	Heavy Tripod	2.44	3.53	2.5
Light Power	2.41	3.25	2.28	Heavy Power	2.41	3.22	2.72
Light Lateral*	4.72	2.81	4.97	Heavy Lateral	5.1	5.31	5.22
Light Tip	2.25	2.88	2.53	Heavy Tip	3.1	4.47	2.22
Light Extension	1.96	3.88	2.37	Heavy Extension	2.9	4.88	2.59
Activities of Daily Living
	BL	NS	STS		BL	NS	STS
Coins	17.81	22.25	21.53	Full Jar	3.13	10.37	3.75
Button Board	8.25	35.2	27.06	Empty Tin	2.53	4.15	2.82
Cutting	18.15	27.47	25.59	Tray Lift	3.97	7.25	5.5
Page Turning	8.18	11.97	5.19	Key	4.82	9.25	6.03
Jar Lid	2.93	3.3	2.38	Zip	4.83	10.59	7.31
Jug Pouring	10.16	12.37	8.93	Screwdriver	10.1	25.31	15.31
Carton Pouring	11	11.35	9.72	Door Handle	2.24	3.53	2.75
				SHAP score	81	58	71

Table 2. SHAP results for the participant during the naïve session (NS), followed by the structured training session (STS) 3 months later, compared to his baseline (BL). *The participant only underperformed the light lateral task in the structured training session in comparison to the naïve session. The overall SHAP score is out of 100.

Discussion

Our findings suggest for the participant in this study that structured training helped improve control of a multifunctional prosthetic hand during one single session. The structured program used here was a combination of imitation, repetition and reinforcement of hand movements that the participant was not able to complete with his traditional prosthetic hand.

Although the participant scored higher with his traditional prosthesis in the SHAP test, it is worth noting that he typically wore that device between 12-15 hr per day over a period of 15 months. As documented by the baseline SHAP score, it is clear that he had learned and become accustomed to his traditional prosthesis after a very long learning period. The difficulty in switching to the multifunctional hand after being so accustomed to his traditional prosthesis was emphasized by the sharp drop in performance observed in the naïve session. This was expected, as evidence suggests that as an individual learns new motor skills, they develop an internal model of the actions being performed.²⁵ When there is some form of perturbation in this internal model, such as changing to a new prosthesis requiring new control inputs, the after effects of learning take some time to dissipate while a new internal model is created.²⁶ Nevertheless, a single session of structured training allowed the participant to outperform his usual device in some of the tasks requested by the SHAP test, and to reach on overall score close to that obtained with the traditional device. The use of structured training as outlined in Section 6 of the protocol may be the critical step that might have enabled the participant to achieve proficient control.

Learning a new task for amputees is complicated by the absence of nerve receptors around the joints and in the muscles which are sensitive to positional and movement changes.²⁷ These proprioceptors enable able-bodied humans to know where their hands are in relation to their body without the use of sight.²⁸ When a limb is lost, these proprioceptors are lost, leading vision to play a stronger role in control than in normal conditions. Amputees must not only relearn how to control hand movements, but also have to do so using a device that provides no feedback other than that obtained visually. This makes the learning process more difficult.

As such, any training strategies that use prostheses that provide no tactile or proprioceptive feedback must place an emphasis on visual feedback. In our case, we attempted to do so using imitation of the desired movements. The complexity of imitation is exemplified by the distributed nature of the neural process.^29,30 Separate regions in the frontal, temporal and parietal lobes are believed to be responsible for perceiving the motion of others^31,32 and then integrating this information into an appropriate motor response.^9,33,34 It is likely that during the participant’s development into adulthood, and prior to the amputation, the neural circuitry required to perform learnt hand movements had become clearly defined, so much so that natural hand movements were fast and instinctive. The distortion of anatomy following the amputation may have required new neural circuits to be formed to enable control of his traditional prosthetic limb. The improvement in SHAP score following the structured training session, suggests that these neural circuits were malleable enough to adapt to the new prosthetic control strategy, despite the lack of experience.

It is worth noting that the participant commented that the act of imitation allowed him to internally visualize hand movements and to generate the appropriate muscular contractions. He found this more intuitive than solely matching his contractions to visual representations on a computer screen. It is also known that amputees prefer to learn from other prosthetic users.¹¹ The device and control algorithms used in this study were both novel. As such there were no previous experienced amputees who could act as demonstrators. Future improvements to this protocol would thus benefit from having an experienced amputee demonstrating the actions to be imitated.

While this study showed the benefit of structured training, the design was not sufficient to determine whether imitation, repetition, reinforcement or the combination of all three learning strategies contributed to the final outcome measure. Instead, this case study lays the foundation for further work to examine the neural circuitry involved in advanced prosthetic control.

Materials

Name	Company	Catalog Number	Comments
Michelangelo Hand	Otto Bock Healthcare Products GmbH, A	8E500=L-M
AxonRotation	Otto Bock Healthcare Products GmbH, A	9S503
Wrist Flexor	Otto Bock Healthcare Products GmbH, A		prototype unit
AxonMaster	Otto Bock Healthcare Products GmbH, A	13E500
Electrode	Otto Bock Healthcare Products GmbH, A	13E200=50AC
ScissorFenceElectrodeCarrier	Otto Bock Healthcare Products GmbH, A		prototype unit
Acquisition Software	Otto Bock Healthcare Products GmbH, A		prototype unit
Carbon shaft	Otto Bock Healthcare Products GmbH, A		prototype unit