Characterization of the Sense of Agency over the Actions of Neural-machine Interface-operated Prostheses

Summary

Here we present a protocol which characterizes the sense of agency developed over the control of sensate virtual or robotic prosthetic hands. Psychophysical questionnaires are employed to capture the explicit experience of agency, and time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.

Abstract

This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the neural-machine interface (NMI) control of sensate virtual or robotic prosthetic hands. The formation of agency is fundamental in distinguishing the actions that we perform with our limbs as being our own. By striving to incorporate advanced upper-limb prostheses into these same perceptual mechanisms, we can begin to integrate an artificial limb more closely into the user's existing cognitive framework for limb control. This has important implications in promoting user acceptance, use, and effective control of advanced upper-limb prostheses. In this protocol, participants control a virtual prosthetic hand and receive kinesthetic sensory feedback through their preexisting NMIs. A series of virtual grasping tasks are performed and perturbations are systematically introduced to the kinesthetic feedback and virtual hand movements. Two separate measures of agency are employed: established psychophysical questionnaires (to capture the explicit experience of agency) and a time interval estimate task to capture the implicit sense of agency (intentional binding). Results of this protocol (questionnaire scores and time interval estimates) can be analyzed to quantify the extent of agency formation.

Introduction

As robotic prostheses become increasingly advanced, the importance of relevant sensory feedback will continue to grow. Sensory feedback affects how humans perceive, interact with, and even integrate machines into their body schema. Recent NMI techniques can now provide prosthetic limb users with intuitive control and achieve sensations associated with touch^{1,2,3,4,5,6,7} and kinesthesia (movement sense)^8,9 in missing limbs. When this sensory information is paired with the visual information provided by watching the artificial limb during operation, we have access to key elements that inform the distinction of self-versus-other. Leveraging this access may help bring prosthetic limb users a step closer to operating an artificial limb as a part of their body, rather than just a tool.

Body awareness and the sense of being embodied arises from the establishment of agency (the experience of authorship over a limb's actions) and ownership (the feeling that a limb is a part of the body)^10,11. Ownership is primarily mediated through the integration of touch and visual information¹². Agency emerges from the integration of intent, movement sensation (kinesthesia), visual information, and predictive cognitive models¹¹. During the performance of a voluntary action, agency is formed when the sensory consequences of that action align with the performer's intent and predictions from the performer's internal models¹³. Agency is separate and distinct from ownership. The concept of limb ownership has been studied frequently in prosthesis literature¹⁴. A sense of limb ownership forms in NMI participants when touch feedback is spatially and temporally appropriate, as measured explicitly through questionnaires or implicitly through changes in residual limb temperature, or temporal order judgments¹⁵. However, fewer opportunities have existed to explore agency in the context of NMI¹⁶. Recent work with NMI participants has demonstrated that agency can be purposefully promoted and is separate from the experience of ownership⁸.

Agency is particularly significant in the operation of robotic prostheses as it is a cognitive link to the control of the artificial limb's physical actions through experiences of causality, the feeling of controlling the artificial limb or causing something to happen¹⁷. Robotic prostheses are advanced computerized machines that the user must cooperate with to effectively complete tasks. Some prosthetic limbs have incorporated autonomous functions, such as grip-slip detection and correction; yet these systems have seen limited adoption as functionality running outside the user's control can be viewed as frustrating if not appropriately implemented^8,18. This presents a fundamental challenge that is echoed throughout applications of human cooperation with autonomous machines. That is, humans often trust their own actions over those resulting from a collaboration with computers or machines, and this trust directly influences an operator's likelihood to use the autonomous functions^19,20. As humans, we innately trust ourselves and our bodies to perform the actions we intend; when this is achieved, we establish an intrinsic sense of agency. Interestingly, the formation of agency is impacted in cooperative human-computer actions. During human-human cooperative tasks, a shared sense of agency may be formed over movement²¹; yet, the literature suggests that shared agency is encumbered during human-computer cooperation^22,23. These challenges are reflected in prosthetic upper-limb use, and the rejection rates of robotic devices remain high, with 23% - 39% of users discontinuing their use²⁴. In fact, many prosthesis users still prefer body-powered systems²⁵. These systems remove the computerized machine from the control loop and more intimately couple the user's body movement to the prosthesis movement via wire cables. This further reinforces the importance of cognitive integration in the use of advanced prosthetic devices. We suggest that NMI systems can provide a number of the necessary sensory and motor pieces to help move artificial limbs closer to establishing a cooperative sense of agency, and this will be instrumental in promoting the acceptance and the true integration of these computerized machines with their users.

Agency may be measured in a number of ways. The simplest measures use psychophysical questionnaires or scales that explicitly ask participants to whom or what they attribute an event^17,26,27. This relies on an individual's existing perception of "self" by requiring participants to make inferential judgments of self-attribution (i.e., to explicitly judge whether "I" or another entity was responsible for an action or event). Implicit measures provide insight into the background cognitive processes that occur during motor action and sensory events. This view of agency attempts to measure that which is not explicitly perceived by an individual. Typically this is achieved by having participants characterize a perceived difference in self- and externally-generated actions, for example having participants report the duration of time they perceived to occur between a self- and externally-generated event^17,28. During the performance of self-generated actions, agency implicitly manifests as a perceptual compression in time between actions and their sensory consequences, known as intentional binding²⁸. When individuals report the time they perceived to occur between an action and its outcome, a shorter perceived duration of time corresponds to a more strongly formed sense of agency^29,30. Interestingly, it has been demonstrated that explicit and implicit measures may not directly correlate as they are likely characterizing different perceptual mechanisms¹⁷ that together inform the sense of agency. As such, establishing a more comprehensive understanding of agency formation during prosthesis use will likely require experimental protocols employing both explicit and implicit measures.

This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the NMI control of sensate virtual or robotic prosthetic hands. Two techniques to measure agency during the performance of a sensorimotor object-grasping task are highlighted. Established psychophysical questionnaires are employed to capture the explicit experience of agency, while time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.

The scope of this protocol is to evaluate the sense of agency in the context of an NMI that provides physiologically relevant active motor control and kinesthetic feedback. These techniques are generalizable to virtual or physical prosthetic NMI systems. There are minimal restrictions on the populations that may be recruited to perform this protocol. For instance, the mobility of the participant's upper limbs cannot be bilaterally affected (they must have one sound limb), and they must possess the cognitive ability to make time-based judgments and articulate experienced sensations.

Protocol

This protocol has been previously approved and follows the guidelines of the Cleveland Clinic’s human research ethics committee.

1. Hardware and Software of the NMI

1. Establish each individual participant’s NMI control and feedback so that when they attempt to perform a movement, they see and feel a virtual prosthesis complete that movement.
   1. Generate a hand kinesthetic percept through the participant’s NMI and capture the kinematics of the perceived motion by having the participant demonstrate what they feel using their intact hand.
      NOTE: Techniques to characterize kinesthetic percept kinematics have been illustrated in other works⁸ and may be achieved using a data glove or an optical motion capture system.
   2. Use a virtual hand/prosthetic simulation to reproduce the kinematics of the movement percept.
   3. Set up hardware to capture the intentional hand movement control signals from the participant’s NMI.
   4. Map this control signal to the activity of the virtual prosthesis.
   5. Create a master control program that coordinates the acquisition of the NMI control signal, the movement of the virtual prosthesis, and the generation of kinesthetic NMI feedback in real-time.

2. Experimental Setup

1. Seat the participant and position a monitor horizontally (i.e., on its back, facing upward) on a table in front of them.
2. Display the virtual prosthesis on the monitor and adjust its size and location so that it is positioned congruently with the location of their missing limb.
3. Render objects (e.g., floating balls) in the virtual environment to serve as stop points for the close and open positions of the hand (endpoints of movement).
4. Configure the master control program so that when the virtual digits make contact with the virtual stop points, an auditory tone is played after an adjustable time delay (300, 500, 700, or 1,000 ms).

3. Experimental Conditions

1. Build an input file for the master control program that specifies the settings for each trial, including the auditory tone delay, whether the NMI feedback is turned on/off, the speed and direction of the virtual hand movement, and the delay between the command and the virtual hand movement.
   1. Create two control conditions, a baseline and a passive condition.
      1. For the baseline condition, configure the kinematics and control of the virtual hand to match the NMI kinesthetic percept.
         NOTE: The baseline condition represents the ideal congruency of motor intent, movement kinematics, and kinesthetic feedback.
      2. Program the passive condition to perform a virtual hand movement when triggered by the investigator (removing the control from the user) while still providing the participant with the NMI kinesthetic percept.
         NOTE: The passive condition captures the theoretical worst-case agency conditions (i.e., movement in the absence of control [without intent], similar to one’s body being passively moved).
   2. Program additional conditions designed to parse out the contributions to agency of motor intent, kinesthetic sensation, and temporal mismatch with the displayed kinematics of the virtual prosthesis. Consider using the following five conditions.
      1. Opposite movement: the NMI kinesthetic feedback indicates that the hand closes while the hand visualization opens.
      2. Too fast: the hand visualization closes faster than indicated by the NMI kinesthetic feedback.
      3. Too slow: the hand visualization closes slower than indicated by the NMI kinesthetic feedback.
      4. Onset delay: the hand visualization closes 1 s later than indicated by the NMI kinesthetic feedback.
      5. No feedback: the hand visualization closes without any NMI kinesthetic feedback.

4. Performance of the Experiment

1. Instruct participants to drive the hand from the open to the closed position without stopping and to report their estimation of the time delay from when the virtual digits contacted the virtual stop points to when they heard the auditory tone.
   NOTE: Participants may use any representation of time between 0 and 1 s that makes the most sense to them (e.g., milliseconds, fractions of seconds, a 0 - 10 scale).
2. Initiate each trial by pressing a start button on the master control program, which moves the virtual hand to the start position, signaling the beginning of the trial. This cues the participant to drive the virtual hand to the virtual stop points, which causes an auditory tone to play after a randomized delay (300, 500, or 700 ms).
   1. Record the participant’s verbally reported estimation of the time delay interval.
3. Organize trials into experimental blocks.
   1. Begin with two practice sessions and exclude them from the final analysis.
      1. In the first practice session, have the participant drive the hand to the movement endpoint and play the auditory tone 1,000 ms after the virtual digits reach the virtual stop points for 10 trials.
         1. Participants do not need to report the estimated intervals for this practice session.
            NOTE: This step is necessary to orient the participants to how long a single second feels.
      2. In the second practice session, again, have the participant drive the hand to the movement endpoint. Randomize the auditory tones so that the 300, 500, and 700 ms delay intervals are presented at least 5x each.
         1. Ask participants to report the estimated delay intervals.
         2. Do not inform the participant of how close their estimates of the delay intervals are to the actual delay during these practice trials or subsequent trials in the experimental block.
            NOTE: This step is important as participants will likely be inexperienced in making time judgments on a scale of fractions of a second, and the testing procedure may not be intuitive to the unpracticed test participant.
   2. Move to experimental sets of 15 trials for each condition. Present the conditions in a randomized order and administer a questionnaire at the end of each condition.
      1. Instruct the participants to reflect upon the latest set of trials and complete the eight-statement agency questionnaire (includes four questions to quantify the explicit experience of agency and four control questions [example provided in the Supplementary File])^8,26.
         1. Randomize the questionnaire statements to provide at least five unique question orders to be randomly presented to the participants.
   3. End the experimental block with a set of 15 trials for the passive condition and administer a questionnaire after completing these trials.
      NOTE: Administer the passive trials at the end of each experimental block to avoid interfering with an established sense of agency.
4. Complete four experimental blocks with different randomized orders of experimental conditions.
5. Provide multiple opportunities over the duration of testing for the participants to take a break. There is no minimum time or time limit for these breaks, but ensure the participant is not physically or mentally fatigued prior to continuing the testing.

Representative Results

The experimental protocol was performed with three amputee participants operating a sensate virtual prosthesis via their NMI⁸ (Figure 1). The setup used a participant-controllable virtual hand moving through preprogrammed kinematic profiles using the MuJoCo HAPTIX physics engine³¹. The virtual hand was displayed on a horizontal monitor in front of the participants at a location spatially congruent with their missing limb. The NMI participants had previously undergone a surgical neural rewiring procedure (targeted reinnervation), which was coupled with standard prosthetic limb myoelectric (EMG) control strategies to provide intuitive control of the virtual hand³²; thus, the participants could drive the virtual hand by 'thinking' about opening and closing their missing hand. Strategic vibrations of the participants' surgically rewired muscles induced illusory perceptions of hand movement, providing a platform for kinesthetic sensory feedback⁸. Through custom software, EMG hand control signals and virtual prosthesis renderings were integrated with the output of a vibration feedback device. When the participant initiated a movement of the displayed virtual hand, the vibration would induce a corresponding matched sensation of a complex grip movement in the missing hand.

Figure 1: An example setup that satisfies the requirements to characterize agency. This setup provides the user with intuitive control and kinesthetic feedback of a displayed virtual hand. Virtual hand control and feedback are achieved through the neural-machine interface using myoelectric control and vibration stimulation (eliciting illusory movement percepts of the missing limb) of the amputees' reinnervated musculature. Control and feedback are coordinated through a data acquisition system and computer running custom software. Virtual hand kinematics are displayed to the user on a horizontal monitor. Please click here to view a larger version of this figure.

Figure 2A is provided for the comparison of explicit measures of agency under each of the feedback conditions. Here, the average score for the four agency questions (and four control questions) are plotted for each participant and by each feedback condition. In Figure 2B, these individual participant scores are averaged and plotted for each feedback condition, with the error bars representing the average standard deviation. An average rating greater than 1 indicates an agreement with a given statement and 0 indicates neutrality of agreement²⁶. Higher agreement ratings (≥ 1) for the agency questions indicate a greater experience of agency. The responses to the control questions should be negative or neutral (≤ 0) and a score between 0 and 1 is taken as inconclusive. The 'baseline', 'too fast', and 'no feedback' condition demonstrated the lowest average interval estimates indicating the strongest sense of agency formed, whereas the 'passive' and 'opposite movement' conditions demonstrated the weakest sense of agency.

Figure 2: Explicit measures of agency under each feedback condition. (A) The average score for the four agency and four control questions provided to each participant under each feedback condition. (B) The average scores across participants under each feedback condition. The error bars represent the standard deviation. In both plots, an average rating greater than +1 indicates agreement and, for the agency questions, the formation of agency, while 0 indicates neutrality. This figure has been modified from Marasco et al.⁸. Please click here to view a larger version of this figure.

For a comparison of intentional binding in each feedback condition, Figure 3A shows the time interval estimates for each participant, averaged according to feedback condition. Differences between the actual and perceived time intervals were then averaged across the three participants and are presented in Figure 3B relative to the baseline feedback condition. The error bars denote the average standard deviation. Lower time interval estimates (Figure 3A) and larger negative differences (Figure 3B) are an indication of a stronger implicit sense of agency. The 'too fast' condition followed by the baseline condition demonstrated the lowest average interval estimates, indicating the strongest sense of agency formed, whereas the 'opposite movement' condition demonstrated the weakest sense of agency.

Figure 3: Implicit measures of agency via time interval estimates under each feedback condition. (A) The average interval estimates for the delay interval between the completion of the virtual hand close and the auditory tone plotted for each participant across each randomly presented actual interval. The results are plotted for each feedback condition, and a lower time estimate indicates a stronger sense of agency. These panels have been modified from Marasco et al.⁸. (B) The average difference (across participants and delay intervals) between the actual time delay and the participant's estimated time interval relative to the baseline feedback condition. The results are plotted for both. Here, a more negative value indicates a stronger average sense of agency, and CI denotes 95% confidence interval (CI). Please click here to view a larger version of this figure.

Figure 4 allows for a comparison of explicit and implicit agency measures. The average difference between actual and perceived time intervals are plotted relative to the results of the baseline feedback condition and with respect to the averaged agency questionnaire scores for each feedback condition. In this presentation of data, moving from left to right on the x-axis indicates a decrease in the explicit experience of agency and moving from bottom to top on the y-axis indicates a decrease in the implicit sense of agency. As in Figure 2 and Figure 3, the 'too fast' condition demonstrated the strongest formation of agency, both explicitly and implicitly.

Figure 4: Average explicit and implicit measures of agency for each feedback condition, combining the results presented in Figure 2B and Figure 3B. The average explicit agency results are plotted on the x-axis and the average interval estimates are plotted on the y-axis. The error bars denote standard deviation. Moving from left to right on the x-axis indicates a decrease in the explicit experience of agency, and moving from bottom to top on the y-axis indicates a decrease in the implicit sense of agency. This figure has been modified from Marasco et al.⁸. Please click here to view a larger version of this figure.

Discussion

Here a methodological framework is presented to characterize the experience of agency formed while operating sensate prostheses via NMIs. In this context, agency is particularly relevant as it bridges physical action to the background cognitive processes that shape perception. Through a participant's prosthesis and NMI, we have direct access to a number of key elements that establish the sense of agency: intent, motor output, and movement sensation. Of importance to advanced prosthetic limb control, the tools provided in this work leverage this direct access to help unlock an understanding of how these elements may promote the user's sense of control over, and the cognitive integration of, the actions of their prosthesis.

The techniques highlighted are flexible in that they can be employed with any NMI research and clinical prosthetic system so long as they meet the criteria for real-time perceptually relevant control and kinesthetic feedback. The advantage inherent to many NMIs is the potential for intuitive control achieved by leveraging the neural pathways that remain postamputation. This allows for the measurement of residual physiological activity that once accompanied intact limb movement, which can, in turn, be decoded and mapped to the appropriate virtual or prosthetic limb movement. Therefore, most NMI techniques should satisfy the requirement for perceptually relevant control, provided that the recorded neural activity and the accompanying digital interface can produce reliable output signals that can be appropriately mapped to the virtual hand. The experimental setup also requires a system providing investigators with the ability to actively initiate kinesthetic sensations in real-time with the displayed virtual hand kinematics. This is a critical requirement as a sense of agency over movements is established when we engage in an action and appropriate sensory feedback is returned during the completion of that action¹³. Again, as long as this criterion is met, most any NMI kinesthetic feedback system will be appropriate.

The techniques presented here have the advantage of evaluating both explicit and implicit cognitive-perceptual measures of agency. There is evidence to suggest that each may be a result of separate cognitive mechanisms that, together, form a complete sense of agency¹⁷; however, there is still not a complete understanding of this relationship. The results from these measures are quantitative and easily interpreted. Decreases in time interval estimates suggest that a stronger implicit sense of agency was formed. Similarly, higher questionnaire scores on agency statements indicate a stronger explicit experience of agency. It is suggested that these quantitative values can provide a basis to evaluate and tune NMI control and sensory feedback. For example, in a previous work⁸ that is reported here in Figure 4, participants often reported smaller perceived time intervals and explicitly reported stronger perceived agency when a virtual hand was displayed that closed slightly faster than the kinesthetic sensation they experienced. This indicates that the user felt a stronger sense of control over the actions of the hand, as reported explicitly, but also suggests that the cognitive processes that establish this sense of control more strongly associate with this faster kinematic display. As such, an adjustment to the NMI control scheme of a clinical prosthesis to accommodate faster hand closing may help improve the user's perceptions of control over their physical device and encourage the user to identify their device's actions as self-generated.

The techniques presented may also be employed to form a more complete understanding of how multiple sensory modalities may influence perceptions of ownership over artificial limbs. For example, touch sensory feedback (or other sensory modalities) may be incorporated into the paradigm presented here to evaluate their possible individual roles in potentiating the sense of agency. Additionally, the techniques presented here may be paired with measures of ownership to more comprehensively characterize the interrelationships between agency, embodiment, and individual sensory modalities. The methods provided may also have broader applicability beyond NMI-controlled devices. Similar experimental tasks could be implemented with complex control systems (such as myoelectric pattern recognition), traditional myoelectric prostheses, and body-powered systems, as well as systems without NMI sensory feedback. This may allow for a unique perspective in understanding how cognitive processes respond to less 'natural' control and feedback paradigms and provide insights into how agency and perceptions of control may act during the operation of more traditional research or clinical prosthetic systems.

As robotic prostheses grow increasingly sophisticated, so too does the need for effective control and cognitive integration of these devices. Sensation is a pathway to addressing a number of critical barriers, and being able to assess the underpinning mechanisms that process movement sensation and information is an important piece. The tools provided here can help facilitate the integration of devices with users by characterizing the explicit and implicit formation of agency. These techniques help quantify the benefits of the innate access to intuitive motor control and sensation that NMIs may provide and can offer a platform for assessment and tuning, to ultimately improve the user's perception of being in control of their artificial limb.

Materials

Name	Company	Catalog Number	Comments
LabVIEW 2015, Service Pack 1, Version 15.0.1f2 64-bit	National Instruments, Austin, TX, USA	Full or Pro Version	We wrote custom software in LabVIEW to coordinate virtual prosthesis control with kinesthetic feedback as well as to present experimental conditions and record data.
8-Slot, USB CompactDAQ Chassis	National Instruments, Austin, TX, USA	cDAQ-9178
±60 V, 800 kS/s, 12-Bit, 8-Channel C Series Voltage Input Module	National Instruments, Austin, TX, USA	NI-9221
100 kS/s/ch Simultaneous, ±10 V, 4-Channel C Series Voltage Output Module	National Instruments, Austin, TX, USA	NI-9263
Custom Wearable Kinesthetic Tactor	HDT Global, Solon, OH, USA	N/A	This item was custom made. Other methods of delivering kinesthetic feedback are acceptable as long as the participant feels the sensation of the hand moving in real-time with the movements of the virtual hand.
MuJoCo Physics Engine, HAPTIX Version	Roboti LLC, Redmond, WA, USA	mjhaptix150	Newer versions of MuJoCo should be acceptable as well. We used the MPL Gripper Model.
Myobock Electrodes, powered by Otto Bock EnergyPack in MyoBoy Battery Receptacle	Ottobock, Duderstadt, Germany	electrodes: 13E200=60 battery: 757B21 battery receptacle: 757Z191=2	Any setup that provides an amplified, filtered, and rectified EMG or neural control signal could be used.