Characterization of the Sense of Agency over the Actions of Neural-machine Interface-operated Prostheses

Summary

Here we present a protocol which characterizes the sense of agency developed over the control of sensate virtual or robotic prosthetic hands. Psychophysical questionnaires are employed to capture the explicit experience of agency, and time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.

Abstract

This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the neural-machine interface (NMI) control of sensate virtual or robotic prosthetic hands. The formation of agency is fundamental in distinguishing the actions that we perform with our limbs as being our own. By striving to incorporate advanced upper-limb prostheses into these same perceptual mechanisms, we can begin to integrate an artificial limb more closely into the user's existing cognitive framework for limb control. This has important implications in promoting user acceptance, use, and effective control of advanced upper-limb prostheses. In this protocol, participants control a virtual prosthetic hand and receive kinesthetic sensory feedback through their preexisting NMIs. A series of virtual grasping tasks are performed and perturbations are systematically introduced to the kinesthetic feedback and virtual hand movements. Two separate measures of agency are employed: established psychophysical questionnaires (to capture the explicit experience of agency) and a time interval estimate task to capture the implicit sense of agency (intentional binding). Results of this protocol (questionnaire scores and time interval estimates) can be analyzed to quantify the extent of agency formation.

Introduction

As robotic prostheses become increasingly advanced, the importance of relevant sensory feedback will continue to grow. Sensory feedback affects how humans perceive, interact with, and even integrate machines into their body schema. Recent NMI techniques can now provide prosthetic limb users with intuitive control and achieve sensations associated with touch^{1,2,3,4,5,6,7} and kinesthesia (movement sense)^8,9 in missing limbs. When this sensory information is paired with the visual information provided by watching the artificial limb during operation, we have access to key elements that inform the distinction of self-versus-other. Leveraging this access may help bring prosthetic limb users a step closer to operating an artificial limb as a part of their body, rather than just a tool.

Body awareness and the sense of being embodied arises from the establishment of agency (the experience of authorship over a limb's actions) and ownership (the feeling that a limb is a part of the body)^10,11. Ownership is primarily mediated through the integration of touch and visual information¹². Agency emerges from the integration of intent, movement sensation (kinesthesia), visual information, and predictive cognitive models¹¹. During the performance of a voluntary action, agency is formed when the sensory consequences of that action align with the performer's intent and predictions from the performer's internal models¹³. Agency is separate and distinct from ownership. The concept of limb ownership has been studied frequently in prosthesis literature¹⁴. A sense of limb ownership forms in NMI participants when touch feedback is spatially and temporally appropriate, as measured explicitly through questionnaires or implicitly through changes in residual limb temperature, or temporal order judgments¹⁵. However, fewer opportunities have existed to explore agency in the context of NMI¹⁶. Recent work with NMI participants has demonstrated that agency can be purposefully promoted and is separate from the experience of ownership⁸.

Agency is particularly significant in the operation of robotic prostheses as it is a cognitive link to the control of the artificial limb's physical actions through experiences of causality, the feeling of controlling the artificial limb or causing something to happen¹⁷. Robotic prostheses are advanced computerized machines that the user must cooperate with to effectively complete tasks. Some prosthetic limbs have incorporated autonomous functions, such as grip-slip detection and correction; yet these systems have seen limited adoption as functionality running outside the user's control can be viewed as frustrating if not appropriately implemented^8,18. This presents a fundamental challenge that is echoed throughout applications of human cooperation with autonomous machines. That is, humans often trust their own actions over those resulting from a collaboration with computers or machines, and this trust directly influences an operator's likelihood to use the autonomous functions^19,20. As humans, we innately trust ourselves and our bodies to perform the actions we intend; when this is achieved, we establish an intrinsic sense of agency. Interestingly, the formation of agency is impacted in cooperative human-computer actions. During human-human cooperative tasks, a shared sense of agency may be formed over movement²¹; yet, the literature suggests that shared agency is encumbered during human-computer cooperation^22,23. These challenges are reflected in prosthetic upper-limb use, and the rejection rates of robotic devices remain high, with 23% - 39% of users discontinuing their use²⁴. In fact, many prosthesis users still prefer body-powered systems²⁵. These systems remove the computerized machine from the control loop and more intimately couple the user's body movement to the prosthesis movement via wire cables. This further reinforces the importance of cognitive integration in the use of advanced prosthetic devices. We suggest that NMI systems can provide a number of the necessary sensory and motor pieces to help move artificial limbs closer to establishing a cooperative sense of agency, and this will be instrumental in promoting the acceptance and the true integration of these computerized machines with their users.

Agency may be measured in a number of ways. The simplest measures use psychophysical questionnaires or scales that explicitly ask participants to whom or what they attribute an event^17,26,27. This relies on an individual's existing perception of "self" by requiring participants to make inferential judgments of self-attribution (i.e., to explicitly judge whether "I" or another entity was responsible for an action or event). Implicit measures provide insight into the background cognitive processes that occur during motor action and sensory events. This view of agency attempts to measure that which is not explicitly perceived by an individual. Typically this is achieved by having participants characterize a perceived difference in self- and externally-generated actions, for example having participants report the duration of time they perceived to occur between a self- and externally-generated event^17,28. During the performance of self-generated actions, agency implicitly manifests as a perceptual compression in time between actions and their sensory consequences, known as intentional binding²⁸. When individuals report the time they perceived to occur between an action and its outcome, a shorter perceived duration of time corresponds to a more strongly formed sense of agency^29,30. Interestingly, it has been demonstrated that explicit and implicit measures may not directly correlate as they are likely characterizing different perceptual mechanisms¹⁷ that together inform the sense of agency. As such, establishing a more comprehensive understanding of agency formation during prosthesis use will likely require experimental protocols employing both explicit and implicit measures.

This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the NMI control of sensate virtual or robotic prosthetic hands. Two techniques to measure agency during the performance of a sensorimotor object-grasping task are highlighted. Established psychophysical questionnaires are employed to capture the explicit experience of agency, while time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.

The scope of this protocol is to evaluate the sense of agency in the context of an NMI that provides physiologically relevant active motor control and kinesthetic feedback. These techniques are generalizable to virtual or physical prosthetic NMI systems. There are minimal restrictions on the populations that may be recruited to perform this protocol. For instance, the mobility of the participant's upper limbs cannot be bilaterally affected (they must have one sound limb), and they must possess the cognitive ability to make time-based judgments and articulate experienced sensations.

Protocol

This protocol has been previously approved and follows the guidelines of the Cleveland Clinic’s human research ethics committee.

1. Hardware and Software of the NMI

1. Establish each individual participant’s NMI control and feedback so that when they attempt to perform a movement, they see and feel a virtual prosthesis complete that movement.
   1. Generate a hand kinesthetic percept through the participant’s NMI and capture the kinematics of the perceived motion by having the participant demonstrate what they feel using their intact hand.
      NOTE: Techniques to characterize kinesthetic percept kinematics have been illustrated in other works⁸ and may be achieved using a data glove or an optical motion capture system.
   2. Use a virtual hand/prosthetic simulation to reproduce the kinematics of the movement percept.
   3. Set up hardware to capture the intentional hand movement control signals from the participant’s NMI.
   4. Map this control signal to the activity of the virtual prosthesis.
   5. Create a master control program that coordinates the acquisition of the NMI control signal, the movement of the virtual prosthesis, and the generation of kinesthetic NMI feedback in real-time.

2. Experimental Setup

1. Seat the participant and position a monitor horizontally (i.e., on its back, facing upward) on a table in front of them.
2. Display the virtual prosthesis on the monitor and adjust its size and location so that it is positioned congruently with the location of their missing limb.
3. Render objects (e.g., floating balls) in the virtual environment to serve as stop points for the close and open positions of the hand (endpoints of movement).
4. Configure the master control program so that when the virtual digits make contact with the virtual stop points, an auditory tone is played after an adjustable time delay (300, 500, 700, or 1,000 ms).

3. Experimental Conditions

1. Build an input file for the master control program that specifies the settings for each trial, including the auditory tone delay, whether the NMI feedback is turned on/off, the speed and direction of the virtual hand movement, and the delay between the command and the virtual hand movement.
   1. Create two control conditions, a baseline and a passive condition.
      1. For the baseline condition, configure the kinematics and control of the virtual hand to match the NMI kinesthetic percept.
         NOTE: The baseline condition represents the ideal congruency of motor intent, movement kinematics, and kinesthetic feedback.
      2. Program the passive condition to perform a virtual hand movement when triggered by the investigator (removing the control from the user) while still providing the participant with the NMI kinesthetic percept.
         NOTE: The passive condition captures the theoretical worst-case agency conditions (i.e., movement in the absence of control [without intent], similar to one’s body being passively moved).
   2. Program additional conditions designed to parse out the contributions to agency of motor intent, kinesthetic sensation, and temporal mismatch with the displayed kinematics of the virtual prosthesis. Consider using the following five conditions.
      1. Opposite movement: the NMI kinesthetic feedback indicates that the hand closes while the hand visualization opens.
      2. Too fast: the hand visualization closes faster than indicated by the NMI kinesthetic feedback.
      3. Too slow: the hand visualization closes slower than indicated by the NMI kinesthetic feedback.
      4. Onset delay: the hand visualization closes 1 s later than indicated by the NMI kinesthetic feedback.
      5. No feedback: the hand visualization closes without any NMI kinesthetic feedback.

4. Performance of the Experiment

1. Instruct participants to drive the hand from the open to the closed position without stopping and to report their estimation of the time delay from when the virtual digits contacted the virtual stop points to when they heard the auditory tone.
   NOTE: Participants may use any representation of time between 0 and 1 s that makes the most sense to them (e.g., milliseconds, fractions of seconds, a 0 - 10 scale).
2. Initiate each trial by pressing a start button on the master control program, which moves the virtual hand to the start position, signaling the beginning of the trial. This cues the participant to drive the virtual hand to the virtual stop points, which causes an auditory tone to play after a randomized delay (300, 500, or 700 ms).
   1. Record the participant’s verbally reported estimation of the time delay interval.
3. Organize trials into experimental blocks.
   1. Begin with two practice sessions and exclude them from the final analysis.
      1. In the first practice session, have the participant drive the hand to the movement endpoint and play the auditory tone 1,000 ms after the virtual digits reach the virtual stop points for 10 trials.
         1. Participants do not need to report the estimated intervals for this practice session.
            NOTE: This step is necessary to orient the participants to how long a single second feels.
      2. In the second practice session, again, have the participant drive the hand to the movement endpoint. Randomize the auditory tones so that the 300, 500, and 700 ms delay intervals are presented at least 5x each.
         1. Ask participants to report the estimated delay intervals.
         2. Do not inform the participant of how close their estimates of the delay intervals are to the actual delay during these practice trials or subsequent trials in the experimental block.
            NOTE: This step is important as participants will likely be inexperienced in making time judgments on a scale of fractions of a second, and the testing procedure may not be intuitive to the unpracticed test participant.
   2. Move to experimental sets of 15 trials for each condition. Present the conditions in a randomized order and administer a questionnaire at the end of each condition.
      1. Instruct the participants to reflect upon the latest set of trials and complete the eight-statement agency questionnaire (includes four questions to quantify the explicit experience of agency and four control questions [example provided in the Supplementary File])^8,26.
         1. Randomize the questionnaire statements to provide at least five unique question orders to be randomly presented to the participants.
   3. End the experimental block with a set of 15 trials for the passive condition and administer a questionnaire after completing these trials.
      NOTE: Administer the passive trials at the end of each experimental block to avoid interfering with an established sense of agency.
4. Complete four experimental blocks with different randomized orders of experimental conditions.
5. Provide multiple opportunities over the duration of testing for the participants to take a break. There is no minimum time or time limit for these breaks, but ensure the participant is not physically or mentally fatigued prior to continuing the testing.

Results

The experimental protocol was performed with three amputee participants operating a sensate virtual prosthesis via their NMI⁸ (Figure 1). The setup used a participant-controllable virtual hand moving through preprogrammed kinematic profiles using the MuJoCo HAPTIX physics engine³¹. The virtual hand was displayed on a horizontal monitor in front of the participants at a location spatially congruent with ...

Discussion

Here a methodological framework is presented to characterize the experience of agency formed while operating sensate prostheses via NMIs. In this context, agency is particularly relevant as it bridges physical action to the background cognitive processes that shape perception. Through a participant's prosthesis and NMI, we have direct access to a number of key elements that establish the sense of agency: intent, motor output, and movement sensation. Of importance to advanced prosthetic limb control, the tool...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
LabVIEW 2015, Service Pack 1, Version 15.0.1f2 64-bit	National Instruments, Austin, TX, USA	Full or Pro Version	We wrote custom software in LabVIEW to coordinate virtual prosthesis control with kinesthetic feedback as well as to present experimental conditions and record data.
8-Slot, USB CompactDAQ Chassis	National Instruments, Austin, TX, USA	cDAQ-9178
±60 V, 800 kS/s, 12-Bit, 8-Channel C Series Voltage Input Module	National Instruments, Austin, TX, USA	NI-9221
100 kS/s/ch Simultaneous, ±10 V, 4-Channel C Series Voltage Output Module	National Instruments, Austin, TX, USA	NI-9263
Custom Wearable Kinesthetic Tactor	HDT Global, Solon, OH, USA	N/A	This item was custom made. Other methods of delivering kinesthetic feedback are acceptable as long as the participant feels the sensation of the hand moving in real-time with the movements of the virtual hand.
MuJoCo Physics Engine, HAPTIX Version	Roboti LLC, Redmond, WA, USA	mjhaptix150	Newer versions of MuJoCo should be acceptable as well. We used the MPL Gripper Model.
Myobock Electrodes, powered by Otto Bock EnergyPack in MyoBoy Battery Receptacle	Ottobock, Duderstadt, Germany	electrodes: 13E200=60 battery: 757B21 battery receptacle: 757Z191=2	Any setup that provides an amplified, filtered, and rectified EMG or neural control signal could be used.