Visualization Method for Proprioceptive Drift on a 2D Plane Using Support Vector Machine

Summary

This article describes a novel method to estimate proprioceptive drift on a 2D plane using the mirror illusion and combining a psychophysical procedure with an analysis using machine learning.

Abstract

Proprioceptive drift, which is a perceptual shift in body-part position from the unseen real body to a visible body-like image, has been measured as the behavioral correlate for the sense of ownership. Previously, the estimation of proprioceptive drift was limited to one spatial dimension, such as height, width, or depth. As the hand can move freely in 3D, measuring proprioceptive drift in only one dimension is not sufficient for the estimation of the drift in real life situations. In this article, we provide a novel method to estimate proprioceptive drift on a 2D plane using the mirror illusion by combining an objective behavioral measurement (hand position tracking) and subjective, phenomenological assessment (subjective assessment of hand position and questionnaire) with a sophisticated machine learning approach. This technique permits not only an investigation of the underlying mechanisms of the sense of ownership and agency but also assists in the rehabilitation of a missing or paralyzed limb and in the design rules of real-time control systems with a self-body-like usability, in which the operator controls the system as if it were part of his/her own body.

Introduction

In recent years, research about the sense or experience of the self-body, that is, one's own body, has increased in the context of embodiment. Embodiment refers to the idea or concept of having a physical or virtual body that can interact with the environment, such as reaching, grasping, and touching. For instance, humans can touch an object or another human positioned in the environment by moving their own body, in this case their own arm and hand. Nowadays, this interaction or communication is not limited to using one's own natural body. Due to inventions and development of human-like robots or avatars in the virtual world, the natural human body can be substituted by an artificial body, such as a humanoid, remote control robot, electric prosthesis, or computer-graphics avatar in virtual reality. For example, researchers developed a robot whose operator can "grasp" an object placed in front of the robot via its mechanical body, even if the robot is placed far away from the operator's body position^1,2. Similar to this example, if a human could perform an action via an artificial body, which body would hold the attribution of the operator's self-body?

We can easily find topics related to this discussion on the attribution or projection of "self" from our own natural body to an artificial, non-flesh-and-bone body. One example can be found in the medical field; for example, in the field of medical rehabilitation, treatments that "trick" the patient's self-body sensation using mirrors are being explored for reducing pain and improving motor function of a missing or paralyzed limb, called mirror therapy^3-6. In this therapy, the mirrored image of the unaffected body part or limb can mislead the patient's brain into believing that the missing or paralyzed limb corresponds to the one displayed in the mirror and lead to the feeling that it is still in its former condition (i.e., before the accident). It is still under discussion how this illusion affects the brain's resilience related to body representation. In addition to this type of discussion on our natural body, we can find similar discussions on embodiment, especially of human-system interaction design issues in the field of engineering. The sense of self for an artificial or virtual body has been well investigated in the context of telepresence, brain-machine interface, and brain-computer interface^1,2,7-9. Some researchers reported that a humanlike robot, which can transfer the tactile sensation from its robot hand to the operator's hand, can capture the operator's sense of self-body to the robot as well as the sense of being at a place where the robot is positioned rather than where the operator actually exists, called tele-existence¹. Other researchers reported that a virtual avatar reflecting the operator's body movements strongly transfers the operator's sense of self-body from the operator's own body to the virtual body⁹. These findings indicate how users can project their sense of self-body into an artificial body, such as a humanoid, remote control robot, electric prosthesis, or computer-graphics avatar in virtual reality, even if the artificial body is not directly connected to their brain and body.

Basic scientific research on this type of self-body sensation for non-flesh-and-blood, artificial body-like objects examined the underlying brain mechanisms for the experience of self-body using the rubber hand illusion (RHI)^10-13 and mirror illusion (MI)^14-16 in the medical and engineering fields as well as in psychophysics and neuropsychology. The RHI is the sensation that a rubber hand belongs to one's own body and is evoked by simultaneously stroking a visible rubber hand and the participant's obscured hand. In the MI, a hand image in a mirror positioned along the midsagittal axis visually captures the participant's perceived position of the unseen opposite hand. Moreover, synchronous movements of the reflected and unseen hand evoke the strong sensation as if the reflected hand image were the unseen opposite hand. According to the research on these illusions, the consistency between multimodal information and the prediction and sensory feedback about body movements seems to play an important role for the judgment of self-body attribution. Thus, these two illusions can be simple but powerful evidence and tools for scientists to investigate the brain mechanisms underlying our sensation of being tricked or believing that some artificial object or image can subjectively be our own body part, and that our self-body sensation does not have to be tied to our natural physical body.

In all these studies listed above, the discussion has been based on the concept of "self" consisting of two types of sensation proposed by the philosopher Gallagher¹⁷: the sense of ownership and the sense of agency. The sense of ownership refers to the sensation that an observed body part is one's own. The sense of agency corresponds to the sensation that body movement is self-caused. These two sensations are defined as the minimal self, that is, an immediate sense of the self¹⁶. According to this concept, the attribution of the "self" for the natural, damaged, virtual, and mechanical bodies can be evaluated by the same indexes: the sense of ownership and agency. In order to use this sensation for scientific evaluation, the question arises of how to measure the sense of ownership and agency robustly. Currently, the estimation of the sense of ownership and agency mainly relies on questionnaires, originally proposed by Botvinick⁹. In addition to questionnaires, we can attempt to measure them in quantitative ways. For instance, the skin conductance response (SCR) has been used as a physiological index of ownership in cases where the rubber hand is suddenly cut by a knife¹⁸. The SCR is calculated by measuring the electrical characteristics of the skin and is a sensitive and valid indicator for arousal¹⁹. Since this method is typically applied for single trials per participant, measuring SCR is not suitable as a physical index during psychophysics experiments that require repetitive measurements within participants. One of the most successful behavioral indexes for the sense of ownership is proprioceptive drift. Proprioceptive drift is the change in the perceived position of the unseen real hand toward the position of an object that looks like a hand, such as the rubber-made prosthesis or computer graphics^10-13. Since this change can be estimated repetitively and robustly by measuring the distance between the unseen real hand and the visual image of the hand, proprioceptive drift is a suitable physical index for psychophysical measurements. However, this usage needs to be evaluated carefully, because recent discussions have questioned whether proprioceptive drift can always be used as a behavioral index of ownership¹².

Typically, proprioceptive drift is measured in only one of the three directions, such as height, width, or depth. Proprioceptive drift has rarely been measured in multiple directions due to the difficulty of estimating and visualizing multi-dimensional data. This metrological limitation is not critical for basic research exploring the mechanisms that process multisensory information, because experimental conditions can be easily designed and controlled to limit the measured dimensions. However, in daily life, our hands move freely in 3D to follow our intentions. In this situation, it is difficult and inadequate to measure a participant's behavior with questionnaires, which severely limits movement and positions of the hands. Thus, considering the potential applications for sense of ownership and agency in engineering and rehabilitation, a measurement that includes multiple directions and allows free hand movement is needed to evaluate the spatial relationship between visual and proprioceptive feedback in daily life situations. If such measurement were possible, the measured distance between real and observed hands could be utilized as a guideline for the sense of self-body. This could not only become an indicator for the progress of rehabilitation but also a criterion for the spatial offset between the manipulated target in the display and the operating hand. The question remains as to how this measurement can be implemented reliably and effectively.

To address this question, we introduce a novel method to estimate proprioceptive drift, which corresponds to the shift from the position of the participant's unseen real hand to that of a visible hand-like object, on a 2D plane using the mirror illusion by combining a psychophysical procedure and an analysis using machine learning. Compared to a rubber hand, the hand image in a mirror strongly captures the participant's perceived position of the unseen real hand. Moreover, a mirrored image immediately reflects voluntary hand movements for hand placement. Thus, a mirror image was selected as the visual feedback of the participant's hand. In addition, to measure proprioceptive drift similar to daily life situations, the participants positioned their hidden hand trial-by-trial at their will, and the number of trials was increased. Although any combination of directions could have been used, the combination of height and depth was chosen due to the ease of placing the mirror vertically. To check consistency between our method and previous research¹³, two visual conditions were implemented: with and without visual feedback. In the condition with visual feedback, the mirror was positioned along the midsagittal plane to create a reflected image of the left hand, as if it were seen as the right hand. In the condition without visual feedback, a matte blackboard was used in order to hide the participant's real right hand. We assessed the effectiveness of this novel method by comparing the results to those obtained with a questionnaire on the sense of ownership and agency.

Protocol

All aspects of the experiment were approved by the Ethical Committee of Tokyo Institute of Technology.

1. Experimental Setup

1. Material and Setup for Measuring Proprioceptive Drift.
   1. Obtain a stand that can hold a 100 x 100 cm plate vertically (Figure 1).
   2. Obtain a chair on which the participant can sit comfortably during the experiment.
   3. Obtain a 100 x 100 cm acrylic mirror and matte blackboard.
   4. Obtain the position tracker (for example, SLC-C02, Cyverse) to track the participant's right hand position. Spatial resolution should be about 1.5 mm to allow sufficient number of samplings to be used for the machine learning.
   5. Obtain an infrared LED and retroreflective markers that will be used to indicate the position of the stand and the participant's right hand, respectively (see steps 1.1.11 and 3.2.6).
   6. Obtain the foot pedal for the participant's response.
   7. Create the custom-made program, which can record and simultaneously display the participant's response and right hand position and play a beep as feedback of the participant's response when the foot pedal is pressed. In these experiments, the participant's right hand position was collected using the motor capture device and its custom-made program according to manufacturer instructions.
      NOTE: According to a previous paper¹⁶, the program was developed with a software development toolkit. The custom-made program developed by the software development toolkit can be adapted for other brands of motion capture devices.
   8. Use a metronome to provide timing signals for the training of the hand movement, that is, tapping the surface of the mirror or blackboard. See step 3.1.1 for precise training instructions.
   9. Use noise-cancelling headphones to reduce the possibility that the participant can hear sound cues for the hand position.
   10. For the visual feedback condition, attach the mirror to the stand. For the condition without visual feedback, attach the blackboard to the stand.
   11. Place the infrared LED at the top-left of the mirror or blackboard.
2. Material and Setup for Measuring Sense of Ownership and Agency.
   1. Repeat the procedure from step 1.1.1 to step 1.1.11.
   2. Create or obtain the questionnaire assessing sense of ownership and agency (e.g.,^10,13,16). Table 1 shows examples from this questionnaire used in the previous study¹⁵.
   3. Use a monitor or tablet PC to display the questionnaire to the participant.

2. Participants

1. Recruit about 10 right-handed participants with normal or corrected-to-normal vision.
   Note: The number of participants can be adjusted according to the experimental goals and the number of repeated trials per participant.
2. Obtain written informed consent for participation prior to the start of the experiment.

3. Experiment Procedure

1. Training Phase for the Hand Movement.
   1. Train the participants to synchronously tap with both hands on the mirror or blackboard at a certain tempo using the metronome. Instruct the participants to perform the tapping movement by keeping the heel of the hand in contact with the mirror or board. At the beginning of training, start the metronome at a tempo of 60 beats per min and then instruct the participant to move both hands synchronously according to the sound of the metronome.
   2. Ensure that the timing of the participant's hand movement is close to one cycle per sec (approximately 1 Hz) by comparing it to the sound of the metronome several minutes after the beginning of tapping.
2. Estimation of Proprioceptive Drift in the Participant's Midsagittal Plane.
   1. Use a counter-balanced order of conditions across participants.
   2. Mount the mirror or blackboard on the stand based on the conditions: with visual feedback, mount the mirror; without visual feedback, mount the blackboard.
   3. Ensure that participant is sitting very close to the mirror or blackboard, which is positioned along the midsagittal plane of the participant (Figure 1).
   4. Make sure that participant can see the mirror image of the left hand but cannot see the real right hand.
   5. Instruct participant to pay attention to the image of the left hand in the mirror during the experiment.
   6. Put the retroreflective markers on the participant's right index fingertip and wrist.
      1. Since the markers are only put on the participant's right hand, make sure that the haptic sensation of the participant's right hand due to the attached markers is not altered substantially compared to the left hand by querying the participant orally.
   7. Put the noise-cancelling headphones over the participant's ears.
   8. Instruct the participant to move the left hand approximately 30 cm vertically and 30 cm horizontally from the lower right corner of the mirror and to maintain this left hand position during the experiment. This position is set as the origin of the surface of the 2D plane.
   9. Instruct the participant to place the right hand at will on the other side of the mirror or blackboard and to maintain its position until the end of the trial.
   10. Instruct the participant about the task as follows:
       1. At the beginning of each trial, instruct the participant to push the middle button of the foot pedal. At this time, the system will sound a beep through the headphone as feedback of the pedal press.
       2. After hearing the beep, instruct the participant to start tapping with both hands synchronously at 1 Hz on the board, which is the mirror in the condition with visual feedback or the blackboard in the condition without visual feedback.
       3. After more than six hand movements, instruct the participant to stop the movement at the preferred time and answer the question about the right hand position by pressing the right or left button on the foot pedal. The right button is a yes and the left is a no. The question is, "Do you feel that the right and left hand are in the same position?" At this moment, the participant will hear a beep as feedback for the pedal press again.
          NOTE: If the participants ask about the meaning of the "the same position," tell them that "same position" means that the subjective height and depth of the right hand are equivalent to that of the left hand.
       4. Instruct the participant to move their right hand to another position of their choice. Then, start the trial again. This cycle will continue for up to 200 trials per condition.
   11. Ensure that the participant can understand the task and ask the participant to restart the task.
   12. During the task, check that the timing of the participant's tapping stays at approximately 1 Hz by viewing the movement compared to the metronome.
       Note: The sound of the metronome can be heard only by the experimenter.
   13. After finishing about 100 trials, let the participant take a break.
   14. Perform the experiment for the other conditions (with or without visual feedback) on separate days.
3. Estimation of Sense of Ownership and Agency in the Mirror Condition.
   1. Define the right hand's positions for collecting the participant's responses on the questionnaire about sense of ownership and agency. For example, in a previous publication¹⁶, there were 13 prefixed right hand positions. These points were arranged every 7 cm up to ± 21 cm from the origin.
   2. Carry out the same procedure for the above estimation as listed from step 3.2.2 to step 3.2.7.
   3. Instruct the participant to place the right hand following the experimenter's guide and maintain its position until finishing one trial.
   4. Instruct the participant about the task as follows:
      1. At the beginning of the trial, push the middle button of the foot pedal. At this time, the participant will hear the beep as feedback of the pedal press.
      2. Then, start to tap the right and left hands synchronously at 1 Hz.
      3. After more than six times of tapping, stop tapping when the experimenter indicates. Then, answer the questions about sense of ownership and agency displayed on the monitor using a 7-point Likert scale with ratings ranging from -3 ("totally disagree") to +3 ("totally agree") with 0 indicating neither agreement nor disagreement ("uncertain").
      4. Move the right hand to the position that the experimenter indicates. Then, start the trial again. This cycle will continue up to the number of right-hand positions that the experimenter defines.
   5. Ensure that the participant can understand the task and ask the participant to start the task.

4. Data Analysis

1. The Analysis of Proprioceptive Drift in the Participant's Midsagittal Plane.
   1. Obtain the statistical tool that contains the machine learning application, especially support vector machine (e.g., R, MATLAB). Use support vector machine (SVM) as the classifier to extract the borders of the participant's responses. A previous publication provides an explanation for the algorithms of the classifier (see Chapter 7)²⁰. In this article, we explain the method using R (version 3.1.2).
   2. Install the package named "kernlab"²¹, which contains the analysis using SVM in the R application.
   3. Mark the area that shows proprioceptive drift of the hand as follows (Figure 2 describes the schematic representation of the data analysis flow). See Supplemental software code and data sample for further explanation of this data analysis.
      1. Calculate the relative right hand positions from the origin. Discard data with errors (e.g., missing position data or participant's responses) from the analysis.
      2. Make a probabilistic model of the participant's "yes" responses in 2D space using the SVM. Use the data from the responses as a symbolic description of the model. Use the data of the right hand position as the parameters of the model. Use the commonly used radial basis function kernel as the kernel for the SVM. In order to avoid the arbitrary analysis, calculate sigma (i.e., parameter used for changing the weight of each data point) by automatic sigma estimation.
      3. Ensure that the model is correctly fitted by checking that the training errors of the model are under 0.2. Using the probabilistic model, define the area in which the p-value of the participant's "yes" responses was estimated to be over 0.5.
   4. Average each participant's data to make an area that shows proprioceptive drift.
      NOTE: As it is difficult to average the border of the "yes" and "no" response area estimated by the p-values of the responses in 2D space, two types of average are recommended. One method is to average the p-values for the participant's responses in 2D space, which is the method used prior to estimating the border. The other method is to average the area size, which is used after estimating the border.
2. Analysis of the Questionnaire Data and Area Size.
   1. Obtain the statistical tool to assess the significance of the position and the categories of the questionnaire (e.g., SPSS or R).
   2. Assess normal distribution of all data using the Shapiro-Wilk test, and apply the appropriate non-parametric test when one or more of the corresponding data sets failed to meet the criteria for normal distribution (e.g., Wilcoxon signed-rank test, Friedman test).
      Note: If a non-parametric method that suits the experiment is lacking, use a parametric method and explain the reasoning. In a previous study¹⁶, a two-way repeated-measures ANOVA analyzed the questionnaire data, as there was no non-parametric substitute for this analysis.

Representative Results

Representative results from a previous study are presented to illustrate the method¹⁶. Figure 3A shows that the area shapes where the participant could not detect the spatial offset between left and right hand position differed between the conditions with (mirror) and without (blackboard) visual feedback. Figure 3B shows that area sizes in the condition with visual feedback are significantly larger than in the condition without visual feedback (Wilcoxon signed-rank test: Z = -2.803, p = 0.005). These results suggest that the required offset between visual and proprioceptive feedback to maintain proprioceptive drift is approximately 10 cm and this value changes by direction (Figures 3 and 4). Vertical offset appeared larger than horizontal offset. In Figure 5, the vertical and horizontal spatial distribution of the questionnaire score for body ownership and agency showed a unimodal distribution. Their peaks were at the origin, where the participant's right hand was almost at the same position as that of the mirrored hand image. In contrast, the spatial distribution for the score of the control statements was almost flat and under -1. A two-way repeated-measures ANOVA revealed the main effects for the categories and positions that the participants indicated in the questionnaires (Horizontal: Category: F(3,27) = 11.12, p <0.001; Position: F(6,54) = 10.27, p <0.001; Vertical: Category: F(3,27) = 24.21, p <0.001, Position: F(6,54) = 7.298, p <0.001). The interactions between category and position were also significant (Horizontal: F(18,162) = 9.42, p <0.001; Vertical: F(18,162) = 8.00, p <0.001). These results imply that the sense of ownership and agency decreased when the spatial offset between the participant's real obscured right hand and the mirrored hand image increased. In Figure 6, the comparison between visualization of proprioceptive drift and questionnaire results for sense of ownership and agency shows that the offset areas to maintain these phenomena are concentric and almost overlap.

Figure 1: Overview of Setup. The experimental setup includes a stand that holds either the mirror or the blackboard, a chair, a response device (foot pedal), and recording devices for the participant's hand position. The top figure shows the schematic representation of the setup from the direction where the participant can see the mirror image of his/her left hand. The bottom figure shows the view of the backside of the mirror. Please click here to view a larger version of this figure.

Figure 2: Data Analysis Flow. (A) An example of the participant's response and right hand positions. (B) Schematic representation of the "yes" response model estimated by SVM. (C) Result of the border analysis. (D) Average areas across participants. Please click here to view a larger version of this figure.

Figure 3: Comparison of Area Shapes and Sizes between the Conditions with and without Visual Feedback. (A) Comparison of area shapes. The origin of the graph is the position of the left hand, that is, the position of the mirrored hand image in the visual feedback condition. The vertical and horizontal axes show the participant's right hand position as proprioceptive feedback of the hand. (B) Comparison of area sizes. Vertical axis shows the area sizes where participants could not detect the spatial offset between left and right hand position. Horizontal axis shows the condition with or without mirrored hand image as visual feedback (right: with visual feedback, left: without visual feedback). Please click here to view a larger version of this figure.

Figure 4: Individual Data. For most participants, the shape of the border in the mirror condition was larger than in the blackboard condition. This suggests that the visualization method using SVM succeeded in showing the effect of visual capture by the body image in the mirror. In contrast, for participants D, H, and J, there were few differences across the conditions, indicating that there might be individual differences for the effect of visual capture. Please click here to view a larger version of this figure.

Figure 5: Questionnaire Results. (A) Spatial distribution of the questionnaire score along the vertical axis. (B) Spatial distribution of the questionnaire score along the horizontal axis. Ownership and agency scores were highest at the origin relative to the vertical and horizontal positions. Please click here to view a larger version of this figure.

Figure 6: Comparison between Classification and Questionnaire. (A) Spatial distribution of the questionnaire scores for ownership and agency along the horizontal axis. (B) Spatial distribution of the questionnaire scores for ownership and agency along the vertical axis. (C) Estimated area where the participants could not detect the offset between mirrored hand image and obscured real right hand. Please click here to view a larger version of this figure.

Category
Ownership	1. I felt as if I was looking at my own right hand.
	2. I felt as if the hand in the mirror was part of my body.
	3. It seemed as if I were sensing the movement of my right hand in the location in which the hand in the mirror moved.
	4. I felt as if the hand in the mirror was my hand.
Ownership control	5. I felt as if my real right hand was turning into the hand in the mirror.
	6. It seemed as if I had more than one right hand.
	7. It appeared as if the hand in the mirror was drifting toward my real hand.
	8. It felt as if I had no longer a right hand, as if my right hand had disappeared.
Agency	9. The hand in the mirror moved just like I wanted my right hand to, as if it were obeying my will.
	10. I felt as if I was controlling the movement of the hand in the mirror like I would control that of my right hand.
	11. I felt as if I was causing the movement created by the hand that I saw as my right hand.
	12. Whenever I moved my right hand, I expected the hand in the mirror to move in the same way.
Agency control	13. I felt as if the hand in the mirror was controlling my will.
	14. I felt as if the hand in the mirror was controlling my movements.
	15. I could sense the movement from somewhere between my real right hand and the hand in the mirror.
	16. It seemed as if the hand in the mirror had a will of its own.

Table 1: Questionnaire consisting of 16 Statements Classified into Four Categories. This questionnaire was adapted and translated into Japanese from questionnaires used in the rubber-hand illusion experiments^10,13.

Supplemental Files. Sample Code and Data Set for the Analysis Method using SVM. This code can be carried out using R (version 3.1.2) and conduct the same analysis as in the current paper. Please click here to download this file.

Discussion

We demonstrate a method to estimate proprioceptive drift in a 2D plane during the mirror illusion using SVM and to compare the result with questionnaire responses for sense of ownership and agency. This novel method revealed that the required offset between visual and proprioceptive feedback to maintain proprioceptive drift is approximately 10 cm and that this offset closely overlaps with the offset required to maintain the feeling of ownership and agency.

Note that the most critical step of this method is described under 3.2.5, an instruction to the participants about their attention. In the previous pilot study¹⁶, the effect of visual capture by a mirrored hand image was rarely observed when the participants subjectively reported that they paid more attention to their body. It is still unclear whether attention contributes to this phenomenon. The relationship between attention and sense of ownership and agency has rarely been discussed due to the difficulty to control or measure the participants' attention. To our knowledge, there is only one study²² that directly investigated the effect of attention on the sense of ownership. To avoid complex discussions about how to control and measure attention, careful instruction regarding the participants' attention is required.

The other critical point is that the participants can select the hand position for the next trial at their own will in the session measuring proprioceptive drift. In traditional psychophysics, the degree of freedom of the participants' behavior tends to be limited and controlled to make the experiment more robust and reproducible. For example, in the experiments measuring proprioceptive drift and sense of ownership and agency using the rubber hand illusion paradigm^10-13, the sampling points of the participants' hand position were pre-defined along one of the three directions, such as height, width, or depth, to avoid difficulties of measuring and visualizing multi-dimensional data. Especially since questionnaires take time for each sampling point due to the number of categories (e.g., ownership, agency, and control statements), the number of sampling points are more limited compared to the methods recording human behaviors. In this method, in order to ensure consistency with other studies, the measurement of the sense of ownership and agency using a questionnaire was limited and prefixed as well. This limitation related to the participants' behavior is not critical for basic research exploring the brain mechanism of the sense of ownership and agency, because it is possible to design and control the experimental conditions precisely and reproducibly by including these limitations in the experimental conditions. However, considering the potential applications for sense of ownership and agency in engineering and rehabilitation, this limitation can be critical to estimate human behavior in 3D under daily life conditions. Under these circumstances, where participants can move their hand and choose its position more freely, it is hard to estimate proprioceptive drift using traditional psychophysical methods. To overcome this problem, an SVM, a type of machine learning, was adopted for the estimation and visualization of proprioceptive drift. This technique can collect and analyze massive data, such as multi-dimensional data containing participants' responses and hand positions in 2D with free selection. Using this technique, the participants' responses sampled at various hand positions were classified and the area analyzed where participants did not notice the distance between the mirror image of the left hand and the hidden actual right hand.

This method has a critical limitation, which is that the average area showing proprioceptive drift across participants currently does not display individual differences. This is due to the restriction of the visualization method to display more than three-dimensional data on a 2D surface. The data for the average area showing proprioceptive drift would contain the 2D position data, the probability of the participant's response, and the individual difference. To include individual differences, a visualization method that can show four-dimensional data would be required.

The future direction of this method is to expand the dimensions from 2D to 3D and to include temporal features. Although there are still some difficulties in visualization method for more than three-dimensional data, these expansions with additional dimensions will help to understand how the brain processes multisensory information from multiple modalities, such as vision and proprioception. To accomplish this expansion, further research is required.

From an application perspective, this method could help engineers to strategically design usability or feeling of control for real-time controlling systems, such as robots, surgical robot systems, and virtual reality systems. In these systems, the user's sensation during operation is often estimated by questionnaires ex post facto. Therefore, it is challenging to implement the user's sensation during operation when making a system prototype. If we could reveal the limitation of the spatiotemporal discrepancy between multisensory feedback to maintain the sense of ownership and agency, it would help to provide the rules about multisensory feedback from machine systems to maintain self-body-like usability, in which the operators control the systems as if they would control their own body. This method reveals the spatial offset between visual and proprioceptive feedback to maintain a sense of ownership and agency. Based on this result, the participant's feelings towards the mirrored hand image can be controlled by manipulating the distance between hand image and hidden real hand. Changing the spatiotemporal relationship across multisensory feedback from various actions would be the first step to strategically control the user's feeling during operation.

This idea can be adopted not only for engineering but also for rehabilitation. Several research efforts have aimed at improving motor functions and pain in people with missing or paralyzed limbs. However, since the main theme of traditional rehabilitation is for patients to regain function in their daily lives, little of this research has managed to improve the patients' feelings in a missing or paralyzed limb. Considering the impact on patients' quality of life, improvement of their sensations is crucial for living with a missing or paralyzed limb. This method can provide a quantitative estimation about the sense of ownership and agency in a missing or paralyzed limb by measuring the offset between the patients' subjective limb position and the position of the real or visible limb. This would help the physical therapist to quantitatively estimate the patients' feeling of their limbs.

To conclude, this paper provides a novel method to visualize the spatial offset between visual and proprioceptive feedback to maintain proprioceptive drift along the midsagittal plane. Based on previous research using this method¹⁵, the spatial offset that participants could not detect extended to approximately 10 cm. Moreover, this value matches the range where participants feel a sense of ownership and agency for the hand image in the mirror. These findings will help to further investigate underlying mechanisms of the sense of ownership and agency from a multimodality perspective of sensory information. Moreover, this method can provide a tool for the rehabilitation of missing or paralyzed limbs and for the design of real-time control systems, as the quantitative indicator shows the spatial offset between visual and proprioceptive feedback to maintain self-body-like sensation or usability. In this way, the development of computational techniques, such as machine learning, makes it possible to build estimation and visualization methods that were previously impossible due to the conventional statistical limitations. This type of metrological progress could reveal human behavior and brain mechanisms producing the sense of self in more natural situations.

Materials

Name	Company	Catalog Number	Comments
Acric mirror
Matte blackboard
custom-made stand			e.g. wood pole or PVC(poly vinyl chloride) pipe
Chair
Foot pedal	P.I. Engineering	Classic X-keys USB, and PS/2 Foot Pedals	Other response device can be avaliable.
Position sensor	CyVerse	SLC-C02	Other position sensor can be avaliable.
Custom-made retroreflectivemarker		The marker provided by the motion capture vendor can be available.
Noise canselling head phone	bose	Quiet Comfort 3	Other head phone can be avaliable.
PC	Mouse computer	NG-N-i300GA	Other PC can be available.