An Objective and Child-friendly Assessment of Arm Function by Using a 3-D Sensor

Summary

An objective measure of muscle functions is challenging especially in children. Based on a commercially available digital 3-D sensor, a child-friendly gaming test was developed to assess upper limb function for clinical trials.

Abstract

Progressive and irreversible muscle atrophy characterizes Spinal Muscular Atrophy (SMA) and other similar muscle disorder diseases. Objective assessment of muscle functions is an essential and important, although challenging, prerequisite for successful clinical trials. Current clinical rating scales restrain the movement abnormalities to certain predefined coarse-grained individual items. The Kinect 3-D sensor has emerged as a low-cost and portable motion sensing technology used to capture and track people's movement in many medical and research fields. A novel approach using this 3-D sensor was developed and a game-like test was designed to objectively measure the upper limb function of patients with SMA. The prototype test targeted joint movement capability. While sitting in a virtual scene, the patient was instructed to extend, flex, and lift the whole arm in order to reach and place some objects. Both kinematic and spatiotemporal characteristics of upper limb movement were extracted and analyzed, e.g., elbow extension and flexion angles, hand velocity, and acceleration. The first study included a small cohort of 18 ambulant SMA patients and 19 age- and gender-matched healthy controls. A comprehensive analysis of arm movement was achieved; however, no significant difference between the groups were found due to the mismatch of patient's capability and the test difficulty. Based on this experience, a second version of the test consisting of a modified version of the first game with increased difficulties and a second game targeting muscle endurance were designed and implemented. The new test has not been conducted in any patient groups yet. Our work has demonstrated the potential capability of the 3-D sensor in assessing such muscle function and suggested an objective approach to complement the clinical rating scales.

Introduction

Comprehensive assessment of muscle function is a critical evaluation in many neuromuscular diseases and an important prerequisite for successful clinical trials. Clinical rating scales are used increasingly as a standardized assessment tool and as a well-established outcome measure¹. However, they rely heavily on the subjective judgment of the clinicians and might result in substantial variations leading to inter- and intra-rater inconsistency² or generated numbers that do not meet the criteria for precise measurements¹. In addition, many neuromuscular diseases largely affect children and most of these rating scales are lengthy and boring, which impose extra challenges. An example of a neuromuscular disease is Spinal Muscular Atrophy (SMA), which is a fatal neuromuscular disease characterized by progressive muscle weakness³. Depending on the clinical phenotypes, some patients live with wheelchairs (type 2), and some can stand and walk unaided (type 3)⁴. There is an increasing demand for a more sensitive and objective assessment tool to measure the muscle function in terms of joint movement range, muscle strength, muscle fatigue and so on, in order to track disease progression and drug efficacy.

The rapid progress in motion sensing technology has made it possible to analyze the movement characteristics at comparatively low costs, among which the value of the 3-D sensor (Kinect) in capturing whole-body movement in a marker-free way has been widely examined. By using the integrated infrared sensor and the implemented machine learning algorithms, the body locations of the tracked persons are inferred through the 3-dimensional positions of 20 anatomical landmarks called body joints or points including head, neck, hands, wrists, elbows, shoulders, spine, hips, knees, ankles, and feet⁵. The temporal resolution is up to 30 Hz, which is sufficient for most physical movement except for some pathological motor symptoms such as tremors. The spatial accuracy of the 3-D sensor has been validated extensively with the ground truth⁶ or the gold standard, which is a marker-based 3-dimensional motion analysis system^{7,8,9,10,11,12}. Good concurrent validity and reproducibility have been revealed by different tests, especially from the frontal view¹² and for gross movements⁷. In order to objectively assess the upper limb function for children with SMA, we designed and implemented a game-like test based on a 3-D sensor to measure the capability of joint movement.

Protocol

The test was performed in an observational, longitudinal study at University Children's Hospital of Basel (UKBB) with both adults and children. More demographics and clinical information about the study can be found in the previous publication^13,14. All the procedures have been approved by the local ethics committee Ethikkomission UKBB and conducted according to the principles expressed in the Declaration of Helsinki. Written informed consent and compliance with the study protocol according to International Conference on Harmonisation (ICH) and local regulations were provided by all the subjects or the legally authorized representatives for children below the age of legal consent.

NOTE: A prototype game "Wardrobe" aimed at measuring the joints' range of movement was designed and implemented. It was then exported as an executable application that can run on any Windows 8 or higher operating system, as long as the necessary 3-D sensor drivers are also installed. The prototype game based on Kinect sensor v1 or the second version (see Discussion) based on Kinect sensor v2 can be provided upon request (see Supplemental Coding Files).

1. Setup and preparation for the test

1. Install the 3-D sensor (e.g., Kinect) drivers and the designed application on the computer.
2. Place the computer on a suitable surface, like a table, with a height of 0.5 - 1 meter.
3. Place the 3-D device also on the same surface aligned to the middle of the computer, so that the height of the 3-D camera is fixed by the table about 0.5 - 1 meter above the ground. Adjust the elevation angle of the 3-D sensor manually as needed to correctly capture the subject (see 2.4.2 below).
4. Connect all the cables properly, including the 3-D adapter to the computer and the power cable to the power supply.
5. Place a height-adjustable chair in front of the table with the computer and the 3-D sensor at about 2 meters.

2. Conducting the test

1. Start the computer and turn on the sound to an optimal volume. If automatic data transmission is desired, make sure the Internet is connected.
2. Instruct the subject to sit on the chair.
3. Start the application on the computer and enter the subject ID on the first page.
4. Click on the "START" button on the first page to enter into the second page, which shows the "Wardrobe" game:
   1. Note the skeleton figure (upper body only) on the screen, which represents the body of the subject in front of a big wardrobe. When the figure is not seen, instruct the subject to wave and move forwards and backwards if he is in a wheelchair or stand up until the 3-D sensor captures the person.
   2. Read the instructions on the screen and adjust the distance, height, and lateral position of the chair and the subject accordingly until all the instructions are in green font.
      NOTE: The distance, height, and lateral position of the chair can be adjusted by the subject himself or a helper who should step out of the field afterwards. If the optimal height cannot be achieved, manually adjust the elevation angle of the 3-D camera. The optimal position is 2 meters away from the device, with the neck of the subject displayed in the middle of the screen.
   3. Press the TRAIN button to start the training session without data recording.
      NOTE: If the subject is already familiar with the game and knows what to do, press the START button directly and skip the training.
   4. Let the subject follow the instructions on the screen and perform the following movements:
      1. Extend the requested arm (right or left) to reach the flickering virtual object until the object is in the virtual hand.
      2. Flex the same arm and physically touch the indicated points on the body in order to place the object.
      3. Continue the arm extension and flexion movements until all 20 objects are reached and placed.
         ​NOTE: There are two rounds with 10 objects in each round. Within a round there are 5 objects on the right side that require the right arm followed by 5 objects on the left side for which the left arm is asked. This concludes the training session.
   5. Press the "START" button to perform the same tasks again from the beginning, as in step 2.4.3, but this time recording the locations of the 9 upper body points into an encrypted file. The game automatically finishes by either placing all the 20 objects or when the pre-defined time (4 min) runs out.
      NOTE: It is also possible to quit the game at any time in case the subject is too weak or any other situations happen.
5. Enter into the last page automatically which shows one spider plot indicating the joints' ranges from the "Wardrobe" game.
6. Exit the game by pressing the END button or repeat the test by pressing the REPEAT button.

3. After the test - data handling

1. Save the recorded and encrypted data and log files on the hard disk, and optionally transfer via internet to a data analyst. A detailed description of data handling and analysis has been provided in a previous publication¹³.
   1. Confirm the completeness and validity of the data by checking the log file and comparing clinical reports with the 3-D records.
   2. Compute and extract 27 movement characteristics as numeric features, like joint angles, velocity and acceleration, hand path length, body compensation movement, reachable space, and so on.
   3. Calculate, plot and select the features in order to understand and interpret the 3-D sensor data as well as for the following statistical analysis.
   4. Perform PCA (Principal Component Analysis), ANOVA (analysis of variance), correlation analysis and linear mixed effect models as statistical analysis.

Representative Results

Using the procedure presented above, different movement features are plotted and analysis approaches are explored in order to gain a comprehensive understanding of the movements.

In Figure 1, the traces from the 9 upper body points are plotted restricting to the X and Y axes which represents a 2-dimentional projection (X represents horizontal position of the subject, Y represents the vertical position while Z is the distance to the 3-D sensor). Figure 1 shows what the 3-D sensor actually measures, the spatial locations of body points. By juxtaposing one SMA patient and one healthy control over the 4 visits, it is shown that both the patient and the control completed the tasks with clearly recognizable trajectories. There were certain subject-dependent characteristics that stayed consistent over all the visits, e.g., the trajectory of the hands of the patient. In comparison, the control subject had relatively less trunk movement over time, which is indicated by the smaller crowds of red, yellow, and pink lines (the head, neck, and torso points, respectively). Neither of the two subjects showed any unsymmetrical movement features.

Figure 2 shows some representative features extracted from the time series locations of the body points. Compared to Figure 1, Figure 2 gives an idea of what potential information could be extracted and visualized from the raw 3-D sensor data in order to understand the underlying movement characteristics. Figure 2a is a segmented hands trace plot indicating elbow extension and flexion phases of two rounds from a single subject. The trajectory resembles itself quite well between the two rounds. For the three lower objects, both hands tend to overreach, but this is not the case for the upper two objects. Figure 2b plots the histogram of the hand velocities (while the hand is moving; the resting state is excluded). There is no significant difference between the left and right hands for this subject. Figure 2c shows the total path lengths of the head, neck, and torso points, which can be considered as a kind of trunk compensation movement per object position. For objects 3 and 8, which are placed at the higher corner, the trunk movement is relatively greater compared to the other positions. Moreover, the head has moved more obviously than the other two points. Figures 2d, 2e, and 2f are boxplots showing the overall distribution from all the 4 visits for some features. Only the first seven patients with all 4 visits are shown for clarity. Figure 2d shows the total hand path length divided by the individual arm length measured by the sensor in order to compensate for the inter-subject arm length difference. During the movements carried out to reach and place objects, the hand path length is around twice the arm length for one object; therefore, for 20 objects the total path length is around 40 times the arm length. Obvious inter-subject differences are observed, for example, in trunk compensation movement between patients 2 and 3 (Figure 2e) or in median hand velocity between patients 1 and 3 (Figure 2f).

Full clinical related analysis and results can be found in our earlier publication¹³.

Figure 1: Representative trace plots of the 9 upper body points during the test. The top 4 plots are from one SMA patient and the bottom 4 plots are from one healthy control. This figure has been modified from our previous publication¹³. Please click here to view a larger version of this figure.

Figure 2: Representative summary analysis results: (a), (b), (c) are from a single subject and (d), (e), (f) are the summary from the first 7 SMA patients with complete 4 visits. In (d), (e), and (f), the bottom and top of the box are the first and third quartiles, and the horizontal line inside the box is the median. The lengths of the whiskers (extending vertical lines from the box) are defined as the lowest point within 1.5 interquartile ranges (IQR) of the lower quartile, and the highest point within 1.5 IQR of the upper quartile. The diamonds represent the outliers outside the whiskers. (a) Segmented hands trace plot. Solid lines represent the elbow extension phase when the hands reach out for the objects while dashed lines represent the elbow flexion phase when the hands place the objects on the body. Each color represents the objects at the same position from the two rounds. (b) Histogram of hand velocity during the extension and flexion movements. (c) Total compensation movement lengths from head, neck, and torso points for each individual object. (d) Boxplot of total hand trajectory path length for right and left hands respectively over 4 visits. The y axis shows the total hand path lengths divided by the individual arm lengths. (e) Boxplot of total trunk compensation movement including head, neck, and torso points for movement from right and left side movement respectively over 4 visits. (f) Boxplot of median hand velocity during the movement for right and left hands respectively over 4 visits. Please click here to view a larger version of this figure.

Discussion

The proposed Kinect-based assessment provided an objective and comprehensive movement analysis while providing a child-friendly, low-cost, and portable solution with respect to the traditional clinical rating scales or sophisticated marker-based video systems. With a gaming test that lasted less than 5 minutes, multiple body points were intensively examined at the same time, and many spatiotemporal and kinematic characteristics were analyzed with high accuracy including velocity, joint angles, and so on. The entire set-up and training efforts were also much less demanding compared to the rating scales or video systems.

The most critical step in this approach was the test design. In order to capture the underlying movement symptoms, the designed task should cover the corresponding capability spectrum and avoid floor or ceiling effects. In the example of this specific indication, the common physical symptoms of SMA include muscle weakness, limited joint range, muscle stiffness, fatigue, and so on. The proposed test is appropriate for these symptoms involving the limited joint range, which was typical for SMA type 2 patients. Unfortunately, the prototype was only tested as a first attempt in a planned study that recruited only SMA type 3 patients. Since the capability of those patients were above what the current test could measure (ceiling effect), the desired results could not be achieved. For this ambulant patient group, muscle fatigue and body transfer would be a better measure.

Based on this experience, a second version of the test consisting of a modified version of the "Wardrobe" game and an additional "Boat-Rowing" game were designed. The modified version of the "Wardrobe" game has three difficulty levels. In the first level, the objects are placed near the body so that the subject does not need to fully extend the arm, which targets the weak patient group, who can only sit in the wheelchair and cannot fully extend their arms without support. In the second level, the objects are placed at the distance of whole arm length, which targets the patient group who can lift and extend the arms without any support. In the most difficult level, the objects are placed slightly out of the arm range; therefore, the subject needs to move the upper trunk as an extension. The third level targets the ambulant patient group where the axial and proximal movement is also measured. During the positioning phase of the game, the arm length of the subject is automatically measured and then used to calculate the locations of the objects in the following game; therefore, the difficulty levels are automatically adjusted to the individual's ability. When the capability limit of each individual is reached and the objects cannot be reached or placed, the level will be skipped either automatically after a certain time or manually by the operator by pressing the "SKIP" button on the screen. The "Boat-Rowing" game targets muscle endurance and it requires the subject to repeat an arm rolling movement as fast as possible for 1 minute. In a future study, the second version is intended to cover the patient spectrum from SMA type 2 to type 3, since the tasks measure the capability of patients with limited arm movement ability to patients with full arm function and limited axial movement.

The other side of the test design is the consideration of the floor effect. Due to the limited spatial and temporal resolution, the 3-D sensor is only able to capture accurately gross movements like walking, arm waving, and so on. For fine movement detection, including figure tapping or hand turning, more sensitive digital devices such as mobile phones or wrist wearables are required. As discussed, the key for the success of such an application is to build up the correct match between underlying disease symptoms, device capabilities and designed task.

Some other considerations during the test design include age group, learning effect, languages, and so on. Since SMA affects mainly children, the test should be as simple and clear as possible while maintaining an attractive gamification trait. In our design, cartoon figures and hand-drawn objects were used. The task imitated the self-dressing behavior which is usually acquired by children after two or three years of age. The movements were kept simple such that subjects could understand and perform the test after a short training phase and learning effects were avoided, which was measured and discussed in our previous publication¹³.

When performing the test protocol and the data analysis, some issues might arise similarly to other 3-D sensor applications. These issues include sunshine interference, special clothes, more than one subject in the field of view, and irregular sampling times. We found one case where a subject in black clothes was not detected by the 3-D sensor in a sunny room, even when the subject was not directly in the sunshine. When more than one subject appears and disappears from the field of view, the assignment of ID numbers to detected skeletons might jump, which burdens the analysis. Even though the 3-D sensor outputs the signal at the frequency of 30 Hz in theory, the actual output may have gaps of up to one hundred ms. Therefore, it is important to track and export the timestamp.

Our first test was performed on the first version of the 3-D sensor, which has currently been replaced by a second version, and our modified version is implemented based on this second version. Between versions, the underlying drivers are different, and also the application interface (API) has changed. There are no other significant differences when migrating the application. Since both application versions can be provided freely upon request by the authors and the sensor drivers can be downloaded from the Kinect website, this is no concern to the user.

Using the 3-D sensor, we developed an innovative, quantitative, and objective upper extremity function assessment tool incorporating children-friendly game-like technology. The feasibility was explored and analyzed. Our work demonstrated the potential power of 3-D sensor as an alternative and complementary approach to movement assessment.

Materials

Name	Company	Catalog Number	Comments
Microsoft Kinect for Windows v1 sensor	Microsoft	N/A	The first version of the test was developed on Kinect sensor v1 which is not sold any more. But the second version was developed on the Kinect sensor v2 which can be tested in a similar way by using Microsoft Kinect for Windows v2 sensor (GT3-00003) together with Microsoft Kinect Adapter (9J7-00009)
DELL XPS 2720 All-In-One PC with windows 8 operating system, 16G RAM, Intel Core i7 and 64-bits	DELL	N/A	In our setup, a All-in-one computer was used, but in fact any laptop or computer which fullfills the following requirements and a big screen for the subjects to see will work: windows 8 or higher operating system; 64-bit processor; dual-core 3.2 GHz or faster processor; dedicated USB 3.0 bus; 2 GB RAM