Web-based Clinician Guide to Record Compatible Video of Standardized Drinking Task Kinematics for Computer Vision Analysis

Our research explores precision rehabilitation of the upper limb, and we really focus on using affordable tools that people can access, like video cameras, wearable sensors, and then open-source AI.So technologies we currently use include AI technology, like Computer Vision, and then also sensor technology, which would be, like, video cameras and wearable sensors. And these technologies have really been game changers for our field because they let us measure recovery more frequently, more objectively, and potentially with finer grain resolution to capture things that we currently may be missing. To begin, position an adjustable height chair with a seat back so the subject can sit with palms resting face down on the table and the wrist crease aligned to the table edge.

Adjust the seat height to achieve approximately 90 degrees of flexion at both hips and knees. Modify the table height so that the subject maintains an upright posture with arms resting comfortably at the sides and elbows flexed at 90 degrees. Align the front edge of the place mat with the table edge directly in front of the subject.

Ensure that the target square box on the place mat is centered with respect to the subject's midline. Then place a 250-milliliter plastic cup in the square box on the place mat, locating it 30 centimeters from the table edge and directly in front of the subject. Fill the cup with 100 milliliters of drinking water.

Now position the stereo camera to capture a frontal overhead view of the subject and cup. Adjust the camera position to ensure the target volume is fully visible. For calibration, Open the Calibration Activity module by navigating to the web application dashboard and selecting the Calibration Activity Guide from the menu.

Before starting the recording, hold the checkerboard pattern at approximately chest level, perpendicular to the tabletop surface. Initiate recording by clicking the Start Recording icon in the web application. When queued to begin the calibration activity, move the checkerboard pattern forward toward the camera in a smooth and controlled manner.

Then reverse the motion and move the checkerboard pattern backward, away from the camera, maintaining smooth and controlled motion. Repeat the forward and backward movement until image capture is complete. Review the recorded calibration video.

Ensure the entire checkerboard pattern remains visible throughout the video, and that the lighting provides clear visibility. If the recording is unsatisfactory, click the Retry option to reinitiate the video recording for another attempt. Upon successful calibration, click the Save Calibration Images icon.

When the Save prompt appears, choose the desired destination folder to store the calibration images. After confirming the images have been saved, select Confirm Save to proceed. Instruct the subject to use a specified hand, either right or left, for the initial recording.

Ask the subject to practice the Drinking Task activity to become familiar with the procedure. Open the Drinking Task activity module by navigating to the dashboard menu of the web application and selecting the Drinking Task Activity Guide. Before recording, inform the subject about the three-second visual and audio countdown provided by the app, followed by a cue to begin the task.

Initiate the video recording by clicking the Start Recording icon, and instruct the subject to perform the drinking task when cued. When cued to begin the drinking task, let the subject perform the activity using the specified hand. Then instruct them to return the cup to the original outlined area, and the hand to the original starting position.

Review the recorded video of the drinking task activity. Ensure that the recorded video shows the subject's face, torso, upper limbs, and the cup within all frames. Verify that the subject completes the entire drinking task, picking up the cup, taking a sip, returning the cup to its original position, and moving their body and hands back to the start position.

To conduct additional trials with the same subject, select the Repeat Another Trial option and continue recording and reviewing new trials using the same instructions. After all desired trials have been recorded, select the Finished with All Trials option to initiate video file saving. In the Notes field, enter comments relevant to future data processing, analysis or interpretation.

Then select Package and Add Notes to proceed. Click the Save Drinking Task Videos icon and choose a folder destination when the prompt appears. Once the video file save is complete, click Confirm Save to finalize.

The Computer Vision workflow extracted three-dimensional posed data from two-dimensional videos by applying calibration, pose detection, lifting procedures, and smoothing filters. Using the extracted 3D pose data, kinematic metrics were computed, including number of movement units, trunk displacement, and movement time. The wrist velocity profiles during the drinking task showed different numbers of movement units between the left and right upper extremity in a subject with chronic post-stroke hemiparesis.

Upper limb biomechanics no longer requires the complex multi-camera setups that have historically been required for these measurements. And we've shown that a simpler approach, using things like video cameras and Computer Vision, can enable us to get those same measurements with equal validity. So this work improves access to upper limb biomechanics, and our real goal is to accelerate research of precision rehabilitation and also to potentiate the translation of this work to clinic, for patients.

So my future research will really focus on upper limb biomechanics of the hand with fine motor skills. And our additional goal then is to develop interventions that use biomechanical feedback to help improve patient's outcome.