Capturing Representative Hand Use at Home Using Egocentric Video in Individuals with Upper Limb Impairment

Summary

A protocol is proposed to capture natural hand function of individuals with hand impairments during their daily routines using an egocentric camera. The goal of the protocol is to ensure that the recordings are representative of an individual's typical hand use during activities of daily living at home.

Abstract

Impaired hand function after neurological injuries can have a major impact on independence and quality of life. Most existing upper limb assessments are carried out in person, which is not always indicative of hand use in the community. Novel approaches to capture hand function in daily life are required to measure the true impact of rehabilitation interventions. Egocentric video combined with computer vision for automated analysis has been proposed to evaluate hand use at home. However, there are limitations to the duration of continuous recordings. We present a protocol designed to ensure that the videos obtained are representative of daily routines while respecting participant privacy.

A representative recording schedule is selected through a collaborative process between the researchers and participants, to ensure that the videos capture natural tasks and performance, while being useful for hand assessment. Use of the equipment and procedures is demonstrated to the participants. A total of 3 h of video recordings are scheduled over two weeks. To reduce privacy concerns, participants have full control to start and stop recordings, and the opportunity to edit the videos before returning them to the research team. Reminders are provided, as well as help calls and home visits if necessary.

The protocol was tested with 9 stroke survivors and 14 individuals with cervical spinal cord injury. The videos obtained contained a variety of activities, such as meal preparation, dishwashing, and knitting. An average of 3.11 ± 0.98 h of video were obtained. The recording periods varied from 12-69 d, due to illness or unexpected events in some cases. Data was successfully obtained from twenty-two out of 23 participants, with 6 participants requiring assistance from the investigators during the home recording period. The protocol was effective for collecting videos that contained valuable information about hand function at home after neurological injuries.

Introduction

Hand function is a determinant of independence and quality of life across clinical populations with upper limb impairments^1,2. Capturing the hand function of individuals with hand impairments at home is vital to evaluating the progress of their ability to carry out activities of daily living (ADLs) during and after rehabilitation. Most clinical hand function assessments are conducted in a clinical or laboratory environment, rather than at home^3,4. Existing clinical hand function assessments that seek to capture the impact on ADLs at home are questionnaires and rely on subjective self-reported ratings^5,6,7. An objective evaluation to assess the ultimate impact of rehabilitation on hand function at home is still unavailable.

In recent years, many wearable technologies have been developed and implemented to capture upper limb function in real-world environments. Wearable sensors such as accelerometers and inertial measurement units (IMUs) have been commonly used to measure upper limb movements in daily life. However, these devices typically do not distinguish whether the detected epochs belong to functional upper limb movements^8,9, defined as purposeful movements intended to complete a desired task. For example, some wearable sensors are sensitive to the presence of upper limb swings during walking, which is not a functional movement of the upper limb. Furthermore, although wrist-worn accelerometers capture upper limb movements, they cannot capture the details of hand function in real-world environments. Sensorized gloves allow capturing more detailed information about hand manipulations¹⁰, but they may be cumbersome for people whose hand function and sensation are already impaired. Wearable approaches have also been proposed to capture finger movements through magnetometry or finger-worn accelerometers^11,12,13, but the functional interpretation of those movements remains challenging¹⁴. Thus, although previously proposed wearable devices are small and convenient to use, they are insufficient to describe the details and functional context of hand use.

Wearable cameras have been proposed to fill these gaps and capture details of hand function during ADLs at home for neurorehabilitation applications^{15,16,17,18,19}. Automated analysis of egocentric videos using computer vision has considerable potential to quantify hand function in context, by providing information both about the hands themselves and about the tasks carried out in real ADLs²⁰. On the other hand, the duration of continuous recordings is typically limited to approximately 1 to 1.5 h by battery, storage, and comfort considerations. Here, within these constraints, we present an egocentric video collection protocol intended to obtain data that is both representative of an individual's daily life as well as informative for hand function evaluation.

Protocol

The study was approved by the Research Ethics Board of the University Health Network. Signed informed consent was obtained from each participant before enrollment in the study. Signed informed consent was also obtained from any caregivers or household members appearing in video recordings.

1. Verification of the protocol applicability to the individual

NOTE: This protocol is intended to be applied to individuals with impaired but not completely absent hand function (specific criteria can be adapted to the population and/or question of interest).

1. Ask participants whether their affected hands impact their ability to carry out ADLs.
   NOTE: It is recommended to ask participants to give some examples of tasks that they can and cannot perform independently with their affected hands.
2. Verify that the total score on the Montreal Cognitive Assessment (MoCA) is above 21, in order to avoid potential difficulties understanding and following protocol procedures.

2. Determination of the daily routine of participants

1. Ask participants to recall their daily routines over the past two weeks. Document which daily tasks are performed, for how long, and at approximately what time.
2. In collaboration with participants, select 3 timeslots of 1.5 h each during which to record videos. Select timeslots that are spread over different days of the week, and take place when ADLs involving the hands are typically carried out in sequence.
   NOTE: The selected ADLs must be representative of each participant's typical activities, and be perceived by them as meaningful. Scheduling recording periods on different days is intended to increase the variety of recorded ADLs and to promote the capture of useful and meaningful data.
   ​NOTE: Recording timeslots are scheduled for recording efficiency, but participants should understand that they have full control of when to start and to stop recordings.

3. Agreement on recording schedules and target video content with participants

1. Obtain agreement of each participant on the recording schedules, after discussing any concerns that they may have.
2. Set a goal of 3 h of videos over two weeks. Inform participants that insufficient videos may lead to extending their recording periods.

4. Emphasis of the importance of performing ADLs naturally

1. Instruct participants to focus on capturing realistic routines, instead of specifying particular activities to record. The intent of the instruction is to discourage participants from artificially recording specific activities in greater amounts than is typical for them.

5. Notification of potential privacy issues during recordings at home

1. Ensure that participants understand that all recordings should take place inside their homes, not in public places to avoid privacy issues.
2. Give some examples that may raise privacy concerns, such as bathing, dressing/undressing, and checking confidential information. Remind participants to be aware of mirrors, which may show their faces in the recordings.
3. Suggest that participants avoid the presence of other people such as family members or caregivers as much as possible in the videos.
   ​NOTE: In the context of research studies, in cases where the presence of other people is unavoidable, informed consent should be obtained from those individuals.

6. Camera and tablet instruction

NOTE: If participants indicate during the initial contact that they require caregiver assistance for many of their daily needs, the caregiver is encouraged to also attend the study visit and be trained on the use of the equipment, so that they can later assist the participant.

1. Demonstrate how to use an egocentric camera (Table of Materials) to participants.
   1. Demonstrate how to turn the camera on and off.
   2. Demonstrate how to control recordings (start, pause, stop) using the camera.
2. Demonstrate how to use a tablet (Table of Materials) with the preinstalled camera app to control the recordings, if applicable.
   ​NOTE: The demonstration includes controlling the recordings from the camera app, as well as replaying and editing (e.g., trim or delete) the recorded videos. A camera remote was initially considered (Supplemental Files), but in practice was not used because participants were comfortable using the camera or tablet to start and stop recordings.
   1. Demonstrate how to turn on and off the tablet.
   2. Demonstrate how to connect the tablet to the camera through the camera app.
   3. Demonstrate how to control the recordings from the camera app.
   4. Demonstrate how to review recorded videos from the camera app.
   5. Demonstrate how to trim or delete the videos from the camera app.
3. Demonstrate how to don and doff the camera using an elastic headband adjustable to the participant's head.
   NOTE: See Figure 1.
   
   Figure 1. Wearable camera setup. (A) Positioning of the egocentric camera. (B) Viewing angle from the camera. Please click here to view a larger version of this figure.
   1. Place the camera on the participant's forehead. Adjust the headband to wear the camera comfortably and steadily.
   2. Ensure an optimal angle of the camera with respect to the forehead.
   3. Ask participants to record a short segment of video while moving the hands in front of them and manipulating an object (e.g., the tablet).
   4. Review the recorded video and ensure the two hands were clearly visible in the central region of the scene while conducting manipulation tasks.
   5. Practice the use of the camera and tablet with participants and their caregivers, until they demonstrate proficiency.

7. Giving the equipment

1. Give the kit with all the equipment to participants to record their ADLs at home. In addition to the camera and tablet, the kit includes extra camera batteries, battery chargers for both camera and tablet, charging cables, headband for the camera, and a printed set of guidelines for using the camera (See the Supplemental Material).

8. Experimental troubleshooting and followup

1. Provide contact information of the researchers to help solve obstacles during the actual recordings at home. After one week, researchers call the participants to document the recording progress and solve any potential technical issues.

9. Retrieval of equipment and videos

1. Retreive all the equipment and videos from participants in person or through pre-paid mail parcels.
2. Ensure that participants agree to share all the videos returned. Participants are encouraged to review all of the collected videos before returning them to the research team, and to delete any portions that they do not wish to share.
3. For research studies, review the returned videos and check if anyone appears in the video without having given their consent. If so, send consent forms or call the individuals who appear in the videos to gain their consents for use of the videos. If the individuals are not reachable, the portions of the videos in which they appear are deleted by the researchers.

Results

Participant demographics and inclusion criteria
A sample of 23 participants was recruited for these studies: 9 stroke survivors (6 men, 3 women) and 14 individuals with cSCI (12 men, 2 women). Summary demographic and clinical information for the recruited sample are reported in Table 1.

Age (years)

Discussion

We presented a protocol for recording videos of ADLs at home using wearable cameras in individuals with upper limb impairments, such as cSCI and stroke. The protocol is flexible and can be goal-directed to capture hand function performance in specific ADLs or to track the progress of rehabilitation remotely in people living at home. The egocentric vision paradigm has great potential for remote monitoring of hand function in individuals living in the community, and for optimizing rehabilitation once people are discharged ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Egocentric camera	GoPro Inc., CA, USA	GoPro Hero 4 and 5	A camera that records from a first-person angle.
Battery chager and batteries	GoPro Inc., CA, USA	MAX Dual Battery Charger + Battery	Extra batteries for the camera and battery charger
Camera charger	GoPro Inc., CA, USA	Supercharger	This charger is connected to the camera directly without disassembling the camera frame.
Camera frame	GoPro Inc., CA, USA	The Frame	The hinge of the camera frame can be used to adjust the angle of view of the camera.
Headband for the camera	GoPro Inc., CA, USA	Head Strap + QuickClip
SD card	SanDisk, CA, USA	32GB microSD
Tablet	ASUSTeK Computer Inc., Taiwan	ZenPad 8.0 Z380M	The tablet is installed with the GoPro App in order to connect with the camera.