Capturing Representative Hand Use at Home Using Egocentric Video in Individuals with Upper Limb Impairment

Summary

A protocol is proposed to capture natural hand function of individuals with hand impairments during their daily routines using an egocentric camera. The goal of the protocol is to ensure that the recordings are representative of an individual's typical hand use during activities of daily living at home.

Abstract

Impaired hand function after neurological injuries can have a major impact on independence and quality of life. Most existing upper limb assessments are carried out in person, which is not always indicative of hand use in the community. Novel approaches to capture hand function in daily life are required to measure the true impact of rehabilitation interventions. Egocentric video combined with computer vision for automated analysis has been proposed to evaluate hand use at home. However, there are limitations to the duration of continuous recordings. We present a protocol designed to ensure that the videos obtained are representative of daily routines while respecting participant privacy.

A representative recording schedule is selected through a collaborative process between the researchers and participants, to ensure that the videos capture natural tasks and performance, while being useful for hand assessment. Use of the equipment and procedures is demonstrated to the participants. A total of 3 h of video recordings are scheduled over two weeks. To reduce privacy concerns, participants have full control to start and stop recordings, and the opportunity to edit the videos before returning them to the research team. Reminders are provided, as well as help calls and home visits if necessary.

The protocol was tested with 9 stroke survivors and 14 individuals with cervical spinal cord injury. The videos obtained contained a variety of activities, such as meal preparation, dishwashing, and knitting. An average of 3.11 ± 0.98 h of video were obtained. The recording periods varied from 12-69 d, due to illness or unexpected events in some cases. Data was successfully obtained from twenty-two out of 23 participants, with 6 participants requiring assistance from the investigators during the home recording period. The protocol was effective for collecting videos that contained valuable information about hand function at home after neurological injuries.

Introduction

Hand function is a determinant of independence and quality of life across clinical populations with upper limb impairments^1,2. Capturing the hand function of individuals with hand impairments at home is vital to evaluating the progress of their ability to carry out activities of daily living (ADLs) during and after rehabilitation. Most clinical hand function assessments are conducted in a clinical or laboratory environment, rather than at home^3,4. Existing clinical hand function assessments that seek to capture the impact on ADLs at home are questionnaires and rely on subjective self-reported ratings^5,6,7. An objective evaluation to assess the ultimate impact of rehabilitation on hand function at home is still unavailable.

In recent years, many wearable technologies have been developed and implemented to capture upper limb function in real-world environments. Wearable sensors such as accelerometers and inertial measurement units (IMUs) have been commonly used to measure upper limb movements in daily life. However, these devices typically do not distinguish whether the detected epochs belong to functional upper limb movements^8,9, defined as purposeful movements intended to complete a desired task. For example, some wearable sensors are sensitive to the presence of upper limb swings during walking, which is not a functional movement of the upper limb. Furthermore, although wrist-worn accelerometers capture upper limb movements, they cannot capture the details of hand function in real-world environments. Sensorized gloves allow capturing more detailed information about hand manipulations¹⁰, but they may be cumbersome for people whose hand function and sensation are already impaired. Wearable approaches have also been proposed to capture finger movements through magnetometry or finger-worn accelerometers^11,12,13, but the functional interpretation of those movements remains challenging¹⁴. Thus, although previously proposed wearable devices are small and convenient to use, they are insufficient to describe the details and functional context of hand use.

Wearable cameras have been proposed to fill these gaps and capture details of hand function during ADLs at home for neurorehabilitation applications^{15,16,17,18,19}. Automated analysis of egocentric videos using computer vision has considerable potential to quantify hand function in context, by providing information both about the hands themselves and about the tasks carried out in real ADLs²⁰. On the other hand, the duration of continuous recordings is typically limited to approximately 1 to 1.5 h by battery, storage, and comfort considerations. Here, within these constraints, we present an egocentric video collection protocol intended to obtain data that is both representative of an individual's daily life as well as informative for hand function evaluation.

Protocol

The study was approved by the Research Ethics Board of the University Health Network. Signed informed consent was obtained from each participant before enrollment in the study. Signed informed consent was also obtained from any caregivers or household members appearing in video recordings.

1. Verification of the protocol applicability to the individual

NOTE: This protocol is intended to be applied to individuals with impaired but not completely absent hand function (specific criteria can be adapted to the population and/or question of interest).

1. Ask participants whether their affected hands impact their ability to carry out ADLs.
   NOTE: It is recommended to ask participants to give some examples of tasks that they can and cannot perform independently with their affected hands.
2. Verify that the total score on the Montreal Cognitive Assessment (MoCA) is above 21, in order to avoid potential difficulties understanding and following protocol procedures.

2. Determination of the daily routine of participants

1. Ask participants to recall their daily routines over the past two weeks. Document which daily tasks are performed, for how long, and at approximately what time.
2. In collaboration with participants, select 3 timeslots of 1.5 h each during which to record videos. Select timeslots that are spread over different days of the week, and take place when ADLs involving the hands are typically carried out in sequence.
   NOTE: The selected ADLs must be representative of each participant's typical activities, and be perceived by them as meaningful. Scheduling recording periods on different days is intended to increase the variety of recorded ADLs and to promote the capture of useful and meaningful data.
   ​NOTE: Recording timeslots are scheduled for recording efficiency, but participants should understand that they have full control of when to start and to stop recordings.

3. Agreement on recording schedules and target video content with participants

1. Obtain agreement of each participant on the recording schedules, after discussing any concerns that they may have.
2. Set a goal of 3 h of videos over two weeks. Inform participants that insufficient videos may lead to extending their recording periods.

4. Emphasis of the importance of performing ADLs naturally

1. Instruct participants to focus on capturing realistic routines, instead of specifying particular activities to record. The intent of the instruction is to discourage participants from artificially recording specific activities in greater amounts than is typical for them.

5. Notification of potential privacy issues during recordings at home

1. Ensure that participants understand that all recordings should take place inside their homes, not in public places to avoid privacy issues.
2. Give some examples that may raise privacy concerns, such as bathing, dressing/undressing, and checking confidential information. Remind participants to be aware of mirrors, which may show their faces in the recordings.
3. Suggest that participants avoid the presence of other people such as family members or caregivers as much as possible in the videos.
   ​NOTE: In the context of research studies, in cases where the presence of other people is unavoidable, informed consent should be obtained from those individuals.

6. Camera and tablet instruction

NOTE: If participants indicate during the initial contact that they require caregiver assistance for many of their daily needs, the caregiver is encouraged to also attend the study visit and be trained on the use of the equipment, so that they can later assist the participant.

1. Demonstrate how to use an egocentric camera (Table of Materials) to participants.
   1. Demonstrate how to turn the camera on and off.
   2. Demonstrate how to control recordings (start, pause, stop) using the camera.
2. Demonstrate how to use a tablet (Table of Materials) with the preinstalled camera app to control the recordings, if applicable.
   ​NOTE: The demonstration includes controlling the recordings from the camera app, as well as replaying and editing (e.g., trim or delete) the recorded videos. A camera remote was initially considered (Supplemental Files), but in practice was not used because participants were comfortable using the camera or tablet to start and stop recordings.
   1. Demonstrate how to turn on and off the tablet.
   2. Demonstrate how to connect the tablet to the camera through the camera app.
   3. Demonstrate how to control the recordings from the camera app.
   4. Demonstrate how to review recorded videos from the camera app.
   5. Demonstrate how to trim or delete the videos from the camera app.
3. Demonstrate how to don and doff the camera using an elastic headband adjustable to the participant's head.
   NOTE: See Figure 1.
   
   Figure 1. Wearable camera setup. (A) Positioning of the egocentric camera. (B) Viewing angle from the camera. Please click here to view a larger version of this figure.
   1. Place the camera on the participant's forehead. Adjust the headband to wear the camera comfortably and steadily.
   2. Ensure an optimal angle of the camera with respect to the forehead.
   3. Ask participants to record a short segment of video while moving the hands in front of them and manipulating an object (e.g., the tablet).
   4. Review the recorded video and ensure the two hands were clearly visible in the central region of the scene while conducting manipulation tasks.
   5. Practice the use of the camera and tablet with participants and their caregivers, until they demonstrate proficiency.

7. Giving the equipment

1. Give the kit with all the equipment to participants to record their ADLs at home. In addition to the camera and tablet, the kit includes extra camera batteries, battery chargers for both camera and tablet, charging cables, headband for the camera, and a printed set of guidelines for using the camera (See the Supplemental Material).

8. Experimental troubleshooting and followup

1. Provide contact information of the researchers to help solve obstacles during the actual recordings at home. After one week, researchers call the participants to document the recording progress and solve any potential technical issues.

9. Retrieval of equipment and videos

1. Retreive all the equipment and videos from participants in person or through pre-paid mail parcels.
2. Ensure that participants agree to share all the videos returned. Participants are encouraged to review all of the collected videos before returning them to the research team, and to delete any portions that they do not wish to share.
3. For research studies, review the returned videos and check if anyone appears in the video without having given their consent. If so, send consent forms or call the individuals who appear in the videos to gain their consents for use of the videos. If the individuals are not reachable, the portions of the videos in which they appear are deleted by the researchers.

Representative Results

Participant demographics and inclusion criteria
A sample of 23 participants was recruited for these studies: 9 stroke survivors (6 men, 3 women) and 14 individuals with cSCI (12 men, 2 women). Summary demographic and clinical information for the recruited sample are reported in Table 1.

	Age (years)	Duration after injury (months)	Etiology	Level of Injury		Upper Limb Function Assessment (Mean ± SD)
				Level of Injury	AIS	UEMS: Right Hand	UEMS: Left Hand
cSCI (N=14)	55.9 ± 7.1	56.4 ± 58.9	12 Traumatic	C4 – C8	A – D	18.1 ± 6.2	18.6 ± 6.4
			2 Non-traumatic
Stroke (N=9)	56.8 ± 19.3	94.4 ± 134.4	5 Ischemic			FMA-UE: Affected Hand	ARAT: Affected Hand
			4 Hemorrhagic			45.6 ± 17.3	37.1 ± 19.1
AIS: ASIA Impairment Scale; UEMS: Upper Extremity Motor Score
FMA-UE: Fugl-Meyer Assessment for Upper Extremity; ARAT: Action Research Arm Test

Table 1. Demographic and clinical information of the recruited participants.

The mutual inclusion criteria for both groups in these studies were: 1) over 18 years old, 2) impaired but not absent hand function, 3) absence of other neuromusculoskeletal disease affecting upper limb movements, 4) no deformity of upper limbs joints, and 5) absence of pain when moving upper limbs. Additional inclusion criteria for each of the two groups were as follows.

For individuals with cSCI: 1) neurological level of injury between C4 and C8 according the International Standards for the Neurological Classification of Spinal Cord Injury (ISNCSCI), 2) American Spinal Injury Association Impairment Scale (AIS) grade A-D. 3) Traumatic or non-traumatic injuries, 4) A unilateral ISNCSCI Upper Extremity Motor Score (UEMS) between 10 and 23 for at least one limb.

For individuals with stroke: 1) at least 6 months post-stroke, 2) total score on the Action Research Arm Test (ARAT) greater than 10, 3) Montreal Cognitive Assessment (MoCA) score above 21 to avoid potential difficulties understanding and following study procedures.

Content and length of recorded video
The videos used for the analysis presented here were from 22 of the 23 participants. The remaining participant (man with cSCI) returned the camera without any usable data after being out of touch with the research team for close to 6 months, and is not included in the remainder of the analysis. As such, the proposed protocol was successful for 95.7% of the participants. On average, participants recorded more than 5 activities. Activities included in the video recordings were meal preparation, eating, dishwashing, physical activity, and knitting (Figure 2). An average of 3.11 ± 0.98 h of video were obtained per participant, after discarding segments where other individuals were present who did not provide consent or where the data was not usable at all (e.g., camera was left recording on a table). In addition, the daily average video recording length per participant was 60 ± 33 min. Three participants postponed the recordings due to illness. Most participants recorded video following the agreed schedules, but they reported feeling tired and uncomfortable wearing the camera for over 1 h due to its weight and heat against the forehead. Additional recordings were scheduled to fulfill the 3 h video duration target when necessary. The average recording period required to acquire 3 hours of video was 22.3 ± 12.9 d. The recording periods ranged from 12 to 69 d, counting from the day the participant was given the camera to the day they returned it. In 4 instances, the length of videos obtained was lower than the 3 h target due to health conditions or scheduling constraints. Two participants took more than 2 months to record 2 h of videos due to family responsibilities. Another participant recorded nearly 2 h of videos over two weeks and then decided to return the camera because of travel plans. Another participant accidentally deleted all recorded video, and after being scheduled for additional recordings was able to provide only 1 h of video due to family responsibilities.

In addition to variations in duration, hand visibility influenced the quality of the videos. Some recorded activities did not show the hands clearly, such as physical training and reaching for objects in over-head cabinets. Hands were not shown in 3 ADLs from two participants with stroke. Regarding physical training, one participant was using an elastic band for upper limb exercises and playing tennis in the backyard. Hands in the video were not always visible since the arm movements were large and fast. Apart from the activities requiring upper limb movements with a wide range, most recorded ADLs were carried out within a workspace between the waist and shoulder, with the hands visible in the recordings.

Figure 2. Examples of two frequently recorded ADLs from the videos obtained. (A) Meal preparation. (B) Dish washing. Please click here to view a larger version of this figure.

Assistance required
Six out of twenty-two (27%) participants required assistance from the research team during the recording periods at home. On average, each help call took 5-10 min. No home visits were necessary. The difficulties encountered related to battery changing, Bluetooth connection, Wi-Fi connection, reviewing and trimming videos on the tablet, and controlling the recordings from the tablet. To solve these issues the following strategies were adopted:

As changing batteries may be difficult for individuals with impaired hand function, the researcher suggested to use the charging cable to recharge the camera. For technical issues related to camera-tablet connections (e.g., Wi-Fi and Bluetooth problems) as well as for reviewing videos on the tablet, the researchers verbally guided participants through the procedures step by step. For the issues related to trimming videos, assistance was provided when participants returned videos to the researchers in person to avoid unintentionally deleting other recorded videos. In case participants were not able to use the tablet to start and stop the recordings, they were told to use the camera alone, by clicking the recording button to control the start and stop of the video recordings.

Privacy concerns
Consistent with previous findings²¹, most participants and their family members did not report privacy concerns about recording ADLs at home. One family member of a participant was unwilling to be included in the video, and the participant was able to avoid the situation. In total, one participant was assisted with trimming the returned videos. Qualitative analyses of participants' views on the collection of egocentric videos at home will be reported elsewhere, however, privacy considerations did not result in any obstacles to the data collection in these studies. In this protocol, the ADLs were recorded in home environments in order to avoid privacy concerns in public spaces. Apart from the participants, 15 bystanders (including caregivers and family members) of participants were included in the returned videos and consented to their inclusions in the study. Among the 15 bystanders included in the videos, 6 of them were not identifiable since their faces were not shown. In addition, approximately 20 min of videos from 2 family members of participants with stroke were discarded because portions of the videos showed their faces without their consents.

Supplemental Material.  Please click here to download this file.

Discussion

We presented a protocol for recording videos of ADLs at home using wearable cameras in individuals with upper limb impairments, such as cSCI and stroke. The protocol is flexible and can be goal-directed to capture hand function performance in specific ADLs or to track the progress of rehabilitation remotely in people living at home. The egocentric vision paradigm has great potential for remote monitoring of hand function in individuals living in the community, and for optimizing rehabilitation once people are discharged from inpatient care. Despite some limitations related to comfort during extended use (e.g., over 1 h), egocentric cameras have considerable potential to capture the details of hand use during functional ADLs and to provide contextual information that cannot be obtained through accelerometers and IMUs. Potential applications of this technology include serving as the basis for new outcomes measures quantifying independent hand use at home, as well as analyzing postural information to identify grasping strategies or compensatory postures and inform therapy planning. Although wearable cameras allow us to capture natural hand use at home, the cameras cannot record continuously for several hours due to battery life, storage, and comfort limitations, though the rapid evolution of wearable technology may reduce these limitations in the near future. In order to capture representative routines of participants despite limited recording duration, the protocol presented here relies on a collaborative process between the researchers and participants to select an appropriate recording schedule.

Thanks to the data collected from individuals with cSCI and stroke survivors, we were able to refine the protocol and identify pitfalls and gaps that could be improved in future studies. For example, the recording period usually took longer than the anticipated two weeks to obtain 3 hours of video data. In our studies, these situations were usually caused by unexpected events (e.g., illnesses, unexpected travels out of town, etc.) which prevented participants from recording videos according the planned schedule. In these cases, we provided support to the participants via weekly phone calls, not only to check on the recording progress, but also to troubleshoot any technical issues that could arise from a non-frequent use of the technology. In total, 15 caregivers of participants were involved in helping to record videos at home. In addition to technical issues, there are two general recommendations to adopt when recording video data using wearable cameras in individuals with hand impairments: 1) Emphasizing when participants should record, rather than what they should record, in order to avoid recording artificially long durations of a single activity and increase the variety of ADLs. 2) Keeping the camera on for longer continuous recordings as long as it is comfortable to do so, in order to increase recording efficiency and capture naturalistic sequences of activities.

This protocol can be used to capture natural hand function in the context of a variety of activities. In the collected videos, some participants included not only necessities of daily living (i.e., eating) but also leisure activities. Giving participants the freedom to record activities of their daily routines, rather than specifying a strict list of ADLs, allowed us to achieve two important aims: first, we made sure that activities were conducted in a manner similar to the natural conditions; second, we were able to record a wide variety of activities. Nevertheless, participants were aware of the camera presence on their head and this feeling might have pushed some of them to record more activities than their normal routine.

To achieve efficient and meaningful video data collection in natural living conditions, we had to find a balance in the guidelines and information given to the participants prior to the video recordings. Giving too much direction about the most interesting activities to record might lead the participants to record only those activities, thus failing to capture their daily routines and performance of natural ADLs. On the other hand, not giving enough information might cause confusion in the participants, who might not record enough video or lose interest in the study. To solve these issues and find an optimal trade-off for successful data collection, a thorough training session with participants is essential, in order to understand their routines and to make sure they clearly understand the purpose of the study, which here focused on recording their natural routines rather than collecting a standard set of activities.

Three limitations of the proposed protocol can be identified. First, individuals with cognitive issues may have difficulty recalling their daily routines or learning how to use the equipment. Second, individuals with low hand function may require a caregiver to help with setting up the recordings at home. Third, since participants have the opportunity to trim or delete recordings, it is possible that they might remove examples of unsuccessful ADLs and bias the data obtained. Nonetheless, this possibility is made less likely by the fact that the recorded ADLs were regularly performed by the participants, and therefore they usually had established strategies to carry out the tasks successfully. Carefully communicating that the goal of using this protocol is to capture typical ADL performance and that variations in function are expected may decrease participants' desire to trim unsuccessful tasks.

Collecting videos of natural ADLs is essential for understanding the hand function of individuals with hand impairments at home. From the collected videos, researchers and clinicians are able to see the typical tasks performed at home, the use of the hands during these tasks, as well as the hand postures and compensation strategies that are adopted during object manipulations. The information obtained from videos is valuable in rehabilitation and allows for the development of innovative and intelligent tools for the automatic analysis of hand function at home. In fact, computer vision algorithms have been developed with the aim of quantifying hand usage (e.g., number and duration of hand-object interactions for each hand) in individuals with cSCI and stroke ^16,17,18. With the constant improvement of computer vision techniques for egocentric hand analysis and the availability of more collected video data, additional details regarding the quality of hand manipulations can also be obtained²⁰. Eventually, the current protocol and the available technology will lead to a comprehensive description of hand use in individuals with upper limb impairments living in the community, with the end goal of optimizing their recovery and improving their quality of life.

Materials

Name	Company	Catalog Number	Comments
Egocentric camera	GoPro Inc., CA, USA	GoPro Hero 4 and 5	A camera that records from a first-person angle.
Battery chager and batteries	GoPro Inc., CA, USA	MAX Dual Battery Charger + Battery	Extra batteries for the camera and battery charger
Camera charger	GoPro Inc., CA, USA	Supercharger	This charger is connected to the camera directly without disassembling the camera frame.
Camera frame	GoPro Inc., CA, USA	The Frame	The hinge of the camera frame can be used to adjust the angle of view of the camera.
Headband for the camera	GoPro Inc., CA, USA	Head Strap + QuickClip
SD card	SanDisk, CA, USA	32GB microSD
Tablet	ASUSTeK Computer Inc., Taiwan	ZenPad 8.0 Z380M	The tablet is installed with the GoPro App in order to connect with the camera.