a-view-their-own-capturing-egocentric-view-infants-toddlers-with-head

A View of Their Own: Capturing the Egocentric View of Infants and Toddlers with Head-Mounted Cameras

Infants and toddlers view the world, at a basic sensory level, in a fundamentally different way from their parents. This is largely due to biological ...

Indiana University

Wearable devices are becoming part of everyday life, from first-person cameras (GoPro, Google Glass), to smart watches (Apple Watch), to activity trackers (FitBit). These devices are often equipped with advanced sensors that gather data about the wearer and the environment. These sensors enable new ways of recognizing and analyzing the wearer's everyday personal activities, which could be used for intelligent human-computer interfaces and other applications. We explore one possible application by investigating how egocentric video data collected from head-mounted cameras can be used to recognize social activities between two interacting partners (e.g. playing chess or cards). In particular, we demonstrate that just the positions and poses of hands within the first-person view are highly informative for activity recognition, and present a computer vision approach that detects hands to automatically estimate activities. While hand pose detection is imperfect, we show that combining evidence across first-person views from the two social partners significantly improves activity recognition accuracy. This result highlights how integrating weak but complimentary sources of evidence from social partners engaged in the same task can help to recognize the nature of their interaction.

Viewpoint Integration for Hand-Based Recognition of Social Interactions from a First-Person View.

Hands appear very often in egocentric video, and their appearance and pose give important cues about what people are doing and what they are paying attention to. But existing work in hand detection has made strong assumptions that work well in only simple scenarios, such as with limited interaction with other people or in lab settings. We develop methods to locate and distinguish between hands in egocentric video using strong appearance models with Convolutional Neural Networks, and introduce a simple candidate region generation approach that outperforms existing techniques at a fraction of the computational cost. We show how these high-quality bounding boxes can be used to create accurate pixelwise hand regions, and as an application, we investigate the extent to which hand segmentation alone can distinguish between different activities. We evaluate these techniques on a new dataset of 48 first-person videos of people interacting in realistic environments, with pixel-level ground truth for over 15,000 hand instances.

Article	Year
Viewpoint Integration for Hand-Based Recognition of Social Interactions from a First-Person View. Proceedings of the ... ACM International Conference on Multimodal Interaction. ICMI (Conference)\| PubMed ID: 28966999	2015
Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions. Proceedings. IEEE International Conference on Computer Vision\| PubMed ID: 29225555	2015

Sven Bambach

Biography

JoVE Journal Publications

Other Publications