Evaluation of a Smartphone-based Human Activity Recognition System in a Daily Living Environment
December 11th, 2015
A standardized evaluation method was developed for Wearable Mobility Monitoring Systems (WMMS) that includes continuous activities in a realistic daily living environment. Testing with a series of daily living activities can decrease activity recognition sensitivity; therefore, realistic testing circuits are encouraged for valid evaluation of WMMS performance.
The overall goal of this activity recognition methodology is to provide a standardized evaluation method for testing wearable mobility monitoring systems by including continuous activities in a realistic daily living environment. This method can help answer key questions in the physical rehabilitation, biomedical engineering, and biomechanics fields, such as whether wearable sensor systems that record how a person moves, provide accurate and reliable results. The main advantage of this technique is that the evaluation protocol uses a control course that reflects real life daily living environments Begin by securely attaching a phone holster to the participant at their right front hip.
Start the application on the mobility measurement smartphone that is used to collect the sensor data. Then place the smartphone in the holster with the back of the device facing outward. Next, start a digital video recording on a second smartphone for anonymity.
Record the comparison video on the second smartphone without showing the participant's face. However, ensure all activity transitions are recorded during the activity circuit. Follow the participant and use the second smartphone to record all of the participant's actions.
Begin by providing the following instructions to the participant from a standing position. Shake the smartphone to indicate the start of the trial. Continue standing for at least 10 seconds and then walk to a nearby chair and sit down.
Stand up and walk 60 meters to an elevator stand and wait for the elevator to arrive and then walk into the elevator. Next turn and walk into the home environment. Walk into the bathroom and simulate brushing teeth, simulate combing hair, washing hands, and then drying hands using a towel.
Following that, walk to the kitchen and take dishes from a rack and place them on the counter. Fill a kettle with water from the kitchen sink and place the kettle on the stove. Then afterwards, place bread in a toaster.
Next, walk to the dining room and sit at the dining room table. Simulate eating a meal at the table. Then stand and walk back to the kitchen sink to rinse off the dishes and place them in a rack.
Walk 50 meters to a stairwell. Open the door and enter the stairwell. Walk up 13 steps and open the stairwell door to the hallway.
Turn right and walk down the hall for 15 meters. Then instruct the participant to turn around and walk 15 meters back to the stairwell. Open the door, enter the stairwell and walk down 13 steps and exit the stairwell.
Next, walk into a room and lie on the bed. Then get up and walk 10 meters to a ramp. Walk up the ramp, turn around, and then walk down the ramp.
Continue walking into the hall and open the door to the outside. Next, walk 100 meters onto the paved pathway. Turn around and walk back towards the room.
Then walk into the room and stand at the starting point. Continue standing and then shake the smartphone to indicate the end of the trial. Once the trial has completed, stop the recording from the second smartphone and remove the first smartphone with the holster from the participant.
Finally, stop the data logging application on the first smartphone. Copy the acquired motion data files and video files from both phones to a computer for post-processing. This protocol discussed an evaluation method for wearable mobility monitoring systems or WMMS during continuous activities in a daily living environment.
Sensitivity and specificity were measured for three types of classifications, mobility, or immobility. Sit, stand, lie, or walking and sit, stand lie, walking, climbing stairs or small standing movement. Although this protocol will result in more conservative results for the WMMS systems than previous reports, the results reported here will better reflect outcomes in everyday practice.
Once mastered, this technique can be completed in 10 to 15 minutes if it is performed properly. While attempting this procedure, it's important to remember to provide clear instructions to the study participant and to record all activity transitions so that they can be logged from the video. Following this procedure, other computational methods can be performed on the sensor output to develop better activity classification methods After its development.
This technique paves the way for researchers in the field of human activity recognition to explore wearable mobility monitoring in an environment that can translate to the real world. After watching this video, you should have a good understanding of how to set up a continuous test environment and capture relevant human movement data for evaluating human activity monitoring applications. Don't forget that people with disabilities will have different mobility and balance capabilities, so precautions should be taken to avoid falls or overexertion when performing this procedure.
View the full transcript and gain access to thousands of scientific videos
Overview
This article presents a standardized evaluation protocol for Wearable Mobility Monitoring Systems (WMMS) by simulating continuous daily living activities in a realistic environment. The method aims to assess the accuracy and reliability of WMMS in recognizing various human activities, providing valuable insights for rehabilitation, biomedical engineering, and biomechanics research.
Key Study Components
Area of Science
- Human activity recognition
- Wearable sensor technology
- Rehabilitation medicine
- Biomedical engineering
Background
- Wearable Mobility Monitoring Systems are increasingly used to track and classify human movement in real-world settings.
- Accurate activity recognition is essential for clinical decision-making and rehabilitation monitoring.
- Existing evaluation protocols often lack ecological validity, as they do not reflect continuous, real-life activity transitions.
- A standardized, realistic protocol is needed to benchmark WMMS performance effectively.
Purpose of Study
- To develop and demonstrate a standardized evaluation method for WMMS using continuous daily living activities.
- To assess WMMS classification accuracy for various activity states in a realistic environment.
- To provide a protocol that can be adapted for future research and clinical applications.
Methods Used
- Fifteen able-bodied participants performed a sequence of daily living tasks in a controlled circuit.
- Participants wore a smartphone at the right front hip to collect accelerometer and gyroscope data.
- All activities and transitions were video-recorded for gold standard comparison, with manual logging of activity onset times.
- Three classification sets were analyzed: (i) mobility vs. immobility, (ii) sit, stand, lie, or walking, and (iii) sit, stand, lie, walking, climbing stairs, or small standing movement.
- Sensitivities, specificities, and F1 scores were calculated for each classification set.
Main Results
- Mobility vs. immobility classification achieved 86.30% ± 7.2% sensitivity and 98.96% ± 0.6% specificity.
- The second classification set (sit, stand, lie, walking) had 88.35% ± 7.80% sensitivity and 98.51% ± 0.62% specificity.
- The third set (including climbing stairs and small standing movement) showed 84.92% ± 6.38% sensitivity and 98.17% ± 0.62% specificity.
- F1 scores were 86.17 ± 6.3, 80.19 ± 6.36, and 78.42 ± 5.96 for the three sets, respectively.
Conclusions
- WMMS performance is influenced by the evaluation protocol as well as the underlying algorithms.
- The described protocol yields more conservative but realistic performance estimates compared to previous methods.
- This approach can be tailored for evaluating human activity recognition systems in clinical and research settings, especially in rehabilitation medicine.
Frequently Asked Questions
Executive Industry Relevance
Standardized evaluation of wearable mobility monitoring systems (WMMS) in realistic daily living environments is critical for establishing predictive confidence in human activity recognition technologies. This protocol enables robust target validation for sensor-based activity classifiers, directly impacting translational research and preclinical model development in rehabilitation and biomedical engineering. Reliable activity recognition supports risk-adjusted advancement of digital health solutions across the biopharma portfolio.
Strategic Applications in Biopharma R&D
Early Discovery & Target Validation
- Enables rigorous hypothesis testing for sensor-based activity classification algorithms.
- Supports biological de-risking by benchmarking classifier performance against gold standard video-logged data.
- Facilitates functional validation of activity recognition targets in ecologically valid settings.
Screening & Assay Development
- Prepares validated activity datasets for downstream algorithm refinement and benchmarking.
- Standardizes assay conditions by using a continuous, real-world activity circuit.
- Enables reproducible, quantitative outputs such as sensitivity, specificity, and F1 scores for classifier evaluation.
Translational & Preclinical Research
- Aligns activity recognition outputs with clinically relevant mobility endpoints for rehabilitation research.
- Ensures continuity from discovery-stage classifier development to preclinical validation in real-world scenarios.
- Provides a foundation for risk-adjusted decisions on advancing digital mobility monitoring tools.
Pipeline & Workflow Integration
This evaluation protocol bridges early discovery, screening, and translational research by providing a standardized workflow for WMMS validation in daily living environments.
- Discovery Biology: Supports hypothesis testing and pathway clarification for activity recognition algorithms.
- Screening: Delivers reproducible, quantitative metrics for classifier performance across multiple activity states.
- Analytics: Generates sensitivity, specificity, and F1 score outputs for robust condition comparison.
- Translational Research: Connects classifier validation to clinically meaningful mobility monitoring endpoints.
- Enterprise Reuse: Establishes a reusable protocol adaptable to diverse activity recognition systems and populations.
Operational & Enterprise Impact
- Scientific Value: Increases predictive confidence and reduces mechanistic ambiguity in activity recognition.
- Operational Value: Promotes standardization, reproducibility, and scalability of WMMS evaluation.
- Strategic Value: Informs go/no-go decisions and enhances capital efficiency for digital health R&D.
- Portfolio Impact: Enables risk-adjusted prioritization of mobility monitoring technologies for clinical and research applications.
Implementation Considerations
- Requires expertise in sensor data acquisition and manual video annotation for gold standard comparison.
- Needs smartphones with accelerometer and gyroscope capabilities and secure data handling infrastructure.
- Demands cross-team standardization of activity protocols and data logging procedures.
- Adaptable to various model systems but must account for participant safety and mobility limitations.
- Protocol duration and participant compliance may limit throughput in large-scale studies.
