Assessing the Accuracy of Fitness Smartwatch Data for Cardiovascular and Physical Activity Monitoring: A Validation Study in Digital Health
February 21st, 2025
Wearable technology has low cost and offers convenient monitoring of physiological data. However, the accuracy and reliability of these devices require cautious assessment to ensure their effectiveness and safety for users. This report describes the validation process of a commercial smartwatch in monitoring physiological data and physical activity.
The research aims to validate their accuracy of low cost-fitness smartwatches by comparing their data with gold standard measurements used in clinical settings. This protocol evaluates the accuracy and reliability of fitness smartwatches, addressing gaps in understanding their real-world performance as their popularity grows. Our findings will provide valuable insights into the accuracy and reliability of fitness smartwatch data, which is crucial for researchers and healthcare professionals using these devices in studies or clinical settings.
To begin, have participants wear the fitness smartwatch on their non-dominant wrist. Attach participants to the apparatus of the polysomnography or PSG machine. With the fitness smartwatch, take two measurements of blood oxygen saturation level while the participant is in a supine position.
Ensure the participant sleep for a maximum of six hours. After the participant awakens, use the fitness smartwatch to take two additional blood oxygen saturation level measurements while they remain in a supine position. Export the sleep trends from both the PSG machine and the fitness smartwatch into a csv file.
Include data on deep sleep duration, light sleep duration, rapid eye movement time, total sleep duration, heart rate, and blood oxygen saturation levels. For the validation of fitness smartwatch step measurements, first have participants wear the fitness smartwatch on their non-dominant wrist and hold a smartphone in their other hand for footstep video recording purposes. Set the step count on the fitness smartwatch to zero, then start the smartphone recording, and have the participant perform the step test.
Record the number of steps measured by the fitness smartwatch. For the three-minute walk test or 3-M WT, instruct the participant to walk at a normal and steady pace for three minutes on a flat surface. For the stair climb or SC test, have the participant take two flights of stairs in ascending and descending order at a normal and steady pace.
Manually calculate the number of steps observed in the video recording. Record this data into a csv file, then export the step counts from the fitness smartwatch into the same file. Import the required packages into Python's workspace, then import the measurement data into Python's workspace.
Replace the file name measurement_data. csv with a file name of choice. Use the drop in a function to remove any empty values for each measurement data.
Perform the Shapiro-Wilk normality test to determine if the data has a normal distribution using the SciPy stats Shapiro package in Python. Now, perform a paired sample T-test using the SciPy stats T-test TestRail package in Python if the data has a normal distribution. If not, perform the Mann-Whitney Wilcoxon test using the SciPy stats Wilcoxon package.
Replace measure1_toolA and measure1_toolB with respective column names of choice in the code. Then use the co and d method to measure the magnitude of the difference between instruments. Employed Bland-Altman analysis to assess the agreement between two instruments, including mean, or median difference, standard deviation, and 95%confidence interval of the bias.
Plot the Bland-Altman analysis for visualization using the Matplotlib pie plot package in Python. Shallow sleep time showed a large bias of 27.77 minutes with wide limits of agreement, indicating significant variability in measurements by the smartwatch compared to PSG. Total sleep time was overestimated by 41.5 minutes, with wider limits of agreement, indicating occasional mismatches in tracking sleep duration.
REM time showed minimal bias and standard deviation, suggesting good interchangeability between the smartwatch and PSG for this parameter. Wake time was overestimated by the smartwatch, with clear clustering and upward trend in differences, indicating measurement bias. Mean heart rate exhibited minimal bias and good clustering near the bias line, indicating strong agreement.
Minimum heart rate displayed some outliers, suggesting difficulty in detecting low heart rates, though overall agreement was reasonable. Maximum heart rate exhibited notable dispersion, reflecting variability in tracking peak values by the smartwatch. Blood oxygen saturation showed minimal bias and narrow limits of agreement, reflecting high agreement between the devices.
The smartwatch significantly underestimated steps for 3-M WT by an average of 31.33 steps and SC steps by 11.
View the full transcript and gain access to thousands of scientific videos
Overview
This study evaluates the accuracy and reliability of low-cost fitness smartwatches in monitoring physiological data. By comparing smartwatch data with gold standard measurements, the research addresses the effectiveness of these devices in real-world settings.
Key Study Components
Area of Science
- Wearable technology
- Physiological monitoring
- Health data accuracy
Background
- Wearable technology is becoming increasingly popular for health monitoring.
- Accuracy and reliability of these devices are critical for user safety.
- There is a need to validate the performance of fitness smartwatches.
- This study aims to fill gaps in understanding their real-world effectiveness.
Purpose of Study
- To validate the accuracy of fitness smartwatches.
- To compare smartwatch data with clinical gold standard measurements.
- To provide insights for researchers and healthcare professionals.
Methods Used
- Participants wore fitness smartwatches on their non-dominant wrist.
- Participants were connected to a polysomnography (PSG) machine.
- Two measurements of blood oxygen saturation were taken.
- Measurements were conducted while participants were in a supine position.
Main Results
- The study provides insights into the accuracy of smartwatch data.
- Findings highlight the reliability of fitness smartwatches in monitoring physiological data.
- Results are essential for clinical and research applications.
- Validation process demonstrates the effectiveness of these devices.
Conclusions
- Fitness smartwatches can be reliable tools for physiological monitoring.
- Further research is needed to enhance device accuracy.
- Study findings support the use of smartwatches in clinical settings.
Frequently Asked Questions
Executive Industry Relevance
Validation of digital health wearables is critical for ensuring data integrity in biopharma research pipelines, especially as these devices are increasingly used for remote monitoring and decentralized studies. This study highlights the necessity of rigorous benchmarking against gold-standard measurements to support reliable physiological data capture for cardiovascular and activity endpoints. Accurate device validation directly impacts translational confidence and risk-adjusted decision-making in digital biomarker development.
Strategic Applications in Biopharma R&D
Early Discovery & Target Validation
- Supports objective assessment of physiological endpoints relevant to cardiovascular and sleep-related targets.
- Enables mechanistic de-risking by clarifying device measurement limitations and variability.
- Informs target validation strategies by identifying reliable digital readouts for downstream studies.
Screening & Assay Development
- Facilitates preparation of validated digital endpoints for integration into screening and monitoring workflows.
- Promotes assay standardization by benchmarking device outputs against clinical gold standards.
- Enables reproducible and quantitative measurement of heart rate, SpO2, and sleep parameters for compound evaluation.
Translational & Preclinical Research
- Aligns digital biomarker development with disease-relevant physiological measurements validated in controlled settings.
- Supports continuity from early discovery through preclinical validation by establishing device reliability thresholds.
- Reduces translational risk by identifying device-specific measurement biases and limitations.
Pipeline & Workflow Integration
This validation protocol positions wearable device data as a potential bridge from early discovery to translational research, contingent on rigorous benchmarking and ongoing quality control.
- Discovery Biology: Enables hypothesis testing for digital endpoints by comparing device outputs to gold-standard measures.
- Screening: Provides reproducible, quantitative data streams for integration into digital health screening platforms.
- Analytics: Delivers statistical outputs (e.g., bias, limits of agreement) to inform cross-study comparisons and device selection.
- Translational Research: Establishes validated digital biomarkers for use in preclinical and clinical research when supported by robust agreement metrics.
- Enterprise Reuse: Offers a standardized validation workflow adaptable to new wearable technologies and evolving digital endpoints.
Operational & Enterprise Impact
- Scientific Value: Increases predictive confidence in digital biomarker data and reduces mechanistic ambiguity in remote monitoring studies.
- Operational Value: Drives standardization and reproducibility in device-based data collection across research sites.
- Strategic Value: Enables informed go/no-go decisions for digital endpoint adoption, improving capital efficiency and reducing late-stage risk.
- Portfolio Impact: Supports risk-adjusted prioritization of digital health tools for integration into clinical and translational pipelines.
Implementation Considerations
- Requires expertise in biostatistics and digital health analytics for robust validation and interpretation.
- Demands access to gold-standard instrumentation (e.g., PSG, manual step counting) for benchmarking.
- Necessitates cross-team standardization of data collection, export, and analysis protocols.
- Must consider device-specific limitations and variability across physiological parameters and populations.
- Ongoing validation is essential as device firmware and algorithms evolve over time.
