This article is part of the 2018 NMJ Oncology Special Issue. Download the full issue.
Reference
Peake JM, Kerr G, Sullivan JP. A critical review of consumer wearables, mobile applications, and equipment for providing biofeedback, monitoring stress, and sleep in physically active populations. Front Physiol. 2018;9:743.
Objective
To summarize features of wearable health technologies and evaluate their suitability for consumer use by assessing whether the data has been validated, is reliable, and does for consumers what the manufacturers claim.
Design
Review of commercially available wearable health technology devices. Investigators identified devices for inclusion in the review by searching the internet and databases of scientific literature (eg, PubMed) using key terms such as “technology,” “hydration,” “sweat analysis,” “heart rate,” “biofeedback,” “respiration,” “muscle oxygenation,” “sleep,” “cognitive function,” and “concussion.”
Study Parameters Assessed
The researchers examined the websites of commercial technologies for links to research, and where applicable, they sourced published research literature. They then divided technologies into the following categories:
- Devices for monitoring hydration status and metabolism
- Devices, garments, and mobile applications for monitoring physical and psychological stress
- Wearable devices that provide physical biofeedback (eg, muscle stimulation, haptic feedback)
- Devices that provide cognitive feedback and training
- Devices and applications for monitoring and promoting sleep
- Devices and applications for evaluating concussion
Primary Outcome Measures
The investigators evaluated the available information based on 4 measurements:
- What does the technology claim to do?
- Has the technology been independently validated against some accepted standard(s)?
- Is the technology reliable and is any calibration needed?
- Is the technology commercially available or still under development?
Key Findings
The researchers identified and evaluated 89 devices; some were commercially available and others were not. They found the vast majority (82/89) had never been formally validated. Only 10% had been used in research settings.
Nearly all the devices (87/89) had no published reliability testing. Calibration of the devices fell into 1 of 3 categories: 1) not reported; 2) reported as “self-calibrating;” and 3) manufacturers stated there was no need for calibration.
Regarding sleep tracking devices specifically, of the 15 wearables reviewed, only 3 had validation information (UP, FitBit Charge2, OURA), and none of those had undergone reliability testing. The 1 device that did have reliability data (FitBit Flex) did not report any validation testing.
Practice Implications
This study was the first to evaluate various types of wearable devices to determine if data produced by wearable devices is valid and reliable.
The wearable device industry is growing annually at 15% and is expected to be worth $51.50 billion globally by 2020.1 Undoubtedly, clinicians have encountered and will continue to encounter patients who use wearables. Understanding the technology, especially regarding a given device’s claims, is essential to having thoughtful conversations with patients, who are likely to use data from their device as “medical information” in clinic visits.
However, the vast majority of devices evaluated by the researchers failed on all accounts. The manufacturers did not validate the data the devices produce, most didn’t calibrate the data to ensure consistency of data readings over time, and they also did not conduct reliability testing or disclose how reference limits were created. Therefore, unless otherwise confirmed, clinicians should not assume that data generated by wearable devices is accurate.
For example, the 2 devices that had validation data (UP and FitBit Flex) were compared to the gold standard for sleep studies, a polysomnography. Each device correlated with total sleep time and time in bed, but they did not correlate with deep sleep, light sleep, or sleep efficacy.2,3
Despite the current limitations on accuracy, a potential benefit of wearable technology is that it can create an opportunity to help raise patient awareness of certain health concerns. For example, a sleep tracker could provide a starting point for a helpful conversation about sleep and how to improve it. Through that conversation the clinician can better assess sleep quality and quantity. This may lead to conversations on how to improve sleep or help determine if a more formal sleep study is warranted.
The silver lining, however, is that these devices may bring opportunities to encourage healthier patient behaviors while verifying (or debunking) potential health problems through more rigorous and validated testing methods.
A 2018 systematic review and meta-analysis published in the American Journal of Health Promotion evaluated the efficacy of wearable devices to improve physical activity in patients with diagnosed cardiometabolic disease.4 Primary outcomes included physical activity as measured by steps per day and moderate to vigorous physical activity [MVPA], which can include activities such as jogging, lap swimming, tennis or racquetball, bicycling, aerobics, and dancing.
Thirty-five studies involving 4,528 volunteers met the inclusion criteria. The pooled data showed significant increases in physical activity and MVPA in volunteers who used wearable devices.4 This study supports the notion that wearable devices can promote physical activity by making patients more aware of their activity levels.
Similarly, wearable devices that monitor other health parameters, such as stress and emotions, heart rate, and blood oxygen levels, may provide an opportunity to focus further clinical evaluations on patient concerns and help them either validate or refute the data from the given device.
While the current study revealed that manufacturers are bringing to market products that have not been validated, researchers are conducting post-market studies to test the accuracy and reproducibility of specific devices. A 2019 study by Nelson and Allen evaluated the heart rate accuracy of Apple Watch 3 and Fitbit Charge 2 and compared the data they produced to an ambulatory ECG (Vrije Universiteit Ambulatory Monitoring System).5 The authors concluded, “The Apple Watch 3 and the Fitbit Charge 2 provided acceptable heart rate accuracy (<±10%) across the 24 hour and during each activity, except for the Apple Watch 3 during the daily activities condition.”5
The caveat to Nelson and Allen’s review is that devices may perform well only under specific conditions. However, it may not be possible to know those limitations from the manufacturer’s literature alone. This leaves clinicians in a difficult situation. Without manufacturers divulging the limitations of their devices, it’s not possible to know under which specific use conditions devices might be more or less accurate.
The silver lining, however, is that these devices may bring opportunities to encourage healthier patient behaviors while verifying (or debunking) potential health problems through more rigorous and validated testing methods.
