Scientific Review - Many Garmin Watches Extremely Inaccurate

actually the medical brainwave device or EEG as its more commonely known cannot be considered gold standard - that is a sleep clinic. The device he uses is only around 80% accurate compared to a sleep clinic which uses more than just brainwaves.

Source on 80%.

Droogles said:

Source on 80%.

According to the manufacturer of the device, it looks like it's more like only 75%:

Source: dreem.com

HTH

gaijin said:

According to the manufacturer of the device, it looks like it's more like only 75%:

When using a true ECG, not a wrist HR sensor, to evaluate the Firstbeat performance, the agreement is only between 63% and 66%.

https://assets.firstbeat.com/firstbeat/uploads/2019/11/Firstbeat-Sleep-Solution_white-paper_short.pdf

It means that 2 times out of 3, when the Firstbeat ECG HRV- and movement-based algorithm classifies a 5 second period of sleep, the result concurs with the experts' polysomnography classification (who apparently have a ~80% agreement rate themselves according to this paper).

1/3 mismatch rate is "reasonably good", perhaps, as the paper states, but Garmin watches are not ECG devices, so we should expect a lower match.

Net net, sleep phases detection is at best "reasonably good", per Garmin.

How to these percentage translate in reality for me?

Let's take the awake/sleep state detection. The agreement should be 94% according to the Firstbeat paper.

"The resulting Firstbeat method differentiated sleep state
from awake state with 94 % sensitivity (true-positive rate)
and 63 % specificity (true-negative rate"

In my case, the awake/sleep detection is less then 50%. First, it doesn't detect all the time I spent awake during the night wondering if the morning report is going to work :-). Second, and the worst, it finds me in deep sleep while watching TV or gaming, or reading FirstBeat white papers in front of the computer.

To make sure the watch is not going to destroy my previous night of sleep data with a phantom nap, and not going to tax my sleep score with poor sleep during TV, gaming or reading, I have to take the watch OFF, and put it back on before going to sleep. How idiotic is that?

Why isn't there a way to turn automatic sleep detection off and enable it manually when I go to bed?

Hm, I can start Sleep Mode manually from my Epix 2 Watch. Sometimes I do that, because I go to sleep before the acutal 'Scheduled Time'. 

That's just a 'do not disturb' mode for the watch.

This has been a great read, with some heated edges!!! You all are doing a great job of talking me out of buying the new MARQ Gen 2!!

personally, I always fall back to that little disclaimer…

“not a medical device”

And, I do find that looking at the device trend graphs, instead of the specifics, work fairly well for me.

But all great arguments!!!

(Note to self… keep your $2100)

This has been a great read, with some heated edges!!! You all are doing a great job of talking me out of buying the new MARQ Gen 2!!

personally, I always fall back to that little disclaimer…

“not a medical device”

And, I do find that looking at the device for trend info and graphs, instead of the specifics, work fairly well for me.

But all great arguments and points!!

(Note to self… keep your $2100)

David Epix said:

Hm, I can start Sleep Mode manually from my Epix 2 Watch

Yes, but this is not linked to sleep detection. This mode is just less power / more discretion & convenience from the watch while you sleep.

Starting the sleep mode doesn't start a sleep cycle for sure, and, per my testing, changing the schedule around doesn't influence the sleep detection.

Sleep mode and sleep detection seem to be better linked on the "ending" side. I often have to terminate sleep detection. I see the watch still thinks I am asleep when I want to see the HRV status, or the sleep widget for example. I can terminate sleep then, and that will terminate the sleep mode as well. Not 100% sure the other way works all the time (ie terminating the sleep mode might not terminate sleep detection).

I'd be happy if I had the option to disable automatic sleep detection, and link the start of sleep detection with the sleep mode (which I know I can start and stop manually). It would be even better if I was not forced to schedule the sleep mode at all, but leaving it manual.