This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Race predictions way off

I have switched from Forerunner 245 to Forerunner 965 recently (yes, Christmas present :)). On the day that I switched, the race predictions dropped enormously. Now, ten days later, the predictions are still way off, they are worse than all my PR's.

Does the Forerunner 965 use a different algorithm than the 245? And why does a newer model show worse predictions?

Top Replies

All Replies

  • No it doesn't. I have had my 965 for 6 months now. I have recorded every activity on it. When I first got the watch my VO2 max immediately dropped by 2 points and my race predictions all got much slower. I had moved from the 745. I know as a fact my fitness has improved, evidenced by solid race results, but the predictions got slower and slower. I did a HM in 1:34. My prediction was 1:37. Straight after the event, the prediction changed... to 1:38. It's just trolling, pure and simple.

    I have a problem with the stamina reading as well. During a 10k race, it said I was at 0% at 4k. I ran a negative split. During the above HM, my stamina reading was 0% at 7k. Again, I ran a negative split. 

    My HR data is correct. The max HR is the highest recorded by the watch.  

  • Age and gender is certainly used for the relative VO2 max benchmarking (the color coding of the VO2 Max scale on the watch). Just like weight, they are not needed for the absolute evaluation of ventilation.

    Yes, that's what I always thought, but you see lots of people insisting that age has to be correct in order to get a reliable VO2 Max estimate. My point being, I think there's a lot of misunderstanding and misinformation out there on this topic.

    Clearly age and gender would be used to for the color-coding and "top X% for your age and gender" labelling of VO2 Max. I assume that comes directly from the Cooper tables that Garmin publishes in its manuals (especially since the app help says "VO2 Max data used with permission from The Cooper Institute")

    Garmin's uses HR and HRV data to estimate ventilation (VO), and uses machine learning to predict your VO2 Max from sub-maximal efforts given altitude and temperature data after filtering and qualifying the data by taking into account movement vs rest. Weight is not needed here.

    However, when the watch gives you a VO2 max number, it has been normalized by your weight. While absolute ventilation is measured in liters of oxygen per minute (L/mn), the number that the watch gives is the relative ventilation, which you can use to compare with others

    Regarding the difference between absolute VO2 Max (the absolute rate of oxygen uptake) and relative VO2 Max (oxygen uptake normalized by weight): my question is what exactly is Garmin estimating when it compares speed and HR? You're saying that it's estimating absolute VO2 Max, but I always thought it estimated relative VO2 Max.

    I know that when the Daniels formula is used to estimate VDOT from a race result, VDOT is similar to relative VO2 Max (in that it can be used to compare runners of different weights.) My understanding is that VDOT is simply relative VO2 Max relabelled to make it clear that it was derived from a race result as opposed to being measured in the lab.

    Similarly, I would think that if Garmin derives VO2 Max from your speed at given HR (by extrapolating from submaximal efforts to a maximal effort), it would be estimating *relative* VO2 Max, in the same way that the Daniels formula estimates VDOT.

    Intuitively (to me), if losing non-functional weight already allows me to run faster (thus increasing my absolute and relative VO2 Max), then that would be reflected in the Garmin algorithm by measuring a faster speed, which would result in a higher VO2 Max. To me, if the algorithm had to take weight into account, that would almost be like double-counting the weight change. (i.e. The weight change would affect two inputs: speed and weight itself.)

    On the flip side, what if I lost weight and I ran the same speed as before at a given HR (for whatever reason - maybe I lost muscle, too.) If weight was a direct input to the Garmin algorithm, then my weight loss should result in a higher relative VO2 Max estimate (since the absolute VO2 Max would stay the same, due to speed/HR staying the same, but relative VO2 Max = absolute VO2 Max / weight), which doesn't make sense to me if I actually can't run faster. For example, there's the lookup tables which go directly from relative VO2 Max (or VDOT) to race results (basically the inverse of the Daniels lookup tables from race results to VDOT), with the implication that higher VO2 Max = faster running.

    TL;DR My understanding is that relative VO2 Max itself is correlated with speed (at a maximal effort for a certain distance), and Garmin's algorithm is estimating VO2 Max by extrapolating speed at maximal effort from sub-maximal efforts, which suggests to me that Garmin is estimating relative VO2 Max itself.

    On the other hand, If I did a real VO2 Max test in a lab where someone strapped a mask to my face and measured my actual oxygen intake, then it would be clear to me that it's absolute VO2 Max which is being measured.

  • That sounds frustrating.

    When I moved from 935 to 955, my race predictions actually got more realistic. I've never seen a stamina of 0%. I think the lowest I've seen is 40%.

    I won't say that I take Garmin's metrics super-seriously, but they've never been complete garbage for me, other than the fact that my race predictions used to be overly optimistic.

  • On the other hand, If I did a real VO2 Max test in a lab where someone strapped a mask to my face and measured my actual oxygen intake, then it would be clear to me that it's absolute VO2 Max which is being measured

    Same thing for the watch. It just uses HR, HRV and on/off information to estimate VO2.

    "Recently, a new neural network-based method has been introduced for the analysis of physical (and mental) workload from R-to-R interval (RRI) recordings during work which does not require individual laboratory calibration between HR and VO2 (Firstbeat Technologies, 2005). The new method is based on both HR and heart rate variability (HRV) from which the software calculates additional information on respiratory frequency and on/ off-dynamics. This information allows avoiding the problems related to non-metabolic increase in HR and inconsistencies in HR/VO2 relation during dynamically changing work intensities"

    https://www.firstbeat.com/wp-content/uploads/2015/10/smolander_et_al_2007.pdf

  • On the other hand, If I did a real VO2 Max test in a lab where someone strapped a mask to my face and measured my actual oxygen intake, then it would be clear to me that it's absolute VO2 Max which is being measured

    Same thing for the watch. It just uses HR, HRV and on/off information to estimate VO2.

    "Recently, a new neural network-based method has been introduced for the analysis of physical (and mental) workload from R-to-R interval (RRI) recordings during work which does not require individual laboratory calibration between HR and VO2 (Firstbeat Technologies, 2005). The new method is based on both HR and heart rate variability (HRV) from which the software calculates additional information on respiratory frequency and on/ off-dynamics. This information allows avoiding the problems related to non-metabolic increase in HR and inconsistencies in HR/VO2 relation during dynamically changing work intensities"

    https://www.firstbeat.com/wp-content/uploads/2015/10/smolander_et_al_2007.pdf

    That white paper refers to estimation of VO2 based on HRV, and makes no reference to speed, based on my quick skimming/searching. I do not think that is the algorithm which is used by Garmin watches to estimate VO2 Max.

    My understanding of the Firstbeat algorithm on Garmin watches is that:

    - HR (not HRV) and speed is measured at (possibly) submaximal efforts (e.g. regular steady-state runs, workouts or races).

    It's worth noting that HRV from optical sensors is still not considered "good enough" for certain applications, like the HRV Stress test. (From my reading, chest straps measure HRV while optical sensors only estimate HRV.) Similarly, the Elite HRV app requires a chest strap, and won't work with an Apple Watch's optical sensor.

    - Segments which are do not have useful data are discarded (like when you're standing still at a red light).

    - The relationship between recorded HR and speed, along with max HR, is used to extrapolate speed at a maximal performance for a given distance from the recorded data

    Again, if speed is correlated with relative VO2 Max (not absolute VO2 Max) (*), and the key to estimating VO2 Max is observing the relationship between speed and HR, to me it makes sense that Garmin is estimating relative VO2 Max and not absolute VO2 Max.

    (*) It's well known that all the lookup tables going from VO2 Max to race predictions and vice versa refer only to relative VO2 Max (or sometimes, VDOT).

    Note that the following paper is 10 years newer than the paper you linked, it makes no reference to HRV, and it's linked on the firstbeat page which clearly markets their algorithm that's used in Garmin devices.

    https://www.firstbeat.com/en/science-and-physiology/fitness-level/

    Wearable technology devices make it possible to measure physical activity in exciting new ways. Combining sensor data for speed (GPS or pod) and HR allows us to estimate VO2max accurately. The Firstbeat algorithm analyzes the relationship between HR and exercise speed at multiple points during a training session. Error correction built into the algorithm ensures that only reliable and representative data from each session are incorporated into the fitness level estimation. Consequently, fitness levels can now be measured during every qualified training session when HR and speed data is available.

    https://www.firstbeat.com/wp-content/uploads/2017/06/white_paper_VO2max_30.6.2017.pdf

    Typically VO2max is measured directly by analyzing inspired and expired breathing gases in a laboratory setting during maximal exertion, and expressed either as absolute maximal amount of oxygen per minute (L/min) or as relative to the individual’s weight as the maximal milliliters of oxygen the person uses in one minute per kilogram of body weight (ml/kg/min).

    I quoted this to show that Firstbeat has noted the difference between absolute and relative VO2 Max, for the purposes of this paper. Later on the paper, it's implied (but not outright stated), that the algorithm is estimating *relative* VO2 Max, as units of "ml/kg/min" are quoted more than once.

    Of course that doesn't preclude what you said about the algorithm possibly first estimating absolute VO2 Max, then dividing by weight to derive relative VO2 Max, but there's no reference to that anywhere in the paper. If you search for "weight" in paper, there's no explicit reference to needing weight as an input or to using weight in the algorithm.

    Calculation steps
    The following calculation steps are used for VO2max estimation:
    1) The personal background info (at least age) is logged
    2) The person starts to exercise with a device that measures heart rate and speed
    3) The collected data is segmented to different heart rate ranges and the reliability of different data segments is calculated
    4) The most reliable data segments are used for estimating the person's aerobic fitness level (VO2max) by utilizing either linear or nonlinear dependency between the person's heart rate and speed data

    Note the lack of reference to weight above. Below, note the usage of "ml/kg/min" units.

    VALIDATION OF THE FIRSTBEAT MODEL
    The Firstbeat VO2max method has been developed against laboratory measured VO2max values (Figure 3) and validated with different exercise modes. The accuracy of the method when applied for running is 95% (Mean absolute percentage error, MAPE ~5%), based on a database of 2690 freely performed runs from 79 runners whose VO2max was tested four times during their 6-9 -month preparation period for a marathon. In a vast majority of the measurements, the error was below 3.5 ml/kg/min and the error was evenly distributed around the mean value. For perspective, the error in a typical indirect submaximal test is 10-15% and in a direct laboratory test about 5%

    ...

    PRACTICAL USE OF THE FIRSTBEAT METHOD

    Next, more details about the key practical use cases of the Firstbeat VO2max method are presented.

    As described, VO2max can be expressed as ml/kg/min, but that value can be confusing for persons unfamiliar with the topic.

    I realize that none of that "proves" anything, but it's clear to me that following the initial explanation of absolute vs relative VO2 Max, every subsequent mention of VO2 Max refers to relative VO2 Max. Given that Firstbeat says nothing about using weight to convert an estimated absolute VO2 Max to relative VO2 Max, as well as my intuition about the fact that speed is used to estimate VO2 Max, I can only conclude that the Firstbeat algorithm is estimating relative VO2 Max.

    Again, I will use my hypothetical example where a runner loses weight but their speed remains the same (at a given HR) -- let's say they lost both fat and muscle, causing them to miraculously run at exactly the same speed as before..

    If the algorithm is calculating absolute VO2 Max, then their estimated absolute VO2 Max will stay the same, but their calculated relative VO2 Max will increase, due to the weight loss. But that can't be right, because that runner's 10k race pace (for example) will remain unchanged. All other things being equal, if your VO2 Max increases, then your race pace should also increase.

    If the algorithm is estimating relative VO2 Max, then the estimated relative VO2 Max will not change, which is exactly the result I would expect.

  • One way to validate which hypothesis is correct would be for a runner to wear two identical, brand new Garmins synced to the same chest strap (if possible). Each Garmin would be synced to a different account (both of which would be brand new.) Each watch would have different weights set in the user profile (and the weights would not change over time.) If I were doing this, I would have a huge gap between weights (like 30-40 lbs). All other physiological parameters, especially max HR, would be identical between the profiles.

    (Then again, since all of the estimation is done on the watch, it really isn't necessary to sync either watch with a Garmin account at all.)

    The runner would run outdoors for two weeks or however long it takes to get a VO2 Max estimate.

    If the estimates are the same for both watches, then I'm right (the algorithm is estimating relative VO2 Max.) If the estimates are different, then the algorithm is estimating absolute VO2 Max.

  • That white paper refers to estimation of VO2 based on HRV

    The point was to reiterate how Firstbeat is estimated ventilation (HR, HRV). Under exercise, the ventilation alreay reflects the weight, age, gender influences.

    The second aspect is to establish the model between ventilation, pace and duration. This is the model that would be used ot establish race predictions. Running at a given pace requires an amount of oxygen per min (oxygen cost relationship); also, if you want to run for a certain time, there is a limit to the percentage of VO2 Max you can use during this time (Daniels/Gilbert drop dead formula). Using these relationships, it is possible to establish which pace you can sustain for a given distance given your ventilation capabilities. 

    Garmin says they maintain these relationships using predictive analytics that cover the linear and non-linear relationships between ventilation and pace. They predicts the maximum pace in ideal conditions you can sustain to cover a given distance given your VO2 profile. 

    Most of the VO2 Max estimation formulas i have seen are expressed as relative so that they can be used by any individual. At the same time, I have read papers where the unit chosen was the absolute one to establish relationships between ventilation and blood lactate concentration for example.

    Thinking of it, the topic of units for the VO2 Max is a bit of a red herring in our discussion since weight is known data. 

  • Reading the above posts, it seems like the two main metrics used to determine VO2max (on which the race predictor is based, I understand), are speed and HR. There are so many flaws with that! How does the watch know what shoes I'm wearing, race carbons or heavy trails, for speed is affected greatly by that. If you take a number of people, all with very close finishing times in a race, their VO2 max's will be very different. If I run 1k intervals hard on fresh legs, I will be able to get into zone 5 by the end of the rep, by which time I will be going at a good speed. If I run a long progression run, I could push it towards the end and get into zone 5 also, but I will be tired and comparatively slower. So, if Garmin primarily uses speed and HR, it can quite easily be manipulated.

    I have noticed that my LT has changed similarly depending on the type of runs I do - intervals with fresh legs and I get a good pace at LT, long progression run and Garmin tells me I'm slower at LT. I suspect my LT is wrong also. Garmin puts it at 168bpm. However I ran a HM at an average HR of 170, recorded on a chest strap HRM, which I use for every run. Isn't LT supposed to be what you can hold for an hour?

  • So, if Garmin primarily uses speed and HR, it can quite easily be manipulated

    Let's back up a bit. Garmin says about VO2 Max estimates before getting into the details about white papers:

    "Over the course of your activity, the Firstbeat Analytics engine built into your device examines the relationship between how fast you’re moving and how hard your body is working to maintain that pace."[...}

    "Intelligence built into the analysis ensures that only the best, most meaningful portions of your performance data are used."

    "Because the analysis is designed to learn about your own performance parameters and patterns over time, recording more walks and runs with your device improves the reliability of your VO2 max estimate."

    https://www.garmin.com/en-US/garmin-technology/health-science/fitness-tracking/

    What does this mean? The algo is retaining your best, most qualified (data filtering and qualification) running data during each run (sub-maximal or maximal efforts) and compounds this data with previous estimates (learning aspect) to give you a new prediction. Your HR/HRV and ability to sustain a certain pace for a certain time already reflects your shoes, the hills, the temperature, your overall health, your motivation, etc.: it is linked to the physiological cost of the effort.

    So, if Garmin primarily uses speed and HR, it can quite easily be manipulated.

    I am not sure what you mean here, but clearly if you introduce some bias in the data the watch collects, it will have an effect on your estimates.

    In a lab environment, the average error rate of the algorithm is 5%.This is an average. For some people, it will be higher, but over time, with a large number of estimates, it should be around 5%.

    HR Max estimations errors change the average error rate of the VO2 Max algorithm and tend to amplify it:

    https://assets.firstbeat.com/firstbeat/uploads/2017/06/white_paper_VO2max_30.6.2017.pdf

    To be fair, the validation of the algo has been done in a lab, so you are right that real life changing running condiitions, despite the "smart filtering" is more challenging.

    However, since Garmin is using machine learning, notwithstaning the risk of chronic bias, the accuracy should be close to 5% which is excellent.

    What is even more important is that the process is repeatable and predictable. So accuracy is not the most important thing here. We all use things that are inaccurate but very useful because they are predictable and repeatable (I am thinking about the fuel gauge in my car, for example).

    So this was for VO2 Max estimates.

    If you take a number of people, all with very close finishing times in a race, their VO2 max's will be very different.

    Exactly right. In fact research has shown that LT data is much better predictor of endurance race performance than VO2 Max alone. If you take 2 people with the same VO2 Max, the one with the highest VO2 at LTHR as a % of VO2 max will win the race.

    But again, for the race predictor Garmin is using all your running data history to maintain the relationships between ventilation (aka HR/HRV), pace and duration. VO2 Max is a key data point, but only one.

    I have noticed that my LT has changed similarly depending on the type of runs I do - intervals with fresh legs and I get a good pace at LT, long progression run and Garmin tells me I'm slower at LT.

    Yes, if you have LT auto-detection on, you will see variations depending on the intensity and duration of your intervals. You should rely only on the LT built-in test. It is set up as a calibrated ramp and is going to be more accurate.

    As noted above, remember that HR Max also influences the estimation of your LTHR, so whenever you see a change in HR Max, you should perform another LT test to realign the expectations of zones, training effect, etc.

    Garmin puts it at 168bpm. However I ran a HM at an average HR of 170, recorded on a chest strap HRM, which I use for every run

    Commonly the speed and HR of a 10k is said to be very close to the LT values, while the same data for a 5k would in the 105-110% range of them. So, yeah your LT values seem off.

    Isn't LT supposed to be what you can hold for an hour?

    Technically speaking LT is a lactate blood concentration inflexion point.

    Other thresholds (O2 or VCO2 exchange VT2, or maximum steady metabolic state) tend to happen close to that lactate blood concentration and are often assimilated with it. 

    The most interesting thing about LT is that improving your lactate clearance rate is better done by training in Zone2 than by training in or around the lactate threshold itself. Hence all the "rage" about 80/20, polarized training etc.

    www.trainingpeaks.com/.../

  • Thank you for the detailed explanations.

    "Commonly the speed and HR of an HM is said to be very close to the LT values"

    The HM pace was almost 30secs/km faster than Garmin's LT pace. I wouldn't call that close. Incidentally, is the Stamina metric based on LT too? That told me I was at 0% after 7k. I ran a negative split.