This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

955 massively underestimating my ability

My predicted race times and my "stamina" meter are way off and have been over the course of 3 months of consistent use. For example, my race predictor gives an estimated 5k time that is only a few seconds faster than the 5ks it has recorded from tempo runs. When doing any speedwork, my stamina will regularly go to 1% halfway through each interval. At the same time, it does seem to be recording improvements, as my VO2 Max and estimated times steadily improve as my workouts get faster. 

While I haven't raced with this device yet, I'm quite sure it's very wrong, based on past experience with workouts and races. 

What could be causing this issue?

  • Model is based on single metric

    This is not correct,m as described in the FirstBeat (not Fitbit) white paper you listed.

    See also this FirstBeat white paper to see how their VO2 Max is better with on/off intervals with accelerometers and GPS to the HR/HRV/load model: the mean average error (vertical axis) when compared with the lab measured VO2 Max is much lower than HR/Load alone (which is what Runalize uses)

    pulkkinen_at_al_acsm_2004_congress.pdf (firstbeat.com)

    For an independent review of VO2 Max models and Firstbeat's model comparison, I recommend this excellent article here:

    www.muscleoxygentraining.com/.../firstbeat-vo2-estimation-valid-or-voodoo.html

    The thing to understand is that the VO2 Max model has an average error rate (across athletes during validation) of 5%. In the analysis below, the blogger spots an athlete where the error is actually 20%! So your mileage will vary, as in every model.

    5% is excellent. In this study on treadmill running, various other models only get to about 10% for maximal efforts.

    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4721596/

    Your time depends on your base (Garmin doesn't have that information), your volume (Garmin doesn't use that I believe), your technique/ economy

    Do you have links to research linking race times with these parameters? I am curious.

    There is plenty of rresearch linking VO2 Max number to race predictions, and a lot of research showing the accuracy of race prediction with Critical Power based models. This is what Stryd uses : https://www.researchgate.net/publication/373398169_Prediction_of_Half-Marathon_Power_Target_using_the_93-Minute_Running_Critical_Power_Test

    As I said above, the main issue with the Garmin predictions is that they don't come with error rates. If uses could see the expected average error rate, expectations would be better managed.

  • Thanks for the information - really useful, will read that stryd paper (never had stryd, although I do use garmin pod).

    What I meant by a single parameter was that there is decent amount of data that Garmin doesn't know about (although you're right - I did not know that they take into account accelerometer data). My point is that a person that've had a decent base in a gym, preparing muscles for the season, doing loads of box jumps, jumping rope, decent amount of barriers will definitely perform better (way better!) then a person with same speed/ hr/ step information without that base. Also, garmin tends to omit weekly milage information - but that affects a lot, especially in a marathon (altough running volume maybe replaced with excersize pretty effectively). 

    Secondly, race predictor assumes that a runner is "flat" in terms of distances. Personally I perform better for shorter races (5k) then longer ones (marathon). I assume there are people who are slower at 5K (and there shall be normall runners somewhere I assume). That is not in the model (and no way Garmin will knwo that, unless they incorporate byopsy sensor in 975 :) ). 

    Also personally it is hard for me to beleive into that 5% error rate. Really - my mistakes are bigger. I am running more or less same pace (1.34-1.40 for half), but in May it was 56 (loads of 1000m intervals at 3.45) and today it is around 52 (I just do less short intervals - almost none). Topicstarter had 5 min+ mistake for 1:24 race - that is 6% difference. I had vo2max of 51 in spring with that 1.35 time, prediction was around 1.41-1.42 - that is 7% mistake. That is still impressive (no joking) but totally unusable. 

    It may be protocol in the research that all runners in studies follow (while "normal" runners out there doesn;t), or a number of researches that are repeated until there's a successful set mof data (I really don't want to dive into conspiracy) - I dont know, but it looks that mistake is higher then 5%. personally I think that is a protocol. I record warm-up (2k normally), exercises and warm-up sprints as a single activity - that affects vo2max I think. I beleive data will be better if I skip those, but I prefer to have it. In runalyze you can set "do not use this activity in vo2max calculation". So that's the garmin's weak point - understand better how things are calculated and how to use them correctly. People expect prediction to be bang-on - and that 5% mistake is considered to be :"small". But that is 4+ min in 1.30time for half - which is considered "huge". So that's expectation issue.  

  • Also, garmin tends to omit weekly milage information

    This is not correct. The Garmin Race Prediction use your training history (like other metrics like Stamina, Training Effect, Training Load, Encurance Score, etc.)

    "The watch will use VO2 max. estimate data and your training history to provide a target race time. The watch analyzes several weeks of your training data to refine the race time estimates.

    The projections may seem inaccurate at first. The watch will require a few runs to accurately provide an ideal prediction. These times are just predictions and do not factor in variables such as weather, course difficulty, or training regimen."


    https://support.garmin.com/en-US/?faq=HUB4yrzJkg1BbgmozWkBm7

    "Not only do stamina levels carry over from one activity to the next, but activity intensity profiles, durations, distances and training load accumulations also are used to enhance the personalization of your real-time stamina feedback beyond your basic fitness metrics."

    https://www.garmin.com/en-US/blog/outdoor/introducing-the-garmin-real-time-stamina-feature/

    "To determine the training effect of a single exercise, peak EPOC achieved during exercise (see figure 3) and the activity class of an individual must be known (see figure 2). The activity class value representing the activity level of the previous month must be defined to individualize training effect interpretation according to training level."

    https://assets.firstbeat.com/firstbeat/uploads/2015/10/white_paper_training_effect.pdf

    "After 4 weeks of training history, your training load estimate will have more detailed target information to help you balance your training activities."

    https://support.garmin.com/en-US/?faq=SEkNpdGyhR917js0qQL3Q6

    "As a result, both longer- and shorter-term training perspectives are factored into your endurance score calculation to reliably reflect your current situation. The longer-term perspective considers between 2 to 3 months of data, while the shorter-term perspective is limited to the most recent 2 weeks."

    https://www.garmin.com/en-US/garmin-technology/running-science/physiological-measurements/endurance-score/

    Secondly, race predictor assumes that a runner is "flat" in terms of distances

    Can you point to the information describing this? On my race predictions graph, the different races points do not follow strictly parallel lines. For example, my marathon predictions dropped significantly at one point while my 5k prediction didn't move with the same amplitude by far.

    Also personally it is hard for me to beleive into that 5% error rate.

    It is an average. Your error rate may be very different for sure. Also, to clarify, this is the error rate of the VO2 Max model, which is one component of the race prediction. Garmin doesn't publish, alas, the error rate of their predicitons.

    So that's the garmin's weak point - understand better how things are calculated and how to use them correctly. People expect prediction to be bang-on - and that 5% mistake is considered to be :"small". But that is 4+ min in 1.30time for half - which is considered "huge". So that's expectation issue.  

    I very much agree with you there.

  • So that's the garmin's weak point - understand better how things are calculated and how to use them correctly. People expect prediction to be bang-on - and that 5% mistake is considered to be :"small". But that is 4+ min in 1.30time for half - which is considered "huge". So that's expectation issue.  

    I very much agree with you there.

    4+ minutes off for the half-marathon makes the race predictor functionally useless, and indeed worse than useless. 

  • Remember this is a range of average error. It can be more or less for you. Your car speedometer has an average error of 3%, and your car nav a bit more, yet you find them useful.

    As usual, predictions like this are less useful for athletes that commonly race typical distances. Following your PB history is probably more satisfying.

    i am not such runner. So I find it useful to get predictions. As I wrote elsewhere, for my last 5K training plan, the predictions were:

    - Garmin with specific course and date: 22:05

    - Garmin generic 5k: 21:45

    - Stryd: 21:51 +/-26s (sgnifcant range for a 5k)

    - Galloway Magic Mile formula: 22:20

    - Actual time: 22:08 with some gas left (new PB)

    So I knew 22:00 would still be a stretch goal! But this is still the goal I put in my next 5k training plan.

    if you find a race predictor that nails your upcoming races, I am eager to try it and compare.

  • In fact there are many race predictors that are widely available and are massively more accurate, namely, those based on systems like those of Jack Daniels (https://runfastcoach.com/calc2/index.php) or Tom "tinman" Schwartz (https://runfastcoach.com/calc2/index.php). These will both get you very close to a race time prediction based on your times for various types of workouts. (My time for my recent HM was within a couple of seconds per mile of what those calculators predicted.) The frustrating thing is that the Garmin knows about these types of workouts (e.g. lactate threshold, VO2 max) and (usually) detects them—and has much more data about them than you can input into these online race predictors—and still comes up with much worse predictions.

  • Oh for sure lots of similar models.

    I stopped using VDOT and other models available in Runalize when I got Stryd as an early adopter. In fact I do not use Runalize any longer.

    The frustrating thing is that the Garmin knows about these types of workouts (e.g. lactate threshold, VO2 max) and (usually) detects them—and has much more data about them than you can input into these online race predictors

    Agreed. The race predictions have been quite OK for me, but I am not surprised they can be off.

    As we discussed above, with the current algorithms and current implementation, the best foot we can put forward is to input a correct HR Max, run a balanced training program that exposes the watch to maximal effort on flat grounds, and ensure the best data capture possible.

  • 1. Regarding milage - the quote you mentioned does not refer to milage directly. "Training history" may not be milage, but just recorded workouts. I really don't know for sure (so you may be right). 

    2. Regarding "flat racing times" - I judge by tables that were available online (up to at least 935, I assume) that contains "vo2max - race prediction for different distances". It doesn't mean they are flat in times (e.g if you do 5k for 19min, your 10k shall be 5*19=38), but rather flat in terms "average for all people of that shape" (if you get a given vo2max, your 5k time is 19, 10k is, say 39, HM is 1.22). I treat those times as "your potential with proper training", and personally I can possibly get to 5k time and definitely far from HM time (and 10k is somewhat inbetween). There are different people with different muscle construction (with my genetics I am better for shorter distances (I shoulve' bene doing sprints or 800-1500, I assume).

    For example I have a friend with more or less same vo2max (or worse then mine), but he's consistently doing better at halves and marathons. 

    3. Regarding stamina - have not dived deep into that, but it is showing crap for me for some reason. While the whole concept is nice and it worked well in intervals (my feelings totally corresponded with stamina - first intervals being "easy" and last being "struggling" - totally corresponded to when stamina level went to 0 during interval. But during recent 10k race it showed total crap. I was going in a mid-to-working tempo mode, not pushing, but I assume that was above garmin estimation of my abilities. Stamina I was at zero in 7-8 minutes (I was still feeling moderate tempo run - totally fine and enjoing the race). It stayed there (at 0%) for 10 minutes, then went up to 30% (I slowed down due to injury pain) and then  (back to tempo pace) - gradually went down to 10% (end of race). Potential went down and joined stamina at some point (potential looked reasonable). So there's some flaw in the model or data (I believe my threthold is different from what garmin thinks it is - at least for pace). 

  • I think most predictors are based on competitive runners, that is not 20+ min 5k time, but rather 15-16-17 min race time. But generally I agree it shows what class you're in and shows you the pace you may try. 

    I don't use them (at all) - because many of them miss. I improved seriously (say 1.45HM to 1.30-32 HM) when I got a trainer and did a proper base and physical training, and predictions were all way around (with runalyze showing 10+ mins slower etc). I still struggle with improving (actually declining last year), but I don't think aiming for a specific time is a good ide. 

  • I think the reason why garmin is not using that is the fact that it is hard to do on-the-go on the device (power is limited, not 100% history is always kept etc), while fitbit is (as far as i know) licenced the tech to each individual device and algorithms are device-based, not garmin-connect base. Hense these issues (not big ones since you can sync all data to Peaks/ runalyze/ etc). So they consider that "okay" and asume you can use a 3rd party to analyze your workouts.