Metric average data types

I don't know if this will make sense. Unless you've tried to do this. But the Activity.Info "average" data types, for at least Power and HR are integers. And they really should be floats.

Here is why.

Say you want to know the AVG POWER for the preceding 10 minutes (any segment of a longer activity). You have two choices.

A: You can save EVERY current power reading for every second, and calculate the segment's avg power by adding up possibly thousands of values and dividing by the number of values. Ugh

But if the avg metric was saved in ACTIVITY.INFO as a float, there is a much easier way.

B. Simply save the AVG POWER at the start of the segment, and then 10 minutes later, grab the AVG POWER again. And use a simple formula to essentially calculate the weighted value that causes that change in AVG.

Except, since the AVG metric is an integer, that approach is broken.

For example. Say you are 5 hours into a ride, and your AVG POWER was 183, and 10 mins later is 184. What was the POWER during that 10 minutes? You have no clue. Since:

That 183 could have been 182.50 to 183.49. That 184 could have been 183.50 to 184.49

Anyway - probably too late to ask Garmin to fix the precision of AVERAGE metrics. :-(

ekutter-dnu over 6 years ago

With the precision you are looking for, you are also assuming power numbers reported are the exact value at the time from the power meter. I'm sure there is some smoothing going on. And depending on the power meter, they are likely reporting readings on every full pedal stroke which doesn't necessarily correlate with the Ant+ signal. Some power meters do have a proprietary interface to send power numbers much more frequently and with more accuracy. But you'd need to understand exactly what they are reporting. Plus you'd have to write your own Ant processing code to read the data.

I think you are trying to milk accuracy where you just can't really get it.

WillNorthYork over 6 years ago

I'll also point out that floats don't give you the accuracy you want, either, because they can't represent decimals precisely, in general.

For example:
System.println((234.56789 * 100000).toString());
^ This will print out something like 23456788.00, not 23456789.00 as you'd expect.

I have to question whether users will see the value in seeing average power to 8 significant digits / 5 decimal places, even in a perfect world where you had perfect data and a perfect averaging algorithm.

The NEW avg power is simply (234.56789 * 10,800,000 + 203 * 1,250) / 10,800,100 = 234.56424W

Are you saying that saving all 10.801 integers, the average would be more accurate? In fact, I think it would be less accurate, since you'd assume all 10,801 values were captured at the same time gap between samples.

This method of calculating the total average seems okay to me, although a bit unnecessary. Because of the limitations of floating point math (and all the other things discussed), you will never get an exact value. You are also losing precision due to rounding/truncation of intermediate values (i.e. the previous average).

I think most people would say that to calculate the total average, it's sufficient to keep a running total T and number of samples N:

newAverage = (T + newSample) / N
(Assuming that T won't overflow)

Notice how you're keeping the total and not the previous average, again because you don't want to lose precision by reusing the results of an imprecise division.

To do rolling average for a fixed window correctly, you don't need to save all the samples for the entire activity. Your window is 10 minutes, so you only need to store 600 samples....

In Monkey C I think this will be about 3000 bytes for the floats, plus a handful of bytes for the array overhead. Easily doable in a data field that's just performing calculations.

DaveBrillhart over 6 years ago

ekutter.... sorry... I'm convinced the rolling average is IDENTICAL to the average taken from saving every reading and doing an average over "n" readings. Here is proof... a listing of 27 random values between 50 and 800. The average of those 27 is identical to taking the rolling average only considering the prior average and the current reading. Note that I am NOT concerned with absolute accuracy.... I do want floating point accuracy that is much better than the "integer" value returned by the CIQ average. Thanks for motivating me to prove this to myself. Memory is an issue for devices like the Edge 520. And more concerning is the compute burden of running a 600 (or almost a 3000 item array - see next paragraph) value sum every second. Battery drain is surprisingly impacted by these kind of frequent computation loops.

Let me explain why I need a more accurate PWR AVG than an integer. Say I want to let a cyclist know the AVG POWER he is putting out up a climb that starts 8 hours into his ride and lasts 45 mins. Well, I take the overall ride avg power at the start. And again at the end of the 45 min climb. One simple calculation will give that to him. Say his avg power after 8 hours, entering the climb is 205W, and at the end of the climb is 207W. Can you tell me his climb power? Yes, but you'd be wrong. You'd tell me his avg power was 228.33W. Except his actual climb power is really somewhere between: 217 and 239. Because the actual averages might have been (204.50 to 207.49) or (205.49 to 206.50)... Therefore your estimated climb power is very likely very wrong. All I need are a couple extra digits of precision.

This forum can't seem to accept embedded pics any longer. here is my Google Sheets test:

https://docs.google.com/spreadsheets...it?usp=sharing community.garmin.com/.../1430911.jpg

ekutter-dnu over 6 years ago

Sorry, but you are talking about two completely different things here. Your example is not a rolling average. It is a total average. A rolling average would be the average of the last N data points, not the average of all the data points.

Your method works just fine if all you want is the lap average, or overall average. It doesn't work if you say want the last 30 second average.

What is it you actually want?

DaveBrillhart over 6 years ago

Ah - ok... what I need is a floating point overall average power for the entire activity... since Activity.Info only provides an integer overall power average.

Given a more precise FP average... Then I can calculate an accurate average power for any segment in the ride.... starting and ending at any point, of any duration. By only grabbing the FP overall average at the start and at the end of the segment. That might be a 10 min segment or a 2 hour segment. Starting 1 hour or 20 hours into the ride.

I use this for my LAPSTATS data field. Hit LAP at any point during your ride, and it displays your current LAP avg power. So I just grab the starting overall avg power at the start of your LAP, and then report on the LAP's current avg power as the lap evolves. At any point, I just grab the current overall average power, and compare it to the overall avg power at the start of the lap.

ekutter-dnu over 6 years ago

Correct, creating your own average where you keep adding to it is relatively easy as you point out. It's the rolling bit that gets complicated. To figure out the average, you just keep a running total of the (power * interval) and a total duration. Then you can compute the average over that period by dividing the total by the duration. I do exactly that for laps as well. When a lap starts, I have a lap running total and lap start time. Gets a little more complicated if you want to pull out zeros or pause time from those averages. If you need more precision, use floats. Less efficient but if you aren't needing every data point, it's not a big deal. Doubles are objects and not simple elements so the overhead for an array of doubles is pretty big.

If you are wanting to do averages over any random period after the fact, you do need to store some value for every data point. If those are the totals described above its an easy calculation:

(listPowerTotal[end index] - listPowerTotal[start index]) / (duration of segment from start index to end index).

That is almost certainly more efficient than just storing all the power numbers and running through calculating the sum on the fly.

But if all you care about is overall average and lap avg, just use to separate sets of counters, with the overall being reset at start, and the lap being reset at every lap.

WillNorthYork over 6 years ago

When a lap starts, I have a lap running total and lap start time. Gets a little more complicated if you want to pull out zeros or pause time from those averages.

Just my opinion, but since...
- Over long periods of time, there's not much difference between counting the number of samples and counting the number of seconds
- Garmin does not include pause time in its averages (or min/max) for things like heart rate. If you want your averages to match Garmin's averages, you have to exclude pause time.
- You always have to exclude nulls - absence of data. To use the example of average HR: your current HR is null - not zero- when you're not wearing your strap/watch, and you can see that Garmin's lap/total average does not change one bit when you take your watch off. It would be wrong to include nulls in the average as if they were zeros, to say nothing of actual zeros (which I don't think is possible with HR on a Garmin.) If you want your average to match Garmin's you have to exclude nulls.

- So whether or not you choose to ignore zeros, you always ignore nulls and pause time, and the amount of work / code complexity for calculating an average while throwing out any kind of sample is pretty much the same.
- I cannot think of a single legit physical scenario when you would want to include a null sample in an average, unless you want to interpret the absence of a sample as the sample being 0....

...the easiest thing to do is to forget about time altogether and just keep a running count. Otherwise you have to worry about storing the time of the last sample you actually used and total time you've seen valid samples, and to weight your average appropriately, which IMO, is a subtlety that is just not going to be noticeable over 30 minutes to 3 hours.

e.g.
(Replace square brackets below with round brackets).

var N = 0;
var total = 0.0;

var ignoringZeros = false; // set to true to ignore zeros

// Lap or overall average
function cumulative_average[var latestSample]
{
    // ignore pause time, nulls, and [optionally] zeros
    if [timerIsRunning && latestSample != null && [latestSample != 0 || !ignoringZeros]]
    {
        total += latestSample;
        N++;
    }

    if [N != 0]
    {
        return total / N;
    }
    return null;
}

As you pointed out, this can be either an overall average or lap average - the only difference is when you reset total and N.

If your app is complex and your code space is tight, this would be the simplest way to do things.

Storing the lap start time is useful for displaying/calculating lap time and lap pace though, obviously. (Lap or overall average pace would be distance / time, and not the same kind of calculation as average HR or power.)

IMO you do want to keep the number of samples for your lap/overall averages, if you care about excluding nulls and pause time (which I do) and optionally, excluding zeros. Obviously the assumption is that compute() is called approximately once per second. If there was some scenario where compute() could somehow be delayed for five minutes, then your way (which is more general) would be necessary.

WillNorthYork over 6 years ago

I use this for my LAPSTATS data field. Hit LAP at any point during your ride, and it displays your current LAP avg power. So I just grab the starting overall avg power at the start of your LAP, and then report on the LAP's current avg power as the lap evolves. At any point, I just grab the current overall average power, and compare it to the overall avg power at the start of the lap.

As ekutter this mentioned, this is completely unnecessary. Just calculate the cumulative average (total / numberOfSamples) and reset total and numberOfSamples at the beginning of every lap. If you are concerned about precision, remember that you lose precision every time you do a division (your result will be a float or a double), so it's best to avoid re-using intermediate results in your final calculations, even if you calculated those intermediate results yourself and didn't grab them from ActivityInfo.

If you really care about the fact that compute() is not called every second, then you would have to weight your running total/average appropriately.

Given a more precise FP average... Then I can calculate an accurate average power for any segment in the ride.... starting and ending at any point, of any duration. By only grabbing the FP overall average at the start and at the end of the segment. That might be a 10 min segment or a 2 hour segment. Starting 1 hour or 20 hours into the ride.

If you really want the "segment average" (e.g. average from 0:30 to 2:30 or from 2 km to 5 km), then you don't need a rolling average and you can use the simple cumulative average calculation, even weighting it by time if you want. And you still don't need the average at the beginning and end of the segment. You just reset your total and count at the beginning of each segment.

On the other hand if your app displays the average for "the last 2 hours" [the start time for your window is constantly changing], then you need a rolling average.

If you do want to calculate an average over a rolling window of 2 hours, that's probably too much data for your app to store (about 36000 bytes). In that case, you could divide the window into 1 minute buckets, and take the average of your 1 minute averages....

xuxumatu over 4 years ago in reply to jim_m_58

Hi him i have used this method for calculating 30 seconds average power

var aAvg = 30;

avgPower = ((avgPower * (aAvg-1) + rawPower) / aAvg).toNumber();

but after a lot of tests i recently noticed that avgPower is 29 down that it should be

i´m not using the

x++;

could it be the reason??

Travis.ConnectIQ over 4 years ago in reply to xuxumatu

The problem is that you're truncating avgPower to the next lowest integer value every time you do this calculation.

Lets just assume for a second that avgPower is 150 and the new sample is 162... ((150 * 29 + 162) / 30) => ((4350 + 162) / 30) => (4512 / 30). If you do an integer divide, the result is 150. If you do a floating-point divide you keep the decimal and you get 150.4. In the next second you would be throwing away (0.4 * 29) => 11.6 from the power value.

To do this right, you need to do the arithmetic with floating point values, and then the presentation with an integer.

hidden var avgPower;
hidden const aAvg = 30;

function compute(info) {
    var rawPower = info.currentPower;
    if (avgPower == null) {
        // ensure that avgPower is a Float so we get fractional parts
        avgPower = rawPower.toFloat();
    } if (rawPower != null) {
        // if avgPower is Float, all of these calculations will produce
        // a Float as well, and no precision will be lost
        avgPower = (avgPower * (aAvg - 1) + rawPower) / aAvg;
    }
    
    // without modifying our Float, get the average value as a whole
    // number. this will truncate to the nearest whole number below
    // the actual value.
    return avgPower.toNumber();
    
    // this would do proper rounding... 150.3 => 150, but 150.7 => 151
    // return Math.round(avgPower);
}