Newbie: How to dump raw FIT data to text (not FitToCSV.bat)

Hi km408,

'number' values, containing 17 digits looks funny at first, but:

Garmin stores all latitude and longitude values as 32-bit integer values.
This unit is called semicircle.
So that gives 2^32 possible values.
And to represent values up to 360° (or -180° to 180°), each 'degree' represents 2^32 / 360 = 11930465.
So dividing latitude and longitude (int32) value by 11930465 will give the decimal value.

Example:
Stored semicircle latitude value is 373473280.
Divided by 11930465 gives 31.30416794 => N31.30417.

On the other hand, distance and altitude values are stored as float values.

I'm using  FIT SDK for several years to create my 'own' FIT reader and writer, depending on fit type (Activity, Locations, Totals, ... from and for a Garmin Edge 1000), but I have no idea, how to implement such a simple and universal program to dump 'raw' FIT data 

Why:

Inside a fit file informations are stored as messages and each message can contains different values, depending on the message type.

A record message, contains information about ONE (1!) geo point and can contains a Timestamp, PositionLat, PositionLong, Altitude, HeartRate, Cadence, Distance, Speed, Power, ...
At the moment up to 74 different values in different units.

So for each value you need a corresponding function to get 'raw' data (C# example):

///<summary>
/// Retrieves the PositionLat field
/// Units: semicircles</summary>
/// <returns>Returns nullable int representing the PositionLat field</returns>
public int? GetPositionLat()
{
Object val = GetFieldValue(0, 0, Fit.SubfieldIndexMainField);
if(val == null)
{
return null;
}

return (Convert.ToInt32(val));

}

Hope this helps.

Best regards

Roland

Most values in a FIT file are stored as unsigned integers, either 8, 16, or 32 bits depending on the required precision. The FIT Profile includes a scale and offset for each field that is used to convert the unsigned integer values to/from float point values. The FIT CSV Tool converts the unsigned integers read from the file to floats using the scale and offset in the profile. The FIT CSV Tool does not manipulate the data otherwise. The Profile.xlsx file that comes with the SDK lists the data type, scale, and offset for each field. The Java, C# and C++ make the application of the scale and offset transparent, but if you are using the C SDK then you need to apply the scale and offsets manually.

In the Record message, distance and altitude fields are stored as uint32 and uint16 (more recently uint32) respectively. This is to reduce file size by not storing more precision than needed. For the purposes of fitness devices, you do not need to store distance to anything less than centimeters. Using unsigned integers along with a scale, and sometimes and offset, is how this is accomplished.

As an example, altitude values are stored as either uint16 or uint32 values. Originally it was uint16, but when people started to use fitness devices at higher altitudes it was changed to a uint32. In both cases an offset of 500 and a scale of 5 is used to convert the unsigned integer values to/from floats.

A concrete example, the lowest point in Death Valley is around -86.01456 meters. To convert that to a uint16 you add the offset, multiply by the scale, cast to a uint16, and store that value in the file.

i.e. (-86.01456m + 500) * 5 = 2069

Then to convert back to a float you apply the scale and offset in revers.

2069 / 5 – 500 = 86.2m

The 5 and -500 allow you to store elevation values from Death Valley to the top of Mt Everest with 20cm resolution using a uint16. When altitude is stored as a uint32, it is still 20cm resolution, but you can go higher.

The distance field has an offset of zero and a scale of 100, which gives you 1cm resolution.

For lat/long values the semicircle unit is used. A semicircle is a signed int32 which has a range of +/- 2^31 [-2,147,483,648 to 2,147,483,647] which gives you 2^32 values [4,294,967,296]. The circumference of the Earth at the equator is 4,007,500,000cm, which makes 1 semicircle is slightly less than 1cm.

From the other commenter's example:
Stored semicircle latitude value is 373473280.
Divided by 11930465 gives 31.30416794

Eight decimal places is about 1.11mm resolution.

Hi Ben FIT,

thanks for the detailed information, how especially distance and altitude values with specific values ​​for scale and offset are implemented.

Now I understand the line

newMesg.SetField(new Field("Altitude", 2, 132, 5, 500, "m", false, Type.Uint16));

inside Profile.cs => CreateRecordMesg()

thanks and best regards.

Roland

Thanks, Songokuonbike and Ben, for the tutorial information.  But that is not what I am looking for.  And I really do not need it.

FYI, Ben, you wrote:  ``The FIT CSV Tool converts the unsigned integers read from the file to floats using the scale and offset in the profile. The FIT CSV Tool does not manipulate the data otherwise.``  So you are saying that the "calculated" values are in the FIT file itself, not calculated by the tool (except for some conversion).  That is interesting.  I'll look more closely at the so-called profiles (protocol).

But there is indeed some "manipulation" going on.  I know this because the CSV file shows altitude, distance and speed with the precision of 1, 2 and 3 decimal places respectively, whereas the corresponding TCX file (downloaded from Garmin Connect) derived from the same FIT file shows those fields with 17 significant digits of precision.  In other words, at the very least, FitCSVTool.jar is rounding those fields to less precision than the data in the FIT file (or the calculations derived from that data).

And Songokuonbike, I do not find the 17-sig-digit numbers "funny" or daunting.  That is simply the least precision for the decimal approximation that is sufficient to represent all 64-bit binary floating-point values without loss of precision when converting either direction between binary and decimal, according to the IEEE 754 specification (*).

-----

But we digress.  Again, what I am looking for is a turnkey program and BAT interface, if the FIT SDK provides it directly or indirectly (I do not accept ad hoc programs from third parties), that simply decodes the binary records into text form with any interpretation at all.

As I noted, FitCSVTool.jar does not, in fact, do exactly that, at least not in all cases.  And I am hoping that there might be additional records or fields that FitCSVTool.jar is not converting.

With nearly 30 years of experience in O/S development in C, I am more than qualified to decode the FIT file myself.  But having retired more than 15 years ago, my C language skills are rusty (and of course I would use C++ or C# now), I do not even have those compilers, and I never learned Java -- although I could now, since it came with the FIT SDK.  So it is no longer second nature for me.  Hence the desire for a turnkey solution.

(The fact is:  I would limp along with Excel VBA to decode the binary file.   )

Again, I am trying to resolve some inconsistencies in the CSV file that I already have from FitToCSV.bat.

Perhaps I'll present them in another thread.  I might get some answers here before I get a useful response from Garmin engineering (WIP).

Thanks for taking an interest in my question.  But please try to stay on topic.

-----
(*) For example, whereas the exact decimal representation of the binary approximation of 2.345 is 2.34500000000000,0195399252334027551114559173583984375 (**), it is sufficient to write 2.34500000000000,02, the rounded 17-sig-digit approximation, because the next closest binary values (2.345 -/+ 2^-51) are 2.34499999999999,98 and 2.34500000000000,06 (rounded).  The latter values (and others) can be the result of calculations (e.g. 3 - 0.655), yet they all look like 2.345 when rounded to 15 significant digits, as Excel displays (but not "stores"; a common misstatement).  Consequently, in languages like Java, C variants and Excel VBA, 2.345 = 3 - 0.655 is false.

This is what I do nowadays in my copious spare time:  explain to people why 10.1 - 10 = 0.1 is false (!).  

(**) I use period for the decimal point and comma to demarcate the first 15 significant digits.

Okay, what I hear Ben and Songokuonbike saying is:  there is no turnkey program in the FIT SDK that decodes the raw FIT data without interpretation; for example, the uint32 value for altitude instead of the calculated decimal value (*).  Instead, the SDK provides the tools for me to develop my own program, as I knew.  I was just hoping to avoid that, being lazy.

Thanks.  LMK if I misunderstood.

-----

(*) I thought perhaps fit.jar is that turnkey program.  But I cannot find documentation on it, and I could not make it work the way that I expected (hoped).

What is fit.jar, and how do we use it?

If there is an explanation online, I would appreciate it if you simply directed me to the webpage.  TIA.

-----

I am well adept at 64-bit binary floating-point arithmetic, and I need no explanation.  But for posterity, I will correct myself briefly.

First, I asserted that altitude is rounded to 1 decimal place in the CSV file.  But in fact, it is saved with 17-sig-digit precision in the CSV file.  I made the "rookie" mistake of opening the CSV file directly in Excel instead of importing the file and assigning the Text format to at least some columns, if not all.

Second, for distance and speed, which are rounded to 2 and 3 decimal places in the CSV file, the "greater" precision in the TCX file is an illusion.  The 17-sig-digit precision is the result of calculating the rounded values with single-precision, then "converting" them to double-precision.

Thanks, Ben.  It was a stab in the dark.