First list of FIT decoding libraries and basic benchmarking

thanks for that explainer

altitude is super easy to handle with just stacking field numbers to process

then there is weirdness like this, where the native cadence is just wrong (probably not wearing the watch on arm but carrying it otherwise) and the real cadence is external, have to figure out best way to detect/handle that

record (20, type: 2, length: 65 bytes):
  timestamp (253-1-UINT32): 2020-08-20T10:00:00 (966800000)
  position_lat (0-1-SINT32): 52.0000000
  position_long (1-1-SINT32): 4.0000000
  distance (5-1-UINT32): 30.42 m (3042)
  enhanced_speed (73-1-UINT32): 8.161 km/h (2267)
  enhanced_altitude (78-1-UINT32): 4.6 m (2523)
  vertical_oscillation (39-1-UINT16, INVALID): 65535
  stance_time_percent (40-1-UINT16, INVALID): 65535
  stance_time (41-1-UINT16, INVALID): 65535
  vertical_ratio (83-1-UINT16, INVALID): 65535
  stance_time_balance (84-1-UINT16, INVALID): 65535
  step_length (85-1-UINT16, INVALID): 65535
  xxx87 (87-1-UINT16): 0
  xxx108 (108-1-UINT16): 1417
  heart_rate (3-1-UINT8): 90 bpm (90)
  cadence (4-1-UINT8): 24 rpm (24)
  temperature (13-1-SINT8): 32 deg.C (32)
  activity_type (42-1-ENUM): running (1)
  fractional_cadence (53-1-UINT8): 0.50 rpm (64)
  0_1_RP_Power (1-1-UINT16): 70
  1_0_Power (0-1-UINT16): 306
  1_2_Cadence (2-1-UINT8): 86
  1_3_Ground_Time (3-1-UINT16): 234
  1_4_Vertical_Oscillation (4-1-FLOAT32): 6.125
  1_11_Air_Power (11-1-UINT16): 13
  1_8_Form_Power (8-1-UINT16): 43
  1_9_Leg_Spring_Stiffness (9-1-FLOAT32): 5.125
hrv (78, type: 4, length: 11 bytes):
  time (0-5-UINT16): {0.666 s (666), 65535, 65535, 65535, 65535}
gps_metadata (160, type: 3, length: 9 bytes):
  enhanced_altitude (3-1-UINT32): 1.2 m (2506)
  enhanced_speed (4-1-UINT32): 12.942 km/h (3595)

Hi,

Your point on fastest known distance is a very interesting topic, which I have been looking at a lot over the year. One of the feature of my iPhone app (which required the FIT parser) is precisely to do these kind of analysis, this is just beyond the parsing issue per say.

I would love to find out smart and fast way to do it. I build curves that are for every X seconds or every X meters the best achieved metrics over the course of the activity, the month, year, etc... I find these quite tricky to do both correctly (I have all kind of quirks/noise) and efficiently (quite slow overall)

I wrote about some of the trouble I had doing it here.

I'd love insight how to make them fast and without the noise, even if over time I fixed and achieved some speed up, it's still imperfect.

If you want to help fix my code, the core of the logic is in here (the app is open source)

-(GCStatsDataSerie*)movingBestByUnitOf:(double)unit
                            fillMethod:(gcStatsFillMethod)fill
                                select:(gcStatsSelection)select
                             statistic:(gcStats)statistic
                        fillStatistics:(gcStats)fillStatistic

I tried FitFileExplorer, it's nice! (btw, also nice that it already supports the M1!) But is it meant to support only some FIT-files?

With the command line tool I'm trying to build, for a large VIRB Ultra 30 file I get:

SUMMARY
---------------------------------------------------
Header

      size: 14
  protocol: 16
   profile: 2010
  datasize: 18962954
    dotfit: ['.', 'F', 'I', 'T']
       crc: Some(14112)
---------------------------------------------------
Data

 Global ID | Message type                 | Count
...................................................
       161 | camera_event                 |     24 *
       219 | UNDEFINED_MESSAGE_TYPE_219   |      1
        49 | file_creator                 |      1
       104 | UNDEFINED_MESSAGE_TYPE_104   |    104
       160 | gps_metadata                 |  60114 *
       210 | one_d_sensor_calibration     |      1
        21 | event                        |      1
        18 | session                      |      1
       208 | magnetometer_data            |  20405
        19 | lap                          |      1
       165 | accelerometer_data           |  20405
       167 | three_d_sensor_calibration   |     59
       164 | gyroscope_data               |  20405
        23 | device_info                  |      3
         0 | file_id                      |      1
        20 | record                       |   6209
       162 | timestamp_correlation        |      1 *
        34 | activity                     |      1
        22 | UNDEFINED_MESSAGE_TYPE_22    |      2
       209 | barometer_data               |   6209
...................................................
                                    Total:  133948 
---------------------------------------------------
UUIDs in file

  1. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_1_69_2017-10-15-12-02-01.fit
  2. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_2_69_2017-10-15-12-02-01.fit
  3. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_3_69_2017-10-15-12-02-01.fit
  4. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_4_69_2017-10-15-12-02-01.fit
  5. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_5_69_2017-10-15-12-02-01.fit
  6. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_6_69_2017-10-15-12-02-01.fit
  7. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_7_69_2017-10-15-12-02-01.fit
  8. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_8_69_2017-10-15-12-02-01.fit
  9. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_9_69_2017-10-15-12-02-01.fit
 10. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_10_69_2017-10-15-12-02-01.fit
 11. VIRBactioncameraULTRA30_Expansive_1920_1440_29.9700_3937280306_344591dc_11_69_2017-10-15-12-02-01.fit
---------------------------------------------------
Result

  Message types 160, 161, 162 (*) present: YES
  Parsed in full, no errors
---------------------------------------------------

(160, 161, 162 required for our workflow)

But for the same VIRB file, FitFileExplorer only lists:

file_id
file_creator
event
device_info
record
lap
session
activity

The VIRB (at least the later variants) keeps the 10Hz GPS data in gps_metadata/160 for example (seems the Fenix 6x only keeps enhanced_alitude + enhanced_speed in there), which isn't listed. Some GPS data in record as well for the VIRB at 1Hz, presumably averaged/enhanced or something.

EDIT: Btw, if you need absolute timestamps for VIRB data you need to augment all FIT timestamps with the timestamp_correlation/162 message, presumably logged at GPS satellite sync (?).

Hi there.

Yes, the version of FitFileExplorer on the App Store only has the messages defined in the sdk example (in the profile.xlsx the one with example = 1)

It should be easy to regenerate with other types from profile.xlsx. I will have a look when I get a chance.

Also by the way the app is open source https://github.com/roznet/connectstats 

I don't have a Virb. so you think it would be possible to make an example file available somewhere for testing?

Ah, I tried to make my tool agnostic as to what the fit-file contains and extract anything (whether it actually can, I don't know of course - compressed timestamp header aren't supported yet for example). On the other hand this means I have to find out how to actually use or process what I have extracted after the fact. Oh well.

Hmm tried to upload an example FIT directly to the forum but got an error. Anyway here's a dropbox-link (not sure how long it will last and I'll remove it when you've downloaded):

https://www.dropbox.com/s/g3com51zfdh5uzy/2017-05-29-13-05-42.fit?dl=0

This from a VIRB Ultra 30. It's only 100k as opposed to the 15MB+ files I work with, but It seems to contain most kinds of VIRB data, such as sensor data and the message types mentioned above.

EDIT: So far I have found no developer data in any of the many VIRB files that I have for work, if that helps. E.g. no field_description/206 etc.

If you want to compare future results I get:

SUMMARY
---------------------------------------------------
Header

      size: 14
  protocol: 16
   profile: 2010
  datasize: 101735
    dotfit: ['.', 'F', 'I', 'T']
       crc: Some(11130)
---------------------------------------------------
Data

 Global ID | Message type                 | Count
...................................................
       162 | timestamp_correlation        |      1 *
       209 | barometer_data               |    194
        21 | event                        |      1
        18 | session                      |      1
       161 | camera_event                 |      4 *
       210 | one_d_sensor_calibration     |      1
       104 | UNDEFINED_MESSAGE_TYPE_104   |      4
       164 | gyroscope_data               |     53
        20 | record                       |    194
        19 | lap                          |      1
       165 | accelerometer_data           |     53
        22 | UNDEFINED_MESSAGE_TYPE_22    |      2
       219 | UNDEFINED_MESSAGE_TYPE_219   |      1
       160 | gps_metadata                 |    581 *
        49 | file_creator                 |      1
         0 | file_id                      |      1
       208 | magnetometer_data            |     53
       167 | three_d_sensor_calibration   |      5
        23 | device_info                  |      3
        34 | activity                     |      1
...................................................
                                    Total:    1155 
---------------------------------------------------
UUIDs in file

  1. VIRBactioncameraULTRA30_Video_1920_1080_29.9700_3937280306_338eb533_1_44_2017-05-29-13-05-42.fit
---------------------------------------------------

what I find interesting/exciting about that data is the barometer info, I've not seen that in any other FIT file for any other device, even the native watch barometer does not appear to be recorded? fascinating a camera of all things records it and not the watch or even a tempe

Oh I assumed at least the watches would have the barometer [explicitly] logged outside of the virb, but can't actually find anything in bricerosenzweig's example files now that you mention it. Is it perhaps only implicitly logged via e.g. altitude? Interesting way of logging those values in bursts each sec (same as the 3d sensor messages), at least for a newbie such as myself, compared to gps_metadata that just logs one message each 100ms. I've just managed to use the calibration messages to calibrate the other sensor values (barometer is mentioned in the calibration section of the sdk docs, but I have found no corresponding calibration message for virb data). I'm learning a lot. About binary data and the programming language I've chosen.

yeah it's disappointing how much sensor data is not written to FIT despite it easily having the ability to store it but the legacy of FIT on very low power CPU devices probably mean "write as little as possible

even the temperature information from the watch internals is overwritten when there is a matching external sensor like the tempe pod - though I've been meaning to look directly at the FIT files in the hope that the data from tempe is actually written as developer data and then the SDK has been rewriting the native fields but originally they might still be there

ie. accelerometer data would be awesome but obviously not in FIT at least not in Garmin watches

another interesting performance tip: unpack is performance death for a zillion tiny fields

it's not even just the looping, I'd did some analysis last night, the more you manage to bypass unpack, the faster it all goes

when you write your own decoder though, you can do some tricks

rather than decoding the entire FIT file, you can optionally step through it, only decoding the local definition fields to get the data/developer field sizes so you know how much further to jump ahead to bypass a data block

example: decoding a year's worth of FIT file and processing them for analysis/display takes 39 seconds consistently on my computer

but if I put in bypass code to only decode types 18 (session) and 140 (undocumented firstbeat metrics) and jump ahead in the file data for every other type, I can process that same full year in THREE seconds

when you decode the local definition, add up the total field sizes at that time and store it in the local definition array, then later when you hit the data, do the bypass jump instead of decoding, tada

I even see a few seconds performance increase by replacing unpack 'C' with ORD (types 0,2,10) but you have to be sure it's only a single uint8 and not array (type 13)

I do not understand why 4 bytes are often reserved in field sizes for C 1-byte types, is it a legacy thing where they didn't know if they'd move to uint16 or 32 later? But that makes no sense since the FIT file doesn't change once written.

Means there's a lot of wasted whitespace in FIT files, could in theory be repacked further but there's no point unless someday we get sub-1-second recording (unlikely).