Where can I find definitions for the large proportion of messages types generated by my Garmin devices which are not defined in Profile.xlxs?

In the FIT files which I download from Garmin Connect which are generated by either my Epix gen2 or my Fenix 6 watch, there are 11 of 30 message types which do not have a mesg_num defined in Profile.xlxs. Does anyone know where these are defined and how you could generate an SDK which actually works without a full definition?

For my purposes, I don't actually need to decode and encode the unknown fields because I only want to fix some mesg_num = record = 20 messages.

However, fields that will not decode cannot be re encoded into an output file and Garmin import throws an error message if you try to upload a FIT file that is missing these undefined fields.

On the other hand, Strava doesn't care that these fields are missing.

I noticed that the Muktihari's Golang FIT SDK provides a raw decode encode facility which would allow passthrough of unknown fields.

However, I'd like to use the Garmin C++ FIT SDK if it were possible.

Also, the Goland SDK notes in its profile doc that:

"supplementary items related to "Global Profile" that are not
declared in the Profile.xlsx itself but are provided by Garmin
or its affiliates website, such as common file types,
are added manually."

I couldn't find these manual additions.

As an example of the astounding number of undefined messages in a file generated by my Garmin watch see below:

- Undocumented FIT_MESG_NUM according to FIT File Viewer
- In a run activity file from 2/28/2024
- Out of 30 message types 11 are undocumented
- mesg 288 3 rows of timestamps + 3 integer fields
- mesg 327 4 rows of timestamps + 6 integer fields
- "GPS Events" 34 rows of timestamps + event type + mode
- mesg 147 3 rows of 11 fields
- mesg 79 1 row of timestamp + 41 fields
- "Other Settings" 1 row of 50 fields
- mesg 325 104 rows of timestamps + 6 integer fields
- mesg 104 5 rows of timestamps + 4 integer fields
- mesg 233 196 rows of 1 field
- mesg 140 1 row of timestamp + 65 fields
- mesg 133 3 rows of timestamps + 6 fields

  • I haven't encountered this exact issue, I wonder if reaching out directly to Garmin support or posting on their developer support channels might yield some insight. Sometimes they have internal documentation or updates that aren't widely publicized.

  • There has always been unknown messages and fields in the FIT SDK, and the expectation is that there always will be unknown messages and fields. The comment in the Golang repo is probably referring to things that have been reverse engineered and then posted to this, or another, forum. 

  • Yup, I did send an email asking this question to Garmin developer support. No response yet. Digging into the history, I get the impression that the SDK may have been acquired along with the 2006 acquisition of Datastream who invented ANT. Perhaps, SDK maintenance has been minimally funded since then. I'll certainly share any insight they might eventually give me. 

  • Yes, I read your previous post about there always being unknown messages. However, given the current structure of the C++ SDK, unknown messages are a major problem since unknow messages and fields are discarded by the decoder and cannot be re encoded by the encoder such that you cannot repair small problems in a FIT file without making additional unintended deletions. So, either the SDK should support passing unknown fields through in raw format or definitions should be added to the Profile for all messages. Basically, I would like to understand why Garmin has no intention of finishing the SDK. NOTE: the repair tool in the Java SDK does NOT repair FIT files. But, rather rewrites some basic elements of a corrupt FIT file into valid FIT file. This valid FIT file would typically be missing many elements of the original FIT file.

  • I would revisit your decoder code. The example decoder only listens for known Message types (and even then, not all known types), but I don't see a reason why you could not use the generic message listener to get all messages, which you can then modify and re-encode into a new fit file.

    I use the java SDK and have defined my own listener to grab specific known and unknown message types, altered field values and re-encoded them into new fit files.

    I've had a quick look at the C++ SDK, can't see why the same cannot be achieved. 

  • I did not say that decoded messages could not be re encoded in a FIT file. In fact, they can and I have done so using the C++ SDK. The Java SDK is essentially identical in structure and produces the same result. The difficulty is that not all messages are decoded and therefore not all messages can be re encoded. Try it. See if when you decode and re encode a FIT file using a generic listener which just Writes decoded message if you get back the file you read in. You do not. In fact, you get back a file missing the unknown messages. This is not a problem if you are just uploading the new FIT file to Strava since Strava ignores these Garmin specific unknown messages anyway. But, for my device, Garmin Connect insists that at least some of the unknown messages encoded by my watch exist or it calls the file "invalid". And, I want to fix the files on Garmin Connect not just on Strava.

    class FilterListener
        : public fit::MesgListener
    {
    public:
        shared_ptr<fit::Encode> encode;
        void OnMesg(fit::Mesg& mesg)
        {
            mesg.RemoveExpandedFields();
            if (mesg.GetNum() == FIT_MESG_NUM_RECORD) {
                // TODO: Rewrite mesg to include fixed heartrate values
            }
            encode->Write(mesg);
    
        }
    };
    
    int Recode(char* filename)
    {
        // Open file to decode
        fstream inFile;
        inFile.open(filename, ios::in | ios::binary);
        if (!inFile.is_open())
        {
            printf("Error opening file %s\n", filename);
            return -1;
        }
    
        fit::Decode decode;
        if (!decode.CheckIntegrity(inFile))
        {
            printf("FIT file integrity failed.\nAttempting to decode...\n");
        }
    
        // Open the file to encode
        std::fstream outFile;
        outFile.open("badFileRecode.fit", std::ios::in | std::ios::out | std::ios::binary | std::ios::trunc);
        if (!outFile.is_open())
        {
            printf("Error opening file badFileRecode.fit\n");
            return -1;
        }
    
        try
        {
            // Create a generic listener which encodes all messages types without
            // changes except for RecordMesg which may be encoded after fixes
            FilterListener listener;
            // Create a FIT Encode object
            listener.encode = make_shared<fit::Encode>(fit::Encode(fit::ProtocolVersion::V20));
            fit::Encode encode(fit::ProtocolVersion::V20);
            // Write the FIT header to the output stream
            listener.encode->Open(outFile);
            // Add the FilterListener to the MesgBroadcaster's generic MesgListener list
            fit::MesgBroadcaster mesgBroadcaster;
            mesgBroadcaster.AddListener((fit::MesgListener&)listener);
    
            // Decode the input file and encode a fixed file
            decode.Read(inFile, mesgBroadcaster);
    
            // Update the data size in the header and calculate the CRC
            if (!listener.encode->Close())
            {
                printf("Error closing encode.\n");
                return -1;
            }
    
            // Close the file
            outFile.close();
    
            printf("Encoded FIT file badFileRecode.fit.\n");
            return 0;
        }
        catch (const fit::RuntimeException& e)
        {
            printf("Exception decoding or encoding file: %s\n", e.what());
            return -1;
        }
        catch (...)
        {
            printf("Exception decoding or encoding file");
            return -1;
        }
    }
    

  • If Garmin Connect is saying that a file is invalid, is it not due to a missing message that is not available in the public FIT SDK.There is another issue with the file. There are third parties, indoor training apps, that upload files to Garmin Connect and those files are created with the public FIT SDK.

  • True, perhaps the discarded fields are a red herring and I should just write a valid file with all the records but not worry about keeping all the original messages. It would be nice if Garmin Connect would give some detail as to what is "invalid".

  • The FIT CSV Tool has a -t option that runs a set of verification checks on the file. That might help narrow down the issue with your modified files that will not upload to Connect.
    https://developer.garmin.com/fit/fitcsvtool/commandline/ 

  • Good thought. I tried it and all tests passed except the last optional one. "One or more Device Info messages contain a null Manufacturer Id." I'm guessing "optional" means Garmin won't complain on upload for that. But, perhaps a Decode::CheckIntegrity on my output file would be more stringent. I'll try that next.

    Running FIT verification tests...
    Message Count: 3917
    FileId Message Exists - Level: REQUIRED Status: PASSED
    FileId Message Is First - Level: REQUIRED Status: PASSED
    FileId Message Type Is Activity - Level: REQUIRED Status: PASSED
    FileId Message Manufacturer Id Exists - Level: REQUIRED Status: PASSED
    FileId Message Time Created Exists - Level: REQUIRED Status: PASSED
    Activity Message Exists - Level: REQUIRED Status: PASSED
    Activity Message Timestamp Exists - Level: REQUIRED Status: PASSED
    Activity Message Local Timestamp is Valid - Level: REQUIRED Status: PASSED
    Activity Message total_timer_time is Equal to the Sum of Session Messages total_timer_time Values - Level: REQUIRED Status: PASSED
    Activity Message Session Count Is Equal To Actual Session Count - Level: OPTIONAL Status: PASSED
    Session Message Exists - Level: REQUIRED Status: PASSED
    Session Message Timestamps are Valid - Level: REQUIRED Status: PASSED
    Session Message Start Time Is Valid - Level: REQUIRED Status: PASSED
    Session Message Total Timer Time and Total Elapsed Time are Valid - Level: REQUIRED Status: PASSED
    Session Message First Lap Index and Num Laps are Valid - Level: REQUIRED Status: PASSED
    Session Message total_timer_time is Equal to the Sum of Lap Messages total_timer_time Values - Level: REQUIRED Status: PASSED
    Session Message total_elapsed_time is Equal to the Sum of Lap Messages total_elapsed_time Values - Level: REQUIRED Status: PASSED
    Session Message Sport Exists - Level: REQUIRED Status: PASSED
    Session Message Sub Sport Exists - Level: OPTIONAL Status: PASSED
    Session Message Are Sequential and Abut - Level: REQUIRED Status: SKIPPED
            Check requires two or more Session messages, found 1.
    Session Message Valid Message Index - Level: REQUIRED Status: PASSED
    Lap Message Exists - Level: REQUIRED Status: PASSED
    Lap Message Valid Message Index - Level: REQUIRED Status: PASSED
    Lap Message Start Time and Timestamp are Valid - Level: REQUIRED Status: PASSED
    Lap Message Are Sequential and Abut - Level: REQUIRED Status: PASSED
    Record Message Timestamps Fall Within Session Message Times - Level: REQUIRED Status: PASSED
    Record Messages Are in Chronological Ascending Order - Level: REQUIRED Status: PASSED
    Device Info Message Timestamps are Valid - Level: REQUIRED Status: PASSED
    Device Info Message Device Index is Valid - Level: REQUIRED Status: PASSED
    Device Info Message Manufacturer Id is Valid - Level: OPTIONAL Status: WARNING
            One or more Device Info messages contain a null Manufacturer Id.