Unable to convert csv to fit file

It says "Ignored 47 unknown messages", it seems most your messages are mostly being skipped since it's a Product Profile specific messages, or we can say manufacturer specific messages, different manufacturer has different meaning. FitToCsv tool can only convert csv to fit only if the messages are specified in Global Profile (defined in Profile.xlsx).

When we convert fit to csv, we lose the information about value type, we can't reconstruct the value correctly without that information. Playing a guessing game would be a bad idea, The only thing we can do is generate custom FIT SDK by registering Product Profile of your watch, then rebuild that FitToCsv tool.

CMIIW.

Thanks for your reply muktihari

I understand the fact of loosing informations converting fit to csv, and a reverse conversion not possible without these missing informations (type).

What I don't understand is :

1. why Profile.xlsx does not have the informations related to the Garmin custom POI ? Because the FIT SDK is not a Garmin SDK but a Generic one ?

2. I opened the Profile.xlsx file and it seems not easy to complete ... I found others discussions about this topic (ex https://forums.garmin.com/developer/fit-sdk/f/discussion/301794/where-to-get-a-complete-list-of-fit-profile-and-message-definitions) but not easy ...

My first need was to update the name of all my POIs, exporting the fit file to csv (then reverse convert to fit and file replace in the watch) ; but it seems easier to do it directly on the watch ..!

Thanks again muktihari

The FIT SDK is generic one, manufacturer need to create their own custom SDK.

FWIW, some information may be intentionally be hidden and only available for internal use. Or, you need to join certain developer program from the manufacturer that generate the FIT file to be able to access that information. Ref: forums.garmin.com/.../1201166

The only thing I can do to help you is this:

I make a program to convert FIT to CSV and then convert it back CSV to FIT without losing any unknown data, by make a custom values: message number and field number in the unknown name and print type as units:

Data,4,unknown(140),unknown(253),"1095032787",uint32,unknown(2),"5380209",sint32,

This way we can reconstruct the unknown data, however you still need to identify which field that you want to edit yourself. Please see https://github.com/muktihari/fit/tree/master/cmd/fitconv

Usage:

1. Convert FIT to CSV

fitconv --verbose file.fit

file.csv will be created

You need to use --verbose when converting FIT to CSV to allow converting unknown data back to FIT.

2. Convert CSV to FIT

fitconv file.csv

file.fit will be created (replace existing), make sure to backup the original file somewhere before proceed.

Hopefully it helps and If this useful, consider leaving a GitHub star to the project. Thanks :)

Hi, have you ever been able to recreate original byte-to-byte identical FIT file from CSV?

Some time ago I tried to modify official Java SDK to achieve that, but failed. Please, see this discussion: FitCSVTool can't recreate original FIT file, is that really true?

As a creator of your own SDK implementation, you may know things better, than I do. Do you have any insights how to edit FIT files without data loss?

If you refer to byte-by-byte identical FIT file, we can't, even if we just modify a single character in a string without changing the field definition, then reconstruct the mesg as the original arrival order a long with its identical mesg definition (without redefining the local mesg num), and write the original FileHeader, the resulting CRC checksum for the data will be different, since the data change, so the file will be different.

But if you refer to modify without data loss, we absolutely can as long as we have these information about unknown data: Mesg number, Field number, the value and its data type. That's why I create similar tool to slightly modify the CSV.

Data,4,unknown(140),unknown(253),"1095032787",uint32,unknown(2),"5380209",sint32,

We know this is a Message with Mesg Num 140, having Field[0] {num: 253, type: uint32, value: 1095032787}, and Field[1] {num: 2, type: sint32, value 5380209}, we can encode it back to FIT file.

For array value, there is a pattern in the CSV that every element of the array are divided by | symbol, we can reconstruct them as well.

Data,9,unknown(65281),unknown(2),"15|6|0|0|0|0|0",uint32,

In my opinion, FileHeader, local message types, byte-order / endianess, and even CRC checksum, all of them does not matter, they can change (at certain point they MUST change e.g. CRC), what's matter is the data and its integrity.

By asking about byte-by-byte identical conversion I meant, whether it is possible convert FIT to CSV, and then this CSV straight back to FIT, having differences only in header in PROFILE_VERSION (aka "SDK version, that is used for encoding") and checksum.

Consider the following scenario: record some FIT file using Garmin device (that is aware of some internal FIT messages, that are unavailable in official SDK), convert it to CSV (either using FitToCsv or your fitconv), convert this CSV back to FIT, convert this new FIT again to CSV. If I do this, I'll get slightly different FIT and CSV files at each iteration, where some values outside the header and checksum will differ - that is what I call "data loss", as there is no way (that I'm aware of) to convert CSV to FIT producing the same sequences of bytes for every value in every field of every message. As far as I understand, decoding is deterministic, so I'd expect CSVs to be the same at every iteration, unless there's a data loss in between conversions.

muktihari said:

In my opinion, FileHeader, local message types, byte-order / endianess, and even CRC checksum, all of them does not matter

I fell uncomfortable when I see that after FIT->CSV->FIT conversion new FIT file size is few tens of kilobytes less than the original one.

I hoped, that you know, how to get the same FIT and CSV files after each iteration of conversions, and that it is possible to edit one single value in file (either through intermediate conversion to CSV or by directly changing values in debugger of your favorite IDE in case of having reliable FIT read-write mechanism without any other bytes modifications, except for header and checksum).

It looks like your SDK can't do this too. Thanks for your time.

9811827 said:

whether it is possible convert FIT to CSV, and then this CSV straight back to FIT, having differences only in header in PROFILE_VERSION (aka "SDK version, that is used for encoding") and checksum.

And PROTOCOL_VERSION and DATA_SIZE might be different, as encoding can be nondeterministic, the algorithm for message definition interleave might be different from the original file, this affect DATA_SIZE a lot, that's why we have normal header with 0-15 interleave and compressed header with 0-3 interleave, this affect how we redefine message definitions, resulting different file size. But if the FIT file is created by the same program, let's say fitconv, the encoding become more deterministic, you will get the same result every iteration. The creation of the original data is possible and it's the reason why fitconv exist since FitToCsv could not do it.

9811827 said:

I fell uncomfortable when I see that after FIT->CSV->FIT conversion new FIT file size is few tens of kilobytes less than the original one.

Unless you points out what is missing, I can't assist you, the few kilobytes might be the message definition interleave that's have been redefined, but this does not affect the data.

9811827 said:

It looks like your SDK can't do this too. Thanks for your time.

I'll appreciate it if you try the program before make a conclusion, but it's up to you to decide. Hopefully it help.

muktihari said:

And PROTOCOL_VERSION and DATA_SIZE might be different, as encoding can be nondeterministic, the algorithm for message definition interleave might be different from the original file, this affect DATA_SIZE a lot, that's why we have normal header with 0-15 interleave and compressed header with 0-3 interleave, this affect how we redefine message definitions, resulting different file size.

OK, thanks for the info. Not that I fully understand FIT protocol internals, I thought (and still do, but just haven't found the right tool yet and failed to write such tool myself) that it is possible just to do read-write without changing anything: if input file has FIT V2 protocol, then just write file using it, if PROFILE_VERSION has some specific value, then keep it, if it uses ARCH_ENDIAN_BIG, then use it too, if messages are in particular order - keep them in that order, the same is for fields and values, preserving exactly the same bytes, etc. In other words - I just want to have a reliable FIT file format mapper to object instances in program's memory, where they can easily be manipulated at high level without dealing with individual bytes and written back to file in the same order with the same data.

muktihari said:

Unless you points out what is missing, I can't assist you, the few kilobytes might be the message definition interleave that's have been redefined, but this does not affect the data.

Fair. I can easily replicate the described behavior for my own FIT files, but let's test fitconv on DC Rainmaker's files from Fenix 7X review, that are available here: https://analyze.dcrainmaker.com/#/public/4dbb2f17-63df-4175-544f-98f311af66f6. Scroll down and press "Download Set Files" to get "Fenix7X-Optical-RightWrist-8127818303.fit" file. The difference in size is not so dramatical as it is for my files, but it is not zero. I did several iterations, and both FIT and CSV files are different each time, and I see differences in "Data" lines of CSV after, and none of FIT files (the original one and all reencoded) are the same. I used "fitconv -verbose Fenix7X-Optical-RightWrist-8127818303.fit" and "fitconv Fenix7X-Optical-RightWrist-8127818303.csv" commands for conversions.

muktihari said:

I'll appreciate it if you try the program before make a conclusion, but it's up to you to decide.

That is the first thing I did before bothering you. Thank you.

EDIT: Here is the example of publicly available file, that loses 14kb after FIT-CSV-FIT conversion: https://github.com/Schrottie/fitfile2db/raw/main/testdata/cycling_with_radar/10901565277_ACTIVITY.fit.

I'm sorry I was going to retract my previous comment and replace it with the new one since I found another bug and I want going to resolve it. Wait a moment, will update.