Unable to convert csv to fit file

Would you like to try again in v0.23.4? Here is the result of my experiments:

File renamed for simplicity

Fenix8-PostHD-Optical-2024-08-15-18-23-35.fit: Fenix.fit is the original file, the rest is the iterations:

-rw-r--r-- 1 mukti mukti 129895 Sep 20 00:34 Fenix-2.fit
-rw-r--r-- 1 mukti mukti 129895 Sep 20 00:34 Fenix-3.fit
-rw-rw-r-- 1 mukti mukti 129940 Nov 30 2107 Fenix.fit

10901565277_ACTIVITY.fit: ACTIVITY.fit is the original file, the rest is the iterations:

-rw-r--r-- 1 mukti mukti 301079 Sep 20 01:11 ACTIVITY-2.fit
-rw-r--r-- 1 mukti mukti 301079 Sep 20 01:11 ACTIVITY-3.fit
-rw-rw-r-- 1 mukti mukti 299907 Sep 20 00:55 ACTIVITY.fit

This is the best I can do both in knowledge and my available time right now (I will revisit the code again later once I got more time), thank you for trying the program and giving me feedbacks, it helps me improve my program. I hope the latest version give better result.

In 0.23.4 my reencoded file is only 1275 bytes smaller, than the original. I guess, this can be considered as a success result for fitconv.

My implementation based on patched Java SDK give similar difference in size, that is sadly not zero.

As a result of these tests and the info you provided I keep my previous conclusion, that it is not a trivial task to find/implement a converter/editor that will produce exactly the same file as an output (both by size and content), even if no messages and field values modifications are made.

Thanks for the effort to help.

Did you try to count how many message definitions interleave that you made and then calculate the difference? The size difference might be from this. There is also a technique to padding the byte to reduce interleave needed, and my SDK doesn't do that, I use different interleave algorithm that I think is (arguably) better to compress the file. 

By default, it will remove padding byte, so the different is expected if this is the case. I don't intend to produce zero differences as I don't know what to gain from this, I'll leave the task for others.

Anyway, thanks for sharing your experiment as well. Appreciate it. 

muktihari said:

Did you try to count how many message definitions interleave that you made and then calculate the difference?

I'm afraid I don't understand, what's your idea about. My FIT SDK and FIT format knowledge in general is very superficial. I wanted to create my own tool for basic file editing and fixing like removing erroneous lap, etc. I started to use official SDK for that, but immediately stopped any development and research, as I haven't found a way to read-write files without modifications, just like other libraries for JSON, XML, CSV and other formats allow you to do. Without that I don't see a point to implement anything, as it is crucial for me to keep everything, that I don't modify explicitly, intact.

muktihari said:

we have normal header with 0-15 interleave and compressed header with 0-3 interleave

I see, that SDK defines 16-length array for message definitions. One of the ideas, that I came up to, is that SDK's encoder with my modifications fails to write more than 5-6 messages byte-to-byte identically because this array is too small to contain all message definitions from original file, and I tried to save all definitions to some external map and use it for encoding directly, instead of recreating them from scratch, based on cut down SDK message definitions without some fields that are in original file for this message type, but stopped, as I found out, that there are different message definitions with the same "num" and/or "localNum". This looked like a dead end to me.

muktihari said:

By default, it will remove padding byte

I think, I played with this too, as I remember altering values encoding method to don't do any trimming.

muktihari said:

I don't intend to produce zero differences as I don't know what to gain from this

My idea is simple: if I want to change activity type name, I want to change exactly these bytes (and checksum, ofc) to make file indistinguishable, as if this name was ininitally written to the file by device. Right now all tools I tried make reencoded FIT files look like a fruit after being in a blender.

9811827 said:

I'm afraid I don't understand, what's your idea about.

Let's take a look at Fenix8-PostHD-Optical-2024-08-15-18-23-35.fit, there are 45 message definitions written in the original file, my fitconv writes only 44 message definitions. The original write definition for time_in_zone messages twice, while it actually can be write once, there is no need to interleave again. How does this make a different?

A message definition layout (without developer data) consist of:

header + reserved + architecture + mesg num (2 bytes) + n fields + fields (3 byte per field)

time_in_zone has 13 fields, so the calculation will be:

1 + 1 + 1 + 2 + 1 + (13 * 3) = 45 bytes

Let's take a look at the size of the files:

-rw-r--r-- 1 mukti mukti 129895 Sep 20 00:34 Fenix-2.fit
-rw-r--r-- 1 mukti mukti 129895 Sep 20 00:34 Fenix-3.fit
-rw-rw-r-- 1 mukti mukti 129940 Nov 30 2107 Fenix.fit

The different in size is 129940 - 129895 = 45 bytes

So we can conclude that the Fenix file data is identical, the only different is the message definition, as my fitconv redefined them to be more compact.

9811827 said:

I think, I played with this too, as I remember altering values encoding method to don't do any trimming.

I see, I found some FIT file contains string that is not a valid UTF-8 string, I don't know how is that possible, maybe I lack of knowledge about string encoding but FIT specification tell us to use UTF-8 string encoding.

And I don't know how Java SDK (or other SDKs) handles invalid UTF-8 string, but my Go SDK will trim it as it does not comply with FIT specification.

"Open Water\xe8\xe8\xe8\xe8\xe8\xe8\xe8\xe8" => "Open Water"

I think, we need to take this into consideration as well.

With previous comment and this, my point is, we need to breakdown what's considered identical in this context, because people might have different opinion about it, and that's okay as well. I’ll leave it at that for the moment.