FitCSVTool can't recreate original FIT file, is that really true?

Sorry if this is an obvious question, but I was exploring FitCSVTool to convert between FIT and CSV formats and I naively assumed that converting from FIT to CSV and then back again would result in a file identical to the original FIT file.

However, the FIT file is missing all the "unknown" fields and messages.

It would seem to be fairly trivial for FitCSVTool to write a hex value for "unknown" so that the conversion would work. Is there some reason why this isn't done? Is there an alternative to FitCSVTool that works as I would have expected?

0 9811827 12 months ago in reply to Ben FIT

OK.

Am I right, that is my application and not the SDK itself should implement some methods that "convey if a value is valid or not"? If yes, which interfaces should I use?

With decoder.showInvalidValues() results are much-much closer to what I want, but there are still some field and field definition differences.

"20" is missing:

// original
Definition,1,file_creator,unknown,20,,software_version,1,,hardware_version,1,,unknown,1,,unknown,1,,,,,,,,

// reencoded
Definition,1,file_creator,software_version,1,,hardware_version,1,,unknown,1,,unknown,1,,,,,,,,,,,

"32" is missing:

// original
Definition,6,device_info,timestamp,1,,serial_number,1,,cum_operating_time,1,,unknown,1,,unknown,1,,unknown,1,,unknown,32,,unknown,1,,unknown,1,,manufacturer,1,,product,1,,software_version,1,,battery_voltage,1,,unknown,1,,ant_device_number,1,,device_index,1,,device_type,1,,hardware_version,1,,unknown,1,,battery_status,1,,sensor_position,1,,ant_transmission_type,1,,ant_network,1,,unknown,1,,source_type,1,,unknown,6,,unknown,1,,battery_level,1,,

// reencoded
Definition,6,device_info,timestamp,1,,serial_number,1,,cum_operating_time,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,manufacturer,1,,product,1,,software_version,1,,battery_voltage,1,,unknown,1,,ant_device_number,1,,device_index,1,,device_type,1,,hardware_version,1,,unknown,1,,battery_status,1,,sensor_position,1,,ant_transmission_type,1,,ant_network,1,,unknown,1,,source_type,1,,unknown,6,,unknown,1,,battery_level,1,,

"64" is missing:

// original
Definition,10,device_settings,utc_offset,1,,time_offset,1,,auto_activity_detect,1,,unknown,40,,unknown,64,,unknown,1,,unknown,1,,autosync_min_steps,1,,autosync_min_time,1,,active_time_zone,1,,unknown,1,,time_mode,1,,time_zone_offset,1,,unknown,1,,unknown,1,,unknown,1,,backlight_mode,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,activity_tracker_enabled,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,move_alert_enabled,1,,date_mode,1,,unknown,1,,unknown,1,,unknown,1,,display_orientation,1,,mounting_side,1,,unknown,13,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,lactate_threshold_autodetect_enabled,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,number_of_screens,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,15,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,

// reencoded
Definition,10,device_settings,utc_offset,1,,time_offset,1,,auto_activity_detect,1,,unknown,40,,unknown,1,,unknown,1,,autosync_min_steps,1,,autosync_min_time,1,,active_time_zone,1,,unknown,1,,time_mode,1,,time_zone_offset,1,,unknown,1,,unknown,1,,unknown,1,,backlight_mode,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,activity_tracker_enabled,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,move_alert_enabled,1,,date_mode,1,,unknown,1,,unknown,1,,unknown,1,,display_orientation,1,,mounting_side,1,,unknown,13,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,lactate_threshold_autodetect_enabled,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,number_of_screens,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,15,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,

"32" is replaced with "23":

// original
Definition,13,sport,name,32,,unknown,4,,unknown,1,,sport,1,,sub_sport,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,3,,unknown,1,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

// reencoded
Definition,13,sport,name,23,,unknown,4,,unknown,1,,sport,1,,sub_sport,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,1,,unknown,3,,unknown,1,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

A few field values differences:

// original
enhanced_avg_speed,"my-value-1|65.535",m/s,enhanced_max_speed,"my-value-2|65.535",m/s

// reencoded
enhanced_avg_speed,"my-value-1|65.535|65.535",m/s,enhanced_max_speed,"my-value-2|65.535|65.535",m/s

Are these differences OK too?

0 Ben FIT 12 months ago in reply to 9811827

This line is a message definition, which is describing what fields to expect in a file_creator message that uses this definition.


Definition,1,file_creator,unknown,20,,software_version,1,,hardware_version,1,,unknown,1,,unknown,1,,,,,,,,

The 1s and 20 are part of the definition not the data. The unknown,20 is probably a string that is 20 bytes long. In the message itself those 20 bytes were probably all invalid (0xFF) or all a null terminator \0. Which means that the Java SDK would treat that field as invalid and not encode it. There is no data loss, just a slimmed down file.

The showInvalidValues() method sets a flag to preserve invalid values so that they can be displayed in the CSV file. It is not intended for other uses.

0 9811827 12 months ago in reply to Ben FIT

So, it seems, that after all there is no good way to implement something similar to what hex editors allow to do - load file, change one value in one field in one message, and produce the file, that differs the original file by these specific bytes and checksum.

Thank you very much for the replies!