• State Not Answered
  • Replies 7 replies
  • Subscribers 5 subscribers
  • Views 260 views
  • Users 0 members are here
Options
Related

Different content from Java SDK for the same FIT files

JOINcycling
1 month ago

Hi,

We're using the Java version of the FIT SDK for parsing and creating FIT files in our server application.

We have a rather large test suite and have started noticing that some of our tests started failing with the same errors time when we run the complete test suite. When we run the tests individually, they do succeed.

We've added extra logging and found the following:

In the circumstances that the tests go wrong, we RecordMesg objects in our RecordMesgListener that contain heart rate values of 255. For the same file, and the same RecordMesg, the value is null in the correct runs (null is the correct value here).

We see the same thing with values for the power, cadence, enhanced_speed and enhanced_altitude fields, though the values are different there because these values have different types.

Of course, we first suspected that our inputs were off, i.e. that we were sending corrupt data. So we checked the contents of the fit files using an MD5 hash digest. Contents turned out not to be different between runs, so that was not the issue.

Following this, we looked at the SDK's code and found that it is obviously stateful, because the class com.garmin.fit.Decode has a static boolean property invalidDataSize. Because of this we put an excluse lock around the FIT decoding code. This should prevent a problem with invalidDataSize being in a faulty state, but it does not fix the problem above.

Our conclusion is that there is some residual state in the SDK after having parsed a FIT file. We do create new Decode and MesgBroadcaster objects for each FIT file that we parse, so that is not the cause of the issue.

Is this expected behavior, and if so, how should we work around it in a long-running server application?

Thank you,

Ilja Booij

JOIN Sports BV

  • 0 1 month ago

    It is known behavior that an instance of a decoder should not be reused. There is other state besides invalidDataSize that the decoder classes maintain. There is almost no overhead to create a new decoder, so creating a new one per file should not negatively impact a long-running server application.

  • 0 1 month ago in reply to Ben FIT

    Hi Ben,

    I understand that we should not be reusing the decoder. We're actually creating a new com.garmin.fit.Decode and com.garmin.fit.MesgBroadcaster object for every FIT file that we parse.

    Should we be doing more than that?

    Note that com.garmin.fit.Decode has a static field invalidDataSize, which is defined as static, so it always be reused between Decode objects anyway.

    Thank you,

    Ilja Booij

    JOIN Sports BV

  • 0 1 month ago in reply to JOINcycling

    I am not sure what the history of invalidDataSize being a static is. It seems odd but it has been that way since 2009 and I assume there was a reason for it.

    invalidDataSize is set to true when the data size in the file header, bytes 4-7, is zero. That could mean that the file was not closed properly and the decoder should read until the end of the file, ignore the CRC since that could be suspect too if the file was not properly closed, and do not throw an error if the end of the file was reached part way through decoding a message. So when a file was not closed properly read as much as possible from a file and don't throw an error at the EOF. 

    I am trying to think of a scenerio when invalidDataSize==true would cause the data read from a file to change. Is the file where you are seeing HR values of 255 a chained FIT file? That may happen for swim activities where the HR values are stored in the HRM and then sent to the device at the end of the activity and appended to the Activity file created by the device. If another file caused invalidDataSize to be set to true, could cause the decoder to not properly process the chained file with the HR data? That could be possible. This assumes that you are using the HrToRecordMesgBroadcastPlugin to merge the HR values from the chained file into the Record mesgs. The HR values in the Record messages are 255 (aka invalid) and the plugin is merging the HR values from the second file. If the chained file is not properly read, then there is not HR data to merge into the Record mesgs.

  • 0 1 month ago in reply to Ben FIT

    Hi Ben,

    We're already guarding against the invalidDataSize possibly being an issue by having an exclusive lock around the parsing code, so there will be at most 1 FIT parsing action at a time for a single instance of our application.

    The files we have are mostly cycling and running activities.

    What we've done currently is to add code to replace the invalid values with null values when parsing. That solves the issues we're seeing completely, but it feels like the SDK should take care of this.

    I have not gone through all the code in the FIT SDK, but is there more stateful code like invalidDataSize?

    Thanks!

    Ilja

    JOIN Sports BV

  • 0 1 month ago in reply to JOINcycling

    The Decode class has been used in production since 2009, and has not changed much in that time. It could be that there is an issue with invalidDataSize being static, but I think that is a red herring based on what you are seeing. The invalidDataSize property does not have any impact on the decoding of the file until the EOF is reached. So it should not affect how messages in the middle of the file are decoded. There may be something else at play. But there is another option.

    We have a new Decoder class that we can add to the SDK in the next release, which should be in December. We have used it for a few internal projects, so it has some runtime on it. 

  • 0 21 days ago in reply to Ben FIT

    Hi Ben,

    Yes, the invalidDataSize thing is probably a red herring, but I'm still confused by the fact that results on a particular file are different when run in isolation (just parse a single file) vs when parsing multiple files in succession.

    Since having added the code that replace invalid values with nulls, everything is working ok, so we don't have this issue anymore.

    Once the new Decoder class is available, we'll test that to see if that gives different results.

    Thanks for the info!

    Ilja Booij

    JOIN Sports BV