Test.assert, Test.assertEqual, Test.assertEqualMessage don't throw exception

Both in SDK 6.4.2 and 7.1.0 the following  tests should fail (throw an AssertException) but they PASS:

    (:test)
    function barAssertTest(logger as Logger) as Boolean {
        var expected = "FOO";
        var actual = "BAR";
        Test.assert(expected.equals(actual));
        return true;
    }

    (:test)
    function barEqualTest(logger as Logger) as Boolean {
        var expected = "FOO";
        var actual = "BAR";
        Test.assertEqual(expected, actual);
        return true;
    }

    (:test)
    function barEqualMessageTest(logger as Logger) as Boolean {
        var expected = "FOO";
        var actual = "BAR";
        Test.assertEqualMessage(expected, actual, "expected: " + expected + ", actual: " + actual + ".");
        return true;
    }

I run the tests from VSC on fr955 using the following configuration in launch.json:

        {
            "type": "monkeyc",
            "request": "launch",
            "name": "Run Tests",
            "runTests": true,
            "device": "${command:GetTargetDevice}"
        },

First I almost opened a bug report, but this is so huge, I'm sure the problem is in something/somehow I do. Or is it?

  • Looks like yet another CIQ 5.0.0 device regression. If you run the same tests with a non-CIQ 5.0.0 device such as descentmk2, all 3 return ERROR (as expected), with either SDK 6.4.2 and SDK 7.1.0

    (Of course this also works as expected with a really old SDK like 4.2.4. But ofc, CIQ 5 devices only work with SDK 7.1.0 now. Actually, fr955 works with SDKs >= 5.0.0, my bad)

  • Maybe part of what you wrote is right, but one part for sure isn't, because as I wrote I tried this with SDK 7.1.0 and it doesn't throw the exception.

  • Workaround:

    import Toybox.Lang;
    import Toybox.Test;
    
    (:test)
    module Flocsy {
        module Test {
            function assertEqualMessage(expected as Object?, actual as Object?, message as String) as Boolean {
                if (expected == null && actual != null || !expected.equals(actual)) {
                    throw new Test.AssertException(message);
                }
                return true;
            }
        }
    }
    

    BTW I also opened a formal bug-report here: forums.garmin.com/.../bug-test-assert-test-assertequal-test-assertequalmessage-don-t-throw-exception

  • Maybe part of what you wrote is right, but one part for sure isn't, because as I wrote I tried this with SDK 7.1.0 and it doesn't throw the exception.

    Once again I’ve completely failed at communication. At this point I should just delete my account tbh.

    TL;DR CIQ 5 devices bad (e.g. fr955), non-CIQ 5 devices good (e.g. descentmk2). It doesn't matter what SDK you use as long as the SDK supports the device in question.

    To be completely clear:

    For me, the bug only appears with CIQ 5 devices (e.g. fr955), which are only usable with SDK 7.1.0 (regardless of SDK, except fr955 requires SDK 5.0.0 or higher to build *). In this case, when running the tests you posted, all the results are PASS, contrary to expectations.

    (* EDIT but it seems that Garmin actually intended SDK 7.1.0 to be the minimum SDK for CIQ 5 devices. Either way, this problem happens with CIQ 5 devices regardless of SDK.)

    For me, the bug does not appear with non-CIQ 5 devices (e.g. descent2mk), regardless of SDK. In this case, all the test results are ERROR, which is expected.

    Therefore to me it appears to be just like the “Menu2 bug” you reported (which actually appears to be a bug with code auto-generated from layout resources) — it only affects CIQ 5 devices.

    Note that "006-B3258-00" is a part number for descentmk2.

    EDIT: actually, fr955 is usable with SDK 5.0.0 and higher, so I misspoke when I originally said that 7.1.0 is now required for fr955. (I had tried 4.2.4 in my earlier test, and that's where I got the error message about not being able to use that SDK with fr955, but I didn't read it closely enough.)

    Nonetheless, the results are the same for either 6.4.2 and 7.1.0: CIQ 5 devices don't work properly (tests don't return ERROR), non-CIQ 5 devices do work properly (tests return ERROR.)

  • On further reflection, I wonder if the "SDK 5" requirement for fr955 is yet another bug. There's literally no CIQ SDK 5.x, the SDK version went directly from 4.2.4 to 6.2.0. I wonder if the actual intent was to require SDK 7.1.0 for CIQ 5 devices.

    This is the error message you get when trying to build fr955 in SDK 4.2.4:

    ERROR: Device 'fr955' requires an SDK version of 5.0.0 or greater to be compiled. The current SDK version is 4.2.4. Update your SDK using the Connect IQ SDK Manager.

    I'm going to open a bug report for this.

    [https://forums.garmin.com/developer/connect-iq/i/bug-reports/building-ciq-5-devices-e-g-fr955-in-sdk-4-2-4-results-in-error-claiming-sdk-5-0-0-or-greater-is-required---shouldn-t-this-be-sdk-7-1-0]

    If I'm right about this, then Garmin literally does not understand their own versioning system, which is hilarious on many levels

  • Also, if I'm right that Garmin intended CIQ 5 devices to require a minimum of SDK 7.1.0, then it means the other bug you filed about not being able to use Menu2 in CIQ 5 devices with the older SDK isn't really a bug,.

  • I thought about this, that's why I took the time to test it on my real fr965 with CIQ 5.0.0, and it works there, so the Menu2 bug is definitely simulator only

  • I think this makes sense from Garmin's part. When the fr955 came out they added it in such a way that it needed the new SDK. I don't see any problem with it (I could see it possible to compile for fr955 with older SDKs if you don't use new functionality though... So if someone has an old project and everything is set up, then the only thing they would need is to download the device in SDK manager and to add them in the manifest. IMHO this would be reasonable, but I can also understand if it's not that developer-friendly)

  • I guess you’re missing the part where:

    - the error message specifically says “SDK version of 5.0.0 or greater is required” (for CIQ 5 devices)

    - there was no SDK version 5.x, it went directly from 4.2.4 to 6.2.0

    You think it’s just a coincidence the error message says SDK >= 5.0.0 for CIQ 5 devices?

    When the fr955 came out they added it in such a way that it needed the new SDK. I don't see any problem with it

    Right, my point is they obviously *tried* to do it but failed.

    Currently you can build fr955 with anything except SDK < 5.0.0 (such as 4.2.4). Again, there was no 5.0.0, SDKs went straight from 4.2.4 to 6.2.0. 6.2.0 was so long ago and unrelated to CIQ 5 (obviously) that it makes no sense for the cut off to be there.

    I thought about this, that's why I took the time to test it on my real fr965 with CIQ 5.0.0, and it works there, so the Menu2 bug is definitely simulator only

    I never said it wasn’t simulator only. I only said that if the real restriction was “SDK >= 7.1.0 is required”, then the “Menu2 bug” (as you call it) wouldn’t exist. And for the millionth time, it’s not just Menu2. When it crashes on the Menu2Sample, it’s on a line from Rez.mcgen that calls setText() on a layout element with the app name (from string resources).

    I’m sure it could be recreated with apps that use a layout but don’t have menu2.