When running tests, the simulator runs each test as a separate instance, with the (really nice) result that even if a test crashes (eg symbol lookup fails), the remaining tests run, and the crashing test is reported as ERROR rather than FAIL.
As of SDK 4.2.0 Beta 1 this still works, but (on MacOS at any rate) if any of the tests crash, the simulator crashes on exit. Here are some repro steps using my github monkeyc-optimizer project.
% git clone https://github.com/markw65/monkeyc-optimizer Cloning into 'monkeyc-optimizer'... remote: Enumerating objects: 1818, done. remote: Counting objects: 100% (314/314), done. remote: Compressing objects: 100% (145/145), done. remote: Total 1818 (delta 208), reused 258 (delta 167), pack-reused 1504 Receiving objects: 100% (1818/1818), 598.83 KiB | 3.09 MiB/s, done. Resolving deltas: 100% (1307/1307), done. % cd monkeyc-optimizer/test/OptimizerTests % ~/Library/Application\ Support/Garmin/ConnectIQ/Sdks/connectiq-sdk-mac-4.2.0.beta1-2022-10-11-bff57b73c/bin/monkeyc -o bin/test_approachs60_OptimizerTests.prg -f monkey.jungle -y ~/connect-iq/developer_key -d approachs60 -w -l 3 --unit-test WARNING: approachs60: The launcher icon (30x30) isn't compatible with the specified launcher icon size of the device 'approachs60' (40x33). Image will be scaled to the target size. WARNING: approachs60: /Users/mwilliams/www/git/gtest/monkeyc-optimizer/test/OptimizerTests/source/OptimizerTestsLookup.mc:243: Class 'Child' does not initialize its super class 'Base'. % ~/Library/Application\ Support/Garmin/ConnectIQ/Sdks/connectiq-sdk-mac-4.2.0.beta1-2022-10-11-bff57b73c/bin/monkeydo bin/test_approachs60_OptimizerTests.prg approachs60 -t nesting.crashNestedLookupNESTINGA1 % ~/Library/Application\ Support/Garmin/ConnectIQ/Sdks/connectiq-sdk-mac-4.2.0.beta1-2022-10-11-bff57b73c/bin/connectiq % ~/Library/Application\ Support/Garmin/ConnectIQ/Sdks/connectiq-sdk-mac-4.2.0.beta1-2022-10-11-bff57b73c/bin/monkeydo bin/test_approachs60_OptimizerTests.prg approachs60 -t nesting.crashNestedLookupNESTINGA1 ------------------------------------------------------------------------------ Executing test nesting.crashNestedLookupNESTINGA1... Error: Symbol Not Found Error Details: Could not find symbol 'NESTINGA1' Stack: - getNESTINGA1() at /Users/mwilliams/www/git/gtest/monkeyc-optimizer/test/OptimizerTests/source/OptimizerTestsGarmin.mc:81 0x10003a84 - crashNestedLookupNESTINGA1() at /Users/mwilliams/www/git/gtest/monkeyc-optimizer/test/OptimizerTests/source/OptimizerTestsGarmin.mc:196 0x10003484 - evaluate_test_entries_0_to_30() at /Users/mwilliams/www/git/gtest/monkeyc-optimizer/test/OptimizerTests/bin/gen/006-B2656-00/source/UnitTests.mcgen:97 0x10001504 - runTest() at /Users/mwilliams/www/git/gtest/monkeyc-optimizer/test/OptimizerTests/bin/gen/006-B2656-00/source/UnitTests.mcgen:179 0x10001bc9 ERROR ============================================================================== RESULTS Test: Status: nesting.crashNestedLookupNESTINGA1 ERROR Ran 1 test FAILED (passed=0, failed=0, errors=1) <Window pops up announcing that the simulator crashed>
Note that the simulator finishes, and reports the results, and *then* crashes. Also, if you replace "-t nesting.crashNestedLookupNESTINGA1" with just "-t" (to run all the tests), all the tests run, including several that crash, and *then* the simulator crashes on exit.
Also note that in this project, the crashing tests are *supposed* to crash; I'm testing the lookup behavior before and after my optimizer runs, and these are cases where the lookup is supposed to fail.
