Watchdog Tripped Error - Code Executed Too Long -> Workaround for Tests?

Hi all,

in my tests, I sometimes face the error that the tests run too long: "Watchdog Tripped Error - Code Executed Too Long".

Actually, I believe that the production code I am testing is "more or less and in most cases" :p fine from a performance point of view. The failure mainly happens because I kind of parameterize my tests, i. e. within a test method, I define a large array or dictionary with test data and I call my method under test in a loop.

Example:

(:test)
static function testTrim(logger)
{
var testDataArray = [
{ :input => "bla", :expected => "bla"},
{ :input => "bl a", :expected => "bl a"},
{ :input => " bla", :expected => "bla"}
];

for (var i=0; i<testDataArray.size(); i++)
{
var testData = testDataArray;

var trimmedString = FhemStringUtils.trim(testData[:input]);

Test.assertEqual(trimmedString, testData[:expected]);
}

return true;
}
[/CODE]
(btw: This code does not cause the error, it is just an example. It usually happens in more complex scenarios)

Does anyone here have a solution? Maybe another pattern how I can write such tests? Or some way to disable/increase the threshold for tests?

Thanks & best regards,
Florian
  • The value used by the watchdog is based on the device and can be seen in \bin\devices.xml in the SDK directory. For example,
    <watchdog_count>80000</watchdog_count>


    You do not want to change this, but use it for reference. The Fenix Chronos for example has 120000 (it's a 2.x.x device), so it won't trip as soon as others.

    This value defines how much you can do (I believe it's the number of bytecodes executed) before returning to the VM before the watchdog is tripped.

    basically, if you hit it, you have to re-work your code so you don't hit it, as there's no good way to just extend it.

    If you post the code that actually triggers the watchdog,it would be easier to help to avoid it
  • Thanks for the quick reply.

    I did not know about the <watchdog_count>80000</watchdog_count> setting in the devices.xml. I use the 735xt for the execution of my tests and that already uses the "highest" watchdog count for a watch if I checked correctly. But it is still good to know.

    Anyway, in the time between my original post and this reply, I tuned my production code a bit and now I face no more watchdog errors :D. I often do this when I face watchdog errors but since I faced some cases where I had just too many test variants, I thought of asking this in the forum. I had some cases in the past where I could only fix it by consolidating my test variants to avoid the watchdog error though I dont like that too much as I might consolidate some needed test variant out of my tests ;).

    Best regards & thank you!
    Florian

    PS: The tuning here was in my custom string-trim method. Originally, I converted the string to a char array and then looped at all chars. If there was a space/linefeed/etc. before the first "real" char, I just skipped that, otherwise, I added the current char to the output string. I know it is a bad implementation but it was the first idea I had when I wrote it and it worked. What I do now is using a while loop in which I take a substring of the first, second, ... character. As soon I hid a non-space/linefeed/etc. I just return the substring from there to the end. Of course, much faster and also cleaner code. Thus, in the end, I am happy that the watchdog error forced me to improve my code :).
  • I structure my tests much like you do, and I have run into the same problem many times. Having the watchdog enabled when running unit tests doesn't seem to provide benefit. Perhaps it is a worthwhile request to ask for it to be disabled when running unit tests...

    Travis
  • Maybe being able to disable it during specific tests? If the bit you're testing gets into an infinite loop, you'd probably want it then, or to see if the bit your testing would run into it in a full app.
  • There isn't really any reason we need the watchdog on for unit tests, so I'll enter a ticket to look into making a change.

    We can probably just increase it significantly during tests so we can still save your from an infinite loop.
  • Brian - in the case with the OP, wasn't it useful that it tripped during his test, and he rethought his code so it wouldn't trip? If it hadn't tripped in the test, it might not have been caught except outside of the test, which kind of says the unit test didn't test something good to know.

    Maybe just insure the watchdog is reset at the beginning of each test?
  • If the bit you're testing gets into an infinite loop

    If that happens, you can kill the simulator, right?

    wasn't it useful that it tripped during his test, and he rethought his code so it wouldn't trip?

    Is there is a correlation between unit tests that cause the watchdog to trip and inefficient code? AFAICT, you can have a comprehensive unit test for a very efficient function that will be killed by the watchdog just as easily as you can have a primitive test for an expensive function that would timeout.

    Maybe just insure the watchdog is reset at the beginning of each test?

    The watchdog is already reset at the start of execution for each unit test function.

    If a tester wants to be sure their code doesn't run long, they can always use Sys.getTimer() to see how long their code has run and bail out if necessary.

    Travis
  • Wouldn't you rather see a fail, and let other tests run, instead of watching in case you have to kill the sim? I think I would.....

    If I was running some complex tests, I may just fire them off, do something else, and then look at the results when I get back to it, for example.

    There could be cases where you want to disable the watchdog, but others, where you don't, IMHO...

    I'm not sure why you'd want to use Sys.getTimer() when the watchdog will deal with that already, and it would adjust the watchdog based on the device being used as a target.
  • Wouldn't you rather see a fail, and let other tests run, instead of watching in case you have to kill the sim?

    No, but that's me. If I have an individual test that takes a long time to run, I want that test to finish. If I'm not willing to wait for that one test to finish, then I probably don't care about the results of any other tests anyway.

    If I was running some complex tests, I may just fire them off, do something else, and then look at the results when I get back to it

    Sure. If you do this, what do you plan to do if all of the tests have finished but one was killed by watchdog? Do you increase the watchdog timeout so that the test will run to completion and build/run again? If so, what is the point of the watchdog?

    I'm not sure why you'd want to use Sys.getTimer() when the watchdog will deal with that already, and it would adjust the watchdog based on the device being used as a target.

    I'm willing to concede that there is probably some scenario that the user might want to have watchdog kill tests for them.

    Not that we really have any say in this, but it seems using an annotation for this would be perfect. Something like one this would be nice...

    (:test(10)) function test_for_only_ten_seconds(logger) {
    // similar to the minSdk("2.3.0") annotation
    }


    Travis
  • I found the watchdog in the unit tests useful in cases when I did something stupid (like forgot to increase the loopcounter in a while loop), so I would certainly retain some kind of watchdog-functionality in the unit-test part as well, but maybe set it to a longer time?