Watchdog Tripped Error - Code Executed Too Long -> Workaround for Tests?

Hi all,

in my tests, I sometimes face the error that the tests run too long: "Watchdog Tripped Error - Code Executed Too Long".

Actually, I believe that the production code I am testing is "more or less and in most cases" :p fine from a performance point of view. The failure mainly happens because I kind of parameterize my tests, i. e. within a test method, I define a large array or dictionary with test data and I call my method under test in a loop.

Example:

(:test)

	static function testTrim(logger)

	{

		var testDataArray = [

						{ :input => "bla", 		:expected => "bla"},

						{ :input => "bl a", 		:expected => "bl a"},

						{ :input => " bla", 		:expected => "bla"}

					];



		for (var i=0; i<testDataArray.size(); i++)

		{

			var testData = testDataArray;



			var trimmedString = FhemStringUtils.trim(testData[:input]);



			Test.assertEqual(trimmedString, testData[:expected]);

		}



		return true;

	}

[/CODE] 

(btw: This code does not cause the error, it is just an example. It usually happens in more complex scenarios)



Does anyone here have a solution? Maybe another pattern how I can write such tests? Or some way to disable/increase the threshold for tests?



Thanks & best regards,

Florian

nairolf123 over 7 years ago

Not that we really have any say in this, but it seems using an annotation for this would be perfect. Something like one this would be nice...

(:test(10)) function test_for_only_ten_seconds(logger) { // similar to the minSdk("2.3.0") annotation }

I strongly agree here. Then the developer can decide. I would definitely not set a (high) threshold for each test by default, but for some I might do it. I have some tests that test a more complex function than my trim example in this post for like 30+ lines of test data. For those, a custom watchdog timeout would be perfect.

In addition, I would also be happy if the unit test framework had something like the parameterized tests in JUnit. They allow to maintain a set of test data in your test class as code. The JUnit framework then calls each test of the entire class for each line in the test data set. Since each single test execution for each single line of the data set is treated like a single test method execution, the watchdog issue described here would arise more rarely (for me :-)).

Maybe the easiest way for Garmin here would be to somehow open up the Run No Evil test framework by providing a test runner class to manually trigger the execution of particular tests/test classes. If that exists, I might be able to implement something like that on my own...

Thanks & regards,
Florian

jim_m_58 over 7 years ago

Not that we really have any say in this, but it seems using an annotation for this would be perfect. Something like one this would be nice...

(:test(10)) function test_for_only_ten_seconds(logger) { // similar to the minSdk("2.3.0") annotation }

Travis

I like this approach! Simple, and handles the different things mentioned here!

Brandon.ConnectIQ over 7 years ago

For now, we've upped the watchdog timer to allow for more complex tests. This change will be available in the 2.3.1 release. Perhaps we can take a look at the annotation approach in the future, because I think it's an interesting idea.