Sometimes testing is checking that some input values matches expected output values, sometimes testing is a bit more than that.

Test timeout

Why care about test timeouts?

Fast test feedback

I like early testing. I often start development with some test. This is good. But it means also that there will be a lot of failing test runs.

In general not a problem, run a test, get feedback fail/OK, continue with the implementation and repeat.

But some scenarios are not able to provide that fast feedback.

Sometimes feedback might be late

For example a message base systems with some actors where each actor is running in a own thread. Or calls to std::async or other std::future returning function.
Some of them under development and this might mean that tests might not terminate.

It is possible that a test is waiting for something that will not happen.

May it be because of a bug, something is not yet implemented, a wrong test setup or any other of countless reasons.

Than the development scenario becomes run a test, nothing happens, look at the monitor, one, two , three seconds, maybe more, still nothing happens.
Finally a Ctr+C key combo to terminate.

And this is not fast.

A plan B

An other problematic scenario is running such a test on a build server. Of course it is often possible to set a test timeout via a test execution framework, like CTest.

But due to reasons I prefer to not be depended only on third party tools which may - or not - be available and used correct. It’s good to have a plan B.

Create a build in timeout insurance

When I create the test for a scenario I usually have a pretty good idea in which time the test should provide success/failure feedback, 100 milliseconds, half a second, whatever …​

Since C++11 there is everything in the language required to create such a timeout insurance.

The concept

One way to implement it is as followed. A test, or code in general, has a scope.

  • Create a object when entering the scope that needs the timeout insurance.

  • If the scope is left, the destructor of the insurance object is called.

  • If the destructor was called, than everything is fine.

  • If the destructor was not called, there is a problem.

  • A background worker in the object checks periodically if the destructor has been called.

  • If the destructor was not called before some timeout value, terminate the execution.

The usage of such a helper will be reduce to a one liner,
TimeoutInsurance timeout{std::chrono::milliseconds{250}};,
for a test scope that should not last longer than 250 milliseconds.

This is simple and short, and having such a one liner of the begin of a test scope is also a nice documentation.

The implementation

  • A class with a done flag, set to false per default.

  • A thread that periodically checks if the done flag was set to true.

    • If the done flag was set, return.

    • If the done flag was not set and the timeout time was reached, terminate.

  • The done flag will be set to true in the destructor.

  • After setting the done flag the destructor joins the check thread.

A simple sample implementation

I am usually fine with milliseconds resolution and a check every 10 milliseconds to see if the timeout was reached or the test already was successful.

So a simple implementation can look like this.

#include <chrono>
#include <thread>
#include <atomic>
#include <iostream>

class TimeoutInsurance
{
public:

   using milliseconds = std::chrono::milliseconds;

   TimeoutInsurance(milliseconds timeout,
                    milliseconds checkinterval = milliseconds(10)) :
      insurance([this, timeout, checkinterval]() mutable
      {
         using namespace std;
         do
         {
            if (timeout < checkinterval)
            {
               this_thread::sleep_for (milliseconds(timeout));
            }
            else
            {
               this_thread::sleep_for (milliseconds(checkinterval));
            }

            timeout -= checkinterval;

            if (this->done)
               return;

         }while(timeout.count() > 0);

         std::cerr << "timeout\n";
         std::abort();
      })
   {
   }

   ~TimeoutInsurance()
   {
      done = true;
      if (insurance.joinable())
         insurance.join();
   }

   TimeoutInsurance(const TimeoutInsurance&) = delete;
   TimeoutInsurance& operator=(const TimeoutInsurance&) = delete;
   TimeoutInsurance& operator=(TimeoutInsurance&&) = delete;

private:
   std::atomic_bool done{false};
   std::thread insurance;

};

Usage of the TimeoutInsurance class is simple as that:

int main()
{

   TimeoutInsurance timeout(std::chrono::milliseconds{1000});
   char c;
   std::cout << "Enter something fast enough: ";
   std::cin >> c;
   std::cout << "managed to enter '" << c << "' fast enough" << std::endl;

   return 0;
}

The blocking std::cin call will not make this program wait for ever. If a input comes within the given timeout of 1 second, the program will run fine, otherwise it will abort with some noise.

Potential extension of the TimeoutInsurance

Such an insurance can be used also in a program where endless waiting is unwanted. In this case, and especially in multi threaded applications, you might not want to have one background thread per scope guard and extend the given sample code appropriated.

Thanks for reading

As usual, if you find any mistakes, in the code or in my spelling, please let me know.
Ideas for improvements, or any other feedback, is also very much welcome.