Limiting execution time with fast-check 3.6

[elefevre]

With fast-check 2.x, we were using interruptAfterTimeLimit to reduce the overall run time of our tests, while still ensuring that at least one full run would occur (some of our tests write to the database and can get quite slow).

I am currently trying to upgrade to 3. The new behavior of interruptAfterTimeLimit is now to stop the currently running test abruptly. This means that sometimes not even a single run is successful, which is (correctly) interpreted as a failure.

I've tried playing with skipAllAfterTimeLimit and timeout instead, but they both lead to an error, even when a full run has completed successfully, because too many runs have been skipped:

Ran 1 time(s)
Skipped 10001 time(s)

...followed by the 10001 input cases that could have been run, but were not.
(as a side note, this message was confusing to me, as it implies that the test could have run 10001 times, and also it goes away when the test manages to reach numRuns iterations, not 10001)

Anyway, I am still trying to figure out a way to express the following:

run each one of fast-check-based tests at least once, and up to numRuns times, or up to X milliseconds, whichever comes first

Any ideas?

[dubzzz]

Hey,

The new behaviour of interruptAfterTimeLimit has been introduced in 3.5.0. In order to ease the bump, you should probably move first to 3.4.x and then towards 3.5.0.

At the moment, the idea has been to consider that passing interruptAfterTimeLimit (as it used to be the case before 3.5.0) because of a long run was not acceptable, thus anytime the first run passes interruptAfterTimeLimit it breaks the test. I initially thought about a flag making it possible not to stop runs at the middle of their execution. But so far the feature has not been implemented (some kind of middleground between interruptAfterTimeLimit and skipAllAfterTimeLimit) as it would mean that we don't really interrupt as soon as possible but wait the running test to stop.

I'm trying to think about it? 🤔

[elefevre]

Thanks for your response!
I do think that the new behavior has some merit, although its side-effect of making the tests flakier when using the interruptAfterTimeLimit flag is unfortunate.
I'll work on upgrading to 3.4 for now, thanks.

[dubzzz]

making the tests flakier

Actually, the behaviour has been inspired by what test frameworks such as Jest usually do for asynchronous code. For instance, in Jest, by default there is a timeout of 5000ms per run. If an asynchronous test takes longer, then the test gets killed and Jest marks it as failed.

The behaviour is now similar in fast-check. If interruptAfterTimeLimit stops on first run, it means that even a classical unit test written in Jest (or any other runner with timeouts) will kill the test and consider it as failed.

For instance if you run the following test in Jest:

test("never ending", async () => {
  await new Promise(() => {}); // never resolved
});

You'll receive:

 FAIL  ./main.spec.js (5.393 s)
  × never ending (5011 ms)
                                                                                                                                                                                                                                
  ● never ending                                                                                                                                                                                                                
                                                                                                                                                                                                                                
    thrown: "Exceeded timeout of 5000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

Is it flaky? Well it depends on your machine, the machine on which you run the tests in your CI, its current load...

[elefevre]

I see your point, but there is an issue of scope for the timeout.
For example, we have a 10-min timeout for running our entire test suite. This does not mean that we would find acceptable to wait for any given test for 10 mins.

For this, we have a separate setting for each of our ~2000 individual tests, the testTimeout that you refer to for Jest, set to 100000 ms, or roughly 1.5 mins. In practice, the worst single test function takes probably around 10 secs, with quite a few more taking 1+ sec.
Now, with fast-check, we have similar tests, except that they are run dozens of times, which can easily reach a minute or more for a single test, from Jest's point of view, risking a failure.

I had addressed this in fast-check v2 with these two flags:

interruptAfterTimeLimit: 200,
numRuns: 10

As I understood it, this is what I was aiming for when I said this:

run each one of fast-check-based tests at least once, and up to numRuns times, or up to X milliseconds, whichever comes first

However, I have just revisited these flags, and it turns out that removing interruptAfterTimeLimit has no discernable impact on the overall execution time. Maybe because numRuns is good enough in our situation, after all.

So, in the end, although I do believe there is merit in what I originally described, the issue has gone away for us, and we now have no problem upgrading to v3.6. 👍
Thanks!

[dubzzz]

What you were looking for is probably timeout option. It makes sure that each run of the predicate will take at most timeout ms.

interrupAfterTimeLimit will stop and stop launching predicates when the limit gets reached. You can see it as a better numRuns as it will try to put as many runs as possible within time limit.

[elefevre]

I don't quite remember if we tried timeout when we started using fast-check a couple of years ago. However, I did try to use timeout just now with both fast-check 2.25.0 and fast-check 3.6.2, and it makes the test fail, even with a high value (removing the timeout parameter is enough to make the test pass).

Property failed after 1 tests
{ seed: -1964255086, path: "0:0:0:0:0:0", endOnFailure: true }
Counterexample: ["00000000-0000-1000-8000-000000000000",["<1st constant string>","<2nd constant string>","<3rd constant string>"],[]]
Shrunk 5 time(s)
Got error: Property timeout: exceeded limit of 1000000 milliseconds

(jest reports that this particular test function took 150ms)
Not too sure what to make of this error.

But again, we have stopped using any timeout-style parameters, so this is not an issue for us.

[dubzzz]

I just read back the whole thread and here is what I'd suggest if I get well the issue.

For this, we have a separate setting for each of our ~2000 individual tests, the testTimeout that you refer to for Jest, set to 100000 ms

In order to avoid fast-check fc.assert run to go above the timeout on Jest side for a single test I'd recommend to use: interruptAfterTimeLimit: 100000. Actually I'd use something a bit smaller than 100000 to make sure fast-check will interrupt the run before Jest interrupts fast-check itself.

With that option toggled the fc.assert will take at most 100000ms (as your other tests from my understanding).

But if I get well, it's not fully answering to your needs as it means that the whole test execution may still takes up to 100s. For that second need I'd try to use skipAfterTimeLimit: 200 in conjunction of the previous flag.

The resulting behaviour will be:

• fc.assert will never run for more than 100000ms (I recommend to use something slightly lower to make sure fast-check can stop itself before jest stops it)
• any predicate still running after 100000 will be stopped at the middle (if feasible): it will result into a failure of the assert if and only if it was the first execution
• no predicate will start after 200ms

As a conclusion you may want to use something like:

fc.configureGlobal({
  skipAfterTimeLimit: 200,
  interruptAfterTimeLimit: 90000,
})

[elefevre]

Yes, "no predicate will start after xxx ms" is pretty much what I wanted. Setting a maximum duration is nice too, but not necessary for our needs (if fast-check tests do take more than 100,000 ms, we have a problem in our code anyway).

I can't find an option called skipAfterTimeLimit . I assume you mean skipAllAfterTimeLimit?
If I try the options literally as you suggest (or even just { skipAfterTimeLimit: 200 }), I get many failures of this type:

Failed to run property, too many pre-condition failures encountered
{ seed: 474496609 }

Ran 1 time(s)
Skipped 10001 time(s)

I am probably missing something, but I'm not sure what.
(to be clear, we have already been able to remove all such options, and we find the runtime to be acceptable; the only option I kept is ignoreEqualValues: true, because we tend to often run fast-check with values extracted from enums)