Kotest: FreeSpec.invocationTimeout is applied to the container, not to the test
Which version of Kotest are you using
4.3.0
- Reproduction *
@ExperimentalTime
class TimeoutTest : FreeSpec({
val specLevelTimeout = 1.minutes
val specLevelInvocationTimeout = 5.seconds
timeout = specLevelTimeout.toLongMilliseconds()
invocationTimeout = specLevelInvocationTimeout.toLongMilliseconds()
val sleepDuration = 1.seconds
"all tests have spec-level invocationTimeout of $specLevelInvocationTimeout and a timeout of $specLevelTimeout" - {
val testLevelTimeout = 30.minutes
val testLevelInvocationTimeout = Duration.INFINITE
"test has $testLevelInvocationTimeout invocation timeout and $testLevelTimeout timeout"
.config(invocationTimeout = testLevelInvocationTimeout, timeout = testLevelTimeout, invocations = 10) {
println("Running test. About to sleep at ${System.currentTimeMillis()} for $sleepDuration")
Thread.sleep(sleepDuration.toLongMilliseconds())
println("Finished sleeping and running test at ${System.currentTimeMillis()}")
}
}
})
My FreeSpec tests are setup as follows:
- Spec-level timeout of 1 minute
- Spec-level invocation timeout of 5 seconds
- Test-level timeout of 30 minutes
- Test-level invocation timeout of infinite
Expected:
Test pass, because each invocation of the inner test takes less than 5 seconds
Actual:
Running test. About to sleep at 1603861299038 for 1.00s
Finished sleeping and running test at 1603861300041
Running test. About to sleep at 1603861300045 for 1.00s
Finished sleeping and running test at 1603861301045
Running test. About to sleep at 1603861301045 for 1.00s
Finished sleeping and running test at 1603861302050
Running test. About to sleep at 1603861302051 for 1.00s
Finished sleeping and running test at 1603861303053
Running test. About to sleep at 1603861303054 for 1.00s
Finished sleeping and running test at 1603861304056
Test did not complete within 1800000ms
TimeoutException(duration=1800000)
at io.kotest.core.internal.TestCaseExecutor.executeAndWait(TestCaseExecutor.kt:231)
at io.kotest.core.internal.TestCaseExecutor.invokeTestCase(TestCaseExecutor.kt:176)
at io.kotest.core.internal.TestCaseExecutor.executeActiveTest(TestCaseExecutor.kt:145)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1$1.invokeSuspend(TestCaseExecutor.kt:83)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1$1.invoke(TestCaseExecutor.kt)
at io.kotest.core.internal.TestCaseExecutor.executeIfActive(TestCaseExecutor.kt:109)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1.invokeSuspend(TestCaseExecutor.kt:83)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1.invoke(TestCaseExecutor.kt)
at io.kotest.core.internal.TestCaseExecutor.intercept(TestCaseExecutor.kt:97)
at io.kotest.core.internal.TestCaseExecutor.execute(TestCaseExecutor.kt:63)
at io.kotest.engine.runners.SingleInstanceSpecRunner.runTest(SingleInstanceSpecRunner.kt:73)
at io.kotest.engine.runners.SingleInstanceSpecRunner$Context.registerTestCase(SingleInstanceSpecRunner.kt:51)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1$3$1$contextp$1.registerTestCase(TestCaseExecutor.kt:219)
at io.kotest.core.test.TestContext$DefaultImpls.registerTestCase(TestContext.kt:40)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1$3$1$contextp$1.registerTestCase(TestCaseExecutor.kt:217)
at io.kotest.core.spec.style.scopes.TestWithConfigBuilder.config-dVAIICY(TestWithConfigBuilder.kt:52)
at io.kotest.core.spec.style.scopes.FreeScope.config-06zEaDw(FreeScope.kt:61)
at io.kotest.core.spec.style.scopes.FreeScope.config-06zEaDw$default(FreeScope.kt:54)
at kotest.TimeoutTest$1$1.invokeSuspend(TimeoutTest.kt:22)
at kotest.TimeoutTest$1$1.invoke(TimeoutTest.kt)
at io.kotest.core.spec.style.scopes.FreeSpecRootScope$minus$1.invokeSuspend(FreeSpecRootScope.kt:20)
at io.kotest.core.spec.style.scopes.FreeSpecRootScope$minus$1.invoke(FreeSpecRootScope.kt)
at io.kotest.core.internal.ExecutionsKt$executeWithBehaviours$2$1.invokeSuspend(executions.kt:13)
at io.kotest.core.internal.ExecutionsKt$executeWithBehaviours$2$1.invoke(executions.kt)
at io.kotest.core.internal.ExecutionsKt.wrapTestWithGlobalAssert(executions.kt:39)
at io.kotest.core.internal.ExecutionsKt$executeWithBehaviours$2.invokeSuspend(executions.kt:12)
at io.kotest.core.internal.ExecutionsKt$executeWithBehaviours$2.invoke(executions.kt)
at io.kotest.core.internal.ExecutionsKt.wrapTestWithAssertionModeCheck(executions.kt:23)
at io.kotest.core.internal.ExecutionsKt.executeWithBehaviours(executions.kt:11)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1$3$1.invokeSuspend(TestCaseExecutor.kt:222)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1$3$1.invoke(TestCaseExecutor.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:194)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1$3.invokeSuspend(TestCaseExecutor.kt:216)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1$3.invoke(TestCaseExecutor.kt)
at io.kotest.mpp.ReplayKt.replay(replay.kt:18)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1.invokeSuspend(TestCaseExecutor.kt:211)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1$1.invoke(TestCaseExecutor.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:120)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:37)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1.invokeSuspend(TestCaseExecutor.kt:210)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2$1.invoke(TestCaseExecutor.kt)
at io.kotest.engine.ExecutorExecutionContext$executeWithTimeoutInterruption$$inlined$suspendCoroutine$lambda$2.invokeSuspend(ExecutorExecutionContext.kt:56)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at io.kotest.engine.ExecutorExecutionContext.executeWithTimeoutInterruption(ExecutorExecutionContext.kt:55)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2.invokeSuspend(TestCaseExecutor.kt:209)
at io.kotest.core.internal.TestCaseExecutor$executeAndWait$2.invoke(TestCaseExecutor.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:120)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:37)
at io.kotest.core.internal.TestCaseExecutor.executeAndWait(TestCaseExecutor.kt:207)
at io.kotest.core.internal.TestCaseExecutor.invokeTestCase(TestCaseExecutor.kt:176)
at io.kotest.core.internal.TestCaseExecutor.executeActiveTest(TestCaseExecutor.kt:145)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1$1.invokeSuspend(TestCaseExecutor.kt:83)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1$1.invoke(TestCaseExecutor.kt)
at io.kotest.core.internal.TestCaseExecutor.executeIfActive(TestCaseExecutor.kt:109)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1.invokeSuspend(TestCaseExecutor.kt:83)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1.invoke(TestCaseExecutor.kt)
at io.kotest.core.internal.TestCaseExecutor.intercept(TestCaseExecutor.kt:97)
at io.kotest.core.internal.TestCaseExecutor.execute(TestCaseExecutor.kt:63)
at io.kotest.engine.runners.SingleInstanceSpecRunner.runTest(SingleInstanceSpecRunner.kt:73)
at io.kotest.engine.runners.SingleInstanceSpecRunner$execute$2$invokeSuspend$$inlined$invoke$lambda$1.invokeSuspend(SingleInstanceSpecRunner.kt:83)
at io.kotest.engine.runners.SingleInstanceSpecRunner$execute$2$invokeSuspend$$inlined$invoke$lambda$1.invoke(SingleInstanceSpecRunner.kt)
at io.kotest.engine.spec.SpecRunner$runParallel$$inlined$map$lambda$2$1.invokeSuspend(SpecRunner.kt:80)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at io.kotest.engine.spec.SpecRunner$runParallel$$inlined$map$lambda$2.run(SpecRunner.kt:79)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Test did not complete within 60000ms
TimeoutException(duration=60000)
at io.kotest.core.internal.TestCaseExecutor.executeAndWait(TestCaseExecutor.kt:231)
at io.kotest.core.internal.TestCaseExecutor.invokeTestCase(TestCaseExecutor.kt:176)
at io.kotest.core.internal.TestCaseExecutor.executeActiveTest(TestCaseExecutor.kt:145)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1$1.invokeSuspend(TestCaseExecutor.kt:83)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1$1.invoke(TestCaseExecutor.kt)
at io.kotest.core.internal.TestCaseExecutor.executeIfActive(TestCaseExecutor.kt:109)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1.invokeSuspend(TestCaseExecutor.kt:83)
at io.kotest.core.internal.TestCaseExecutor$intercept$innerExecute$1.invoke(TestCaseExecutor.kt)
at io.kotest.core.internal.TestCaseExecutor.intercept(TestCaseExecutor.kt:97)
at io.kotest.core.internal.TestCaseExecutor.execute(TestCaseExecutor.kt:63)
at io.kotest.engine.runners.SingleInstanceSpecRunner.runTest(SingleInstanceSpecRunner.kt:73)
at io.kotest.engine.runners.SingleInstanceSpecRunner$execute$2$invokeSuspend$$inlined$invoke$lambda$1.invokeSuspend(SingleInstanceSpecRunner.kt:83)
at io.kotest.engine.runners.SingleInstanceSpecRunner$execute$2$invokeSuspend$$inlined$invoke$lambda$1.invoke(SingleInstanceSpecRunner.kt)
at io.kotest.engine.spec.SpecRunner$runParallel$$inlined$map$lambda$2$1.invokeSuspend(SpecRunner.kt:80)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at io.kotest.engine.spec.SpecRunner$runParallel$$inlined$map$lambda$2.run(SpecRunner.kt:79)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
My understand of invocationTimeout (as per https://kotest.io/timeout/) is
per individual run of a test
However, according to https://github.com/kotest/kotest/blob/master/doc/styles.md#free-spec, the "test" should be the inner-most body of the FreeSpec, but the invocationTimeout is clearly applied at the container level, not the test level
"containers can be nested as deep as you want" - {
"and so we nest another container" - {
"yet another container" - {
"finally a real test" {
1 + 1 shouldBe 2
}
}
}
}
Note that the exception messages are incorrect, but these are covered separately in https://github.com/kotest/kotest/issues/1807 and https://github.com/kotest/kotest/issues/1809
bug
Woodz
All 4 comments
Invocation timeouts should not apply to container tests.
PR here: https://github.com/kotest/kotest/pull/1814
sksamuel
on 29 Oct 2020
👍1
Fixed in master, will be included in 4.3.1
sksamuel
on 29 Oct 2020
👍1
Thank you for your rapid fix @sksamuel! Really appreciate the support :-)
Woodz
on 4 Nov 2020
You're welcome.
On Wed, 4 Nov 2020, 07:21 Richard Wise, notifications@github.com wrote:
Thank you for your rapid fix @sksamuel https://github.com/sksamuel!
Really appreciate the support :-)—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kotest/kotest/issues/1810#issuecomment-721759706, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFVSGQJC3X4G7DJVSQAH3TSOFPPBANCNFSM4TB2KWKA
.
sksamuel
on 4 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
izhangzhihao
·
3Comments
Reevn
·
5Comments
sksamuel
·
11Comments
myuwono
·
8Comments
code-schreiber
·
9Comments