许多公司采用的一种实践是反复进行不稳定的测试,直到测试连续通过x次为止(可以是总次数中的连续),如果测试执行了n次并且未能至少通过x次,则将其标记为失败。
TestNG使用以下注释支持此功能:
@Test(invocationCount = 5, successPercentage = 40)
我该如何使用JUnit5实现类似的功能?
在JUnit5中有一个类似的注解,叫做@RepeatedTest(5)
,但它不会有条件地执行。
TestTemplateInvocationContextProvider
、ExecutionCondition
和TestExecutionExceptionHandler
扩展点,为您提供了一个简单的示例。
我处理失败测试的方法是将它们标记为“中止”,而不是让它们完全失败(这样整个测试执行就不会认为它是失败的),并且只有在我们无法获得最小数量的成功运行时才会失败测试。如果已经成功运行了最小数量的测试,则我们还会将剩余的测试标记为“禁用”。测试失败记录在ExtensionContext.Store
中,以便可以在每个地方查找状态。
这只是一个非常粗略的示例,肯定存在一些问题,但希望可以作为组合不同注释的示例。最终我用Kotlin编写了它:
@Retry
类似于TestNG示例的注释:
import org.junit.jupiter.api.TestTemplate
import org.junit.jupiter.api.extension.ExtendWith
@TestTemplate
@Target(AnnotationTarget.FUNCTION)
@ExtendWith(RetryTestExtension::class)
annotation class Retry(val invocationCount: Int, val minSuccess: Int)
TestTemplateInvocationContext
是用于模板化测试的上下文对象:
import org.junit.jupiter.api.extension.Extension
import org.junit.jupiter.api.extension.TestTemplateInvocationContext
class RetryTemplateContext(
private val invocation: Int,
private val maxInvocations: Int,
private val minSuccess: Int
) : TestTemplateInvocationContext {
override fun getDisplayName(invocationIndex: Int): String {
return "Invocation number $invocationIndex (requires $minSuccess success)"
}
override fun getAdditionalExtensions(): MutableList<Extension> {
return mutableListOf(
RetryingTestExecutionExtension(invocation, maxInvocations, minSuccess)
)
}
}
TestTemplateInvocationContextProvider
扩展用于@Retry
注释:
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ExtensionContextException
import org.junit.jupiter.api.extension.TestTemplateInvocationContext
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider
import org.junit.platform.commons.support.AnnotationSupport
import java.util.stream.IntStream
import java.util.stream.Stream
class RetryTestExtension : TestTemplateInvocationContextProvider {
override fun supportsTestTemplate(context: ExtensionContext): Boolean {
return context.testMethod.map { it.isAnnotationPresent(Retry::class.java) }.orElse(false)
}
override fun provideTestTemplateInvocationContexts(context: ExtensionContext): Stream<TestTemplateInvocationContext> {
val annotation = AnnotationSupport.findAnnotation(
context.testMethod.orElseThrow { ExtensionContextException("Must be annotated on method") },
Retry::class.java
).orElseThrow { ExtensionContextException("${Retry::class.java} not found on method") }
checkValidRetry(annotation)
return IntStream.rangeClosed(1, annotation.invocationCount)
.mapToObj { RetryTemplateContext(it, annotation.invocationCount, annotation.minSuccess) }
}
private fun checkValidRetry(annotation: Retry) {
if (annotation.invocationCount < 1) {
throw ExtensionContextException("${annotation.invocationCount} must be greater than or equal to 1")
}
if (annotation.minSuccess < 1 || annotation.minSuccess > annotation.invocationCount) {
throw ExtensionContextException("Invalid ${annotation.minSuccess}")
}
}
}
这是一个简单的data class
,代表了重试(在本例中使用ParameterResolver
将其注入到测试用例中)。
data class RetryInfo(val invocation: Int, val maxInvocations: Int)
异常
用于表示重试失败:
import java.lang.Exception
internal class RetryingTestFailure(invocation: Int, cause: Throwable) : Exception("Failed test execution at invocation #$invocation", cause)
主要扩展实现ExecutionCondition
,ParameterResolver
和TestExecutionExceptionHandler
。
import org.junit.jupiter.api.extension.ConditionEvaluationResult
import org.junit.jupiter.api.extension.ExecutionCondition
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ParameterContext
import org.junit.jupiter.api.extension.ParameterResolver
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler
import org.opentest4j.TestAbortedException
internal class RetryingTestExecutionExtension(
private val invocation: Int,
private val maxInvocations: Int,
private val minSuccess: Int
) : ExecutionCondition, ParameterResolver, TestExecutionExceptionHandler {
override fun evaluateExecutionCondition(
context: ExtensionContext
): ConditionEvaluationResult {
val failureCount = getFailures(context).size
// Shift -1 because this happens before test
val successCount = (invocation - 1) - failureCount
when {
(maxInvocations - failureCount) < minSuccess -> // Case when we cannot hit our minimum success
return ConditionEvaluationResult.disabled("Cannot hit minimum success rate of $minSuccess/$maxInvocations - $failureCount failures already")
successCount < minSuccess -> // Case when we haven't hit success threshold yet
return ConditionEvaluationResult.enabled("Have not ran $minSuccess/$maxInvocations successful executions")
else -> return ConditionEvaluationResult.disabled("$minSuccess/$maxInvocations successful runs have already ran. Skipping run $invocation")
}
}
override fun supportsParameter(
parameterContext: ParameterContext,
extensionContext: ExtensionContext
): Boolean = parameterContext.parameter.type == RetryInfo::class.java
override fun resolveParameter(
parameterContext: ParameterContext,
extensionContext: ExtensionContext
): Any = RetryInfo(invocation, maxInvocations)
override fun handleTestExecutionException(
context: ExtensionContext,
throwable: Throwable
) {
val testFailure = RetryingTestFailure(invocation, throwable)
val failures: MutableList<RetryingTestFailure> = getFailures(context)
failures.add(testFailure)
val failureCount = failures.size
val successCount = invocation - failureCount
if ((maxInvocations - failureCount) < minSuccess) {
throw testFailure
} else if (successCount < minSuccess) {
// Case when we have still have retries left
throw TestAbortedException("Aborting test #$invocation/$maxInvocations- still have retries left",
testFailure)
}
}
private fun getFailures(context: ExtensionContext): MutableList<RetryingTestFailure> {
val namespace = ExtensionContext.Namespace.create(
RetryingTestExecutionExtension::class.java)
val store = context.parent.get().getStore(namespace)
@Suppress("UNCHECKED_CAST")
return store.getOrComputeIfAbsent(context.requiredTestMethod.name, { mutableListOf<RetryingTestFailure>() }, MutableList::class.java) as MutableList<RetryingTestFailure>
}
}
接下来,是测试消费者:
import org.junit.jupiter.api.DisplayName
internal class MyRetryableTest {
@DisplayName("Fail all retries")
@Retry(invocationCount = 5, minSuccess = 3)
internal fun failAllRetries(retryInfo: RetryInfo) {
println(retryInfo)
throw Exception("Failed at $retryInfo")
}
@DisplayName("Only fail once")
@Retry(invocationCount = 5, minSuccess = 4)
internal fun succeedOnRetry(retryInfo: RetryInfo) {
if (retryInfo.invocation == 1) {
throw Exception("Failed at ${retryInfo.invocation}")
}
}
@DisplayName("Only requires single success and is first execution")
@Retry(invocationCount = 5, minSuccess = 1)
internal fun firstSuccess(retryInfo: RetryInfo) {
println("Running: $retryInfo")
}
@DisplayName("Only requires single success and is last execution")
@Retry(invocationCount = 5, minSuccess = 1)
internal fun lastSuccess(retryInfo: RetryInfo) {
if (retryInfo.invocation < 5) {
throw Exception("Failed at ${retryInfo.invocation}")
}
}
@DisplayName("All required all succeed")
@Retry(invocationCount = 5, minSuccess = 5)
internal fun allRequiredAllSucceed(retryInfo: RetryInfo) {
println("Running: $retryInfo")
}
@DisplayName("Fail early and disable")
@Retry(invocationCount = 5, minSuccess = 4)
internal fun failEarly(retryInfo: RetryInfo) {
throw Exception("Failed at ${retryInfo.invocation}")
}
}
在IntelliJ中的测试输出如下:
我不确定从TestExecutionExceptionHandler.handleTestExecutionException
抛出一个TestAbortedException
是否应该终止测试,但是我在这里使用它。
你可以尝试使用这个 JUnit 5 的扩展。
<dependency>
<groupId>io.github.artsok</groupId>
<artifactId>rerunner-jupiter</artifactId>
<version>LATEST</version>
</dependency>
示例:
/**
* Repeated three times if test failed.
* By default Exception.class will be handled in test
*/
@RepeatedIfExceptionsTest(repeats = 3)
void reRunTest() throws IOException {
throw new IOException("Error in Test");
}
/**
* Repeated two times if test failed. Set IOException.class that will be handled in test
* @throws IOException - error occurred
*/
@RepeatedIfExceptionsTest(repeats = 2, exceptions = IOException.class)
void reRunTest2() throws IOException {
throw new IOException("Exception in I/O operation");
}
/**
* Repeated ten times if test failed. Set IOException.class that will be handled in test
* Set formatter for test. Like behavior as at {@link org.junit.jupiter.api.RepeatedTest}
* @throws IOException - error occurred
*/
@RepeatedIfExceptionsTest(repeats = 10, exceptions = IOException.class,
name = "Rerun failed test. Attempt {currentRepetition} of {totalRepetitions}")
void reRunTest3() throws IOException {
throw new IOException("Exception in I/O operation");
}
/**
* Repeated 100 times with minimum success four times, then disabled all remaining repeats.
* See image below how it works. Default exception is Exception.class
*/
@DisplayName("Test Case Name")
@RepeatedIfExceptionsTest(repeats = 100, minSuccess = 4)
void reRunTest4() {
if(random.nextInt() % 2 == 0) {
throw new RuntimeException("Error in Test");
}
}
在 IDEA 中查看:
测试方法最少成功四次,然后禁用所有其他测试:
您也可以将 @RepeatedIfExceptionsTest 与 @DisplayName 混合使用
rerunner-jupiter
时,运行测试时就会收到这个错误,即使在测试中没有使用 @RepeatedIfExceptionsTest
,错误如下:java.lang.NoClassDefFoundError: org/junit/jupiter/api/extension/ScriptEvaluationException
。有什么想法如何解决这个问题吗? - user1207289rerunFailingTestsCount
。plugins {
id 'org.gradle.test-retry' version '1.2.0'
}
test {
retry {
maxRetries = 3
maxFailures = 20 // Optional attribute
}
}
@RetryingTest(3)
void flakyTest() {
// this works most of the time
}
@Test
和@RepeatedTest
属性的特性,例如currentRepetition
、totalRepetitions
和success
。 - Naman40%
的成功执行? - mkobit