Jest: Add `jest.retryTimes` metadata to `TestResult` for custom reporters

Created on 15 Feb 2019 · 5Comments · Source: facebook/jest

🚀 Feature Proposal

I love using circus's retryTimes to run several integration tests. It would be very helpful to have access to the number of retries required for the test to pass in a custom reporter. For example, if a test passed on the second run, the TestResult passed to onTestResult would have testResult.retryCount == 1.

Motivation

I use Jest to run several smoke tests using puppeteer in my company's staging and production environments to ensure our services are properly configured to work well together. As with many browser level integration tests, they tend to have false positive failures frequently. retryTimes is a great feature to address these false positives. However, I'd like to track metrics for how frequently each test requires a retry to pass. This will help us understand if our failure rate for each test changes over time.

We recently had a test which went from close to a 0% failure rate to a 30% failure rate. We didn't realize how big the problem was until I did an analysis of the historic logs. I'd like to be able to set up alerts when the error rate changes compared to historic values.

Example

const StatsD = require('hot-shots');
const statsdClient = new StatsD({ port: 8020, globalTags: { env: process.env.NODE_ENV }); 

class StatsdRetryCountReporter {
  onTestResult(_test, testResult) {
    statsdClient.increment(testResult.displayName, {value: testResult.retryCount})
  }
}

module.exports = StatsdRetryCountReporter;

Pitch

I believe jest core is the right place for this feature because retryTimes is part of circus. Additionally, this metric seems very valuable to custom reporters when using retryTimes.

Feature Request Help Wanted good first issue

Source