Nx: Coverage reports are not restored when cache is used

[gund]

Current Behavior

If the cache for tests is enabled and after first run with coverage on the subsequent runs test coverage will not be restored even when passing flag --codeCoverage.

I'm not sure if this affects all the test runners but I experience this with @nrwl/jest.

Expected Behavior

When a test was ran with coverage report then repeated tests with coverage must restore coverage reports.

Steps to Reproduce

• Create or use existing NX repo with test runner used with @nrwl/jest
• Make sure that cache for tests is enabled in nx.json
• Execute the tests with coverage report nx test --codeCoverage
• Remove generated report folder (usually in coverage/**)
• Execute the tests with coverage again nx test --codeCoverage
• Assert that cached output was used and no coverage report is on file system.

_NOTE:_ You may use my public repo for testing as it's public and that is where I hit this error: https://github.com/gund/time-slots

Please also note that I've disabled the cache for tests for time being as my CI was failing due to missing coverage reports (failed job as example where the cached test output was used)

Failure Logs

When the cache is used for tests with coverage:

> @gund/[email protected] test:ci /home/travis/build/gund/time-slots
> nx test --codeCoverage
> nx run time-slots:test --codeCoverage 
>  NX   NOTE  Cached Output:
PASS time-slots packages/time-slots/src/lib/time-slots-excluder.spec.ts
PASS time-slots packages/time-slots/src/lib/time-slots-generator.spec.ts
Test Suites: 2 passed, 2 total
Tests:       3 passed, 3 total
Snapshots:   0 total
Time:        2.613 s
Ran all test suites.
>  NX   CLOUD  Results for time-slots were retrieved from Nx Cloud. See run details at https://nx.app/runs/G0r5rKk5abD

The folder with coverage report is not recovered and codecov tool fails:

> codecov -p coverage/packages/time-slots --disable=gcov
  _____          _  
 / ____|        | |  
| |     ___   __| | ___  ___ _____   __  
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /  
| |___| (_) | (_| |  __/ (_| (_) \ V /  
 \_____\___/ \__,_|\___|\___\___/ \_/  
                                v3.8.0
==> Detecting CI Provider
    Travis CI Detected
==> Configuration: 
    Endpoint: https://codecov.io
{
  commit: '6b383b7c6b0eca2a690ca1676ed04b762d638454',
  branch: 'master',
  package: 'node-v3.8.0'
}
==> Building file structure
internal/fs/utils.js:269
    throw err;
    ^
Error: ENOENT: no such file or directory, scandir '/home/travis/build/gund/time-slots/coverage/packages/time-slots'

Environment

>  NX  Report complete - copy this into the issue template

  nx : Not Found
  @nrwl/angular : Not Found
  @nrwl/cli : 10.3.1
  @nrwl/cypress : Not Found
  @nrwl/eslint-plugin-nx : 10.3.1
  @nrwl/express : Not Found
  @nrwl/jest : 10.3.1
  @nrwl/linter : 10.3.1
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : 10.3.1
  @nrwl/react : Not Found
  @nrwl/schematics : Not Found
  @nrwl/tao : 10.3.1
  @nrwl/web : Not Found
  @nrwl/workspace : 10.3.1
  typescript : 4.0.3

[un33k]

@gund Thank you for reporting this - I also have disabled cache for test and e2e as the artifacts are not being restored. Hoping for a configurable solution, such that we can provide a list of directories for test, build & e2e to be cached and restored on each run.

[Esmilis]

Actually if you add outputPath to test options in workspace.json, it will get cached and restored on subsequent builds

{"example-app":` {
      "root": "apps/example-app",
      ...
      "architect": {
        "build": {
         ...
        },
        "serve": {
         ...
        },
        "lint": {
         ....
        },
        "test": {
          "builder": "@nrwl/jest:jest",
          "options": {
            "outputPath": "coverage/apps/example-app", <----add this
            "jestConfig": "apps/example-app/jest.config.js"
          }
        }
      }

Keep in mind that cache is restored by folder, so don't mix output from different apps/libs within the same one.

[un33k]

@Esmilis Jest has that option, Karma doesn't, and there could be many libs, some use Jest and some use Karma. Otherwise, this would have been a great workaround.

[un33k]

A workaround is to set the outputPath for jest unit tests, and create a script that grabs jest and karma unit tests from workspace.json | angular.json into separate lists and runs them one at a time, using the options --project=jestProj1,jestProj2 for jest and --project-karmaProj1,karmaProj2 --skip-nx-cache for the karma. This way you have at least enabled cloud cache on the jest tests. Works well, specially if you have a very large project with mixed tests and still need some caching till this issue is addressed.

[Esmilis]

@un33k Actually outputPath caching is a task runner feature, and not specific to jest. It will work for Karma too, or any kind of architect output caching

[un33k]

@Esmilis It is not possible for karma - Jest is fine. (nx 10.4.4)

[Esmilis]

This is just a validation issue, It should work when you enter and run it. And if it really does, just need to raise PR to add this to karma schema too :)

[bekos]

It will work if you use outputs. This has already been fixed via 6c1335ac83859b0a39a4767c8787560272b9a1ac and will be part of v11 release (you can already try our beta even now though 😉), with the appropriate migration. Feel free to open a new issue if something doesn't work as expected.

[un33k]

May I recommend to only close tickets based on "it works", and not "it should work". It would be beneficial to leave the tickets open till the fix is released. This way, the next person hits this error, is not going to open a new ticket.

Upgrade to beta is a no-go either.

{
      "version": "11.0.0-beta.7",
      "description": "Adds `outputs` based on builders",
      "factory": "./src/migrations/update-11-0-0/add-outputs-in-workspace",
      "package": "@nrwl/workspace",
      "name": "add-outputs-in-workspace"
}

[Esmilis]

It works for those who know how to work with beta versions:) The rest will need to wait for the official release...

Also i'm pretty sure these tickets are to plan work, and since there is no work left, it makes sense to close it:) Otherwise someone might waste precious developer time in trying to implement it again

[un33k]

For those who end up on this page looking for an answer - NX 11-beta-16 works.

The following migration must succeed.

{
      "version": "11.0.0-beta.16",
      "description": "Adds `outputs` based on builders",
      "factory": "./src/migrations/update-11-0-0/add-outputs-in-workspace",
      "package": "@nrwl/workspace",
      "name": "add-outputs-in-workspace"
}

Then all your projects need to setup an "outputs" list, something like "outputs": ["coverage/<app-or-lib-path>", "<app-or-lib-path/reporters"].

Pretty much anything you want to push up to nx cloud cache and retrieve on cache hit can be put in the above list.
I am sure NX docs will follow soon.