Jest: Coverage report fails with puppeteer evaluate code

[sab24]

🐛 Bug Report

Code coverage fails to evaluate code that is evaluated in another environment.

To Reproduce

Steps to reproduce the behaviour:

1. Create a Nodejs program that include page.evaluate from Puppeteer. This code is evaluated in a separate environment and causes test cases to fail.

Expected behaviour

Proper coverage report with the puppeteer evaluate code excluded. Otherwise some kind of configuration option to exclude remotely executed code.

jest
 PASS  ./test.js
  ✓ testing puppeteer page scrape (1144ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        3.653s
Ran all test suites.

Current behaviour

jest --coverage
 FAIL  ./test.js
  ✕ testing puppeteer page scrape (737ms)

  ● testing puppeteer page scrape

    Evaluation failed: ReferenceError: cov_gpckfapr6 is not defined

      at __puppeteer_evaluation_script__:3:7
      at ExecutionContext.evaluateHandle (node_modules/puppeteer/lib/ExecutionContext.js:121:13)
        -- ASYNC --
      at ExecutionContext.<anonymous> (node_modules/puppeteer/lib/helper.js:108:27)
      at ExecutionContext.evaluate (node_modules/puppeteer/lib/ExecutionContext.js:48:31)
      at ExecutionContext.<anonymous> (node_modules/puppeteer/lib/helper.js:109:23)
      at DOMWorld.evaluate (node_modules/puppeteer/lib/DOMWorld.js:105:20)
        -- ASYNC --
      at Frame.<anonymous> (node_modules/puppeteer/lib/helper.js:108:27)
      at Page.evaluate (node_modules/puppeteer/lib/Page.js:815:43)
      at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:109:23)
      at Object.evaluate [as evaluateContent] (scrape.js:4:24)

-----------|----------|----------|----------|----------|-------------------|
File       |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
-----------|----------|----------|----------|----------|-------------------|
All files  |       50 |      100 |       50 |       50 |                   |
 scrape.js |       50 |      100 |       50 |       50 |             5,7,8 |
-----------|----------|----------|----------|----------|-------------------|
Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        2.781s, estimated 4s
Ran all test suites.
Jest did not exit one second after the test run has completed.

This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.

Link to repl or repo (highly encouraged)

https://github.com/sab24/jestexample

Run by typing
npm i
jest
jest --coverage

Run npx envinfo --preset jest

Paste the results here:

  System:
    OS: macOS 10.14.2
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
  Binaries:
    Node: 11.9.0 - /usr/local/bin/node
    npm: 6.8.0 - /usr/local/bin/npm
  npmPackages:
    jest: ^24.1.0 => 24.1.0 

[jeysal]

If I understand correctly, you want return(document.querySelector('#hplogo').alt); to show up as uncovered, but it shows up as covered. Can you explain why you would want that to happen? It seems like very unintuitive behavior to me.
You could extract the pieces of code that you want to exclude from coverage into their own files that are matched by coveragePathIgnorePatterns if you really need to ignore those lines

[sab24]

Hi @jeysal,

The test cases fail because the variable tracking in the code coverage software go out of scope I think. Therefore my test cases fail while they do succeed without the code coverage flag.

Is it even possible to save the inner part of page.evaluate in another file?

[jeysal]

Oh, now I see what you mean. I didn't understand that the actual test result differed depending on whether coverage is enabled (and I couldn't reproduce it on my Linux machine because I got a different error - but on my Mac I can see the behavior you're describing)

[jeysal]

Could something like https://github.com/istanbuljs/puppeteer-to-istanbul perhaps solve your issue?

[carmanchris31]

I'm having the same issue and unfortunately puppeteer-to-istanbul had no effect. Best option for now might to be to isolate the page.evaluate call into a separate file and selectively disable coverage for that file by using coveragePathIgnorePatterns in your jest config :-/

[StephanBijzitter]

Best option for now might to be to isolate the page.evaluate call into a separate file and selectively disable coverage for that file by using coveragePathIgnorePatterns in your jest config :-/

This worked for me, thanks!
It's not quite perfect... but it'll do.

Took me way too long to figure out the problem was with Jest, rather than Puppeteer or being related to my test being in TypeScript.

Will see if I can edit https://jestjs.io/docs/en/puppeteer to add a disclaimer.

[tom-sherman]

I have also encountered this same issue when working with PouchDB where indexes are written in JS and stringified into the database. The views are evaluated in a sandbox where the coverage collectors aren't defined. See https://github.com/pouchdb/pouchdb/issues/8129

We've resorted to ignoring the file from coverage which works, but isn't ideal.