Cypress: --reporter CLI option doesn't look in root node_modules folder

[c32hedge]

Current behavior:

I was trying to restructure my repo to be similar to https://github.com/cypress-io/cypress-test-nested-projects, with cypress and other node modules installed at the root instead of in the same folder as the cypress.json files.

However, I'm also passing the --reporter option to use cypress-multi-reporters, which now fails with the following error:

Could not load reporter by name: cypress-multi-reporters

We searched for the reporter in these paths:

- /home/<redacted>/<projectRootFolder>/<suiteFolder>/cypress-multi-reporters
- /home/<redacted>/<projectRootFolder>/<suiteFolder>/node_modules/cypress-multi-reporters

The error we received was:

Cannot find module '/home/<redacted>/<projectRootFolder>/<suiteFolder>/node_modules/cypress-multi-reporters'
Require stack:
- /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/reporter.js
- /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/project.js
- /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/modes/run.js
- /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/modes/index.js
- /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/cypress.js
- /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/index.js
- /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/index.js
- 

Learn more at https://on.cypress.io/reporters

Desired behavior:

Cypress should also look further up the folder tree when a reporter package name is passed to the --reporter option, or there should be a CLI option to specify the node modules folder.

Test code to reproduce

I'll work on adding a reproducible example tomorrow. I expect it would reproduce if you tried passing a --reporter option in the existing cypress run logic in https://github.com/cypress-io/cypress-test-nested-projects.

Versions

Cypress: 4.0.1

[c32hedge]

Reproducible example linked. Sequence of commands to run to reproduce:

npm install
npm run start-bar
npm run test-bar

[c32hedge]

I added another commit after attempting to specify the path to the node modules folder in the reporter option value in a cypress.json file as shown below.

Commands to reproduce

npm install
npm run start-foo
npm run test-foo

Cypress.json

{
  "baseUrl": "http://localhost:8080",
  "reporter": "../../../node_modules/cypress-multi-reporters",
  "reporter-options": {
    "configFile": "reporter-config.json"
  }
}

Behavior

In this case, the tests actually run, but afterward the test runner fails with the following errors:

TypeError: test.isPending is not a function
    at XUnit.test (/home/<user>/cypress-test-nested-projects/node_modules/mocha/lib/reporters/xunit.js:182:19)
    at /home/<user>/cypress-test-nested-projects/node_modules/mocha/lib/reporters/xunit.js:106:12
    at Array.forEach (<anonymous>:null:null)
    at Runner.<anonymous> (/home/<user>/cypress-test-nested-projects/node_modules/mocha/lib/reporters/xunit.js:105:11)
    at Object.onceWrapper (events.js:291:20)
    at Runner.emit (events.js:208:15)
    at Reporter.emit (/home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/reporter.js:234:55)
    at Object.onMocha (/home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/project.js:359:18)
    at Socket.<anonymous> (/home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/socket.js:305:32)
    at Socket.emit (events.js:203:13)
    at /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:528:12
    at processTicksAndRejections (internal/process/task_queues.js:75:11)

TypeError: test.isPending is not a function
    at XUnit.test (/home/<user>/cypress-test-nested-projects/node_modules/mocha/lib/reporters/xunit.js:182:19)
    at /home/<user>/cypress-test-nested-projects/node_modules/mocha/lib/reporters/xunit.js:106:12
    at Array.forEach (<anonymous>:null:null)
    at Runner.<anonymous> (/home/<user>/cypress-test-nested-projects/node_modules/mocha/lib/reporters/xunit.js:105:11)
    at Object.onceWrapper (events.js:291:20)
    at Runner.emit (events.js:208:15)
    at Reporter.emit (/home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/reporter.js:234:55)
    at Object.onMocha (/home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/project.js:359:18)
    at Socket.<anonymous> (/home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/server/lib/socket.js:305:32)
    at Socket.emit (events.js:203:13)
    at /home/<user>/.cache/Cypress/4.0.1/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:528:12
    at processTicksAndRejections (internal/process/task_queues.js:75:11)

[c32hedge]

Here's a link to the actual branch since I needed to make another commit. I confirmed that the error in my last comment occurs even if mocha-junit-reporter isn't one of the reporters in the list.

https://github.com/c32hedge/cypress-test-nested-projects/tree/6406

[c32hedge]

Anyone had a chance to look at this? @jennifer-shehane?

[jennifer-shehane]

@c32hedge Where do you have cypress installed? I see it is not included in the package.json file of the project.

When I install cypress as a dev-dep in the project and run the commands you specified - I don't get any errors.

[c32hedge]

@jennifer-shehane cypress is included in the package.json file at the top level of the repository: https://github.com/c32hedge/cypress-test-nested-projects/blob/6406/package.json

This matches how I'm wanting to set it up for my real repository--I want to manage a single cypress version instead of one per "project".

[c32hedge]

This also matches what's described in the readme for cypress-test-nested-projects: https://github.com/cypress-io/cypress-test-nested-projects/blob/master/README.md. And it appears to work fine, if the --reporter option is not specified.

For the below details, it's useful to define some terms in this context:

• project root: A folder that contains a cypress.json file and directories containing test specs, which is passed to cypress using the --project CLI option.
• cypress root: The folder that contains the package.json file and node_modules folder that include Cypress.

I did a little digging in the Cypress code based on the stack trace, and it looks like the code just gives up if it can't find the reporter in the project root.

I think the behavior I would expect is that it would look in the project root first, and then either traverse through parent directories until it reaches cypress root, or else just check cypress root immediately after looking in project root (since it might not be safe to assume that project root is nested inside cypress root).

[c32hedge]

@jennifer-shehane any thoughts? Currently I'm working around this by only using the junit reporter, which is annoying for reading console logs, and it would be really nice to get the cypress-multi-reporter working again.

[stevenvachon]

I'm wrapping cypress so that I can abstract custom configuration and I switched from using both spec and @mochajs/json-file-reporter to only the latter (because of https://github.com/YOU54F/cypress-multi-reporters/issues/68) and now I get this:

Could not load reporter by name: @mochajs/json-file-reporter

We searched for the reporter in these paths:

- /project/node_modules/<wrapper-project>/@mochajs/json-file-reporter
- /project/node_modules/<wrapper-project>/node_modules/@mochajs/json-file-reporter

The error we received was:

Cannot find module '/project/node_modules/<wrapper-project>/node_modules/@mochajs/json-file-reporter'
Require stack:
- /<caches>/Cypress/4.5.0/Cypress.app/Contents/Resources/app/packages/server/lib/reporter.js
- /<caches>/Cypress/4.5.0/Cypress.app/Contents/Resources/app/packages/server/lib/project.js
- /<caches>/Cypress/4.5.0/Cypress.app/Contents/Resources/app/packages/server/lib/modes/run.js
- /<caches>/Cypress/4.5.0/Cypress.app/Contents/Resources/app/packages/server/lib/modes/index.js
- /<caches>/Cypress/4.5.0/Cypress.app/Contents/Resources/app/packages/server/lib/cypress.js
- /<caches>/Cypress/4.5.0/Cypress.app/Contents/Resources/app/packages/server/index.js
- /<caches>/Cypress/4.5.0/Cypress.app/Contents/Resources/app/index.js

I can confirm that the package is located at:
/project/node_modules/@mochajs/json-file-reporter

Using require.resolve('@mochajs/json-file-reporter') patches the issue, but the terminal log is littered with irrelevant crap (from Mocha?).

[YOU54F]

If we need any changes in cypress-multi-reporters just give us a shout 👍 will take some time to look at this over the weekend

[c32hedge]

Is there a better way to get more visibility to this other than adding a comment every few months? Again, it's a pretty annoying issue for us, and it seems like no one on the Cypress team has really looked closely at it 😢.

[emilong]

@c32hedge I think I found a workaround! I looked in https://github.com/cypress-io/cypress/blob/9abed894eb1de3f81535cc13ec2775ed9d966c22/packages/server/lib/reporter.js#L516-L521

Looks like the reporter string is evaluated against the Cypress spec directory and its node_modules subdirectory, but it's also using path.resolve, so relative paths worked for me, e.g.

{
  "reporter": "../node_modules/cypress-multi-reporters"
}

Hope that helps!

[af6140]

as discussed, the loading mechanism does not search ${NODE_PATH}/${reporter_module_name} when reporter module is installed globally.

[c32hedge]

@emilong Interesting. I had tried the relative path approach previously, in my reproducible example, and got errors. However, since creating this issue we switched to calling Cypress using the module API from a script in the root of our repo, and I confirmed that giving reporter a relative path and including it in the run options passed to cypress.run does seem to work. I don't know if the Cypress teams changed something in a more recent version that allowed this to work, if using the module API behaves differently, or if there's something about the cypress-test-nested-projects repo that caused the other error, but in any case I'm back in business for my specific situation. Thanks!