Cypress: Show warning messages in command log

[Bkucera]

It would be useful to allow commands to print a warning instead of an error in the command log.

This would allow commands to alert the user something they have done may lead to unexpected behavior, such as pressing the {home} key while we cannot preform the default action of scrolling. Or calling a cy.wait after a cy.visit.

We could have:

• If there's a soft error during a command, allow a config value to turn it into a warning.
• If there's a warning during a command, add an icon to the command and allow it to be clicked and expanded
• Ability to add a new command log message / event for warnings/errors not linked directly to a command

[flotwig]

A bunch of warnings already exist in the driver, though they just dump to console.warn, where most users won't notice it:

https://github.com/cypress-io/cypress/blob/0cbd95460199176896d7cb935df73777b0e878d7/packages/driver/src/cy/commands/xhr.coffee#L418-L418

https://github.com/cypress-io/cypress/blob/0cbd95460199176896d7cb935df73777b0e878d7/packages/driver/src/cy/commands/request.coffee#L100-L100

https://github.com/cypress-io/cypress/blob/0cbd95460199176896d7cb935df73777b0e878d7/packages/driver/src/cy/commands/agents.coffee#L234-L234

https://github.com/cypress-io/cypress/blob/0cbd95460199176896d7cb935df73777b0e878d7/packages/driver/src/cypress/cookies.coffee#L48-L58

https://github.com/cypress-io/cypress/blob/0cbd95460199176896d7cb935df73777b0e878d7/packages/driver/src/cypress/cy.coffee#L65-L67

https://github.com/cypress-io/cypress/blob/0cbd95460199176896d7cb935df73777b0e878d7/packages/driver/src/cypress/server.coffee#L45-L51

https://github.com/cypress-io/cypress/blob/0cbd95460199176896d7cb935df73777b0e878d7/packages/driver/src/cypress/xml_http_request.coffee#L127-L137

[jennifer-shehane]

Yeah, we mostly use console.warn for deprecations currently.

I like this idea though. I feel like this is what most people would want for the 'uncaught:exception' messages we give. They do not want a test failure, but a warning may be nice.

Relevant to error improvements: https://github.com/cypress-io/cypress/issues/3762

[jennifer-shehane]

Played around with some of the design of this warning.

[jennifer-shehane]

Second iteration for design:

[Bkucera]

@jennifer-shehane I like the yellow background with dark yellow/orange text :+1:

[jennifer-shehane]

K, I need to rework the design to also work for inline commands - so the situation Ben originally described like

pressing the {home} key while we cannot preform the default action of scrolling

[jennifer-shehane]

• Establish pattern that draws the user's attention to something that functionally is fine - does not cause a failure, but that they should be aware of.
• Maybe have warn types - like deprecation, antipattern, misuse - even when it's not associated with a command?

Example: unnecessary assertion

cy.get('button').should('exist')

cy.get('button', { warn: false }).should('exist')

While functionally this won't break - the .should('exist') is unnecessary.

Example: antipattern?

cy.visit(...)
cy.route(...) // definition for a route that has already happened

Another cy.route() example warning: https://github.com/cypress-io/cypress/blob/develop/packages/driver/src/cy/commands/xhr.coffee#L419

We don't display route as a command in the command log, but when there's a warning, we should show it in there - similar to showing .then() when it throws.

cy.then(() => {
  throw new Error('foo')
})

You likely don't want to define the listener for the cy.route() after cy.visit() in this situation.

Example: limitations / unexpected behavior

In certain browsers, the home char sequence wouldn't really behave as you are expecting...

Like caniuse for mdn docs saying what does or doesn't work in browsers.

cy.get('input').type('foo{home}', { simulated: true })

cy.get('input').type('foo{home}', { simulated: true, warn: false })

Example: deprecation warnings

We do these in the console.warn now, but why not have it in the Command Log?

Example: async weirdness/mixing Promises and Cypress

Example: uncaught exceptions

Implementation:

Cypress.log({
  warning: ...
})

Should be expandable, but not expanded by default - unless outside of Interactive Mode.

Have ability to turn off warnings - maybe by type of warning.

• Possibly look at eslint / linting libraries for API inspiration (config verbiage).
• Also look at BlueBird config. http://bluebirdjs.com/docs/api/promise.config.html

Cypress.Warnings.defaults({
  warnOnMixedAsyncUsage: false,
  warnOnUncaughtExceptions: false,
})

Warning event (aka. like URL, XHR events...)

For uncaught exceptions - we do some complex things to associated the uncaught exception with a test / command - we should just create an uncaught exception event and highlight that also in red.

Side discussion As a breaking change - should switch logic to not fail by default on uncaught:exception - make them turn on failure for uncaught exception

Maybe add a code frame for the error events - honestly we should just design the UI of the warnings to be exactly like the 'failed' errors - with a 'docsUrl', 'code frame', 'stack trace', etc.

Dashboard

We could pin warnings to Dashboard results on the specs, "warnings tab?" - code frames - however Dashboard wants to design.

Could be multiple warnings per test - would require API changes.

[jennifer-shehane]

Another application for displaying a 'warning' instead of throwing. A person could screenshot an element that is 0 width and/or 0 height as described here: https://github.com/cypress-io/cypress/issues/5149

I wouldn't necessarily fail the test in this case but would want to show a warning that there will be no screenshot taken.

[cmcnicholas]

The ability to warn and have these not fail a test run but be visible on cypress dashboard would be fantastic, we have several use cases really only specific to our application where we would like to stub tests in preparation for QA in situations where the amount of tests to write is a huge effort.

[YoungElPaso]

The ability to warn and have these not fail a test run but be visible on cypress dashboard would be fantastic, we have several use cases really only specific to our application where we would like to stub tests in preparation for QA in situations where the amount of tests to write is a huge effort.

This is exactly my use case as well. I have a complex mechanism of filtering 'dev' tests vs 'QA' tests so that no test I've written fails against the environment it is filtered for but this is not great for non-developers.

I want to be able to write tests useful for our QA process that aren't yet satisfied but should be. I'll grant, this is a workflow/organization problem, but being able to illustrate to stakeholders that we can write tests that will eventually all pass is a big selling point for using testing like Cypress. Conversely having those tests just 'fail' right away (cause the code isn't yet developed) doesn't enamor the non-technical people to the process. They just see all the red. It's also bad for a CI pipeline situation (although my filtering mechanism is well suited to that).

Anyway, a 'warning' would be useful for some situations although I recognize the potential for abuse (everything becomes a warning).

[Cleudi]

I also see the need for cy.warn. We maintain an extended cypress custom command library which is used by various applications. No one will ever notice eg. deprecation notes in the console log. In contrast, having (orange) warnings in the command log will be noticed for sure (as long as the functionality is not abused).

Maybe useful as well (or instead of cy.warn). Allow passing a custom color to cy.log - so there can be visual distinction between different use cases (eg. deprecation vs. anti-pattern vs. ... ).

[aleitch1]

Yeah, I could REALLY use a case for passing warnings or annotations to students about the tests they're trying to pass! All green or gray means they just don't read the messages.