Cypress: Cypress result shows skipped tests as pending

[Saibamen]

Cypress results (with run command) shows skipped tests as pending

Current behavior:

(Results)
  ┌──────────────────────────────────────────────────┐
  │ Tests:        4                                  │
  │ Passing:      3                                  │
  │ Failing:      0                                  │
  │ Pending:      1                                  │
  │ Skipped:      0                                  │
  │ Screenshots:  0                                  │
  │ Video:        true                               │
  │ Duration:     29 seconds                         │
  │ Spec Ran:     xxx\xxx\Page2b.spec.js │
  └──────────────────────────────────────────────────┘

Result file: (see skipped="1" in Mocha Tests)

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="29.914" tests="4" failures="0" skipped="1">
  <testsuite name="Root Suite" timestamp="2019-01-08T18:14:13" tests="0" failures="0" time="0">
  </testsuite>
  <testsuite name="Page2b" timestamp="2019-01-08T18:14:13" tests="1" failures="0" time="8.691">
    <testcase name="Page2b should see entered data from previous page" time="8.691" classname="should see entered data from previous page">
    </testcase>
  </testsuite>
  <testsuite name="Valid data" timestamp="2019-01-08T18:14:22" tests="2" failures="0" time="21.223">
    <testcase name="Page2b Valid data goes to next page after entering valid data" time="7.554" classname="goes to next page after entering valid data">
    </testcase>
    <testcase name="Page2b Valid data should see previously entered data when I go back from next page" time="13.669" classname="should see previously entered data when I go back from next page">
    </testcase>
  </testsuite>
  <testsuite name="Invalid data" timestamp="2019-01-08T18:14:43" tests="1" failures="0" time="0">
  </testsuite>
</testsuites>

Desired behavior:

Cypress results should show 0 pending and 1 skipped tests

Steps to reproduce: (app code and test code)

cypress.json:

    "reporter": "junit",
    "reporterOptions": {
        "mochaFile": "results/output.[hash].xml"
    },

test:

describe('Page2b', function () {
    beforeEach(() => {
        cy.fixture('correctAnswers.json').as('answers')
        cy.login()
    })

    context('Valid data', function() {
        it('goes to next page after entering valid data', function () {
            // terefere
        })

        it('should see previously entered data when I go back from next page', function () {
            // terefere
        })
    })

    context('Invalid data', function () {
        it.skip('shows required validation texts after clicking on Next link on empty form', function () {
            // terefere
        })
    })

    it('should see entered data from previous page', function () {
        // terefere
    })
})

Note

This same issue when running:

context.skip('Invalid data', function () {
        it('shows required validation texts after clicking on Next link on empty form', function () {
            // terefere
        })
    })

Versions

Cypress 3.1.4, Chrome 71, Windows 10

[chrisbreiding]

I can see how this is confusing. It's not necessarily a bug, but a terminology issue. Cypress considers a pending test to be any of the following:

• it.skip(), context.skip, etc
• it('some test') (no callback)

pending tests are tests you don't plan to run and explicitly mark not to run.

A skipped test is one that you plan to run but is skipped because, for example, a beforeEach hook failed.

beforeEach(() => {
  throw new Error('whoops')
})

// these will all end up 'skipped' because while you plan to run them, the runner skips them because the beforeEach threw an error
it('a test', () => {})
it('another test', () => {})

I would agree that this is confusing because it.skip creates a pending test instead of a skipped test.

It looks like the reporter you're using reports pending tests as "skipped", which seems intuitively correct but is not correct as far as Cypress is concerned because "skipped" has a different meaning.

All that to say that Cypress isn't really reporting the wrong numbers. It's reporting them correctly as it understands them.

That said, it's certainly up for debate whether the current terminology is sufficient and whether it should change to be more intuitive. I imagine it would be a lot of work and perhaps quite difficult to do so, however.

Thoughts, @brian-mann?

[Saibamen]

@chrisbreiding Incorrect.
throw new Error('something') makes test marked as failed

I want to skip test and have all other tests with "green light" - have information about how many tests from single spec file was skipped

[chrisbreiding]

Right, my bad. The first test will be failed, the subsequent ones will be skipped.

I want to skip test and have all other tests with "green light" - have information about how many tests from single spec file was skipped

Those are pending tests as far as Cypress is concerned, but your reporter is calling them skipped.

[morficus]

+1 for this being confusing.

To the untrained eye, the term "pending" implies that the test is still running or has not been ran yet (as in it is queued to run in the future). Meanwhile the term "skipped" would imply that it was not ran and will not be ran. This is especially confusing because the mocha syntax is it.skip.

My initial thoughts would be to:

option 1
rename "skipped" to "excluded" and "pending" to "skipped"

option 2
consider the things currently called "skipped" to be failed tests ... since the reason they did not run would be due to a failing hook. then rename "pending" to "skipped" and keep "pending" for things with no callbacks (or get ride of it all together)

[dsesami]

Any update on this? I really do prefer the option 1 provided by @morficus. It makes _much_ more sense. If it's just string-swapping it's literally two lines.

[dialex]

I also vote for option 1. Please, make this happen 🙏

[brian-mann]

I've written an extensive answer in our internal chat detailing the history as to how we arrived at the nomenclature we use. It's long, and complex, and we're in an awkward position because of what mocha chose to do - it's not something we can fix on our end unless we deviate from mocha completely - which would end up creating its own big can of worms - or introduce new unique test states (see below).

The problem is that we inherit all of the mocha reporters and we use the spec reporter by default. These mocha reporters are coupled to the definition of what a pending test is which mocha determines. Mocha decided to use the it.skip API design, but internally refers to them as pending tests in the reporter.

Deviating from this nomenclature would create a conflict from the default spec reporter, or other custom reporters you may use. You'd see the mocha stats summary indicating numbers that don't match up to our own summary at the end.

Because we don't control mocha reporters - there's not really much we can do about this. We'd either have to deviate from mocha's own API's - such as rewriting it.skip, or we'd have to rewrite and control every single mocha reporter and 3rd party mocha reporter (which is not going to happen).

In addition, Mocha has no concept of accurately depicting truly "skipped" tests. That is the one thing that we ourselves do - when a hook fails we skip the remaining tests in that suite. Sure - I agree we could probably rename this state to another word that would perhaps alleviate some confusion. But no matter what - we inherit Mocha's it.skip and pending state problems.

One idea we've batted around would be to expand the number of test states to help describe and more accurately model the actual runtime behavior of each test.

A more comprehensive list of test states that could be proposed:

• Passed (a test that ran to completion including all its applicable hooks)
• Failed (a test that failed in a hook or its own test body)
• Pending (a test that had no implementation - i.e. no callback function)
• Skipped (a test that was programmatically skipped via it.skip, or this.skip(), or our own programmatic API's)
• Bypassed / Excluded / Aborted / Not Ran (a test that was not run because a hook failed in the current suite thus "bypassing/excluding/aborting" the remaining tests in the suite)

This would enable us to separate out the reason a test didn't run based on the cause.

Unfortunately, the upgrade path here would be pretty harsh, as the current definitions look like this:

• Passed (a test that ran to completion including all its applicable hooks)
• Failed (a test that failed in a hook or its own test body)
• Pending (a test that had no implementation - i.e. no callback function, a test that had it.skip or this.skip())
• Skipped (a test that was not run because a hook failed in the current suite thus "bypassing/excluding/aborting" the remaining tests in the suite)

[dsesami]

Unfortunately, the upgrade path here would be pretty harsh, as the current definitions look like this:

I think most users (me included) would have no issue dealing with the bumpiness of an upgrade in that sense as long as it was well-communicated. Personally, your proposed idea works fine for me. I do see pretty good feedback as far as GitHub thumbs up counts go :smile:

[morficus]

I like the idea of expanding the number of test states available.

[krokofant]

This looks weird in the UI as well as referenced in the #5366 issue which I closed in favor of this issue.

This is the icon for a it.skip("test" ()=>{}) which I complained did not look as a skip icon.

[julian-sf]

@brian-mann Can you more thoroughly explain what you mean by

the upgrade path here would be pretty harsh

Do you mean internally to Cypress development or for users (IE a possible major version upgrade)?

Would it be possible to use a flag to opt into this new test state style so you can push the update to 3.x? You could force the new path in 4.0, which would eliminate the problem of incompatibility from a user perspective. Granted that would add more work on whoever makes the update.

[erwinbooij]

I actually had a bigger problem because of this issue.
The last test in our test run was set to skip and this causes the test run to never end. After 6 hours I had to cancel it myself when I found out it was still running (in the release pipeline)
I guess because it is seen as pending.

[pragapraga]

Is this Issue related to https://github.com/mochajs/mocha/issues/1815? @Saibamen @brian-mann

[Saibamen]

IDK

[akuznetsov-lineate-qa]

When I using

it('test',()=>{
    this.skip();
})

Cypress says:

TypeError: Cannot read property 'skip' of undefined

So, can we really mark the test as Skipped inside?

Cypress version 3.8.3

[dialex]

@akuznetsov-lineate-qa I stumbled upon that too. It took me a while but found how to solve it:

it('test',function(){
    this.skip();
})

The problem is the () => notation, which makes this yield undefined. If you use function() then it all works. Do you confirm?

[akuznetsov-lineate-qa]

@dialex yeah, thanks, but it works partly.

It really works if your test run contains only this one test or this skipped test is last in your test run.
But when your test run contains a lot of tests and skipped test is first (for an example) - this test will be marked as "Skipped"

and then (after the second test is finished) marked as "Failed".

In this particular case Cypress says:

Error: Cypress command timeout of '15000ms' exceeded.

Can you confirm that this behavior is reproduced for you?

[dialex]

No, in my case, it doesn't matter the order of my test. I have several tests being skipped with this.skip(), regardless of their order, and these skipped tests do not affect the execution of other tests. There must be something else in your _setup_ or _flow_ causing te problem 🤷‍♂️

[estefafdez]

Any idea how can I force a test to be marked as skipped then on the suite results? thanks!

[internSW]

I have issue, when record pending are not ending, but don't have error. Help me :((((