Cypress-cucumber-preprocessor: @focus tag doesn't work when @ignore tag is used as an env var

[onderceylan]

Hi there! For starters, thanks for your great work on cypress-cucumber-preprocessor, it makes our lives easier :)

Current behavior

When I add the following env var to make a good use of @ignore tag as it's documented on README, the @focus tag doesn't work anymore due to the implementation of hasEnvTags here: https://github.com/TheBrainFamily/cypress-cucumber-preprocessor/blob/8a5f032e411624f41f3a3cbd1d9604dd2ffc65f4/lib/createTestsFromFeature.js#L24

When @ignore tag is set in cypress.json config file as an env var, cypress-cucumber-preprocessor executes all the tests even there's a @focus tag in place.

Desired behavior

I'm not sure why the use of env tags are coupled to @focus tag functionality but I'd very much like to combine the both tags in my work environment. Using both @ignore and @focus tag at the same time should not be an issue.

I only temporarily use @focus tag in my development workflow with cypress open and there might be flaky tests ignored temporarily with @ignore tag at the same time.

Test code to reproduce

cypress.json

{
  "projectId": "my-app",
  "baseUrl": "http://localhost:4200",
  "env": {
    "TAGS": "not @ignore"
  },
  "fixturesFolder": "cypress/fixtures",
  "integrationFolder": "cypress/features",
  "pluginsFile": "cypress/plugins/index.js",
  "supportFile": "cypress/support/index.js",
  "testFiles": "**/*.{feature,features}"
}

SmartTagging.feature

Feature: Smart Tagging

  As a cucumber cypress plugin which handles Tags
  I want to allow people to select tests to run if focused
  So they can work more efficiently and have a shorter feedback loop

  Scenario: This scenario should not run if @focus is on another scenario
    Then this unfocused scenario should not run

  @focus
  # This doesn't have any effect when 'not @ignore' is set as an env var
  Scenario: This scenario is focused and should run
    Then this focused scenario should run

  @ignore
  # This should be ignored all the time
  Scenario: This scenario should also not run
    Then this unfocused scenario should not run

  # This should be executed always
  Scenario: This scenario is also focused and also should run
    Then this focused scenario should run

Versions

• Cypress version: 4.5.0
• Preprocessor version: 2.3.1
• Node version: 12.16.1

[lgandecki]

Hello.
Sorry about that. That's the currently programmed behavior indeed I'm afraid, I'd be ok with changing how this behaves - if someone wants to create a PR :)

It's a bit tricky, we could ignore the env tags if focus is found, but that could lead to problems if for example the focus was left and pushed to CI.
We could maybe try to do that only when running with cypress open (as focus doesn't make much sense with cypress run anyway)

ideas?

[cellog]

@focus should be treated separately from other tags. It's up to devs to notice that their tests have a @focus tag in the PR review phase. In other words, @focus should always override envVars. @focus can make sense in a cypress run env if you're trying to determine whether a test fails in relation to those around it (relatively common if you don't clean up/cy.wait() for requests) in a CLI env, so I would try to be clever. Trust us to know when we use @focus.

[cellog]

Alternative: in the docs, show us a way to include @focus in the tags. I tried adding it to env vars, and it ended up skipping all tests when there were no @focus tags. Can't see an easy way to say "only run these tags, skip these" except to restart cypress with a @focus-specific cypress.json or special TAGS=@focus (which seems rather silly)

[sandra-ouadghiri]

Still true, still making our life difficult X0! @lgandecki I agree with @cellog on the expectation.
Here's a proposition if this helps providing a conversation or a PR :).

As a test-writer:

• I take responsability for not pushing an @focus (or expect my code reviewer to check that). The plugin should not worry about that.
• I expect @focus to always win
  --> if @focus is found, only thoses tests are executed, everything else is skipped (whatever envTags says)
  --> if a scenario has @focus & is also skipped per envTags rules > @focus wins (once again: focus wins whatever envTags says)
  --> envTags are only applied if no @focus are found

Let me know if I can help.

Versions
Cypress version: 6.5.0
Cucumber Preprocessor version: 4.0.0

[lgandecki]

fair enough. :)
Would one of you guys wanted to try making a PR for this change?
I might do it but can't promise any timelines..

[sandra-ouadghiri]

@lgandecki I might do it but you may regret it XD. I can try tough.

[lgandecki]

haha, well, give it a try and we will see :)

[sandra-ouadghiri]

@lgandecki > I created a branch but was not able to push. So far, only scenarios showing the problem to solve, to see if we agree on those already.

ERROR: Permission to TheBrainFamily/cypress-cucumber-preprocessor.git denied to sandra-ouadghiri.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[badeball]

@sandra-ouadghiri, you should fork the repository, push a new branch to your fork and then create a PR (Github will suggest it once you've pushed). No permissions are required that way.

[san-ouadghiri]

@lgandecki Can we first agree on tests that would state that we succeeded or not in this development?
I'm not familiar to this repo code nor test manner. Any feedback is welcome. When aligned on that, I'll have a look at if I can make this happen or not (^-^ ).
-> https://github.com/TheBrainFamily/cypress-cucumber-preprocessor/pull/525

[sandra-ouadghiri]

@lgandecki if you can review my tests when you have time (draft #525). If ok, I'll try implementing. @cellog @badeball you're also welcome obviously.