Cypress: type / submit in Cypress not triggering HTML validation on elements

Created on 9 Mar 2020 · 7Comments · Source: cypress-io/cypress

Current behavior:

Not finding invalid elements, whether by:

cy.get('<selector>:invalid')

cy.get('[data-name="name"]:invalid').should('have.length', 1);

validity property

cy.get('[data-name="name"]').then(($input) => {
  return expect($input[0].validity.tooShort).to.be.true;
});

Desired behavior:

Correctly indicate that the element is indeed invalid (so that assertions can be based on this).

Test code to reproduce

These tests both fail in showing any problems with invalidity, despite the HTML element being queried having minlength attribute of 3 and our test typing only a single letter ("a"):

it('should catch invalid element by pseudo-selector', () => {
    cy.visit('http://localhost:8000/');
    const tooShortOfAName = 'a';
    cy.get('[data-name="name"]').type(tooShortOfAName);

    // Fails with or without a form click
    // cy.get('[data-name="action2"]').click();

    // Fails here
    cy.get('[data-name="name"]:invalid').should('have.length', 1);
  })

  it('should catch invalid element by JS API', function () {
    cy.visit('http://localhost:8000/');
    const tooShortOfAName = 'a';
    cy.get('[data-name="name"]').type(tooShortOfAName);

    // Fails with or without a form click
    // cy.get('[data-name="action2"]').click();

    // Fails here
    cy.get('[data-name="name"]').then(($input) => {
      return expect($input[0].validity.tooShort).to.be.true;
    });
  });

Here's the minimal repo to reproduce: (note that I've added a cypress install as I think you may want to do in your boilerplate also since your docs refer to installing cypress locally)

https://github.com/brettz9/cypress-test-tiny/tree/not-getting-invalidity

(i.e., on the not-getting-invalidity branch)

Outdated info

I've also tacked on an environmental variable test (same as #2605 ?)

it('should obtain environmental variables', function () {
  cy.log(Cypress.env('secret'))
  expect(Cypress.env('secret')).to.equal('Be good');
});

...which is not consistently getting the secret despite it being set in plugins/index.js

module.exports = (on, config) => {
  config.env = config.env || {};
  config.env.secret = 'Be good';
  return config;
);

...consistent with the approach described for dynamically setting the environmental variables at https://docs.cypress.io/api/plugins/configuration-api.html#Switch-between-multiple-configuration-files or https://docs.cypress.io/api/cypress-api/env.html#From-a-plugin.

Versions

4.1.0, Mac OSX 10.15.2, Chrome 80.0.3987.132 (Official Build) (64-bit)

pkdriver needs investigating cy.type ⌨️ bug

Source

brettz9

👍2

Most helpful comment

So the problem here is not that it's not finding elements with a psuedo-selector of invalid, it's that the actions being performed through Cypress are not triggering the validation checks on the form for some reason.

You can use jQuery to query for input:invalid on the application under test after Cypress performs the type and even the click to submit the form and see that it finds no invalid elements.

Reproducible Example

spec.js

it('does not find invalid input', () => {
  cy.visit('index.html')
  cy.get('input').type('a')
  // cy.contains('Update').click()
  cy.get('input:invalid').should('have.length', 1)
})

index.html

<!DOCTYPE html>
<body>
    <input minlength="3">
    <button type="submit">Update</button>
</body>
</html>

Screen Shot 2020-03-10 at 2 01 45 PM

Events triggered during type.

Screen Shot 2020-03-10 at 1 51 42 PM

If you type manually into the form while the test is running, it will all of a sudden pass.

Screen Shot 2020-03-10 at 2 04 08 PM

jennifer-shehane on 10 Mar 2020

👍4

All 7 comments

I might add my validity checks are akin to that added in https://github.com/cypress-io/cypress-documentation/pull/1919 (at https://docs.cypress.io/faq/questions/using-cypress-faq.html#Can-I-check-that-a-form%E2%80%99s-HTML-form-validation-is-shown-when-an-input-is-invalid )

brettz9 on 9 Mar 2020

My apologies re: the environmental issue; it seems that my issue there may have been not returning the config object. However, the main problem with validity checking remains.

brettz9 on 9 Mar 2020

You can use jQuery to query for input:invalid on the application under test after Cypress performs the type and even the click to submit the form and see that it finds no invalid elements.

Reproducible Example

spec.js

it('does not find invalid input', () => {
  cy.visit('index.html')
  cy.get('input').type('a')
  // cy.contains('Update').click()
  cy.get('input:invalid').should('have.length', 1)
})

index.html

<!DOCTYPE html>
<body>
    <input minlength="3">
    <button type="submit">Update</button>
</body>
</html>

Screen Shot 2020-03-10 at 2 01 45 PM

Events triggered during type.

Screen Shot 2020-03-10 at 1 51 42 PM

If you type manually into the form while the test is running, it will all of a sudden pass.

Screen Shot 2020-03-10 at 2 04 08 PM

jennifer-shehane on 10 Mar 2020

👍4

In case it helps: I have 2 forms, one with a text and password input, and one with a text, password, and numerical input. In both cases only the text input exhibits this bug. The password and numerical inputs behave as expected - form validation popups are appearing. This remains true if I reorder the inputs on the page and change the order I type into them in Cypress.
Node 12.13.0
Cypress 4.8.0
Chrome 83.0.4103.97
Windows 10 Home v1903 build 18362.900

shamgang on 14 Jun 2020

Just to add my own experience: getting this bug on a minlength validation on a password input:
when using cy.get().type() to type text that should be too short, I am able to submit the form both manually and through Cypress.

In essence, it looks as if Cypress is not sending the necessary key events for the form to perform validation.

zedrdave on 10 Jul 2020

I came across this issue as well. The minlength attribute will be ignored. Validation of the pattern
attribute pattern=".{6}" however will be triggered. Unfortunately the resulting message ("Please match the requested format") is very vague and rather confusing for the user. Further adding setCustomValidity manually wouldn't be my favourite option.