Cypress: Provide a way for custom commands to retry on failed should()

[alexkrolick]

Current behavior:

cy.get('#me').should('not.exist') // retries chain
cy.customGetById('me').should('not.exist') // does not retry

https://github.com/cypress-io/cypress/issues/1210#issuecomment-359075731

Desired behavior:

cy.customGetById('me').should('not.exist') // retries custom get if "should" fails

Versions

Cypress 3.1.4

Related

• https://github.com/cypress-io/cypress/issues/2670
• https://github.com/kentcdodds/cypress-testing-library/issues/30
• https://github.com/kentcdodds/cypress-testing-library/issues/23

[Bkucera]

@alexkrolick can you provide the code for customGetById? I'm not able reproduce this issue

[alexkrolick]

See https://github.com/kentcdodds/cypress-testing-library/issues/30 - there are a number of custom commands

[Bkucera]

@alexkrolick I see, you're asking for plugin authors to have access to the retry mechanism, custom commands using cy.get under the hood will still work fine

[alexkrolick]

Yes exactly

[lukeapage]

This workaround works:

cy.getByTestId = (id) => cy.get(`[data-test-id="${id}"]`);

[alexkrolick]

The custom queries we are trying to support aren't wrappers around jQuery attribute selectors; some of them use multiple DOM traversals to match labels to inputs, for example. cy.get isn't a workaround unfortunately. See linked issues.

[Lakitna]

I've been looking at this for a bit now and digging into the source code gave me the command verifyUpcomingAssertions. That's how you can wait for a should to be resolved.

However, this command is not documented and can, therefore, be a bit iffy to work with at the moment.

The way I was able to figure out how to use it was again by looking to the source of the default commands found here: https://github.com/cypress-io/cypress/tree/develop/packages/driver/src/cy/commands

When you get it working it works great, but expect to spend a lot of time tinkering. It helps to use the same basic structure that's used in default commands.

I'm at a point where I got it working perfectly except for the log.end() of the first upcoming assertion, the steps remains 'pending' (blue with spinner) when it should be 'passed' (green).

edit: I got it working now! The basic format should look something like the following. I left out logging, options etc for clarity.

Cypress.Commands.add('aThing', (element, options={}) => {
    /**
     * This function is recursively called untill the timeout passes or the upcomming
     * assertion passes. Keep this function as fast as possible.
     *
     * @return {Promise}
     */
    function resolveAThing() {
        // Resolve a thing
        const aThing = $(element).attr('aThing');

        // Test the upcomming assertion where aThing is the value used to assert.
        return cy.verifyUpcomingAssertions(aThing, options, {
            // When the upcoming assertion failes first onFail is called
            // onFail: () => {},
            // When onFail resolves onRetry is called
            onRetry: resolveAThing,
        });
    }

    return resolveAThing();
});

[Bkucera]

You can make an arbitrary function retry and pass along its return value using a small hack that combines should and invoke

Here's an example of a custom command that makes sure all h2's are in alphabetical order, and it retries:

cy.get('h2')
.should(($els) => $els.fn = () => {
  return $els.toArray().map(el => el.innerText)
})
.invoke('fn')
.should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

or

better yet, add a new command that does this called try:

Cypress.Commands.addAll({
  prevSubject: 'optional',
},
{
  try: (subject, fn) => {
    return cy.wrap({ try: () => fn(subject) }, { log: false })
    .invoke('try')

  },
})

and use it like:

cy.get('h2').try(($els) => {
  return $els.toArray().map(el => el.innerText)
})
.should(headers => {
  const sortedHeaders = headers.concat().sort()

  expect(headers).deep.eq(sortedHeaders)
})

without a cypress parent command:

cy.try(() => {
  const h2s = cy.state('document').querySelectorAll('h2')
  return Array.from(h2s).map(el => el.innerText)
}).should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

[Lakitna]

It's a nice idea to add a command that is basically then + retry. I've run into situation where it would be usefull. Though I don't think try is the best name. It implies error handling and that you need a catch to go with it.

Couple of ideas:

Name it retry

cy.get('foo')
    .retry((elem) => {
        // ...
    });

Name it thenTry

cy.get('foo')
    .thenTry((elem) => {
        // ...
    });

Overwrite then
When you overwrite then you can add an option to enable retrying and make it false by default, something like below. Though try might be one of the few commands that can't be overwritten, I'm not sure.

cy.get('foo')
    .then((elem) => {
        // ...
    }, {retry: true});

[bohendo]

What's the status on this? I've got a two-part cy.get that I'd like to retry until it resolves to a non-zero value. My first attempt:

const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    // cy.wait(2000)
    getBalance().should('not.equal', '0.00')
  })
})

The above runs getBalance() once and then waits on the return value to pass the assertion. It fails via timeout even though my UI eventually has a non-zero balance because getBalance() isn't rerun when it's return value fails the assertion.

I can get the above test to pass by uncommenting the cy.wait(2000) but I'd rather not introduce any potential race conditions.

My second attempt, inspired by the conversation between @Lakitna and @Bkucera above.

Cypress.Commands.addAll({ prevSubject: 'optional' }, {
  retry: (subject, fn) => {
    return cy.wrap({ retry: () => fn(subject) }, { log: false }).invoke('retry')
  },
})

describe('Test', () => {
  cy.retry(getBalance).should('not.equal', '0.00')
})

The above errors out with CypressError: Cypress detected that you returned a promise from a command while also invoking one or more cy commands in that promise. and I've tinkered with it for a while but can't figure out how to avoid this error while still testing what I want to test..

[Lakitna]

There is no official stance on this yet. But that won't stop us!

I've made a quick implementation of this idea using the retry name and tried to document it a bit so you can alter it. I only tested if it retries, I have no idea what its limitations are.

const _ = Cypress._;
const $ = Cypress.$;

/**
 * Basically `then`, but will retry if any upcoming assertion fails
 * @param {JQuery} subject
 * @param {function} fn
 * @param {boolean} [options.log=true]
 * Log to Cypress bar
 *
 * @return {*}
 */
Cypress.Commands.add('retry', {prevSubject: 'optional'}, (subject, fn, options={}) => {
    _.defaults(options, {
        log: true,
    });

    // Setup logging
    const consoleProps = {
        'Applied to': $(subject),
    };
    if (options.log) {
        options._log = Cypress.log({
            $el: $(subject),
            name: 'retry',
            message: fn.name,
            consoleProps: () => {
                return consoleProps;
            },
        });
    }

    /**
     * This function is recursively called untill timeout or the upcomming
     * assertion passes. Keep this function as fast as possible.
     *
     * @return {Promise}
     */
    function resolve() {
        const result = fn(subject);

        // Update logging
        if (options.log) {
            consoleProps.Yielded = result;
        }

        // Test the upcomming assertion where result is the value used to assert.
        return cy.verifyUpcomingAssertions(result, options, {
            // When the upcoming assertion failes first onFail is called
            // onFail: () => {},
            // When onFail resolves onRetry is called
            onRetry: resolve,
        });
    }

    return resolve();
});

The full extend of my tests:

it('retries', function() {
    let c = 0;
    cy.retry(() => ++c)
        .should('equal', 5);
});

[bohendo]

Thank you @Lakitna 🙏 An(other) example using cy.verifyUpcomingAssertions was just what I needed. I think cy.retry is a reserved command though (at least in v3.2.0) because if I try to add it, unrelated parts of my code start blowing up.

But I tinkered a bit, got it working & was able to remove the race condition (ie a cy.wait) I was relying on earlier. Hope the following example is useful for others who are in a similar situation:

// cy.resolve(fn).should(blah) will re-run the promise-returning fn until
// the value it resolves to passes the assertion
Cypress.Commands.add('resolve', { prevSubject: 'optional' }, (subject, fn, opts={}) => {
  const resolve = () => {
    fn(subject).then(res => cy.verifyUpcomingAssertions(res, opts, { onRetry: resolve }))
  }
  return resolve();
});

// an example function that returns a Cypress.Promise
const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    cy.resolve(getBalance).should('not.contain', '0.00')
  })
})

[Lakitna]

You are correct, retry is an existing, undocumented cy command (see screenshot).

I would personally fall back to thentry until I made a way to overwrite then with a retry option.

[Lakitna]

Behold, the birth of the module cypress-commands!

I've made the extension on the then command and published it in a repo (and on npm) where I will add more commands in the future. I have a few commands laying on the shelf I could add.

For more details see the repo at https://github.com/Lakitna/cypress-commands

[jennifer-shehane]

Hey @Lakitna, can you add your commands library to our docs on the plugins page?

[Manuel-Suarez-Abascal]

Hi, I was using the @Lakitna example to implement my own retry mechanism to assert three different modals on our app. I'm using the verifyUpcomingAssertion method, but it seems it retries right away & I can't make it to wait until the assertions fail or pass. This is my code:

Cypress.Commands.add(
  'assertModal',
  ({ selector = '[data-test=popup-container]', headerTitle, shouldClose = true } = {}) => {
    const assertADSModal = () => {
      cy.get('[data-test=acmodal]')
      // a bunch of assertions here
      .........
    };

    const assertDefaultModal = () => {
      cy.get(selector)
       // a bunch of assertions here
      .........
    };

    const assertLegacyModal = () => {
      cy.get('.ui-dialog')
      // a bunch of assertions here
      ......... 
    };

    const triggerModalAssertions = () => {
      cy.get('body').then($body => {
        const isAnyDefaultModalVisible = $body.find(selector).length > 0;
        const isAnyAdsModalVisible = $body.find('[data-test=acmodal]').length > 0;
        const isAnyLegacyModalVisible = $body.find('.ui-dialog').length > 0;

        if (isAnyAdsModalVisible) {
          assertADSModal();
        }
        if (isAnyDefaultModalVisible) {
          assertDefaultModal();
        }
        if (isAnyLegacyModalVisible || selector === '.ui-dialog') {
          assertLegacyModal();
        }

        return cy.verifyUpcomingAssertions(
          $body,
          {},
          {
            onRetry: triggerModalAssertions,
          }
        );
      });
    };

    return triggerModalAssertions();
  }
);

How can I make the verifyUpcomingAssertions to wait until the assertion pass or fail until retries? I have tried with setTimeout & specify the amount of time, but it didn't work.

[pathurs]

For this problem, I ended up creating my own implementation of "retry-ability" (in TypeScript):

interface RetryOptions<T> extends Cypress.Timeoutable {
    interval: number;
    default?: T;
    throw: boolean;
}

export function retry<T>(checker: () => T, confirmer: (result: T) => boolean, options?: Partial<RetryOptions<T>>): Cypress.Chainable<T>;
export function retry<T>(checker: () => T, confirmer: (result: T) => boolean, originalOptions?: Partial<RetryOptions<T>>): Cypress.Chainable<T> {
    const options: RetryOptions<T> = {
        ...{ timeout: Cypress.config('defaultCommandTimeout'), interval: 200, throw: true },
        ...originalOptions
    };

    return cy.wrap(
        new Promise<T>((resolve, reject) => {
            const startTime = Date.now();
            const result = checker();

            if (confirmer(result)) {
                resolve(result);

                return;
            }

            const intervalId = setInterval(() => {
                const currentTime = Date.now();
                const endTime = startTime + options.timeout;

                if (currentTime >= endTime) {
                    if (options.throw) {
                        reject(new Error(`Timed out while retrying after ${options.timeout}ms`));
                    } else if ('default' in options) {
                        resolve(options.default);
                    } else {
                        resolve();
                    }

                    clearInterval(intervalId);

                    return;
                }

                const result = checker();

                if (confirmer(result)) {
                    resolve(result);

                    clearInterval(intervalId);

                    return;
                }
            }, options.interval);
        }),
        { log: false, timeout: options.timeout + options.interval }
    );
}

Usage Example:

retry(
    () => {
        const selector = 'your selector';

        return Cypress.$(selector);
    },
    elements => elements.length > 0,
    { throw: false, default: Cypress.$() }
);