Cypress: Clear session storage workaround is not consistently functioning

[salty-blue-mango]

Current behavior:

While logging in targeting the User Interface I have to clear session storage because we store route history in there. The first time the test is run and the initial test runner browser is opened, but the session storage is not cleared. Refreshing the test without closing the browser lets the test succeed the second time and any time after that as long as you don't close the test runner browser window.

I am using the clear session storage workaround by using
cy.visit(activeEndpoint, { onBeforeLoad: (win) => { win.sessionStorage.clear(); } });

Desired behavior:

The expected behavior is that session storage is cleared using win.sessionStorage.clear() on the initial firing of my login test allowing me to visit the login page and be able to log. It's bothersome to have to run the test a few times to get it to succeed.

Steps to reproduce:

describe('Logs In Using the UI', () => {
    beforeEach('Clear session storage', () => {
        cy.visit(url, {
            onBeforeLoad: (win) => {
                win.sessionStorage.clear();
            }
        });
    });
   it('Logs in to url using the UI', () => {
       cy.get(loginSelectors.usernameInput).type('username');
       cy.get(loginSelectors.passwordInput).type('password');
       cy.get(loginSelectors.loginButton).click();
       waitOnLoginXhr();
   });
});

[shcallaway]

I was under the impression that Cypress clears local storage and cookies (ie session state) before each test. However, I have noticed that since switching to 3.0 we occasionally see session state "leak" between tests, which causes flakiness.

I can provide a link to our (private) dashboard if it helps.

[egucciar]

@shcallaway i am suddenly seeing the same thing right now after upgrading to 3.1.0. Thinking about rolling back to 3.0.1, or trying this in the support file (it works):

afterEach(() => {
  cy.window().then(window => window.localStorage.clear());
})

Any idea what cypress team's official stance on this is? I think this is a major error due to most of my test cases relying on the assumption that state isnt leaking and us using localStorage quite extensively.

And this code just icky, but it seems bad for cypress to just stop supporting a feature that it claims it supported without this being a breaking change.

EDIT: Spoke to soon, it seemed to work in open but fails to work in run. Unfortunately my hand is forced to downgrade.

EDIT2: Downgrading didnt work. It seems to be an issue on my MacOS. This issue has not occurred in CI (Linux based)

[brian-mann]

It's likely an entirely different issue altogether (nothing has changed regarding how we handle this) but there are problems with the current implementation that we are aware of and will fix. This document does a great job explaining how state can leak and what we must do to prevent that from happening.

Basically if you're not using a cy.visit() before each test - it will happen, and there are situations in which requests callbacks can still be called due to timing, which is why it must be solved in a different way than how we do it today.

https://github.com/cypress-io/cypress/issues/686

To be clear - we're aware of how to solve this, and this rolls into a bigger issue we're working on now - which is test retries. To make test retries work, we ensure that tests can retry in an idempotent fashion, which means we must enforce that there is always a fresh, clean test state. We are no longer considering API's that allow users to opt out of these guarantees.

[egucciar]

Thanks for the update. I can assure you we are visiting in the before each. I am not observing the issue on Linux (circleci) only on my local macos, both on open and run.

Since I've only ever really leverage CI to run the full suite, this if the first time I've encountered the issue since these tests were created multiple months ago

[brian-mann]

The workarounds for this are pretty brutal and aren't great but you can do a visit after each test to a blank page on your app, or a visit BEFORE the real visit, which should do absolutely nothing but send empty <html> contents.

This would force all network requests to be flushed from the original app and ensure that its completely torn down - then you clear the localstorage, and then do the real visit (or let the test move on). I prefer having this at the beginning because if its at the end of the test, then any test you work on will always end up on a blank screen - although you could conditionally do that only in CI.

[egucciar]

Well from my perspective , since this is currently only happening to me as of today on my local macos environment and had never occured in CI, i can manage to avoid all nasty workarounds assuming that my observations are correct in that this is only an issue for me in MacOS. The same exact tests and code pass on circle wonderfully but fail 100% of the time on local due to leaking state in localStorage. And normally when deving we work with .only so it's not affecting normal deving either. For the most part as long as our team relies on circle or running one test at a time on the local, we can avoid nasty work arounds in the meantime. I hope this extra data point assists in solving the issue since it's very strange to me how it's been working so well in Circle this whole time and still works despite failing on a local run. Thanks for the suggestions which I may need to use if just relying on CI to run all tests in a suite ceases to be enough for me right now.

[brian-mann]

If you're seeing consistent local failures then it is worth exploring and putting together a reproducible repo. There is an inordinate amount of complexity involving way too many variables to speak to until they can be reduced down to the minimum set involved in reproducing the problem.

At that point we can assess if there's a change we need to make that would avoid the workarounds.

[egucciar]

Sounds good. I will include it on the lower priority side of things since so long as my tests are still passing in CI the priority of this problem for me is dramatically reduced. Thanks so much for Weighing in and I'll try to get a reproducer together when time permits.

[egucciar]

After making no changes to the app, this is beginning to happen on CircleCI as of now. We will need to create the reproducer however I have higher prio tasks. Clearing localStorage manually is not perfect prior to test case running since sometimes scripts get instantiated before we got the chance to clear. Please advise if we have a way to always get the window and clear out storage b4 a script begins to run

[egucciar]

Unfortunately my attempts to make this reproducible have failed though we clearly need the workarounds in our giant app test suite :(

edit: rerunning without the workarounds to see how it goes.

[egucciar]

@brian-mann OK creating a reproducible repo was actually really tough but i have one. I can share the link. The only downside is that it has to visit our Feature URL. Because of this, the entire source code of the App remains hidden from you guys. Unless this poses a problem, I'd be willing to share the Repository.

What I'm observing

function setBaseUrl() {
  Cypress.config('baseUrl', 'featureBranchURL');
}

describe('Local Storage Reproducer', () => {
  let value;
  beforeEach(() => {
    setBaseUrl();
    cy.request({
      method: 'POST',
      url: 'http://nlv4d.mocklab.io/json2', // a mock API setup to respond in 1234 MS
    })
    cy.visit('/', {
      onBeforeLoad: (win) => {
        value = win.localStorage.getItem('app'); // Our Apollo GraphQL Client storage
        console.log('localstorage', value);
      }
    });
    cy.shadowGet('upd-cta-bar', { timeout: 30000 }); // wait for the app to load
  })
  context('testing', () => {
    it('1', () => {
      expect(value).to.be.null; // Passes!
    })
    it('2', () => {
      expect(value).to.be.null; // Fails!
    })
  })
});

Findings:

Removing the cy.request or reducing the response time to be a few MS (as opposed to 1 whole second) Allows the test to PASS.

Trying to recreate the issue without Our application is hard. There must be a race condition between app and Cypress and GraphQL. :/

EX: The above code produces this failure:

Removing cy.request causes PASSING Test. We have no idea why this is happening this way. The mock API Is minimal to ensure we arent messing with anything. Just takes 1s long. I reproduced our cy.request in mock API lab to see the minimum set of options needed to create failure -> It literally just has to take longer than 1 S to respond and it causes failure.

Also, if we nav away from the app before we assert the app loaded, the Apollo client doesnt have time to set local storage, so it also passes.

[dudewad]

Any chance there's an update on this? I'm experiencing the same issue and this has been open for a year and a half. This seems pretty important since setting local storage (or in my case session storage, which, since there's no clearSessionStorage on the API I assume it does it implicitly) is a fairly common thing.

[dudewad]

Actually -- never mind. I found that you guys actually haven't implemented clearing session storage which was my issue. This issue covers the workaround for anyone else who runs into this issue thinking that they might be the same call.

[single-stop-justin]

What is the latest on this?

[jennifer-shehane]

We've outlined some work to be done to address some of the concerns of 'session storage' in this issue: https://github.com/cypress-io/cypress/issues/8301

It outlines a proposal for a 'session' API, to quickly summarize:

With cy.session, you'll be able to perform UI interactions before taking a "snapshot" of all the session related data and "restore" it before each test.

The session related data would include sessionStorage.

I recommend reading the entire proposal in https://github.com/cypress-io/cypress/issues/8301 and following there for any updates. Please feel free to express any concerns or questions in that issue concerning the API and add a 👍 for general support.

[jennifer-shehane]

Closing in favor of https://github.com/cypress-io/cypress/issues/8301