Cypress: Fixture returns outdated/false data

[andreasremdt]

Current behavior:

Reading and writing fixtures seems to not work as expected (please let me know if this is my error). Across two different tests (within the same spec) the returned value from cy.fixture is outdated and should have been updated by a previous call to cy.writeFile. This looks to me like a caching issue?

Desired behavior:

It should always return the latest data from the fixture and not something outdated.

Steps to reproduce: (app code and test code)

1. Create a empty spec and paste the below code.
2. Run the spec and see the output in the dashboard.

describe('fixture', () => {
  it('step 1', () => {
    // Create the fixture first
    cy.writeFile("cypress/fixtures/test-temp.json", {
      id: 1,
      name: 'Step 1'
    });

    // Let's see the data, it should be fine
    cy.fixture("test-temp").then(data => cy.log(data));

    // Update the fixture again
    cy.writeFile("cypress/fixtures/test-temp.json", {
      id: 1,
      name: 'Step 2'
    });
  });

  it('step 2', () => {
    // Let's wait 5 seconds just to be sure
    cy.wait(5000);

    // The returned data is { id: 1, name: 'Step 1' }
    // Should be { id: 1, name: 'Step 2' }
    cy.fixture("test-temp").then(data => cy.log(data));
  });
});

Versions

Cypress 3.4.0 & 3.3.2
MacOS Mojave
Chrome 75

UPDATE: No need to create 2 different tests, it also happens inside the very same test.

[jennifer-shehane]

Yeah, this looks like a bug to me - if it's not, then it's really unexpected behavior.

I tried using an alias to reference the fixture and also tried using ugly thenables through the Cypress chains to ensure they were being run after another - it always references the original content of the fixture, even though I can see that the content of the fixture has changed.

Reproducible failing test

describe('fixture', () => {
  it('step 1', () => {
    // Create the fixture first
    cy.writeFile('cypress/fixtures/test-temp.json', {
      id: 1,
      name: 'Step 1',
    })

    // Let's see the data, it should be fine
    cy.fixture('test-temp').then((data) => {
      cy.log(data)
      expect(data.name).to.eq('Step 1')
    })

    // Update the fixture again
    cy.writeFile('cypress/fixtures/test-temp.json', {
      id: 2,
      name: 'Step 2',
    })

    cy.fixture('test-temp').then((data) => {
      cy.log(data)
      expect(data.name).to.eq('Step 2')
    })
  })
})

[WolfeLogic-LHR]

Maybe I'm chaining functions with the expectation of a void promise improperly, but I'm running into the same issue. I'm expecting a fixture to read from a file _after_ the file's been updated.

I produce a "random" test ID with a before() hook

Cypress.Commands.add('randID', () => {
    var text = ""
    var possible = "abcdefghijklmnopqrstuvwxyz0123456789"

    for (var i = 0; i < 10; i++) {
        text += possible.charAt(Math.floor(Math.random() * possible.length))
    }
    cy.writeFile('cypress/fixtures/testID.json', { testID: text }).then((data) => {
        expect(data.testID).to.equal(text)
        cy.fixture('testID.json').then(data => {
            cy.log('TEST ID AFTER WRITE: ', data.testID)
        })
    })

})

Where testID.json ends up with something like:

{
  "testID": "okgt1hp768"
}

I run the actual test and end it with an after() hook:

Cypress.Commands.add('clearTestID', () => {
    let localTestID = ''
    cy.writeFile('cypress/fixtures/testID.json', { testID: " " }).then((data) => {
        expect(data.testID).to.equal(" ")

    })
    cy.wait(3000).then(() => {
        cy.fixture('testID.json').then(data => {
            localTestID = data.testID
            if (localTestID !== " ") {
                cy.wait(4000).then(() => {
                    cy.fixture('testID.json').then(data => {
                        cy.log('TEST ID AFTER WIPE: ', data.testID)
                    })
                })
            }
            else {
                cy.log('TEST ID AFTER WIPE: ', data.testID)
            }
        })
    })

The assertion expect(data.testID).to.equal(" ") passes, and the JSON file reads:

{
  "testID": " "
}

but my log from cy.clearTestID() reads:
TEST ID AFTER WIPE: , okgt1hp768

These

Cypress.Commands.add('clearTestID', () => {
    cy.writeFile('cypress/fixtures/testID.json', { testID: " " }).then((data) => {
        expect(data.testID).to.equal(" ")
    })
})
Cypress.Commands.add('checkForClearID', () => {
    cy.fixture('testID.json').then(data => {
        cy.log('TEST ID AFTER WIPE: ', data.testID)
    })
})

Called as:

after(() => {
        cy.clearTestID().then(() => {
            cy.checkForClearID()
        })
    })

Also do not work.

And again... There's a more-than-reasonable chance I don't understand Cypress' asynchronicity as it relates to promise chaining (or at all?), but I thought I did? Happy to be schooled. But otherwise this is a pretty annoying bug... Or pattern.

[jennifer-shehane]

So I asked the team about this. We are indeed caching the reference to the fixture to save memory. This was intended at the time, but is likely not a great solution now.

We likely need to rework this logic, to not use the cache when the file has been changed. We can detect that the file has changed by initially getting the SHA, compare if SHA has changed, otherwise use cache. Alternatively look at modifiedAt time of the file (if works on all platforms)

Workaround:

You can use cy.readFile() instead of fixture as this will not use the cacheing.

[WolfeLogic-LHR]

Thanks Jennifer. Knowing that cy.fixture() caches the reference and that cy.readFile() reads the current state of the file is a solid workaround.

[x-yuri]

Not only doesn't it detect fixture changes, it also ignores encoding when caching. Having said that, I need it (a fixture in different encodings) for a test repo, where I'm going to show several solutions to the same problem. Not for testing a site.

As for changing fixtures, I'm not sure it's often needed. @andreasremdt Can you elaborate your case? But adding ignoreCache and/or dontCache options would probably resolve the issue. That might be easier to implement than detecting file changes.

Another workaround (that shouldn't go without an explaining comment) is adding slashes:

cy.fixture('upload.png');
cy.fixture('/upload.png');
cy.fixture('//upload.png');

[mwren-mshanken]

@x-yuri

My use case is:

I am stubbing getting an array of data that pre-populates fields on a page, altering the data, then I am clicking a "save button", stubbing the new data into a new fixture, then reloading the page and stubbing in my new data. I do this in multiple tests to test things like clearing all data, populating fields, making sure header elements update based on new data values. This allows me to mock tests to both our get API and the API that updates this data.

An example of the function I call to do all this:

export function saveApplicationAndReload() {
  clickSaveButton();
      cy.wait('@updatedApplication').then((xhr) => {
        cy.writeFile('cypress/fixtures/updatedApplication.json', xhr.request.body.application).then(() => {
          cy.readFile('cypress/fixtures/updatedApplication.json').then(() => {
            cy.fixture('updatedApplication.json').as('updatedApplication')
            cy.route('GET', 'applications/*', '@updatedApplication').as('getUpdatedApplication')
          })
        })
      })
      cy.reload()
      cy.wait('@getUpdatedApplication')
}

Since the fixture file always returns the same, original value, this function only works properly the first time it's called.

Replacing this with a cy.readFile() call works for now, but it took me a lot of time to track down this issue, and find the workaround due to the current implementation.

[nephix]

If you don't want to deal with workarounds and your fixtures rarely change then you can also remove the following directories:

• MacOS: ~/Library/Caches/Cypress
• Linux: ~/.cache/Cypress
• Windows: /AppData/Local/Cypress/Cache

And run in your project folder rm -rf node_modules && npm i
This will cause Cypress to reinstall, but much better than introducing workarounds in many of your tests IMO

[x-yuri]

@mwren-mshanken So you use one fixture for many tests and many purposes? That's not how fixtures are to be used, don't you agree? The solution is probably to use a different fixture name in different tests.

Speaking of the bigger picture... you do an xhr request, save the body in a fixture, and then stub subsequent requests with the fixture? Is it for performance reasons? Does using fixtures in such a way improve performance a lot?

Since the fixture file never gets updated, this function only works properly the first time it's called.

By the way, the fixture file is always updated, it's just that the cy.fixture() always return the first version of a file.

@nephix cy.fixture returns stale data if a fixture changes, what deleting Cypress has to do with it?

[mwren-mshanken]

I have multiple tests that interact with the same data object. In our application loading a page requires the same data to pre-populate fields, then we test various front-end functionality. Part of these tests includes what happens when we alter data, save the changes, then reload the page. This is why we need to:

• Know what the new data is when we save the page
• Store this data
• Reload the page and use this new data

Writing this data when it changes simulates our network layer, which hits the update API, then write the new data to a database. Stubbing this improves performance and keeps with the recommended pattern of having one golden-path test that does full end-to-end testing on all network paths, and stubs that simulate our e2e patterns.

For more information:

https://docs.cypress.io/guides/guides/network-requests.html#Stub-Responses

Suggested Use
Use for the vast majority of tests
Mix and match, typically have one true end-to-end test, and then stub the rest
Perfect for JSON APIs

[nephix]

@nephix cy.fixture returns stale data if a fixture changes, what deleting Cypress has to do with it?

Looks like cypress somehow caches them in the mentioned directories and deleting them therefore "cleares the cache"

[x-yuri]

No, the cache is in memory, as said above.

[nephix]

Definitely false, as the outdated fixture persists across sessions. It's probably serialized and stored somewhere, which is then re-initialized on a new session.

[x-yuri]

@nephix You seem so confident. Can you possibly back up your words? I can't reproduce what you're saying:

The test:

    it('...', () => {
        cy.fixture('1.txt').then(content => {
            cy.log(content);
            cy.writeFile('cypress/fixtures/1.txt', String(Number(content) + 1));
        })
        cy.fixture('1.txt').then(content => {
            cy.log(content);
        });
    });

$ git clone -b cypress-change-fixture https://github.com/x-yuri/issues cypress-change-fixture
$ cd cypress-change-fixture
$ npm i
$ npx cypress open

Then choose 1.spec.js, and you'll see:

Then press "r" (Run All Tests), and you'll see:

Which means fixtures don't persist across sessions. In case they persist for you under some circumstances, please provide exact and easy to follow instructions like I did.

[nephix]

@nephix You seem so confident. Can you possibly back up your words?

Yes have done it several times already since I've posted and wouldn't have posted it if it didn't work for me.

Which means fixtures don't persist across sessions.

Definitely not my observation.

In case they persist for you under some circumstances, please provide exact and easy to follow instructions like I did.

Can't really provide you with a minimal example from my phone, but in my test cases I've initialized them via:

cy.fixture('foo.json').as('foo')

and this.foo always had the same values from the first test run, even though I have changed content in foo.json. Could mean that .as() does some additional magic. Removing cypress and reinstalling it through npm i solved it for me though

provide exact and easy to follow instructions

I already provided instructions above that solve the problem of loaded fixtures being outdated and just wanted to provide another option besides the workaround mentioned here

[x-yuri]

Can't really provide you with a minimal example from my phone

I'm not asking to answer right away, please take your time to make Cypress better.

in my test cases I've...

I've once again followed your instructions, and still can't reproduce. Probably because your instructions were not exact. Can you please spend some time and give us a sure way to reproduce your issue? Ideally, create a repository one can clone, npm i, cypress open and see the result.

I already provided instructions above that solve the problem of loaded fixtures being outdated

Let's make it clear. The issue you're experiencing is not the issue described in the original post. Because your solution doesn't solve the original issue. Although both fall under the "outdated fixture data" category. That's why I'm trying to make you share the exact steps needed to reproduce your issue.

[nayakavik]

Thanks for this thread. Wasted a lot of time on this due to the fact that I am not a JS expert :) I could clearly see in the console in runner that the contents of the file under fixtures folder is getting updated but my assertion was failing due to cached data of the original content of the fixture.
cy.readFile() is a real savior.

I am not sure whether dynamically changing the data of fixture files is a best practice or not - still I hope the bug is fixed soon.