Cypress: Discussion: e2e testing

[compojoom]

We have an ios and android app and we are currently building a progressive web app for it.

For testing the app we use Detox (https://github.com/wix/detox) for our e2e tests and I’m currently evaluating frameworks for e2e tests for our web app. Cypress looks cool, but I have some issues with the concept.

In this video https://m.youtube.com/watch?v=5XQOK0v_YRE @brian-mann says that tests should run independently. One should thoroughly test the login and then for everything else mock a logged in user. The first definition of e2e test that comes online when searching in google is from techopedia:
https://www.techopedia.com/definition/7035/end-to-end-test

Definition - What does End-to-End Test mean?

End-to-end testing is a methodology used to test whether the flow of an application is performing as designed from start to finish. The purpose of carrying out end-to-end tests is to identify system dependencies and to ensure that the right information is passed between various system components and systems.
Techopedia explains End-to-End Test

End-to-end testing involves ensuring that the integrated components of an application function as expected. The entire application is tested in a real-world scenario such as communicating with the database, network, hardware and other applications.

For example, a simplified end-to-end testing of an email application might involve:
Logging in to the application
Accessing the inbox
Opening and closing the mailbox
Composing, forwarding or replying to email
Checking the sent items
Logging out of the application

The first sentence reads “to test whether the flow of an application is performing as designed from start to finish”. With e2e tests we are talking about a flow, which is different from unit tests where we testing different units in isolation.

With detox I’m able to simulate the flow of interaction with our app. The user logs in, then he is able to perform different actions one after another.

After loging in the user creates an article, then he comments on the article etc. The user is not able to comment without first creating an article. There is no point of executing a comment test before creating an article. There is also no point of always logging in the user before each test. Nobody in real life logs in before every action. I understand the reasoning of starting with clear state for each test, but this goes against the way users acutally interact with our applications. There are situations in which the user enters an url directly, but in 99.99% of the time we navigate from page to page by clicking on buttons on the site. The user follows a pattern that the developer has created for him.

Is it possible for cypress to execute tests one after the other without refreshing the state and without actully starting a new browser instance. I would like to create a suite of tests that need to execute in particular order one after the other.

[jordandlaman]

Regardless of framework it is generally encouraged to make your tests within a given spec independent of each other so that you have less flake within the spec when ran. An example being protractor which discourages mocking http://www.protractortest.org/#/style-guide. If each it block requires the subsequent block running and passing you get incorrect reporting results when that does not happen. Especially so if your framework fails the entire spec file on any failed test. From my time using cypress you can simulate entire flows (login - actions - logout) within any given it block.

[bahmutov]

@compojoom nothing stops you from writing _very long_ tests that simulate the entire user story workflow. A single test could really test all possible user actions

• sign up
• login
• add a todo
• mark todo completed
• delete completed items
• write another todo
• reload the page
• confirm the todo is still there
  ... etc etc

just a single test, and probably close to how a user uses the web app, right? And Cypress is _great_ at these long tests: you see every command, you can time travel to each step, you can see additional details for each command, etc. Even if this test fails somewhere in the middle, it will be 100% clear _what_ has failed from screenshot and video.

But where this fails is developer experience and development workflow. Imagine there is a problem in marking a todo as completed. The one and only test fails at that step - does this mean the app is 100% broken? Is reload still working? Can the user still add a second item? You have no idea, because the test failed before those steps were executed.

Now if you want to debug or change your application, this loooong test is a major pain. Because changing something requires sitting through all the steps just to get to the feature that you care about. Even if you have multiple separate tests building the data for the user test by test without destroying it in between - even that is still an anti-pattern in our opinion, because you would need to _wait_ in order to get to the test you are trying to debug.

So we recommend making all tests independent for these two reasons: knowing the status of different features, and faster development. And yet if you want a combined "supertest" just to simulate a longer user interaction - no problem. Refactor your tests to call separate JavaScript functions or make custom Cypress commands, and then have another long test where you just call more of these functions! It would look something like this

const signUp = () => ...
const login = () => ...
const action1 = () => ...
const action2 => () => ...
beforeEach(() => {
  cy.visit(...)
})
it('signs up user', () => {
  signUp()
})
it('allows user to login', () => {
  login()
})
// more individual tests
it('allows to do all actions in single long scenario', () => {
  signUp()
  login()
  action1()
  // some glue code maybe to prepare for action2?
  action2()
  action3()
})

Happy testing!

[brian-mann]

Agree with the thoughts expressed here - and we have also independently discussed whether or not we actually describe ourselves as an "e2e" testing tool. In fact, the reason we call the folder cypress/integration is specifically about not coupling ourselves to be a purely e2e solution.

Because we run in the same run loop you can do things in Cypress that are not possible to do in any other testing tool - and you should take advantage of that to the fullest extent. You can even import your react, vue, angular, etc components directly in your test code and mount them as a real DOM component.

[compojoom]

Thanks for the replies guys.

@bahmutov - yep, exactly. For the described reasons I don't want to write everything in one single test :)

Now if you want to debug or change your application, this loooong test is a major pain. Because changing something requires sitting through all the steps just to get to the feature that you care about. Even if you have multiple separate tests building the data for the user test by test without destroying it in between - even that is still an anti-pattern in our opinion, because you would need to wait in order to get to the test you are trying to debug.

I've noticed this in our detox tests. But in an iOS application you can't just jump to screen C, without going through A and B (unless you do some magic). In order to test the settings you start from the first screen in the app, then navigate to the other, then to the other until you reach the settings. In the browser you could have /#settings url, but it's unlikely that a normal living creature on this earth is going to type it. Most non-tech users I've seen don't even use the address bar, but type in the google search box...

Anyway, I think that we have 2 use cases.

first one - test specific screens of an application.
Second one - test the flow of an application

Cypress seems to concentrate only on the first one.

It's awesome that we can confirm that page B works as expected, but again - are we confident that if you come from page A - it still works? It's highly unlikely for a user to directly enter the URL and interact with page B in the context of single page applications, but yet it seems that this is what we test.

I think that both types of tests are important. I'm not sure how hard this is going to be (since I haven't looked at the internals of cypress), but if we had a conf option for a describe block that could specify that each tests inside of it should execute after the other without restarting the browser and session, then we could cover the use case of flow testing?

[bahmutov]

@compojoom I don't understand how you can say that second test (the flow of the application) is less supported. Take your mobile analogy of screen to screen transitions - the same problem exists in some web applications today - if the routing is NOT using hashbang and is set up client-side only, then going directly to the URL will result in 404.

Example: a web app with client side routing where you need to visit the index page / to click on the "Help" link to get to the client side view /help. If you go directly to the /help the server has no idea what page to return, so it gives 404. Example is vue-router with history mode https://router.vuejs.org/guide/essentials/history-mode.html

So, is this a problem in the E2E test runner that it cannot immediately go to the /help URL to run tests specific to the "Help" page? No, this is a problem with your web application implementation. If you think this is ok, then the tests just adopt like this

beforeEach(() => {
  cy.visit('/')
})
describe('Help page', () => {
  beforeEach(() => {
    cy.contains('nav a', 'Help').click()
    cy.location('pathname').should('eq', '/help')
  })
  it('explains things', () => {
     // make assertions about help page
  })
})

If your application did implement server-side routing fallback for this case, the tests ... would be almost the same! Just the first beforeEach would not be necessary, and the navigation could go straight to /help url

describe('Help page', () => {
  beforeEach(() => {
    cy.visit('/help')
  })
  it('explains things', () => {
     // make assertions about help page
  })
})

Boom, same tests, different reusable setup, etc - it is just JavaScript, and the tests adopt to whatever your implementation supports.

Aside: the common joke I enjoy a lot goes like this "Web apps broke browser's back button. Service workers broke browsers reload button"