Cypress: Bad XHR performance with Google Cloud Firestore. Like really bad.

[bdiz]

Current behavior:

When making requests to Firestore outside of Cypress, XHR responses take less that a couple seconds (at most).
When making requests to Firestore inside of Cypress, XHR responses take more than 60 seconds.

Desired behavior:

When not stubbing, I expect XHRs to have the same performance inside of a Cypress spec as they do when my site is hosted or when using a local server.

Steps to reproduce:

I created an example app which will hit the actual Firestore server. It includes a button to send an XHR and a timer which will run until the response is returned.

git clone https://github.com/bdiz/cypress-firestore-performance.git
cd cypress-firestore-performance
npm install
npm run serve
# click the "Add document to Firestore" button and observe the XHR finish within a couple seconds
# CTRL-C to kill the server
npm run test:e2e
# Run the test.js spec and click the "Add document to Firestore" button and observe the XHR finishes in 30 seconds plus.

Versions

Cypress 3.1.0
Ubuntu 16.10
Chrome 67 (used for hitting local server and inside test runner)

[leyenda]

Issue #1150 refers to this.

[james-bowers]

I think PR #1195 addresses this, it just needs a review

[bentlusty]

The PR didn't fix it, all firestore requests time out

[pvdyck]

@bentlusty please confirm the version used -develop branch ? 3.1.1 ?

[bentlusty]

@pvdyck version used latest develop branch

[pvdyck]

any news ? can we help ?

[orzarchi]

I can confirm this still doesn't work in the latest develop branch.
If anyone requires help reproducing let me know, but I think simple running any kind of firestore query will do the trick - the query never completes.
Hopefully this will be get looked at soon, as we our company will soon be forced to choose another testing solution :(

[pvdyck]

It is working for me, no more delay on firestore queries.
So yes, it could be useful to provide a sample repo with failing tests ?

[lilaconlee]

We're still able to replicate the issue with the repo that was originally provided and with the Firestore quickstart project, but still need some time to figure out a fix.

It's in a queue with a couple of other high priority, but pretty time consuming issues that we'll get to as soon as possible.

[afcastano]

Thank you for looking at this and for this awesome product.
I really want to use it but our web app is heavily using Firestore. Any idea when this could be fixed? I need to decide soon which testing framework should we use... :(

Thanks again =)

[pvdyck]

Also, could you please automate the release process ?
You should really release new versions more frequently. Priority number 1 !
We are quite happy with the fix, but we need to compile a local version and make our own docker image for the CI ... quite a mess

[fredrik-sogaard]

The latest 3.1.1 release seems to have fixed this for us.

[afcastano]

@fredrik-sogaard Where can I get 3.1.1? Is that develop branch?

I cloned develop and ran it using npm start.
Still getting the same error...

[fredrik-sogaard]

It's on npm: https://www.npmjs.com/package/cypress

[afcastano]

@fredrik-sogaard duh... didn't see the new release... anyway... version 3.1.1 still has the problem for me... :(

[afcastano]

Any news on this? I am afraid I can't wait any longer... :'(

[bahmutov]

Andres, do you have full reproduction example that shows the problem?

Sent from my iPhone

On Nov 25, 2018, at 21:13, Andres Castano [e-mail] wrote:

Any news on this? I am afraid I can't wait any longer... :'(

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[afcastano]

Hi @bahmutov, unfortunately not as the source code is private :( , but as @lilaconlee mentioned, it can be replicated by running the Firestore Quickstart Project

[nils235]

Is there an appropriate solution to this issue?

[mslooten]

I think we might be having the same issue. Unfortunately also private source code, but perhaps some observations could be of use?

• Angular
• Firestore
• Cypress

When running the tests, I can see the Firestore requests showing up in the sidebar as XHR. However, they never respond when running in Cypress. When I open the url in my browser, I can see the network calls respond. The app is still working in Cypress, although the requests will show as pending and eventually timeout. It also means I cannot use .wait(), because they never return a value. I suppose I could try to stub them, but of course we rather have a choice.

I'm starting to think it has something to do with the custom protocol in use. It seems Firestore uses something called WebChannels, or at least AngularFire does (https://google.github.io/closure-library/api/goog.net.WebChannel.html).

I could be way wrong here, but I guess it makes sense that IF a custom protocol is used, Cypress won't be able to intercept it the way it does XHR reqs (similar to .fetch perhaps?). Please let me know if there's anything I can do to help move this forward.

[flotwig]

@mslooten if the Network tab says it's going out as an XHR, we should be able to intercept it

Does your company require a proxy to access the Internet, by chance? If so, that's probably what's causing the error. Currently HTTPS requests to third-party domains behind a proxy will time out.

Cypress 3.3.0 will contain fixes for #1469 that will allow this proxied traffic to reach the web and work correctly.

[mslooten]

@mslooten if the Network tab says it's going out as an XHR, we should be able to intercept it

Does your company require a proxy to access the Internet, by chance? If so, that's probably what's causing the error. Currently HTTPS requests to third-party domains behind a proxy will time out.

Cypress 3.3.0 will contain fixes for #1469 that will allow this proxied traffic to reach the web and work correctly.

First, thanks for the quick reply. Appreciate it! This behavior is consistent whether on company or home network (no proxies). However, the application is served from localhost (http://localhost:4200) and Firestore is HTTPS, not sure if that's the same thing though.

I can see the XHR requests being made, I can even see the application updating with the right data (that was obviously returned from Firestore), but the requests stay pending in Cypress. Visiting the same url in my browser (also Chrome), I can see the XHR requests and the responses.

[edwinwalda]

Any update on this? My company's project is entirely serviced by firestore, and we were really hoping to use cypress to do our integration tests.

[travwritescode]

This issue has been under development on and off for a year now. Is there any update on deliverability? My team is testing two apps that are dependent on Firebase and we would really like for Cypress to be the tool we use for these.

[Rednegniw]

I second this. Firestore requests have really big delay with Cypress for me.

[prescottprue]

I have experienced this only periodically in applications like fireadmin.io (here are the tests). The tests for the larger application we run at the company I work for and a number of folks using cypress-firebase have mentioned they see this issue very regularly

[prescottprue]

For those looking for a repro of not being able to see Firestore emulator in Cypress, which seems to be similar/related, I provide one in this issue: https://github.com/cypress-io/cypress/issues/6350.

In an issue opened in firebase-tools, it was mentioned that passing experimentalForceLongPolling: true to firestore settings that seems help. When setting this in the above mentioned repro, the data loads!

There was also the following theory from @wvanderdeijl that seems to make sense:

I think the underlying problem is that Cypress is intercepting all network traffic so it can monitor and sometimes mock. However, the webchannel protocol used by firestore has multiple replies over the same http request. The Cypress code cannot handle this and will only forward the first reply and ignore the rest.

^ Also aligns with what @mslooten mentioned above (so it seems that it isn't specific to AngularFire, but instead just Firestore in general)

[wilhuff]

I work on Firestore, so I can confirm: the default behavior of Firestore's web SDK is to make use of WebChannel's streaming mode. The client makes what looks like an XHR, but then the server will hold the response open for 60 seconds and send as many server-initiated responses as it can during that time window.

The experimentalForLongPolling option forces the server to send only a single response per request. While this works around issues where proxies buffer responses, it also introduces a 1/2 RTT delay between every response. You won't notice this developing your app in the US talking to a Firestore instance in us-central, but RTT from many parts of the world to the same instance is over 250 ms. Don't just blanket enable this option if you can avoid it.

[mesqueeb]

I have this issue as well.

Doing

  if (window.Cypress) {
        Firebase.firestore().settings({ experimentalForceLongPolling: true })
    }

solved it

[RaviH]

Thank you @mesqueeb for an example setup. Thank you @wilhub for confirming it; confirmation from someone close to the code helps. Is there any thoughts from cypress team about working with/around this? Firestore may remove this flag as it's experimental; besides would it be possible for Cypress to handle it auto-magically without user googling around for the problem.

[TDAK1509]

To only enabled in localhost, maybe you can do this

const db = firebase.firestore();

if (location.hostname === "localhost") {
  db.settings({
    experimentalForceLongPolling: true,
  });
}