Cypress: Can Cypress.io record outgoing network requests and let me query them? (Not trying to stub requests)

[tangst]

Desired behavior:

My use case:
I am trying to identify certain outbound network requests on a page, and of the requests that match what I'm looking for, I want to check it string and regex pattern match for certain values to verify the network request is formed correctly.

For example, I need to check all outbound network requests, and any that match https://securepubads.g.doubleclick.net/gampad/ads?* need to be looped through. In each loop, I need to check if a certain string parameter (via regex or Javascript string methods) in the URL is present.

If I wanted to do this test manually, I have two options. In Chrome, I go to the Network tab in Chrome Dev Tools and examine the network requests I'm looking for using its filter field. Likewise, Charles and Fiddler also record network requests, and I can find the requests I need using its filter field.

I'm trying to find out if Cypress.io can give me all the outbound network calls automatically that match a criteria. I was once told on StackOverflow (https://stackoverflow.com/q/50820829/432089) that I should be able to use

     cy.route({ 
         onRequest: (req) => { 
           // Some code or test here, but they don't work
           // I tried cy.log() and expect(true).to.equal(false); and neither has any noticeable effect.
         } 
     });

to automate this. But, when I set up cy.server() and then this cy.route() example up, I don't see anything inside of this callback executing.
Using the example above, all I can see is that the route to https://securepubads.g.doubleclick.net/gampad/ads?* is checkmarked, but I see that one of the calls to https://securepubads.g.doubleclick.net/gampad/ads?* returns 400 by the server from Chrome Dev Tools' network tab. This 400 error doesn't happen if I hit the page normally.

It seems cy.route is for stubbing requests and responses. I don't need to stub any requests or responses. I want an automated way to see and query the network requests going out without modifying them like I would using Chrome Dev Tools, Fiddler, and Charles. Does Cypress.io support this use case currently? I see this ticket (https://github.com/cypress-io/cypress/issues/687) but I don't know if it would address my use case.

Thank you.

Versions

Cypress 3.0.1, Chrome 67, Mac

[jennifer-shehane]

Hey, could you provide the code you set up to listen for the requests? cy.route() should only stub responses if you provide a response option.

[tangst]

Hello,
Here is the test code.

    describe("Desktop - Basic Profile", function () {
        it("should make ad request for 5004 ad position", function () {
            var url = 'REDACTED URL';

           Cypress.on('uncaught:exception', (err, runnable) => {
                return false;  // returning false here prevents Cypress from failing the test
           });

          cy.viewport(1280, 1024);

          cy.log('HELLO! OUTSIDE cy.route!');  // This works and is seen in Cypress

           cy.server();

           /* https://docs.cypress.io/api/commands/route.html#Arguments */
           cy.route('https://securepubads.g.doubleclick.net/gampad/ads?*', {
               onRequest: (req) => {
                    cy.log('HELLO!! INSIDE cy.route!');  // Doesn't show in Cypress
                    expect(true).to.equal(false);  // Doesn't show in Cypress.  I was trying to fail the test, but it doesn't fail.
              }
           });

           cy.visit(url);
     });
    });

[jennifer-shehane]

The cy.route() is syntax set up to accept several variations of options, but what you have passed is essentially cy.route(url, options), which is not one of the ways that the arguments can be passed. Check the syntax in the docs. You want to pass the arguments like cy.route(options), so you would write it this way.

cy.route({
  url: 'https://securepubads.g.doubleclick.net/gampad/ads?*',
  onRequest: (req) => {
    cy.log('HELLO!! INSIDE cy.route!');
    expect(true).to.equal(false);
  }
});

Let me know if this works!

[tangst]

That code was given to me by someone on Stack Overflow, so I copied and pasted without verification. I'm still very new to Cypress.

Let me try this again with correct syntax. I will reply again soon.

[tangst]

I ran the test again with the modified syntax, but nothing inside the onRequest property seems to fire. I have attached a screenshot of the test. The stubbed field is "no" but the entire test passes despite trying to fail the test on purpose. Either that, or I do not know how to write Cypress tests correctly. Ugh.

    describe("Desktop - Basic Profile", function () {
        it("should make ad request for 5004 ad position", function () {
        var url = 'REDACTED URL';

        Cypress.on('uncaught:exception', (err, runnable) => {
            return false;  // returning false here prevents Cypress from failing the test
        });

        cy.viewport(1280, 1024);

        cy.log('HELLO! OUTSIDE cy.route!');

        cy.server();

        /* https://docs.cypress.io/api/commands/route.html#Arguments */
        cy.route({
            url: 'https://securepubads.g.doubleclick.net/gampad/ads?*',
            onRequest: (req) => {
               cy.log('HELLO!! INSIDE cy.route!');  // Doesn't show in Cypress
                expect(true).to.equal(false);  // Doesn't show in Cypress.  I was trying to fail the test, but it doesn't fail.
            }
        });

        cy.visit(url);     
    });
    });

[brian-mann]

I'm going to guess here and say that the doubleclick.net requests are not XHRs... if they are they are likely from inside an iframe OR they are actually <img> requests since that's how most ad networks get around origin policy.

If either of those two things are true, then that's why you're not seeing the onRoute function get called, and that's why you're not seeing it show up in the command log.

Also, you should never be using cy.log or console.log to debug what's going on inside of JavaScript. Always prefer a debugger and keep your dev tools console open.

If you take a screenshot of your network tab that will tell you 100% for sure what's going on.

[tangst]

Here is the relevant doubleclick.net call when I load it up in Chrome with dev tools active. Chrome says the network request type is XHR. Is there something else I'm missing? See screenshot below.

I tried to use debugger inside of the cy.route option object initally, but it didn't work, so I thought I was doing something wrong and went with cy.log().

I appreciate the Cypress.io staff trying to assist me and the rest of the team working with me. We're all trying to use Cypress.io and trying to move away from Webdriver.io/Selenium/Browsermob Proxy.

[jennifer-shehane]

I would also just double check the glob you have set up to listen to this route. Could you test https://securepubads.g.doubleclick.net/gampad/ads?* against the actual url using Cypress.minimatch, it should return true if it is accurately matching.

https://on.cypress.io/minimatch#Examples

[brian-mann]

Yeah I see it in the Command Log which means Cypress detects it. You've definitely written the glob pattern wrong. 100% that is the problem. I think you need multiple globstars for it to match correctly.

[tangst]

I am using Cypress.minimatch() now.

Since the Cy.() methods queue operations, I put the debugger line before any of them while I use minimatch. I also have to put in the actual doubleclick.net call into minimatch to compare with the pattern. The actual call is really long.

     describe("Desktop - Basic Profile", function () {
    it("should make ad request for 5004 ad position", function () {
        var url = 'REDACTED URL';

        Cypress.on('uncaught:exception', (err, runnable) => {
            return false;  // returning false here prevents Cypress from failing the test
        });

        var val = Cypress.minimatch('https://securepubads.g.doubleclick.net/gampad/ads?gdfp_req=1&pvsid=3271753434766291&correlator=1698246360017793&output=json_html&callback=googletag.impl.pubads.callbackProxy1&impl=fifs&adsid=NT&pucrd=CgYgASgBOAF4Aw&jar=2018-06-26-22&json_a=1&eid=21061764&vrg=220&guci=2.2.0.0.2.2.0&sc=1&sfv=1-0-29&iu_parts=8668145%2Cconsumer%2Cwebmd&enc_prev_ius=%2F0%2F1%2F2%2C%2F0%2F1%2F2%2C%2F0%2F1%2F2%2C%2F0%2F1%2F2%2C%2F0%2F1%2F2%2C%2F0%2F1%2F2&prev_iu_szs=4x1%2C728x90%7C970x90%7C970x250%2C300x250%7C300x600%7C300x1050%2C1x8%7C970x251%2C1x1%2C1x6&prev_scp=pos%3D5004%7Cpos%3D101%26amznbid%3D2%26amzniid%3D%26amznsz%3D0x0%26amznp%3D2%26mnet_abd%3D1.04%26mnet_apc%3D11572292%26mnet_asz%3D728x90%26mnet_alt%3D495%26mnet_aat%3DO%26mnetSize%3D728x90%26mnet_rbd%3D0.51%26mnet_rpc%3D119514%26mnet_rsz%3D728x90%26mnet_rlt%3D495%26mnet_rat%3DO%26mnetbidderID%3D%26mnetCrid%3D356136642%26mnet_placement%3D356136642%26mnetbidID%3D28%26mnetAct%3DheaderBid%26mnetScpvid%3D33%26mnetbidPrice%3D0.26%26mnetCID%3D8CU66J63J%26mnetTd%3D%257Cab%253D0%257C%26mnetPageID%3D2%26mnetCV%3D1%26mnetCC%3DUS%26mnetUGD%3D4%7Cpos%3D121%26amznbid%3D2%26amzniid%3D%26amznsz%3D0x0%26amznp%3D2%26mnet_abd%3D1.57%26mnet_apc%3D11572290%26mnet_asz%3D300x250%26mnet_alt%3D498%26mnet_aat%3DO%26mnetSize%3D300x250%26mnet_rbd%3D0.48%26mnet_rpc%3D119514%26mnet_rsz%3D300x250%26mnet_rlt%3D500%26mnet_rat%3DO%26mnet_79bd%3D1.89%26mnet_79pc%3D1163480%26mnet_79sz%3D300x250%26mnet_79lt%3D500%26mnet_79at%3DO%26mnetbidderID%3D%26mnetCrid%3D656615527%26mnet_placement%3D656615527%26mnetbidID%3D4%26mnetAct%3DheaderBid%26mnetScpvid%3D51%26mnetbidPrice%3D1.96%26mnetCID%3D8CU66J63J%26mnetTd%3D%257Cab%253D0%257C%26mnetPageID%3D4%26mnetCV%3D1%26mnetCC%3DUS%26mnetUGD%3D4%7Cpos%3D922%7Cpos%3D901%7Cpos%3D920&eri=1&cust_params=pvid%3D153005169617492530%26ecd%3D0%26sname%3Dcore%26pimc%3D0%26art%3D%26hcent%3D%26leaf%3D%26mcent%3D%26mic%3D%26sec%3D%26scent%3D%26tmg%3D%26tug%3D%26cc%3D133%26env%3D1%26rd%3D1%26segm%3D0%26bp%3D1%26lif%3D0%26lpid%3Db872723d903bcfba4abf9b90cef87ede%26dmp%3Dall%252C40%252C62%252C87%252C117%252C138%252C161%252C171%252C191%252C1503%252C2572%252C2783%252C2785%252C2786%252C2787%252C2788%252C2789%252C2791%252C2792%252C2793%252C2794%252C2795%252C2798%252C2799%252C2800%252C2803%252C2804%252C2805%252C2809%252C2813%252C2815%252C2817%252C2819%252C2823%252C2827%252C2834%252C2842%252C2848%252C2850%252C2862%252C17819%252C30292%252C30302%252C32500%252C32502%252C32505%252C32539%252C32920%252C32922%252C32924%252C32927%252C32928%252C32931%252C32933%252C32938%252C32939%252C32951%252C32957%252C32965%252C32973%252C33004%252C33185%252C105534%252C112593%252C116419%252C127932%252C129793%252C129794%252C131179%252C132261%252C132426%252C132546%252C132877%252C133943%252C134046%252C134944%252C134946%252C134950%252C134953%252C137167%252C137962%252C138665%252C141999%252C144298%252C173168%252C177835%252C178438%252C178439%252C189511%252C193056%252C194024%252C197506%252C216766%252C248948%252C249295%252C249298%252C252391%252C267709%252C267715%252C293961%252C299129%252C301281%252C303644%252C303653%252C308034%252C310013%252C314572%252C317046%252C318069%252C318941%252C319159%252C319161%252C319163%252C319164%252C319168%252C319169%252C319171%252C319172%252C319173%252C319175%252C319176%252C320728%252C321441%252C321443%252C321445%252C321449%252C323381%252C323763%252C324722%252C325023%252C327598%252C327704%252C329175%252C331175%252C332713%252C333959%252C334584%252C334589%252C334596%252C334597%252C334600%252C334602%252C337872%252C340567%252C343444%252C343445%252C343649%252C343855%252C343859%252C344801%252C344811%252C344891%252C345749%252C346035%252C351524%252C361209%252C361210%252C361438%252C361445%252C364251%252C375037%252C380750%252C393587%252C393609%252C412989%252C417478%252C421711%252C421729%252C423616%252C423647%252C423654%252C423655%252C423692%252C423695%252C423697%252C423701%252C423708%252C423718%252C423721%252C423726%252C423735%252C426787%252C426788%252C15%252C57%252C58%252C108%252C126%252C131%252C134%252C144%252C145%252C150%252C167%252C173%252C218%252C264%252C465%252C466%252C482%252C483%252C484%252C485%252C486%252C487%252C552%252C1431%252C1433%252C1434%252C1436%252C1437%252C1441%252C1445%252C1446%252C1447%252C1448%252C1449%252C1451%252C1794%252C2784%252C30030%252C131186%252C134952%26pd_city%3D%26pd_state%3D%26pd_zip%3D%26pt%3D1680%26app%3D26%26uri%3D%252Fdoctor%252Fmichelle-wong-b20cea0d-4ae5-446d-bd33-70d0ef7c401a-overview%26pd_prof%3D29277%26fis%3D0%26fipt%3D0%26excl_cat%3Dssg&cookie_enabled=1&bc=7&abxe=1&lmt=1530051701&dt=1530051701539&frm=20&biw=1440&bih=150&oid=3&adxs=210%2C356%2C936%2C0%2C0%2C0&adys=1375%2C104%2C749%2C3523%2C4649%2C4666&adks=3849460651%2C367367064%2C1866634141%2C2421346861%2C414359294%2C3189317492&gut=v2&ifi=1&u_tz=-240&u_his=1&u_h=900&u_w=1440&u_ah=841&u_aw=1440&u_cd=24&u_nplug=3&u_nmime=4&u_sd=2&flash=0&url=https%3A%2F%2Fdoctor.qa01.webmd.com%2Fdoctor%2Fmichelle-wong-b20cea0d-4ae5-446d-bd33-70d0ef7c401a-overview%3Flid%3D5611276&dssz=74&icsg=163840&mso=2048&std=0&vis=1&dmc=8&scr_x=0&scr_y=0&psz=676x2089%7C1440x132%7C312x260%7C1440x22%7C1440x4701%7C1440x4701&ga_vid=1528501089.1530051702&ga_sid=1530051702&ga_hid=899837885', 'https://securepubads.g.doubleclick.net/gampad/ads?*');

        debugger;

        cy.viewport(1280, 1024);

        cy.server();

        /* https://docs.cypress.io/api/commands/route.html#Arguments */
        cy.route({
            url: 'https://securepubads.g.doubleclick.net/gampad/ads?*',
            onRequest: (req) => {
                expect(true).to.equal(false);  // Doesn't show in Cypress.  I was trying to fail the test, but it doesn't fail.
            }
        });

        cy.visit(url);
    });
    });

The screenshot is where I inspect the the value of val that returns from minimatch. It says true which would contradict what Brian was thinking. If my glob pattern is bad, shouldn't this be false?

[brian-mann]

Try this...

cy.route({
  url: /gampad\/ads\?/,
  onRequest: (req) => {
    debugger
  }
});

[tangst]

Hello,
Ok, this is an actual regex as an argument for URL. That wasn't in the documentation...

Since there are possibly (though unlikely) other requests that could have the pattern 'gampad/ads?', I needed to include the domain name, like so.

The regex version works! The debugger is executed and pauses there when I have the Dev Tools for the Cypress-controlled Chrome browser active. Ok, now I can query the request URL in the req object to verify it has whatever pattern I need to test.

     cy.route({
            url: /securepubads\.g\.doubleclick\.net\/gampad\/ads\?*/,
            onRequest: function onRequest(req) {
                debugger;
                expect(true).to.equal(false); // Now works.  Assertion does fail as expected.
            }
        });

Thank you for your help, but the documentation didn't talk about using a regex argument for URL, so is this some undocumented feature or an oversight?

[tangst]

There are two doublclick.net calls on my test page. When I look in the Cypress-controlled Chrome, the first request is http 200 from the server but the second is http 400 from the server. Both are now type 'script.' Is that correct when Cypress is in control?

[jennifer-shehane]

The docs do mention that the url arg can be a "String, Glob, RegExp" and has an example here: https://on.cypress.io/route#With-Stubbing

We consciously updated the majority of our examples to globs because we found more users having troubles matching RegExp than Globs.

If you are not providing a response to the cy.route(), then the server is just passing through its response. So the server is responding with 200 on the first request and 400 on the second. Cypress is not changing the status unless response was provided.

[tangst]

@jennifer-shehane : Thank you for pointing out that in the docs. The part of the docs you mention with the regex mention is under the heading "With Stubbing". Since I wasn't stubbing, I was looking at your examples under the heading "Without Stubbing" which appeared above and didn't have the regex mention. While I was too singularly focused looking in the Without Stubbing section, it would improve the docs to mention the regex option in this section.

If more people understood Globs than regex, then why did the minimatch() test you asked me to do resolve as true but it didn't work in the route? Is there a bug somewhere in the glob resolution logic used in cy.route with my particular URL example?

Anyway, I got the route detection to work the way I think I want it, thanks to you, @brian-mann , and another person on the Cypress gitter board. This is what I settled on. This will catch the request, and if the request doesn't exist, it will timeout and fail.

     cy.server();

       /* Set route as an alias, so we can chain off it * /
       /* Using non-greedy token to find the 5004 position in the URL.  That tells us the ad configuration is set up correctly */
        cy.route({
            url: /securepubads\.g\.doubleclick\.net\/gampad\/ads\?(.)*?(pos\=5004)/
        }).as('getOmnitureUrl');

        cy.visit(url);

        /* If the URL is found, it passes the test, but we need a way to flag that to Cypress.  So we have an assertion here to pass the test */
        cy.wait('@getOmnitureUrl').then((xhr) => {
            isOmniReqFound = true;
            expect(isOmniReqFound).to.be.true;
        });

[mnagy]

@brian-mann sorry for veering a bit off-topic, but is there a way for Cypress to record non-XHR requests?

[Bkucera]

@mnagy you're referring to full network stubbing, could you mention that here? #687

[jennifer-shehane]

I created a new issue in our docs to improve the docs to mention the regex option in the "Without Stubbing" here: https://github.com/cypress-io/cypress-documentation/issues/774. Our documentation is open source and contributions are welcome. 😄

We also fixed the answer in StackOverflow, as that was a mistype by one of our team members.

We're going to close this as resolved. Let me know if there is still an outstanding issue to reopen.

[Vandivier]

I'm having a related issue so let's consider re-opening. I'm doing an e2e test so I don't want to stub, but I want to observe a query param in a POST and then observe the response. POST url is http://localhost:9000/public-api/auth/cognito

Consider these related images:

As you can see, Chrome tools is observing the request was made, I'm able to debug the response and it has the expected value, and I'm using a number of cy.route patterns but none are catching. Out of those screens I also tried url: /[\w-]*/ and url: /\*/,.

Route code is simple, like this (method is added for POST):

    cy.route({
      url: /\*/,
    });