Gatsby: Tests failing since a few weeks ago

[tremby]

Description

While following the guide at https://www.gatsbyjs.org/contributing/setting-up-your-local-dev-environment/ I hit failing tests at step 6. Master at time of writing is ea117e3d8

Two issues:

• Some tests fail
• Testing framework doesn't exit

Steps to reproduce

Follow the guide as far as yarn test.

Expected result

Tests pass, and the testing program exits.

Actual result

Two tests fail, then the testing program keeps counting up forever, not exiting.

Failing tests are:

• GraphQL field extensions - allows creating a cutom field extension
• GraphQL field extensions - allows creating a custom field extension (used in type builder)

I ran a git bisect. In earlier commits (not sure the threshold for this) the same tests fail but the tester exits, but not before outputting this message:

console.log packages/gatsby-source-wordpress/src/http-exception-handler.js:11
    The request failed with error code "undefined"

Further back in history the tests pass.

The bisect led me to the first commit where the tests fail: 4f9c790b1c1472ed6fb0e020a172766730871e5d

This commit adds the tests and, judging by the commit message, also the code which is being tested.

Environment

  System:
    OS: Linux 4.15 Ubuntu 18.04.2 LTS (Bionic Beaver)
    CPU: (4) x64 Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
    Shell: 5.4.2 - /bin/zsh
  Binaries:
    Node: 10.16.0 - /usr/bin/node
    Yarn: 1.17.3 - /usr/bin/yarn
    npm: 6.9.0 - /usr/bin/npm
  Languages:
    Python: 2.7.15+ - /usr/bin/python
  Browsers:
    Firefox: 68.0
  npmGlobalPackages:
    gatsby-cli: 2 

[tremby]

@stefanprobst you're the author of that commit. Any idea what's up? Presumably it was passing for you or you wouldn't have submitted it.

[missmatsuko]

Having same issue.
Are tests being checked before marking a release?
Or maybe something missing on our local environments that isn't noted in the set up docs.

[stefanprobst]

Tests are passing for me locally, and have been passing in CI.
i do have jest not exiting because of gatsby-dev-cli/src/__tests__/watch.js locally (which does not seem to be an issue in CI though)

could you provide more info on what you're getting after yarn bootstrap && yarn jest?

[tremby]

OK.

Here's the output of each command. I ran them with 2>&1 >file so stderr and stdout should both be in there, in order.

The yarn test command hangs forever. I had it going through tee so I could see when it's done as far as I'm aware, waited a good minute, then exited with ^C.

If you search FAIL in the yarn test output you'll find the failed tests.

See also the message a few lines from the end.

• bootstrap output
• test output

[tremby]

The failed test's diff

    Difference:

    - Expected
    + Received

      Object {
        "test": Object {
    -     "withDefaultArgs": "Cheers!",
    -     "withQueryArg": ":cake:",
    -     "withoutArgs": "Happy birthday!",
    +     "withDefaultArgs": "2019-09-26",
    +     "withQueryArg": "2019-09-26",
    +     "withoutArgs": "2019-09-26",
        },
      }

Sorta seems like some external data structure has changed. Is this test calling out to a third-party source over the network? If something there changed some time in the last few weeks that'd explain the tests suddenly failing even though the Gatsby code hasn't changed.

[edit] oh... but you said it's still passing for you locally. :\

[stefanprobst]

i guess this is an issue with UTC Dates, I will look into this later today!

[stefanprobst]

@tremby if possible, could you check if #15895 fixes the failing tests for you? thanks!!

[missmatsuko]

I tried out #15895 and now have no failing tests.
Still have 3 todo and timer still going up, is that expected?

Test Suites: 224 passed, 224 of 225 total
Tests:       53 skipped, 3 todo, 1907 passed, 1963 total
Snapshots:   568 passed, 568 total
Time:        246s^C

[stefanprobst]

@missmatsuko Thanks for testing! The it.todos are expected, the issue with jest hanging still needs to be fixed in gatsby-dev-cli - unfortunately i'm not super familiar with that package.

[tremby]

Same here: they all pass but it still hangs. I don't have time right now but I may try to bisect to find where that started some time over the weekend.

[stefanprobst]

I may try to bisect to find where that started some time over the weekend.

:+1: thanks! my guess would be #15649 (specifically this)

[sidharthachatterjee]

This was resolved in https://github.com/gatsbyjs/gatsby/pull/15974 and https://github.com/gatsbyjs/gatsby/pull/15975

[stefanprobst]

@sidharthachatterjee would be cool to also merge #15895 thanks!

[sidharthachatterjee]

It's in! Thank you @stefanprobst 🥇

[tremby]

@stefanprobst just curious -- why are the "todo"s expected? What do they mean? They suggest that some tests haven't run (yet?), and that begs the question why not.

[stefanprobst]

@tremby test.todo is a feature introduced in Jest v24 (I think) and it's to remember tests that need yet to be written. it can also mean that it has not been decided how some behavior they are supposed to be testing should be implemented. I'm personally more worried by 78 skipped tests