Gutenberg: Run test suite against WordPress trunk

[danielbachhuber]

Following up on #10412, the test suite needs to be running against WordPress trunk after the 5.0 backport is complete.

Related 9c75e47 0c12b15

[youknowriad]

Not sure how the WordPress docker image is maintained, but if it could provide versions for trunk, 5.0,... we could test against all these versions.

[swissspidy]

That image is maintained by the Docker core team. See https://github.com/docker-library/wordpress.

One can specify a specific WordPress version when using that image and it tries to download it from https://wordpress.org/wordpress-${WORDPRESS_VERSION}.tar.gz

That works for 5.0 and latest, but not for nightly or trunk. I'm sure we could submit a PR for that though.

[youknowriad]

@swissspidy This makes me wonder if we could add a redirect from https://wordpress.org/wordpress-nightly.tar.gz to https://wordpress.org/nightly-builds/wordpress-latest.zip which would make this possible right away without any change in the image?

[swissspidy]

That could work, yes. Not sure if this something the meta team can do or if it needs to be requested at https://make.wordpress.org/systems/.

[aduth]

That works for 5.0 and latest, but not for nightly or trunk. I'm sure we could submit a PR for that though.

Is this a fair summarization of the action item here?

1. Submit upstream enhancement to the Docker image to provide trunk and nightly variants
2. Update references of WP_VERSION in .travis.yml to use trunk in place of latest

[aduth]

Yesterday's release of WordPress 5.2 and subsequent universal Gutenberg build failures (#15502, Trac #47183) should serve to reinforce that tests ought to be run against a prerelease version prior to its final release (even if just once).

It was fortunate that the issues discovered were relatively minor, though they certainly could have been much more severe.

[pento]

With the merge of #17004, tests are now running against trunk.

[aduth]

Nightly downloads are proving to cause some instability in the Travis builds. I have opened #18773 with an overview of the situation, proposing to revert to running the latest stable version instead. If the pull request goes ahead as proposed, we should consider to reopen this issue.