Gutenberg: Env: wp-phpunit not in sync with mapped WordPress version

[noahtallen]

Describe the bug
A few days ago, we noticed phpunit failures in Travis. This was because a change was merged to wordpress core which was propagated to wp-env. Unfortunately, wp-phpunit (the dep we use to install wp phpunit test lib) is pinned at the test code in the last WP version, so the same changes did not propogate to it.

Essentially, the WordPress version running on the wp-env container is _not in sync_ with the version the wp-phpunit test lib expects.

Link to discussion in make WordPress slack (requires signup): https://wordpress.slack.com/archives/C02QB2JS7/p1591979580086000

Solution

The solution we have discussed (cc @TimothyBJacobs, @joemcgill) is to do something like this:

1. Detect the current WordPress version. If wordpress-develop, bail since the test lib is already mapped.
2. Use that WordPress version to generate a ref to wordpress-develop svn which contains the test-lib in sync with the current WP source code. (We can use wp-cli as a basis for this code.)
3. Download tests/phpunit/includes and tests/phpunit/data (either inside the docker image or in ~/.wp-env/$hash/)
4. Map those to the correct location in the phpunit Docker service.
5. Make sure that the phpunit tests reference this code for the test lib.

One benefit of this approach is that no one will have to worry about installing a wp-phpunit dep, which makes phpunit much more accessable in wp-env.

[timnolte]

I'm using a source dependency of wordpress-develop in my project and I'd like to use this Install with the Docker test environment to run PHP Unit. Based on that Slack documentation it seems like this might not be possible OOTB currently. Is that correct?

[noahtallen]

Hm. I'm not exactly sure how to do that, but really the only thing that wp-phpunit does is make the WordPress tests library available inside of docker. you currently need to write most everything else yourself (like the config file and bootstrap file). So if your wordpress source includes the tests library, then I think it's just a matter of loading the tests lib from the right location.

[timnolte]

What I'd like to see is have PHPUnit available inside the wp-env test Docker container.

[noahtallen]

Why not use the phpunit service? https://github.com/WordPress/gutenberg/blob/a94a95b6962bc56156e594ca7755e8bac1ddf3f1/packages/env/lib/build-docker-compose-config.js#L168-L179

Or at least, what is that missing for you? It should pick up on any sources or configuration mapped to the tests volume, and I think it would use the same database

[timnolte]

@noahtallen now that I'm looking through that build script for Docker and seeing how it's used in Gutenberg as well I have a better clue how to make use of the phpunit container. Thanks!