Gutenberg: Framework: Travis build performance

[aduth]

Previously: #14289

This serves as a tracking ticket for various suggested improvements to Travis build performance. See various comments within #14289 for more detail.

• [x] Remove postinstall script from package.json (#14353)
  
  
  
  • Most of our Travis jobs ultimately call npm run build twice, wastefully, because of the combination of the postinstall triggered by an initial npm install, and a subsequent explicit build
  
  
  • npm install is run in many separate jobs, but npm run check-licenses only really needs to be run at most one time, not in each job
  
  
• [ ] Remove npm / JavaScript build from the "PHP Unit Tests" job
  
  
  
  • Avoids lengthy install / build process
  
  
  • As best I can tell, there should be no need to rely on JavaScript produced by npm run build. Previously it may have been needed to use the npm scripts for npm run test-php and npm run test-unit-php-multisite, but these resolve to docker-compose commands anyways, so it seems reasonable enough to call them directly
    
    
    
    
    
    • As of #13521, there is some PHP produced as the result of the Webpack build, though it is not subject to tests
    
    
    
  
  
• [x] Split JavaScript tests into separate tasks (containers) (#15229)
  
  
  
  • Separately resolves #10625
  
  
  • Not all of these tests need to depend on npm run build
  
  
  • In theory, frees up containers sooner by those which can finish early
  
  
• [x] Split end-to-end tests into more containers (#15228)
  
  
  
  • Currently already split across 2; consider 3 or 4
  
  
  • Since a build takes as long as its slowest task, this would be key to shorter builds
  
  
  • See: https://github.com/WordPress/gutenberg/pull/14289#issuecomment-470285204
  
  
• [ ] Leverage parallelization within end-to-end test containers
  
  
  
  • See: https://github.com/WordPress/gutenberg/pull/14289#issuecomment-470285204, https://github.com/WordPress/gutenberg/pull/14289#issuecomment-472462545
  
  
• [ ] Pregenerate the plugin build to use across tasks
  
  
  
  • Related Travis documentation: https://docs.travis-ci.com/user/build-stages/#data-persistence-between-stages-and-jobs
  
  
  • Possible enhancement for gutenberg.run build result (offer plugin zip)
  
  
  • See: https://github.com/WordPress/gutenberg/pull/14289#issuecomment-472462545
  
  
• [x] Reduce build time (#14860)
• [ ] Avoid "reset" set-up and tear-down tasks
  
  
  
  • If we can treat Travis as a fresh, throwaway environment, we can avoid running unnecessary tasks like reset-e2e-tests.sh or afterAll tasks like deactivating plugins, etc.
  
  
  • See: https://github.com/WordPress/gutenberg/pull/15228#issuecomment-487525637 , https://github.com/WordPress/gutenberg/pull/14245#issuecomment-471704737
  
  
• [x] Avoid Puppeteer's Chromium download except when necessary (#15712)
  
  
  
  • Related: #15667
  
  
• [ ] Streamline Docker setup
  
  
  
  • [x] Consolidate docker-compose run commands where possible (example) (#15742)
  
  
  • [ ] Keep containers active while setup process is run (avoid Starting gutenberg_mysql_1 ... done)
  
  
  • [ ] Consider executing setup as shell script within container(s) to avoid repeated docker-compose connections
  
  
  • [ ] Reduce interval that attempted WordPress connection elapses (source), either by lowering sleep or a mechanism to be made aware immediately
  
  

[gziolo]

We also should look into how to optimize for Travis the usage of cross-env SCRIPT_DEBUG=false ./bin/reset-e2e-tests.sh. It seems like it might be a part of ./bin/setup-local-env.js since you never re-run the same job without starting Docker from scratch.

[aduth]

We also should look into how to optimize for Travis the usage of cross-env SCRIPT_DEBUG=false ./bin/reset-e2e-tests.sh. It seems like it might be a part of ./bin/setup-local-env.js since you never re-run the same job without starting Docker from scratch.

I agree. I added a checklist item to the original comment.

[aduth]

I've added a new section "Streamline Docker setup". Observing the Travis end-to-end tasks, it's clear that the Docker setup takes a substantial amount of time. Individual docker-compose run commands in install-wordpress.sh are the main contributor here.

Keep containers active while setup process is run (avoid Starting gutenberg_mysql_1 ... done)

I spent some time trying to sort this out: It seems like we shouldn't be expecting "Starting ..." to be occurring for every individual command, since the mysql Docker container is already up while the commands are being run.

I wasn't able to make progress here, but perhaps someone with a bit more Docker-fu could advise? (cc @youknowriad)

A few things I tried:

• Do we need depends_on ?
  
  
  
  • From my understanding, it doesn't do what you might think it does anyways (wait for availability)
  
  
• links
• networks
  
  
  
  • Related: https://hub.docker.com/_/wordpress#wordpresscli
  
  
• Database environment variables assigned for cli_e2e_tests container
  
  
  
  • Related: https://hub.docker.com/_/wordpress#how-to-use-this-image
  
  

I also wondered: Can the wordpress:cli image be kept up, then use docker-compose exec instead of docker-compose run to avoid starting up a new container for each command? I tried this as well, unsuccessfully due to errors that the local site is not available (which is expected I suppose, since the script itself is responsible for installing the site).

[gziolo]

I tried to remove npm i & npm run build from Travis jobs which run PHP unit tests. It turns out they are necessary to make all tests pass.

More at: https://travis-ci.com/WordPress/gutenberg/jobs/201750063

Related changes: https://github.com/WordPress/gutenberg/pull/15748/commits/53a4ea99c771a22f4a14c358313f724565d67546

We could still seek for a solution which would mock those CSS files instead.

[aduth]

@gziolo It's intended that those tests should automatically stub a file if one does not exist. It may not be working correctly.

https://github.com/WordPress/gutenberg/blob/cc6d90e0fcc1e77a0cfc179ecf3016b7149a2133/phpunit/class-extend-styles-test.php#L67-L71

Also noting that there was some work in #14289 which could serve as prior art for trying to eliminate the need for NPM to be installed in the PHP tasks.

[aduth]

I chatted briefly with @noisysocks about what options we might have for specifically improving the Docker setup. While I'd been inclined to at least explore some options around moving all of the WP-CLI setup into a single script file to be run in the container image, it may be worth further exploration to either pre-build or create a custom container image with all of the setup steps. Looking to discussions like in https://github.com/travis-ci/travis-ci/issues/5358 , it's not clear how well this could be cached, though as in https://github.com/travis-ci/travis-ci/issues/5358#issuecomment-248915326 there are some options to explore. Alternatively, we may consider to publish custom images to Docker Hub.

The ideal outcome here would be that the setup-local-env.sh steps never need to happen (at least not as part of regular usage).

[noisysocks]

I'll have a try at using a custom Docker image—it's something I've been experimenting with on and off.

[aduth]

Noting that as of #18942, the npm run build script will now run tsc to compile and output types for packages. While this is definitely the intended behavior for publishing packages, and is used in checking types in the project, it is redundant or not strictly relevant for many of the jobs where we are currently running npm run build, including:

• PHP unit tests
• PHP unit tests (PHP 5.6)
• E2E tests (Admin) (1/4)
• E2E tests (Admin) (2/4)
• E2E tests (Admin) (3/4)
• E2E tests (Admin) (4/4)
• E2E tests (Author) (1/4)
• E2E tests (Author) (2/4)
• E2E tests (Author) (3/4)
• E2E tests (Author) (4/4)

Especially if those types are built in a "clean" environment, it may take a non-trivial amount of time to complete. It would be nice if we could avoid this by some means.

cc @sirreal

[sirreal]

Good point. Anecdotally, my machine takes >30s to do a cold build of types, and <1s for subsequent builds. Some ideas:

• We could pretty easily skip type building on CI outside of the typecheck job.
• Cache type files. From memory that would be something like:
  
  
  
  • packages/*/build-types
  
  
  • packages/*/*.tsbuildinfo
  
  
  • bin/.cache/*