Gutenberg: @wordpress/env: A zero-config, self contained local WordPress environment for development and testing.

[epiqueras]

17668 was merged as the base for a new WordPress local environment CLI tool for development and testing that avoids the pitfalls of previous iterations.

Previous approaches were slow, required too much configuration, and/or did not work on certain machines/versions of Docker Engine.

wp-env will be a zero config tool for developing plugins, themes, or Core, and it will replace npm run env. Its goal is WordPress development in general, but it doesn't achieve it at the cost of burdening plugin or Gutenberg developers with various configs, commands, and options. It will automatically infer what the user needs from the contents of the folder it is running on and act accordingly.

The base PR supports plugin development, which includes Gutenberg. Here are the next steps needed to make this tool accessible to all types of WordPress developers:

• [x] Provide options for modifying the default ports. #17697
• [x] Support running in themes. #17732
• [ ] Remove colors from --help output.
• [ ] Replace all usage of @wordpress/scripts's npm run env with wp-env.
• [x] Support mounting multiple plugins/themes through a configuration file. #18121

[noahtallen]

Neat!

I saw this comment on the original PR:

That's a fair point on docker-composer.override.yml. Let's wait and see if there's a need for it.

I ended up modifying gutenberg/wordpress/docker-composer.override.yml to add volume mappings for both some themes and a plugin for development purposes. Is that use case still supported, or what would the new correct use case be if we are removing the override file?

version: '3.7'
services:
  wordpress-develop:
    volumes:
      - '/Users/noah.allen/source/wp-calypso/apps/full-site-editing/full-site-editing-plugin:/var/www/${LOCAL_DIR-src}/wp-content/plugins/full-site-editing-plugin'
      - '/Users/noah.allen/source/themes/varia/:/var/www/${LOCAL_DIR-src}/wp-content/themes/varia'
      - '/Users/noah.allen/source/themes/maywood/:/var/www/${LOCAL_DIR-src}/wp-content/themes/maywood'
      - '/Users/noah.allen/source/gutenberg:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg'
      - '/Users/noah.allen/source/gutenberg/packages/e2e-tests/plugins:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg-test-plugins'
      - '/Users/noah.allen/source/gutenberg/packages/e2e-tests/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
  php:
    volumes:
      - '/Users/noah.allen/source/wp-calypso/apps/full-site-editing/full-site-editing-plugin:/var/www/${LOCAL_DIR-src}/wp-content/plugins/full-site-editing-plugin'
      - '/Users/noah.allen/source/themes/varia/:/var/www/${LOCAL_DIR-src}/wp-content/themes/varia'
      - '/Users/noah.allen/source/themes/maywood/:/var/www/${LOCAL_DIR-src}/wp-content/themes/maywood'
      - '/Users/noah.allen/source/gutenberg:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg'
      - '/Users/noah.allen/source/gutenberg/packages/e2e-tests/plugins:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg-test-plugins'
      - '/Users/noah.allen/source/gutenberg/packages/e2e-tests/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
  cli:
    volumes:
      - '/Users/noah.allen/source/gutenberg:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg'
  phpunit:
    volumes:
      - '/Users/noah.allen/source/gutenberg:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg'

[epiqueras]

Hi Noah,

There are 2 ways you can do this:

• You can run the tool in path/to/gutenberg, then go to path/to/gutenberg-wordpress and drop the plugins and themes in there that you need. You can even have nested git directories in there. I do that if I am developing a block for example.

• You can also run the tool in path/to/full-site-editing, then go to path/to/full-site-editing-wordpress and add whatever extra themes or plugins you need there.

Basically, the tool mounts the plugin or theme you run it on, but you can manually add more after that.

[noahtallen]

Hm interesting. Do you mean manually dropping those folders in the right places in the WordPress install, like path/to/full-site-editing-wordpress/wp-content/plugins? (Or perhaps I misunderstand :D)

I don't know how feasible that is since our particular use case is 2 monorepos:

• wp-calypso repo, in which case the plugin is located at wp-calypso/apps/full-site-editing/full-site-editing-plugin. (I would have to build the JS before copying the folder elsewhere, since it depends on scripts inside the monorepo)
• themes repo, which stores several themes, like themes/theme-name (ideally it would be nice to tell wordpress (hey make all these themes available)

(and then, of course, it would be nice to also have access to gutenberg master at the same time :D)

The use case is that developing a singular plugin in our case also means working with themes and some other items.

I'm always a big fan of automation, so I wonder what something might like like to more easily specify which plugins/themes ought to be loaded?

For example, maybe wp-calypso/apps/full-site-editing/full-site-editing-plugin can have a config file which says "also load gutenberg dev plugin, located at x path, and also load this theme, located at y path". And then wp-env start in that directory could facilitate that

[epiqueras]

Yeah, I see how manually dropping them can get complicated. Could you set it to be the output of your build processes maybe?

In any case, I think that this will be required by more people now so it makes sense to have a first class solution.

I like those ideas. What if, we use environment variables PLUGIN_DEPENDENCIES and THEME_DEPENDENCIES to specify extra plugins and/or themes to mount. These can be provided through the shell or a .wpenv file. We should also just add DOCKER_COMPOSE_OVERRIDE since it's explicit and we won't run into the confusion that prompted the decision to not automatically detect the overrides.

What do you think?

[noahtallen]

These can be provided through the shell or a .wpenv file

I think this is a cool idea! I think defining a .wpenv file for a project makes a lot of sense. Maybe it could be json, something like this:

{
  "themes": {
    "theme-name": "path-to-theme"
  },
  "plugins": {
    "plugin-name": "path-to-plugin"
  }
}

or maybe with arrays:

{
  "themes": [
    "path-to-theme",
  ],
  "plugins": [
    "path-to-plugin"
  ]
}

I think these should also be linked instead of copied. Perhaps these could be added to the Docker volumes of the wp container?

[epiqueras]

or maybe with arrays:

Let's go with that.

I think these should also be linked instead of copied. Perhaps these could be added to the Docker volumes of the wp container?

That's the plan :smile:

[noisysocks]

Support running in Core, wordpress-develop. Would Core adopt this and benefit from it? It looks like they have a very specific setup they can tweak often.

What do you mean by a specific setup?

I'm not fussed if Core and Gutenberg has seperate development tooling, but I do think that the approach that local-env takes has some nice advantages that we should try to preserve:

• Switching between repos: Because local-env mounts Core and Gutenberg into the same container it's easy to switch between working in Core and working in Gutenberg. Right now, by default, Gutenberg's @wordpress/env uses the same port as Core's local-env which means that both can't run at the same time.
• Support for cross-codebase workflows: Because local-env uses the development version of WordPress (wordpress-develop@master as opposed to wordpress@master), it allows one to e.g. begin debugging a problem in Gutenberg but then ultimately land the fix in Core.

[epiqueras]

What do you mean by a specific setup?

E.g. They inject a custom NGINX config to handle some nuance around how things are served from either /, /src, or /build.

Switching between repos
Support for cross-codebase workflows

Aren't those two the same thing?

The thing is that wordpress-develop has to be built to be used right? That would incur a big performance hit on the tool and is part of the reason why local-env is slower.

What do you think about introducing a develop mode that does this?

Or just keeping both tools?

[noisysocks]

Aren't those two the same thing?

By _switching between repos_ I mean running both at the same time—not necessarily using the same Docker container. For example, right now if you run wordpress-develop using npm run env:start and then gutenberg using wp-env start, you'll get an error because both try to use port 8889.

The thing is that wordpress-develop has to be built to be used right? That would incur a big performance hit on the tool and is part of the reason why local-env is slower.

What do you think about introducing a develop mode that does this?

That's a good point regarding performance. I'd say a develop mode is worth exploring. Something like wp-env start --dev would be cool to see.

Another alternative might be to document somewhere how one can set up Core's local-env to do cross-repo development using docker-compose.override.yml.

Or just keeping both tools?

I don't think this is a good idea. We risk fragmenting the ecosystem, e.g. tutorials and resources for beginners would differ from one another. It also doesn't align with the WordPress philosophy of "decisions, not options."

[epiqueras]

By switching between repos I mean running both at the same time—not necessarily using the same Docker container. For example, right now if you run wordpress-develop using npm run env:start and then gutenberg using wp-env start, you'll get an error because both try to use port 8889.

Right, but that's not an advantage of local-env or any tool. It's just a symptom of having two tools. The ports are configurable though.

That's a good point regarding performance. I'd say a develop mode is worth exploring. Something like wp-env start --dev would be cool to see.

Another alternative might be to document somewhere how one can set up Core's local-env to do cross-repo development using docker-compose.override.yml.

I don't think this is a good idea. We risk fragmenting the ecosystem, e.g. tutorials and resources for beginners would differ from one another. It also doesn't align with the WordPress philosophy of "decisions, not options."

https://github.com/WordPress/gutenberg/pull/17871#issuecomment-543836795

It sounds like a --dev flag is the way to move forward with this then?

What does everyone else think? @youknowriad @noahtallen

[epiqueras]

Another alternative might be to document somewhere how one can set up Core's local-env to do cross-repo development using docker-compose.override.yml.

I guess the question is: Is this hard enough for the people who need it, to warrant baking it into the tool?

[epiqueras]

https://github.com/WordPress/gutenberg/pull/17871#issuecomment-544640938

I've updated the task list accordingly.

[noahtallen]

It sounds like a --dev flag is the way to move forward with this then?

Hm it seems like it wouldn't affect my use case, right? I likely won't need to be debugging core WordPress, but I would be debugging Gutenberg and other plugins/themes.

[epiqueras]

Hm it seems like it wouldn't affect my use case, right? I likely won't need to be debugging core WordPress, but I would be debugging Gutenberg and other plugins/themes.

Yeah, you probably won't use the flag. And even without it, you can always tweak the local clone of wordpress/wordpress and then patch wordpress/wordpress-develop manually.

[noahtallen]

I put up a PR which could add support for a .wpenv file as we discussed: https://github.com/WordPress/gutenberg/pull/18121. Open to feedback!

[noisysocks]

Support running in Core, wordpress-develop, with a --dev flag.

Was thinking a bit about this. How about, instead of a --dev flag, we expand on the .wp-env.json config file so that wp-env is a tool which can glue together arbitrary combinations of core, plugins and themes?

Some examples of different .wp-env.json configs:

1) Run the current directory as a plugin installed onto the latest trunk version of WordPress. Plugin build files must built manually e.g. by running npm run dev. Use cases: Plugin (incl. Gutenberg) development

{
    "core": "WordPress/wordpress#master",
    "plugins": [ "." ]
}

2) Run a few local plugins and themes installed onto the latest production version of WordPress that the wordpress Dockerfile uses. Plugin or theme build files must be built manually e.g. by running npm run dev. Use cases: Plugin (incl. Gutenberg) development, Theme development, QA

{
    "core": null,
    "plugins": [ "~/projects/gutenberg", "~/projects/advanced-custom-fields" ],
    "themes": [ "~/projects/twentytwenty" ]
}

3) Run a few local plugins and themes installed onto an older latest production version of WordPress. Plugin or theme build files must be built manually e.g. by running npm run dev. Use cases: Testing, QA

{
    "core": "WordPress/wordpress#5.0",
    "plugins": [ "~/projects/gutenberg", "~/projects/advanced-custom-fields" ],
    "themes": [ "~/projects/twentytwenty" ]
}

4) Run a local plugin installed onto a local development version of WordPress. Core build files must be built manually e.g. by running grunt watch. Plugin build files must be built manually by running e.g. npm run dev. Use cases: Core development, Plugin (incl. Gutenberg) development

{
    "core": "~/projects/wordpress-develop/build",
    "plugins": [ "~/projects/gutenberg" ]
}

Similar to the dependencies field in npm-package.json, we can support all sorts of different paths and URLs in core, plugins and themes:

• An absolute path: /path/to/gutenberg
• A relative path: ../gutenberg
• A GitHub URL: WordPress/gutenberg#master
• A Git URL: git://develop.git.wordpress.org#master
• A SVN URL: svn://plugins.svn.wordpress.org/advanced-custom-fields/tags/5.8.7/
• A ZIP or Tarball: https://wordpress.org/wordpress-5.3.1-RC2.zip

When a local path is specified, it is mounted into the Docker container as-is. When a remote path is specified, it is cloned or downloaded into a temporary directory (maybe: ~/.wp-env) and then mounted. When a remote path is a GitHub or Git URL and has been cloned before, a git pull is performed.

SVN, ZIP and Tarball URLs are a nice-to-have and can be implemented further down the track.

What do you think?

[epiqueras]

This is great @noisysocks!

Run a local plugin installed onto a local development version of WordPress. Core build files must be built manually e.g. by running grunt watch.

I like this, because we avoid having to deal with that in this tool.

When a local path is specified, it is mounted into the Docker container as-is. When a remote path is specified, it is cloned or downloaded into a temporary directory (maybe: ~/.wp-env)

Nice, this gets rid of the awkward ../${ name }-wordpress directories.

[noahtallen]

Replace all usage of @wordpress/scripts's npm run env with wp-env

Since I was using scripts to run some of the test commands, I want to make sure we can still use them with @wp-env. Here's a quick look at what might need changed in @wordpress/scripts

scripts/env.js

Question: should we delete these commands entirely, or map them to their counterparts in wp-env to provide backwards compatibility?

There are a few test scripts in @wordpress/scripts that should probably stay there since it better fits the theme of the package. These are scripts/env/test-php.js, scripts/env/lint-php.js, and scripts/test-e2e.js. If we decide to remove the other env commands from the scripts package, then these scripts will need to be moved up one directory. Either way, we should update them to reference the new wp-env container.

scripts/test-e2e.js

This uses a config to give a url/username/password to puppeteer:

https://github.com/WordPress/gutenberg/blob/15eb6d07f3be358f494cef56294a803488eba3d4/packages/scripts/scripts/test-e2e.js#L55-L59

This behavior could stay the same, but it could also call back to wp-env to figure out the default values used to set up the environment.

scripts/env/lint-php.js and scripts/env/test-php.js

The code affected is:

https://github.com/WordPress/gutenberg/blob/15eb6d07f3be358f494cef56294a803488eba3d4/packages/scripts/scripts/env/lint-php.js#L17-L21

https://github.com/WordPress/gutenberg/blob/15eb6d07f3be358f494cef56294a803488eba3d4/packages/scripts/scripts/env/test-php.js#L16-L21

Essentially, we need to make sure that these commands correctly execute on the plugin or theme at cwd, or we need to add something to the config file which lets us specify some sort of test configuration

[epiqueras]

The old env scripts need to be deprecated and the test scripts need to be
rewritten to leverage wp-env in the best way possible.

[noisysocks]

We've iterated a lot on @wordpress/env these past few weeks and I think we're now at the point where we can close this issue and use smaller more focused issues and bug reports for any further work. The [Package] Env label tracks this.

Thanks everyone! ❤️