Build: Feature Request: Orchestrator Plugins

[sdras]

What follows is a collaboration between @jlengstorf and I, asked for by @DavidWells and @ehmicky, to stub out some solutions to #186, and PR #495. We Voltroned out a solution, this is just the tip of the iceberg of what we're thinking about, so if you like this general concept, we'd love to dive deeper with you.

Netlify Build Plugins — Orchestration and Shared Data

This issue proposes solutions to open questions about Netlify Build Plugins:

1. How can developers ensure that build plugins run in a given order?
2. How can developers share data between build plugins?
3. If plugins are being composed in such a way, how can a maintainer easily tell what's going on?

We propose adding functionality for an orchestrator plugin.

An example orchestrator plugin:

module.exports = {
  onInit: async ctx => {
    // this needs to be orchestrated in order
    // only executes the onInit lifecycle hook for these plugins
    await ctx.netlify.pluginExecute('netlify-plugin-1')
    await ctx.netlify.pluginExecute('netlify-plugin-2')
    await ctx.netlify.pluginExecute('netlify-plugin-3')
  },
  onPreBuild: async ctx => {
    // this one only has one plugin that needs it
    // only executes the onPreBuild hook for this plugin
    await ctx.netlify.pluginExecute('netlify-plugin-1')
  }
}

NOTE: Orchestrator plugins would _not_ be allowed to do anything other than execute other plugins.

Individual Plugin Usage

The first plugin sets up and caches data required by other plugins:

module.exports = {
  name: 'netlify-plugin-one',
  onInit: ({ cache }) => {
    const thing = 'very important data';

    // this is what the cache would store from this
    // this cache would be unique to this plugin
    /*
      {
        owner: 'netlify-plugin-one', // auto-set by Netlify
        lifecycle: 'onInit', // auto-set by Netlify
        data: {
          thing: 'very important data'
        }
      }
     */
    cache.create({ thing })
  },
};

The second plugin runs after the first:

module.exports = {
  name: 'netlify-plugin-two',
  onInit: ({ cache }) => {
    if (!cache.data) return // basic error handling

    // or we can also be more specific to the first plugin
    if (cache.data.owner !== 'netlify-plugin-one') return

    const pluginOneStuff = cache.data.thing
    // do things with pluginOneStuff here

  }
};

---

More Details

The pluginExecute method

The API for pluginExecute that was shown in the orchestrator might look something like this:

pluginExecute(
  pluginName,   // string — the package name of the plugin to execute
  {
    lifecycle,  // array — which lifecycle methods to execute
                //   defaults to the lifecycle where the method is called
    config,     // object — any data to make available to the plugin
  }
)

[jlengstorf]

Super excited about this!

One potential opportunity I see is passing data between functions a return value instead of something that can be arbitrarily accessed. I like the explicitness of being able to check the return value vs. needing to scan all the code for cache.set calls.

To pass data between plugins, the lifecycle method should return the data that’s shareable between plugins:

module.exports = {
  name: 'netlify-plugin-database',
  onInit: async ({ context }) => {
    const db = new DBConnection();

    db.connect(credentials);

    // return this because we want other plugins to be able to access it
    return db;
  },
}

An orchestrator plugin will be able to capture the returned values of lifecycle methods and pass them to other plugins via pluginExecute():

module.exports = {
  name: 'netlify-plugin-orchestrator',
  onInit: async ({ context }) => {
    // get `db` that was returned from the DB plugin’s `onInit` handler
    const db = await context.netlify.pluginExecute('netlify-plugin-db');

    // now we can pass the database connection to another plugin
    await context.netlify.pluginExecute('netlify-plugin-other', {
      config: { db },
    });
  },
}

netlify-plugin-other would access the db variable either by making plugins into functions:

module.exports = ({ db }) => ({
  name: 'netlify-plugin-other',
  onInit: () => { /* do something with `db` */ },
});

or we could attach it to the context:

module.exports = {
  name: 'netlify-plugin-other',
  onInit: ({ context }) => {
    // access the `db` data from context
    context.config.db.doStuff();
  },
};

My preference would be plugins as functions because it makes that config stuff available across all lifecycle methods without requiring the same boilerplate to access it in each method.

---

I also think that plugins should have a way to pass their _own_ context as well, but that should be explicitly scoped to the plugin and not set in some global cache — there's too much opportunity for key collisions, one plugin to accidentally break others, etc.

I think an explicitly scoped in-memory cache for use inside plugins would solve the need to pass data between lifecycle methods without running the risk of collisions, etc.

module.exports = {
  name: 'netlify-plugin-database',
  onInit: async ({ context }) => {
    // we need info from two separate plugins
    const db = new DBConnection();

    db.connect(credentials);

    context.netlify.setCacheData({ db })
  },
  onBuild: ({ context }) => {
    const { db } = context.netlify.getCacheData();

    // do DB stuff
  },
};

setCacheData accepts a JavaScript object.

getCacheData returns the JavaScript object.

[ehmicky]

I like it! Thank you so much @jlengstorf and @sdras for working on it so fast :rocket:, I hope I can get to implement this next week!

I have several questions regarding some specific details, but overall I think it's a great solution to both inter-plugin communication ("outputs" #186 #494) and plugin orchestration (#396)!

Before moving on to those questions, I would like to know: what do you think @DavidWells?

[DavidWells]

Interesting idea. Thanks for the write up.

How can developers ensure that build plugins run in a given order?

Currently developers can ensure that builds run in a particular order by the order in which they are defined in the netlify.toml/yml file and further details of execution order can be seen with netlify build --dry command

How can developers share data between build plugins?

This is the million dollar question 😃. I like Jasons idea of simply returning values and passing those along. There are a couple idea around how this works. Need to strike a balance between user DX & plugin author DX that makes outputs nice to use

---

A couple questions

1. If i'm using an orchestrator plugin, how is it defined in the netlify configuration file?
2. Followup to question 1, the plugins netlify-plugin-1, netlify-plugin-2, & netlify-plugin-3 where are they defined?
3. Because the plugin usage is defined in the code, how would I be able to see the workflow. Would netlify build --dry still print the steps?
4. If the orchestrator plugin is defining things in code, would it still be possible to run things in parallel? If so, what does that syntax look like?

---

Achieving something similar with plugins today.

Below is an example of importing plugins and using them programmatically like shown above.

The main difference here is the dependancies are explicitly included as require statements vs the magic strings approach

const pluginOne = require('plugin-one')
const pluginTwo = require('plugin-two')
const pluginThree = require('plugin-three') // just an object (no config)

module.exports = function orchestratorPlugin(config) {
  // initialize plugins with config (if they have config)
  const one = pluginOne(config)
  const two = pluginTwo(config)

  return {
   onInit: async (ctx) => {
      await one.onInit(ctx)
      await two.onInit(ctx)
      await pluginThree.onInit(ctx)
    },
    onPreBuild: async (ctx) => {
      await one.onPreBuild(ctx)
    }
  }
}

A usage example in netlify config file would look like this:

build:
  publish: build

plugins:
  - package: orchestrator-plugin
    config:
      xyz: true
      otherValue: hello

[jlengstorf]

If i'm using an orchestrator plugin, how is it defined in the netlify configuration file?

This would most likely just be a plugin of plugins, so it would be defined as:

[[plugins]]
  type = "netlify-orchestrator-plugin"

Followup to question 1, the plugins netlify-plugin-1, netlify-plugin-2, & netlify-plugin-3 where are they defined?

I'd recommend resolving them by package name, same as the config file does, so:

context.netlify.pluginExecute('netlify-plugin-one')

is resolved the same way as:

[[plugins]]
  type = "netlify-plugin-one"

Because the plugin usage is defined in the code, how would I be able to see the workflow. Would netlify build --dry still print the steps?

It should. Calling via pluginExecute should be functionally equivalent to adding the plugin to the [[plugins]] config.

If the orchestrator plugin is defining things in code, would it still be possible to run things in parallel? If so, what does that syntax look like?

Definitely!

Here's an example of running two plugins in parallel:

module.exports = {
  name: 'netlify-plugin-orchestrator',
  onInit: async ({ context }) => {
    // these plugins can run in parallel to speed things up
    const [db, files] = await Promise.all([
      context.netlify.pluginExecute('netlify-plugin-db'),
      context.netlify.pluginExecute('netlify-plugin-preprocessor'),
    ]);

    // this plugin needs the previous two to complete first
    await context.netlify.pluginExecute('netlify-plugin-other', { config: { db, files }});
  },
}

---

Your example of programmatically calling plugins by importing them directly is pretty much what I expect would be happening under the hood, but with an explicit API.

The explicit API is important, I think, because it gives us the ability to expand on how this works later — if we need to inject something or sanitize env vars or whatever, we can do that inside pluginExecute — we wouldn't be able to do that if devs are making raw calls to plugin.onInit.

[jlengstorf]

We could always add a little syntactic sugar around the Promise.all stuff, too, if Promises are too head-bendy.

[DavidWells]

Both of the use cases described above can be accomplished in user land with vanilla JS.

Here is the parallel example as it can be written today

const pluginOne = require('plugin-one')
const pluginTwo = require('plugin-two')
const pluginThree = require('plugin-three') // just an object (no config)

module.exports = function orchestratorPlugin(config) {
  // initialize plugins with config (if they have config)
  const one = pluginOne(config)
  const two = pluginTwo(config)

  return {
   onInit: async (ctx) => {
     // these plugins can run in parallel to speed things up
     const [db, files] = await Promise.all([
       one.onInit(ctx),
       two.onInit(ctx),
     ])
     await pluginThree.onInit(ctx, db, files)
    },
  }
}

A usage example in netlify config file would look like this:

build:
  publish: build

plugins:
  - package: orchestrator-plugin
    config:
      xyz: true
      otherValue: hello

---

Because these things can be accomplished in plugin author land already, I think we need to hold off on implementing a bespoke context.pluginExecute function until there is a clear need for this.

If we find ourselves in the position where we need to "inject something or sanitize env vars or whatever" we can revisit this and add to the core API

[jlengstorf]

I think the important thing here is to implement and form an opinion on the pattern. If that doesn't require writing more code, that's great!

However, there are some risks with telling people to handle things themselves:

1. It requires a deeper knowledge of the underlying code that powers build plugins, which increases the onboarding difficulty
2. It relies on the discipline of the developers instead of creating guard rails, which has historically resulted in large, hard to manage messes of userland code
3. It opens the door for unintended uses of plugin code, which means downstream updates will be more painful because people did things we didn't expect them to do, but told them they could do (implicitly, at least) and will therefore need to handle all sorts of unexpected edge cases

The benefits of writing a wrapper that handles plugin execution vs. doing it in userland overcome those challenges:

1. There is a well documented API for running another plugin; no knowledge of how build plugins work to e.g. initialize with config required
2. Writing orchestrators automates discipline because _we_ define what is and is not allowed — we build the guard rails
3. Creating an official API (even if all it does is exactly what @DavidWells wrote in the previous example) means we can make downstream changes and not worry about unsupported use cases as much — if you hack a tool, your hacks require you to make changes to support them — which should decrease our support and maintenance burden going forward

For those reasons, I think a lightweight API is a good idea. It could literally be what @DavidWells is proposing, just wrapped in a function:

function pluginExecute(pluginName) {
  const plugin = require(pluginName);

  const config = getConfigFromSomeInteralThing();
  const context = getContextFromSomeInteralThing();
  const currentLifeCycleMethod = getCurrentLifeCycleMethod();

  const instance = plugin(config);

  return instance[currentLifeCycleMethod](context);
}

This gives us an Officially Supported™ stance and makes it harder to do weird things that makes support suffer later.

[ehmicky]

I would like to echo @jlengstorf comments about having an official wrapper / abstraction layer for this, this makes sense to me.

When it comes to emulating pluginExecute() userland by requiring modules and calling them directly: this would not work for several reasons:

1. The pluginConfig is passed to plugins top-level function, not the netlifyConfig. This means an orchestrator plugin would need to look inside the config to retrieve each relevant pluginConfig (if defined).
2. After loading plugins, their exported object (with the name and event handlers properties) is normalized by our engine. For example, we currently rename build methods to onBuild. We will include more normalization in the future for similar backward compatibility reasons. This normalization would not happen if users are directly calling plugins.
3. Same applies to plugin syntax check: we are currently checking that the plugin syntax is correct, which helps plugin authors with helpful error messages as they are creating their plugin. This would not happen if users are directly calling plugins.
4. Same applies to plugin configuration validation. Plugins can define a config property with the shape of their configuration properties. If user passes an invalid configuration property, validation error messages are thrown. This would not happen if users are directly calling plugins.
5. Plugins can define API scopes to restrict which API methods their api instance can use. This cannot be enforced if plugins are required directly. This is more of a side note though, since this feature is still a WIP.

On the other hand, using an abstraction layer such as pluginExecute() allows our engine to still do this type of logic when plugins are being loaded. From that standpoint, this issue proposal cannot be done userland at the moment.

[DavidWells]

I make a working example of how this works today https://github.com/DavidWells/netlify-build-programatic-plugin-usage

• Only one plugin is defined in config: https://github.com/DavidWells/netlify-build-programatic-plugin-usage/blob/695f75fb751c4328baa5f67651df295f6ce0e1fc/netlify.yml#L6-L10
• The orchestrate plugin runs the plugin methods (in this case onInit but it could be any of them or custom non-lifecycle functionality) https://github.com/DavidWells/netlify-build-programatic-plugin-usage/blob/695f75fb751c4328baa5f67651df295f6ce0e1fc/plugins/orchestrate/index.js#L5

You can see the log output here:
https://app.netlify.com/sites/thirsty-rosalind-0029cf/deploys/5e151c053e4e700008f2b399

4:03:02 PM: ┌───────────────────────────────────────────────────────────┐
4:03:02 PM: │ 1. Running onInit command from netlify-plugin-orchestrate │
4:03:02 PM: └───────────────────────────────────────────────────────────┘
4:03:02 PM: 
4:03:02 PM: 1. Plugin one with conf.optionOne = foo
4:03:02 PM: 1. return databaseString mongo:ddbdbdbdbdbdbd:foo
4:03:02 PM: 2. Plugin two with conf.other = bar
4:03:02 PM: 2. return file [ 'bar.html', 'abc.js', '123.md', '789.css' ]
4:03:02 PM: 3. Plugin three onInit
4:03:02 PM: 3. database mongo:ddbdbdbdbdbdbd:foo
4:03:02 PM: 3. files [ 'bar.html', 'abc.js', '123.md', '789.css' ]
4:03:02 PM: Do Stuff in plugin 3!
4:03:02 PM: plugin-orchestrate: Output from plugin 3 = {"other":"values","here":"⊂◉‿◉つ"}

---

One outstanding question is how scopes can operate mentioned above in #5 by @ehmicky.

• One way it might work is making orchestrate plugin here define the correct scopes it's children need.
• Another is removing scopes & allowing plugins to operate on a subset of API endpoints. Scopes while important often are the bane of a smooth DX.

---

It's possible that we will have a utility that will make this easier. (syntax sugar)

The main question around that is what is the correct abstraction. The proposal above limits which methods can be invoked (e.g. context.netlify.pluginExecute('netlify-plugin-db') assuming it should run onInit when the developer really needs to invoke another method).

[jlengstorf]

The main question around that is what is the correct abstraction. The proposal above limits which methods can be invoked (e.g. context.netlify.pluginExecute('netlify-plugin-db') assuming it should run onInit when the developer really needs to invoke another method).

The pluginExecute should run whatever lifecycle it's called from by default. In the initial spec, we also proposed a config option that would allow you to specify which lifecycle method would be run:

pluginExecute(
  pluginName,   // string — the package name of the plugin to execute
  {
    lifecycle,  // array — which lifecycle methods to execute
                //   defaults to the lifecycle where the method is called
    config,     // object — any data to make available to the plugin
  }
)

I know I'm repeating myself a lot here, but telling people to handle stuff like this in userland is a footgun that's going to cost our support team a HUGE amount of time as the ecosystem scales. A clear API boundary lets us control that surface area and give Support a clear "we support this, but not that" definition to protect their time and stress levels.

I would advocate for either building the utility or, if we're going to leave this to userland, not talking about it _at all_ and letting userland work it out on its own.

[DavidWells]

Interesting. Trying to understand the utility shape & how it might be used.

Is this what you mean?

function netlifyPlugin(conf) {
  return {
    name: 'my-plugin-one',
    onInit: async ({ utils }) => {
      const output = await utils.pluginExecute('other-plugin', {
        lifecycle: ['onBuild', 'onPostBuild'],
        config: {
          foo: 'bar',
          baz: 'zaz'
        }
      })
    },
  }
}

If yes, a couple clarifying questions.

1. What does the full corresponding configuration (netlify.yml/toml) look like?
2. What is the return value from utils.pluginExecute that is set to const output if multiple lifecycle methods are run.
3. Is this example of config correct? e.g. is it the top level plugin config?
4. Is other-plugin also somehow defined in netlify.yml/toml file? (related to question 1)
5. If I wanted to execute a local plugin via the string, what might that look like? utils.pluginExecute('../path/to/it/local-plugin-xyz')?
6. If lifecycle is defined and it's multiple lifecycles, do the methods run in parallel?
7. Is the pluginExecute utility on the utils object by default or do users need to npm install @netlify/plugin-utils as a dependency to use utils.pluginExecute? ref

[jlengstorf]

What does the full corresponding configuration (netlify.yml/toml) look like?

I don't think we should allow complex config via TOML/YML — you would install the orchestrator and any sub-config would need to be passed in the orchestrator config

What is the return value from utils.pluginExecute that is set to const output if multiple lifecycle methods are run.

{
  onBuild: onBuildResult,
  onPostBuild: onPostBuildResult,
}

Is this example of config correct? e.g. is it the top level plugin config?

that seems reasonable. as the API gets built out, it may need to include additional meta-config, but I don't really know what that might require because I'm not in the code

Is other-plugin also somehow defined in netlify.yml/toml file? (related to question 1)

I don't think so; this would be like an npm package that uses lodash — internals are abstracted away for convenience

If I wanted to execute a local plugin via the string, what might that look like? utils.pluginExecute('../path/to/it/local-plugin-xyz')?

that could work, or we could define an explicit API for loading them:

const myLocalPlugin = utils.executeLocalPlugin('./path/to/plugin')

If lifecycle is defined and it's multiple lifecycles, do the methods run in parallel?

I don't see how they could — build plugins are run sequentially, so we should assume that a later lifecycle might depend on the output of a previous one

Is the pluginExecute utility on the utils object by default or do users need to npm install @netlify/plugin-utils as a dependency to use utils.pluginExecute? ref

would it cost us anything to make it available by default? if not, I don't know if there's any value to adding steps

[DavidWells]

Thanks for this!

So to summarize, If I understand the comment above correctly:

The API you are suggesting is 2 utils pluginExecute & executeLocalPlugin. These utils would be included in core with no additional installation required

function netlifyPlugin(conf) {
  return {
    name: 'my-plugin-one',
    onInit: async ({ utils }) => {
      const { onBuild, onPostBuild } = await utils.pluginExecute('other-plugin', {
        lifecycle: ['onBuild', 'onPostBuild'], // <== These run in order of lifecycle & not in parallel
        config: {
          foo: 'bar',
          baz: 'zaz'
        }
      })
      console.log('onBuildOutput', onBuild)
      console.log('onPostBuildOutput', onPostBuild)

      // Lifecycle option omitted, default to `onInit` 
      const { onInit } = await utils.pluginExecute('other-plugin', {
        config: {
          foo: 'bar',
          baz: 'zaz'
        }
      })
      console.log('onInitOutput', onInit)

      // Local plugins are executed similarly but with executeLocalPlugin
      const { 
        onBuild: onBuildOutput, 
        onPostBuild: onPostBuildOutput 
      } = await utils.executeLocalPlugin('./path/to/plugin-abc', {
        lifecycle: ['onBuild', 'onPostBuild'], // <== These run in order of lifecycle & not in parallel
        config: {
          zap: 'boom',
          pow: 'bam'
        }
      })
      console.log('onBuildOutput', onBuildOutput)
      console.log('onBuildOutput', onBuildOutput)

      // Lifecycle option omitted, default to `onInit` 
      const { onInit: pluginThreeOutput } = await utils.executeLocalPlugin('./path/to/plugin-3', {
        config: {
          foo: 'bar',
          baz: 'zaz'
        }
      })
      console.log('pluginThreeOutput', pluginThreeOutput)
    },
  }
}

And the corresponding config would look like:

build:
  functions: functions
  publish: build

plugins:
  - package: my-plugin-one
    config:
      hi: foo

Does the above code look correct?

---

I don't think we should allow complex config via TOML/YML — you would install the orchestrator and any sub-config would need to be passed in the orchestrator config

I agree, defining the single plugin (that maybe under the hood is using other things) in the config is the way to go for simplicity.

it may need to include additional meta-config

Interesting, what do you mean by meta-config?

[jlengstorf]

this all makes sense, though now I'm wondering if naming should line up better:

• executePlugin
• executeLocalPlugin

or:

• pluginExecute
• pluginExecuteLocal

(I think I prefer plugin as the util prefix — could also sub-config this as utils.plugins.execute/utils.plugin.executeLocal)

what do you mean by meta-config?

one example could be consolidating the lifecycles (which are what I mean by meta-config):

whateverWeNameThePluginExecutionHelper('plugin-name', {
  data: {
    thing: 'to pass to the plugin conf',
  },
  lifecycles: [ 'onInit', 'onSuccess' ],
  // per our discussion today — this would need to evolve with the design of compat management
  buildBotVersion: '^1.2.17',
})

I'm not sure if there are other things that plugins can configure via YML/TOML, so this would be a questions for @ehmicky and @RaeesBhatti RE what meta-config might be required.

[ehmicky]

I like the proposal. :+1:

The netlify.yml package property distinguishes between local paths and Node modules using the Node.js module resolution algorithm (with the resolve library). There might probably be no need to introduce separate methods since we can determine from the argument whether the user intends to use a Node module or a local path.

To answer your question about the meta-config: at the moment this is the full syntax when loading plugins in YAML:

plugins:
  # Only `package` is required. It can be anything that works with `require()` including a Node module name and a local file path
  - package: netlify-plugin-example
    id: example
    enabled: false
    config:
      foo: bar

I don't think the example netlify-build-programatic-plugin-usage is working, as it is not addressing the issues raised above:

1. pluginThree.onInit(context, db, files) is incorrect since context.pluginConfig is the configuration of netlify-plugin-orchestrate instead of the configuration of the plugin three.
2. This does not work if the plugin three method is called init instead of onInit. When plugins are loaded, we normalize them, which includes backward compatibility fixes such as when we renamed init to onInit. By calling plugins directly, users lose this feature.
3. The plugin syntax is not validated. Normally, when a plugin is loaded but its shape is incorrect, we print helpful error messages. By calling plugins directly, users lose this feature.
4. The context.pluginConfig passed to plugin three is not validated. Plugins can define a config property with the shape of their configuration properties. If user passes an invalid configuration property, validation error messages are thrown. This would not happen if users are directly calling plugins.
5. The context.api passed to plugin three is incorrect, as it uses the scopes of netlify-plugin-orchestrate instead of the scopes of plugin three.

Overall, we need some abstraction layer to keep providing all those features. Direct calls cannot provide with these since they lack indirection.

I like @jlengstorf proposal. I have few minor questions but I think it's more important to address @DavidWells questions first as they are more high-level.

One question I had was about dry runs. @DavidWells did mention it and you answered it, but I am still wondering how this would work. To do a dry run with the current proposal, in order to determine which plugins gets called and in which order, we would need to trigger both the orchestrator plugin and the plugins called by that orchestrator. Doing so would mean it's not a dry run anymore? Did you have an idea on how to fix this specific issue @jlengstorf?

[jlengstorf]

we would need to trigger both the orchestrator plugin and the plugins called by that orchestrator.

I think I'd argue that an orchestrator plugin is a full abstraction of the underlying plugins, so dry runs wouldn't show underlying plugins being used — they're an implementation detail at that point

like, I don't expect the dry run to tell me that a plugin imports lodash or axios — an orchestrator turns plugins into underlying libs to accomplish a higher-level goal, so I feel like there doesn't need to be visibility under the hood here