Build: Plugin Output Proposal

[DavidWells]

Plugin outputs are a way for values to be passed between plugins.

These outputs will allow for more advances use cases of plugins that depend on values from each other.

For example, a mongodb-atlas-plugin might pass a connection string back as an output that a custom-rest-api-function-plugin might use to automatically scaffold a serverless function with said connection string.

To achieve a feature like this, Netlify Build requires additional configuration from the plugin author so outputs and build order can be correctly executed.

Considerations

Plugin outputs must work in these 2 scenarios:

1. Plugins defined in netlify config file
2. Plugins being programmatically invoked ("orchestrator plugins")

Below is a proposal on how these two can work.

1. Plugins defined in netlify config file

Here we use 2 plugins. pluginOne has required inputs and pluginTwo's output values are used in pluginOne.

// Has required config property module.exports = function pluginOne(pluginConfig) { return { // Name of plugin name: 'plugin-one', // Config needed for plugin config: { biz: { type: 'string', required: true, } }, // Lifecycle functions to run onBuild: ({ pluginConfig }) => { console.log(pluginConfig.biz) }, } } // Exposes 1 output during onPreBuild phase module.exports = function pluginTwo(pluginConfig) { return { // Name of plugin name: 'plugin-two', // Outputs returned for DAG resolution outputs: { foo: { type: 'string', // `when` is important for DAG resolution when: 'onPreBuild' } }, // Lifecycle functions to run onPreBuild: (pluginAPI) => { // Output value `foo` is returned from onPreBuild phase return { foo: 'hello' } }, } }

# Netlify Config build: publish: build functions: functions plugins: - package: plugin-one config: biz: ${pluginTwo.outputs.foo} # ^ output referenced from `id` # given to plugin-two - package: plugin-two id: pluginTwo As an aside, the configuration could also be ordered like so: # Netlify Config build: publish: build functions: functions plugins: - package: plugin-two id: pluginTwo - package: plugin-one config: biz: ${pluginTwo.outputs.foo}

What happens on build:

Inputs & outputs dependancies are ordered up front via a DAG and cycles throw an error.

1. Required inputs validated in plugin-one. Output syntax is recognized, the when value is read & lifecycle is checked to verify ordering works.
2. Order works, so onPreBuild functionality from plugin-two runs
3. Then output foo from plugin-two is returned. ${pluginTwo.outputs.foo} then fully resolved to hello.
4. onBuild functionality from plugin-one runs with it's biz config value set to hello
5. Then build ends

---

2. Plugins used programmatically

Here we use 3 plugins. The plugin-one & plugin-two are programmatically used in a third plugin called orchestratorPlugin. Because they are programmatically used, the order of lifecycle methods in plugin-one & plugin-two do not matter. Effectively they are used as normal NPM modules.

Programmatic usage from orchestrator plugin. Because these are used programmatically and are not defined in the Netlify config file, they can be called in any order the user wishes. const pluginOne = require('plugin-one') const pluginTwo = require('plugin-two') module.exports = function orchestratorPlugin(config) { return { // Name of plugin name: 'orchestrator-plugin', // Config needed for plugin config: { optOne: { type: 'string', required: true, }, optTwo: { type: 'string', required: true, } }, // Lifecycle functions to run onInit: async ({ pluginConfig }) => { // initialize plugins with config const one = pluginOne({ biz: pluginConfig.optOne }) const two = pluginTwo({ zaz: pluginConfig.optTwo }) const [outputFromOne, outputFromTwo] = await Promise.all([ one.onPreBuild(pluginAPI), two.onBuild(pluginAPI), ]) // Do custom stuff with outputFromOne / outputFromTwo }, } } module.exports = function pluginOne(pluginConfig) { return { // Name of plugin name: 'plugin-one', // Config needed for plugin config: { biz: { type: 'string', required: true, } }, // Outputs returned for DAG resolution outputs: { wow: { type: 'string', when: 'onBuild' } }, onPreBuild: ({ pluginConfig }) => { console.log(pluginConfig.biz) return { wow: 'nice' } }, } } // Has required config property module.exports = function pluginTwo(pluginConfig) { return { // Name of plugin name: 'plugin-two', // Config needed for plugin config: { zaz: { type: 'string', required: true, } }, // outputs returned for DAG resolution outputs: { wow: { type: 'string', when: 'onBuild' } }, onBuild: ({ pluginConfig }) => { console.log(pluginConfig.zaz) }, } }

build: publish: build plugins: - package: orchestrator-plugin config: optOne: hello optTwo: goodbye

---

What happens on build:

Because they are programmatically used, the order of lifecycle methods in plugin-one & plugin-two do not matter.

1. Required inputs validated from read config file
2. orchestrator-plugin loads & onInit functionality runs with config set to optOne: hello & optTwo: goodbye
3. plugin-one & plugin-two are initialized with config
4. Then plugin-one.onPreBuild & plugin-two.onBuild methods are called
5. Then output from plugin-one.onPreBuild & plugin-two.onBuild are referenced in the code of orchestrator-plugin.
6. Then the build ends

Implementation Proposal

The way values are resolved and lifecycle events are ordered depend on the inputs & outputs of each event listener.

There are probably multiple ways to implement something like this. It would be important to note how mature tools like terraform & cloudformation use DAG as their mechanism for resolving & ordering operations.

I'm proposing we use a DAG algorithm to resolve the order in which plugin lifecycle methods fire.

The DAG will tell us the order in which the build steps should happen. It will also be able to detect cycles and throw an error if plugin outputs don't exist yet where they are being used as plugin config (inputs).

The DAG implementation really only effects plugins that are defined in the Netlify configuration file vs the plugins that are called programmatically (because they are just function calls & resolve themselves in user code)

Proposed Resolution flow:

Related issues

• older outputs example https://github.com/netlify/build/pull/494
• older syntax discussion https://github.com/netlify/build/issues/396
• orchestrator plugins https://github.com/netlify/build/issues/603
• https://gist.github.com/DavidWells/d0d0621a3f49c6a0a17aa6d6963de10b

---

Feedback and comments welcome!

Alternative suggestions on how to get this working would also be interesting to see.

[jlengstorf]

I like the programmatic approach, though I'll echo my concerns from https://github.com/netlify/build/issues/603#issuecomment-566202411 about using plain JS vs. a well-defined API

reading this, I wonder if there's a simpler, safer route: what if we just required plugins that rely on outputs to use the JS approach?

this would skip the requirement for additional config and API surface and have the benefit of encouraging developers to build workflow-centric plugins that limit complexity for the vast majority of end-users who just want to install things, not orchestrate them

[DavidWells]

reading this, I wonder if there's a simpler, safer route: what if we just required plugins that rely on outputs to use the JS approach?

That would be easier. We could achieve this now in userland. Perhaps we should promote this approach first as we iron on further details here.

/* Plugin 1 file */
// Require plugin two and use programatically
const pluginTwo = require('plugin-two')

module.exports = function netlifyPlugin() {
  return {
    name: 'netlify-plugin-xyz',
    onPreDeploy: async (pluginAPI) => {
      const two = pluginTwo({ foo: 'bar' })
      // get output value
      const output = await two.build(pluginAPI)
      // do stuff with output.pokemon value === 'yay'
    }
  }
}

/* Plugin 2 file */
module.exports = function netlifyPluginTwo() {
  return {
    name: 'plugin-two',
    build: async () => {
      // custom logic to do stuff and return data
      return {
        pokemon: 'yay'
      }
    }
  }
}

---

It will be very important for output values to be referencable once addons are also definable in the Netlify configuration file.

It is also critical that we consider this as an implementation detail that we will need to support. E.g. resolving inputs/outputs in the correct order. This is why the DAG is suggested. It's a tried and true algorithm used by many provisioning systems including CloudFormation, ARM, serverless & terraform.

[jlengstorf]

I'm still _really_ nervous about encouraging people to import plugins inside their plugins. I've never seen that go well from a maintenance standpoint. Providing a clear API gives us a way to say "we support this, but not that" — telling people to import plugins makes that line blurrier and harder to communicate.

[DavidWells]

^ Adding link to import plugins issue for reference https://github.com/netlify/build/issues/603

---

@jlengstorf what issues have you seen from the maintenance standpoint?

[jlengstorf]

if there's no clear delineation between "supported" use cases and unsupported use cases, then _every_ question will be sent to support, and it's really hard to define where we get to say, "That's not our problem."

(and even if we _do_ say that, if there's no clear boundary people will be salty because they have no way of knowing what is and isn't supported)

people will start to make up their own patterns, and we'll end up with plugins that depend on plugins that depend on plugins with no way to trace or debug the relationships, and our build pipeline has the potential to fall down at random if some dependency somewhere in the chain changes without telling the topmost plugin what happened

the idea of userland code being able to break our core offering at random is _really_ concerning — unless we're ready to go manually test and update every build plugin whenever we make changes to the buildbot, unchecked userland implementations may break with every patch bump because they might be using something in a way we've never considered

if we say, "You can use this API in these ways, and everything else is considered a hack/monkey patch and may break at any time," then we're safe — we can test our changes against the exposed APIs and if the tests pass, we're good (though I'd argue that a broken pipeline is going to damage trust in Netlify regardless of whose fault it is)

if we say, "Import plugins and do whatever you want with them," we're sending the message that this is a safe and officially supported way to do things — how do we honor our SLAs without testing every change we make against every plugin in the ecosystem? if enterprise users follow our "do whatever you want" directive and do something enterprisey and horrible that they never intend to change, are we now on the hook to maintain backward compatibility to their horrible hack for the rest of time? (IBM 100% has to do this — there are multiple forks of the codebase to maintain enterprise backward compat and it's a nightmare)

Build Plugins are going to be seen as part of Netlify's build pipeline for all intents and purposes, so if we let people ship weird code that we don't know how to test or support, we run the risk of lots of people adopting something that we can't guarantee works, and that can erode trust in our core product — that's a _huge_ risk for us to take (e.g. "Netlify sucks! the builds just randomly break!" because they have some third-party Build Plugin, but no one will know or care that Netlify had nothing to do with the breakage)

creating a narrow, officially supported path gives us room to add more features later as people tell us what they need that our stuff can't support

letting people do whatever they want means we have to maintain compatibility with whatever anyone decides to do — or risk eroding public trust in our core product

does that make sense?

[DavidWells]

This does make sense. Thanks for the response @jlengstorf

the idea of userland code being able to break our core offering at random is really concerning

I agree! That would be a big problem. For this reason the core plugin API must remain stable and only additive changes can be made. We have seen this issue already in the early beta & moving forward toward a v1.0.0. we are only making backwards compatible changes.

Where I'm not following you is how user land code & using plugins (described here) would break our core offering.

Plugins are either versioned dependancies installed from npm or locally control via source control. In this sense, the code should work forever™ unless a core breaking API change from our end is introduced or the dependency is upgraded & has a regression. This is true regardless of the implementation (this example or via a util).

In that programatic usage example, lets pretend they are installed from NPM at a specific version. The user land code and the plugin code they are using is pegged to a specific version and normal dependency management rules apply (semver etc). As long as we maintain forward API compatibility, that javascript code should run as normal.

I appreciate the insights & experience you are bringing to the table. It's very clear, we have an obligation to publish a v1 API and never break this contract with developers & plugin authors.

[jlengstorf]

Where I'm not following you is how user land code & using plugins (described here) would break our core offering.

my caching plugin had a bug (something changed upstream, but it could have just as easily been that I did something wrong and published it), and my builds broke

pegged to a specific version

this would work, but the vast majority of devs (myself included) just upgrade deps when new versions are released because I assume that the maintainers wouldn't ship breaking changes unless they bump the major release — I only think about versions when things break, and at that point it's too late

I think we have to assume that developers aren't going to be vigilant; the research Marisa shared shows that developers are actively disinterested in the code behind build plugins and are looking for things that save them time and eliminate mental overhead — they're looking for shortcuts

for a more practical example of how this can go wrong, Gatsby shipped early plugins that didn't follow a prescribed API path: sharp, the filesystem plugin, etc.

these plugins did weird userland things that aren't sanctioned or encouraged by the Gatsby team (they were also built by the Gatsby team to get something shipped and avoid designing the API for safety/resilience) and as a result they are a consistent source of bugs, confusion, and migration headaches because they require special handling

even as core plugins, "userland" code causes maintenance problems, and that complexity compounds — my guess is that Gatsby v3 is going to break a HUGE amount of things to correct for these decisions, and it'll be painful and create a ton of work and churn that could have been avoided by designing and enforcing good APIs up front

ultimately, I think my concern boils down to this: we can always expand what we expose if we start with a super restricted, controlled API — each new thing we expose would be an exciting feature release; we _can't_ contract what we expose if we tell people that plugins can be used however they want without causing strain on our customers, Engineering, and Support — each problem we retroactively guard for will break someone's workflow and expose us to churn and public blowback

[DavidWells]

my caching plugin had a bug (something changed upstream, but it could have just as easily been that I did something wrong and published it), and my builds broke

This was a core breaking change and a regression. Again apologies for this, an oversight on the release process we are working on. This break would effect all plugins regardless of how they are used (via code or via util).

this would work, but the vast majority of devs (myself included) just upgrade deps when new versions are released because I assume that the maintainers wouldn't ship breaking changes unless they bump the major release — I only think about versions when things break, and at that point it's too late

From what I can tell, this isn't really something we can prevent. This is true of all dependancies plugins & frontend/backend functions deps. If versions get bumped & the author of a dependency broke semver, things will break.

Can you help me understand how this might be different if a utility was introduced?

we can't contract what we expose

I agree with this, this is actually my main hesitation around introducing this as a feature when it's possible for the same use case described can be achieved without this utility.

Side note: This is getting off topic from the outputs proposal described here. Can we call about the utility in https://github.com/netlify/build/issues/603. There are some additional questions on how the proposed util would work.

[jlengstorf]

From what I can tell, this isn't really something we can prevent.

Yeah, this is kind of getting at a core concern I have about build plugins in general. If we can't _easily_ trace what's happening in every build plugin and let people know which one broke and why in extreme detail, people will think Netlify is broken if they install a junk build plugin.

It won't matter that they're wrong or that it's not our fault, because public perception will be a tweetstorm about Netlify being a shitty product because X or Y broke. This is a core problem at Gatsby right now: people install a dozen userland plugins, their site won't build, and they open issues on the Gatsby repo and @ Gatsby on Twitter because — as far as they're concerned — Gatsby is broken.

This lack of quality control is also a huge contributing factor in why WordPress is such a maintenance nightmare.

Can you help me understand how this might be different if a utility was introduced?

Good tracing, extremely clear boundaries, and excellent reporting helps us mitigate the splash damage of bad userland code.

"Build failed with non-zero exit code" seems like our fault.

"Error: the build plugin netlify-plugin-garbage failed. We have disabled this plugin to prevent your build from breaking. See below for the original error message" shows that Netlify is fine and the plugin is broken. A well-defined API lets us capture those errors, return them in a usable format, complete the build as if the plugin wasn't installed, and add a "Published, but with plugin errors" (and maybe optional email alerts, etc.) that surfaces that things worked, but probably not the way they expected.

We could potentially do this without an API by wrapping everything in try/catch blocks, but we lose e.g. structured logging with that approach.

it's possible for the same use case described can be achieved without this utility.

Exposing a wild west approach is committing Support to handle every question because there's no way to figure out _why_ something broke. See above for why.

This is getting off topic from the outputs proposal described here.

I'm not sure I agree — we should tightly control how plugins communicate to each other. I've responded on #603 as well, though.

[ehmicky]

An implementation of the 1. Plugins defined in netlify config file feature is in progress at #720.

[ehmicky]

This is a summary of our latest discussion on the design of this feature.

Producing outputs

Outputs are returned as a plain object from plugin event handlers.

module.exports = {
  name: 'netlify-plugin-producer',
  onInit() {
    return { foo: 'bar' }
  }
}

Consuming outputs (configuration file)

Outputs are consumed by using template variables in the configuration file.

plugins:
  - package: netlify-plugin-producer
    id: example
  - package: netlify-plugin-consumer
    config:
      url: ${outputs:example.foo}

In ${outputs:example.foo}, example is the plugin id and foo the output name.

foo can use a dot notation for nested variables (e.g. ${outputs:example.foo.prop}).

The ${outputs:...} template variable can only be used inside plugins[*].config (like in the example above) or inside lifecycle shell commands (like in the example below).

plugins:
  - package: netlify-plugin-producer
    id: example
build:
  lifecycle:
    onBuild: echo ${outputs:example.foo}

Consuming outputs (plugins)

As any other plugin configuration property, the template variable value is available inside pluginConfig.

module.exports = {
  name: 'netlify-plugin-consumer',
  onBuild({ pluginConfig: { url } }) {
    console.log(url)
  }
}

Declaring produced outputs

Outputs must be declared by producers using the outputs plugin property.

module.exports = {
  name: 'netlify-plugin-producer',
  outputs: {
    foo: { when: 'onInit' }
  },
  onInit() {
    return { foo: 'bar' }
  }
}

The value of when must match the first event handler (onInit) that produces the specific output.

Declaring consumed outputs

Outputs must be declared by consumers using the when plugin property inside config.

module.exports = {
  name: 'netlify-plugin-consumer',
  config: {
    properties: {
      url: { when: 'onBuild' }
    }
  },
  onBuild({ pluginConfig: { url } }) {
    console.log(url)
  }
}

To make the syntax more similar to the outputs syntax, we have the following two suggestions tracked in separate issues:

• rename config to inputs: #724
• rename config.properties.* to simply config.*: #746

Configuration validation

Configuration properties can be validated using the config plugin property.

module.exports = {
  name: 'netlify-plugin-consumer',
  config: {
    properties: {
      url: { type: 'string', minLength: 4 }
    }
  }
}

When using template variables, the value of configuration properties is not known when the build starts, so, when a template variable is used, we will need to delay configuration property validation to the moment the output is produced.

Top-level function

Plugin outputs can conflict with top-level plugin functions.

module.exports = function(pluginConfig) {
  if (pluginConfig.checked) {
    return { name: 'netlify-plugin-consumer', ... }
  } else {
    return { name: 'netlify-plugin-consumer', ... }
  }
}

plugins:
  - package: netlify-plugin-producer
    id: example
  - package: netlify-plugin-consumer
    config: 
      checked: ${outputs:example.foo}

In the example above, the value of pluginConfig.checked will not be resolved yet when the top-level plugin function is called. All we have is the unresolved template variable ${outputs:example.foo}. Its usage is basically erroneous here and might provide with hard-to-debug error messages or issues.

I cannot find a perfect solution for this. The best I can come up with at the moment is any of:
1) Remove top-level functions altogether
2) Use a Proxy to detect which properties of pluginConfig are accessed in the top-level plugin function, and throw if they are accessed but a template variable ${outputs:...} was used
3) Do not pass pluginConfig to the top-level plugin function

If you have any other ideas @DavidWells @RaeesBhatti, please let me know.

Plugins graph

A Directed Acyclic Graph (DAG) can be built from connecting consumed outputs (config) with produced outputs (outputs). This allows for the following:

• Ensuring event handlers are run in order. For example, if there are two plugins with the same event handler, but one is consuming outputs from the other, it should be run last.
• Preventing users from consuming outputs in the wrong order. For example, if a plugin is consuming inside an onInit event handler an outputs produced from an onBuild event handler, we need to throw an error since onInit events always run before onBuild events.
• Figuring out the order in which event handlers are run statically, i.e. without running the build steps. This is important for dry runs (--dry CLI flag).

To ensure this we will need to throw an error when a plugin:

• returns an output in an event handler without declaring it in outputs
• returns no output even though one is declared in outputs

Requiring config

To be able to build the DAG, we need plugins to declare when configuration properties are first used with the when property. If we don't have that information, we cannot build the DAG.

I see two possible solutions:
1) Default to the earliest event which has an event handler defined in the plugin. For example, if the plugin has onBuild and onInit handlers, this would be onInit. The drawback here is that if onBuild was actually the method consuming the output, it will only be able to consume it an output produced by another plugin's onInit method (where technically anything before onBuild should have been fine)
2) Require when to be defined for plugins with more than one event handler. The majority of plugins so far have only one event handler.

Outputs typing

We discussed about typing and validating outputs, but decided to defer this to the next iteration on this feature.

Moving forward

@DavidWells @RaeesBhatti Is this correctly describing what we've discussed? If not, please me know.

Also, to implement this I would need feedback on the points above:

• "top-level function"
• "Requiring config"

Please let me know what you think.

[DavidWells]

This looks good to me. Many thanks for writing this up 🎉

A couple notes:

Config syntax

The config syntax should look like this, described in that issue is how we can have a simpler config AND support advanced JSON schema (in the future). Also included there are the values needed including when as you have described here.

"top-level function"

We should have a rule: "Outputs cannot be used in top level functions & any value used in a top level function should be resolved BEFORE the build initiates."

"Requiring config"

Yeah we should require plugins to define their configuration inputs. This needs to be handled delicately to not immediately break beta people.

---

Getting the inputs (config) and the outputs defined like this will give us a much better way to:

• Automatically generate documentation for plugins
• Incorporate a point/click experience in the UI
• help enforce nice validation / DX warning if users misconfigure something
• & make much more robust use cases possible (without hard forking plugins to achieve things)

Excited to see this move forward 😃

[ehmicky]

The config syntax should look like this, described in that issue is how we can have a simpler config AND support advanced JSON schema (in the future). Also included there are the values needed including when as you have described here.

Tracked in #724 and #746.

We should have a rule: "Outputs cannot be used in top level functions & any value used in a top level function should be resolved BEFORE the build initiates."

One way to do that would be:

1. Use a Proxy to detect which properties of pluginConfig are accessed in the top-level plugin function, and throw if they are accessed but a template variable ${outputs:...} was used.

Please let me know if you have any another idea.

Yeah we should require plugins to define their configuration inputs. This needs to be handled delicately to not immediately break beta people.

My summary included two possible solutions.

1. Default to the earliest event which has an event handler defined in the plugin. For example, if the plugin has onBuild and onInit handlers, this would be onInit. The drawback here is that if onBuild was actually the method consuming the output, it will only be able to consume it an output produced by another plugin's onInit method (where technically anything before onBuild should have been fine)
2. Require when to be defined for plugins with more than one event handler. The majority of plugins so far have only one event handler.

Please let me know if you have any another idea there too.

[DavidWells]

Related input keys https://github.com/netlify/build/issues/821 & the when field

[verythorough]

@ehmicky What's the current status on this issue?

[ehmicky]

There was lots of back and forth between different proposals related to this feature. We finally decided to postpone this feature for the time being, or until it was requested by users.
@erquhart might also have some input on this.

[erquhart]

We can address the entire concern of plugin outputs when and if it's determined they're necessary. As mentioned in #1177 there are already ways to accomplish inter-plugin communication. We can close and start fresh if this is revisited in the future.