Build: Run plugins in parallel

[ehmicky]

Build speed is very important to our users. Should be run plugins in parallel by default?

The following would still happen serially:

• lifecycle hooks, e.g. anything triggered in build always happen before anything in buildFunctions
• when user specifies an array of commands, those would be run in order

However, for a specific lifecycle hook, everything else should be run in parallel.

One downside would be that, by default, plugins would only be able to depend on other plugins that are defined in previous lifecycle hooks. We could introduce a plugin property after to overcome this.

Logs would need to be buffered so that they are displayed to the user as if the plugins had be run serially.

This would provide with a major speed boost.

[sw-yx]

oof. speed is important, but predictable and debuggable execution is more important.

suggest offering a function tag to indicate parallelism eg parallel([plugin1, plugin2]) and that itself runs as a plugin

[ehmicky]

Yes @sw-yx you're right, running plugins in parallel by default is probably a bad idea, for the reasons you mentioned. :(

What about this: plugins run serially unless the plugin.parallel boolean option is set to true. This can be set either by the user (in netlify.yml plugins.pluginName.parallel) or by the plugin author (in the returned plugin object), with the user having priority. When true, this plugin would run in parallel with the other plugins of the same lifecycle hook.

Orchestration and series/parallel order can be a little mind-bending. I think following we should make it as simple as possible to the user. A boolean option is very easy to understand.

What do you think?

[sw-yx]

you’ll find i’m usually not a fan of adding boolean flags :) bc it leaves no room to evolve api if we need a third behavior, plus is hard to manage combinations of booleans unless they are truly orthogonal (which requires a lot of foresight)

wdyt about the “plugin parallelizer” function i proposed above? it takes an array of plugins, but is itself a plugin, and when it is invoked runs its array of plugins in parallel.

[ehmicky]

The boolean flag could be a two-value string option too, that works as well. Although in that case it does look like there won't be a third behavior.

Could you provide an example of what you meant? I did not see how this would look in the netlify.yml or for plugin authors.

[DavidWells]

There are a couple cases to consider:

• Plugin authors might want all lifecycle methods to run in parallel by default
• Plugin authors might want specific (not all) lifecycle methods to run in parallel by default
• Consumers might want to force all lifecycle methods of a given plugin to run in parallel
• Consumers might want to force specific (not all) lifecycle methods of a given plugin to run in parallel

There will likely need to be 2 ways parallelism is defined

1. First and foremost by the plugin author in the plugin itself
2. Overrides from the consumer (from netlify.yml)

Need to ponder how this looks.

Here are some ideas:

Plugin author defining single parallel lifecycle:

module.exports = function exampleOne() {
  return {
    init: () => {
      console.log('I am normal blocking lifecycle')
    },
    postbuild: {
      parrallel: true,
      run: () => {
        console.log('I run and dont block other things!')
      }
    },
  }
}

Plugin author defining all lifecycle method parallel:

(this example is less verbose but perhaps more confusing b/c its not explicit / easy to see everything is parallel)

module.exports = function exampleEverythingParallel() {
  return {
    parrallel: true,
    init: () => {
      console.log('I am normal blocking lifecycle')
    },
    postbuild: () => {
      console.log('I run and dont block other things!')
    }
  }
}

...

[sw-yx]

ohhh. i didnt think about how it would look in the yml. yeah then a flag would work 😂

[ehmicky]

@DavidWells this sounds good. What about merging those two syntaxes:

module.exports = function exampleEverythingParallel() {
  return {
    // This defaults to an empty array
    parallel: ['init', 'postbuild'],

    init: () => {
      console.log('I am normal blocking lifecycle')
    },
    postbuild: () => {
      console.log('I run and dont block other things!')
    }
  }
}

Like this, hooks can remain simple functions?

1. Overrides from the consumer (from netlify.yml)

What about the following?

plugins:
  pluginName:
    enabled: true
    parallel: true # This defaults to whatever the plugin author has set

[DavidWells]

^ this looks good. I like having the same syntax in both spots.

Code

module.exports = function exampleEverythingParallel() {
  return {
    parallel: ['init'],  // only init is parallel, postBuild is sync
    init: () => {
      console.log('I dont block other things!')
    },
    postBuild: () => {
      console.log('I block other things!')
    }
  }
}

YAML

Setting individual things as parallel

plugins:
  pluginName:
    parallel: ['init'] # // only init is parallel, other lifecycle hooks are whatever their default is

Setting everything things as parallel via true

plugins:
  pluginName:
    parallel: true # // if true make all lifecycle hooks parallel

[ehmicky]

I like it! :+1:

A follow-up question: logs of parallel plugins should be shown in serial and predictable order (as opposed to interleaved). To do that, they have to be buffered. To be buffered, plugins should be run in child processes.

Running plugins in child processes should be easy, using Node.js stdio: "ipc" to trigger specific hooks. It also has other advantages (see #166 and #167). What do you think?

[DavidWells]

Its looking more and more like we need child processes.

If we use child processes we need to make sure a few things happen:

1. Terminal colors are preserved from the child processes (we had trouble with this in past)
2. Child processes will play nice with plugin outputs
3. Speed/perf doesn't take huge hit

Child processes might make it easier to scope env vars and other things. So I think its worth checking out

[ehmicky]

This is an attempt to summarize some of the problems with this issue for @raeesbhatti and @Benaiah to join in.

Feature

In order to make builds as fast as possible, we want to run plugins in parallel as much as possible.

Related issue

Plugin outputs (#186) might impact this issue since they add a requirement for some plugins to run serially.

Problem 1

Plugins usually have global side effects. Running them in parallel creates race conditions.

Problem 2

Plugins might access/modify shared resources such as files or global variables, requiring atomic writes. Doing it in parallel creates race conditions.

Possible solutions (each of those solutions have some problems in itself):

• ask our plugin authors to use concurrency-safe libraries. For example write-file-atomic instead of fs.writeFile(). Or providing our own (if it does more than already existing libraries, which I doubt of).
• tell our plugin authors to avoid accessing/modifying shared resources
• tell our users not to mix two plugins that might access/modify the same resources

Problem 3

Logs would be non-deterministic.

Possible solution: buffer logs and output them in a deterministic order

Problem 4

Should this be opt-in or opt-out?

Problem 5

If opt-in, who should decide which plugin to parallelize: the user, the plugin author or both?

Problem 6

If opt-in, should all lifecycle of a plugin be parallellized, or should specific lifecycles be selected instead?