Esbuild: Support the esbuild plug-in system？

[ycjcl868]

esbuild is small but complete in every detail, do you have any plans to support the plugin-in system to extend the web development workflows?

[evanw]

Not right now. I may figure out an extensibility story within esbuild after the project matures, but I may also keep esbuild as a relatively lean bundler with only a certain set of built-in features depending on how the project evolves.

The use case of using esbuild as a library was one I hadn't originally considered. It's interesting to see it start to take off and I want to see where it goes. It could be that most users end up using other bundlers and esbuild is just an implementation detail that brings better performance to those bundlers.

I'm still thinking about how I might add extensibility to esbuild in the back of my mind. Obviously it's made more complicated by the fact that it's written in Go. It would be possible to "shell out" to other processes to delegate the task of transforming input files, but that would almost surely be a huge slowdown because JavaScript process startup overhead costs are really high.

One idea I've been thinking about is to have esbuild start up a set number of JavaScript processes (possibly just one) and then stream commands to it over stdin/stdout. That could potentially amortize some of the JavaScript startup cost (loading and JITing packages from disk). It would be more of a pain to debug and might still be surprisingly slow due to all of the serialization overhead and the single-threaded nature of the JavaScript event loop.

Another idea is to turn the esbuild repository into a Go-based "build your own bundler" kit. Then you could write plugins in Go to keep your extensions high-performance. The drawback is that you'd have to build your own bundler, but luckily Go compiles quickly and makes cross-platform builds trivial. That would likely require me to freeze a lot of the Go APIs that are now internal-only, which would prevent me from making major improvements. So this definitely isn't going to happen in the short term since esbuild is still early and under heavy development.

[rsms]

Finding this interesting!

TL;DR: The subprocess approach sounds like a solid idea.

Some thoughts in regards to the conversation above

It would be possible to "shell out" to other processes to delegate the task of transforming input files, but that would almost surely be a huge slowdown because JavaScript process startup overhead costs are really high.

This is a really interesting approach I think, although not the most portable (i.e. running systems like iOS that do not support subprocess creation.) I'd think about the quality-performance problem here in the same way as with Figma plugins:

• Making it clear to the user that they are using "extra things"
• Warnings when things seems to be slow or hung (like how a web browser running a locked-up javascript might ask the user "Foo is taking a long time to run. Do you want to wait for Foo or kill it?".
• Allowing async operation (not just execution) so that things that don't need to be serialized isn't. For example: a process that rsync's changes to a remote server. This program would want to be triggered on a build but does not directly depend on the build nor the build depending on it.
• Processes are widely understood, supported and have a robust execution model.

One idea I've been thinking about is to have esbuild start up a set number of JavaScript processes (possibly just one) and then stream commands to it over stdin/stdout.

In my experience what makes nodejs programs slow is I/O rather than CPU. For example, loading gatsby.js causes an incredible amount of files and directories to be read etc. TypeScript is an example of a good player making the best they can, avoiding runtime imports, but starting tsc is still painfully slow (thus their daemon/server model, which is a solution just like the "subprocess" idea you have!)

Another idea is to turn the esbuild repository into a Go-based "build your own bundler" kit.

Perhaps a nice option for people who are comfortable with Go. Might "pair well" with a subprocess approach for "putting lego blocks together" vs "build your own lego blocks".
Ideas about stuff to consider:

• Maintenance cost of an API that can't change much. I.e. the API provided for "build your own...". Perhaps making it extremely minimal with just two-three functions for pre- and post-processing with a string file list would get you most of the upsides with a low API maintenance cost?
• How to manage a future fragmented landscape of "many different esbuild versions out there." For example, imagine someone effectively forking esbuild and adding a substantial and desired addition like OCaml transpilation. Ideally they'd just gomod import esbuild but for some reason they had to fork it. Now you end up with users of that version filing issues and so on.

Some thoughts on Go plugins

I've worked with plugins in go in the past and it's a little bit complicated (for good reasons.) Since Go is statically compiled and doesn't have a fully dynamic runtime like for example JavaScript it is a little tricky to load code at runtime and _really_ hard to unload code (replace/update.)

Some things to keep in mind:

• Go's plugin package provides this functionality
• Plugins must be built by the same Go compiler as the program that loads them
• Plugins are compiled with copies of code that it makes use of (i.e. imports.) I _think_ these must be the same versions of any packages linked in the main program and in other plugins, but I could be wrong.
• Plugins can't be unloaded in any meaningful way (since Go uses fairly complex memory management.) However I've had success with adding some structure to what plugins must do, with things like an unload function it must provide and use to zero any references to code outside itself.
• Loading plugins is fast since you just load the code into memory and then "call it".

Some example code from GHP:

Loading plugins:
https://github.com/rsms/ghp/blob/8ab5e52dded3ad7443849bf7a51a82e8e7ee2de2/ghp/servlet.go#L65-L71

Add structure to plugins to allow unloading them (without actually unloading their code.)
https://github.com/rsms/ghp/blob/8ab5e52dded3ad7443849bf7a51a82e8e7ee2de2/ghp/servlet.go#L107-L119

A plugin: (called "servlet" in this project)
https://github.com/rsms/ghp/blob/8ab5e52dded3ad7443849bf7a51a82e8e7ee2de2/example/pub/servlet/servlet.go

[zmitry]

What about something like this https://github.com/hashicorp/go-plugin. Use RPC based plugin system so anyone could write plugin in any language.

[evanw]

@rsms thanks for writing up your thoughts. It was very interesting to read through them, and helpful to learn from your experience. I haven't seen the plugin package used before. I hadn't thought of the Go compiler version problem but that makes them much less appealing.

Maintenance cost of an API that can't change much. I.e. the API provided for "build your own...". Perhaps making it extremely minimal with just two-three functions for pre- and post-processing with a string file list would get you most of the upsides with a low API maintenance cost?

Yes, I was thinking of something extremely minimal. Of course that would come at the cost of performance, which isn't great. I'm not sure if there's a great solution to this.

What about something like this https://github.com/hashicorp/go-plugin. Use RPC based plugin system so anyone could write plugin in any language.

I think something like this is promising. This is basically how esbuild's current API works, except over stdin/stdout. The advantage of this over shelling out is that it lets you amortize the startup overhead of node by keeping it running during the build. You could imagine a more complex API where esbuild has hooks for various points and could call out to node and block that goroutine on the reply. That would let you, say, run the CoffeeScript compiler before esbuild processes the source code of a file.

I plan to explore this direction once esbuild is more feature-complete. My research direction is going to be "assuming you have to use a JavaScript plugin, how fast can you make it". If we can figure that out then that's probably the most helpful form of API for the web development community.

[floydspace]

I would like to vote for this feature.

Currently, I'm migrating some small project from webpack to esbuild and it has graphql schema defined in a file .graphql which is bundled using graphql-tag/loader, of course, I can just change schema definition using the different approach. But It would be nice to have this capability to write esbuild loader without raising a PR in your repository for every single case.

Thank you

[chowey]

If you do implement a plugin system, please consider making it Go-based (or better yet, cross-language as per @zmitry's suggestion).

I think there is a very big opportunity for non-JS-based tooling to transpile JS. As you state in your readme:

I'm hoping that this project serves as an "existence proof" that our JavaScript tooling can be much, much faster.

[zmitry]

I guess for first iteration it would be nice to have just golang api for plugins. I guess if you can run some arbitrary code in golang you could spawn sub process in any language. So even simple golang api would be enough.
I like rpc based approach because it would allow to do plugins hot swap or remote plugins and it's much cleaner approach.

[evanw]

I have an update!

While the final plugin API might need a few rewrites to work out the kinks, I think I have an initial approach that I feel pretty good about. It reuses the existing stdio IPC channel that the JavaScript API is already using and extends it to work with plugins. Everything is currently on an unstable branch called plugins.

It's still very much a work in progress but I already have loader plugins working. Here's what a loader plugin currently looks like:

let esbuild = require('esbuild')
let YAML = require('js-yaml')
let util = require('util')
let fs = require('fs')

esbuild.build({
  entryPoints: ['example.js'],
  bundle: true,
  outfile: 'out.js',
  plugins: [
    plugin => {
      plugin.setName('yaml-loader')
      plugin.addLoader({ filter: /\.ya?ml$/ }, async (args) => {
        let source = await util.promisify(fs.readFile)(args.path, 'utf8')
        try {
          let contents = JSON.stringify(YAML.safeLoad(source), null, 2)
          return { contents, loader: 'json' }
        } catch (e) {
          return {
            errors: [{
              text: (e && e.reason) || (e && e.message) || e,
              location: e.mark && {
                line: e.mark.line,
                column: e.mark.column,
                lineText: source.split(/\r\n|\r|\n/g)[e.mark.line],
              },
            }],
          }
        }
      })
    },
  ],
}).catch(() => process.exit(1))

Any errors during loading are integrated into the existing log system so they look native. There is a corresponding Go API that looks very similar. In fact the JavaScript plugin API is implemented on top of the Go plugin API. The API consists of function calls with option objects for both arguments and return values so it should hopefully be easy to extend in a backwards-compatible way.

Loader plugins are given a module path and must come up with the contents for that module. I'm going to work on resolver plugins next which determine how an import path maps to a module path. Resolver plugins and loader plugins go closely together and many plugins are probably going to need both a resolver and a loader. The resolver runs for every import in every file while the loader only runs the first time a given resolved path is encountered.

Something that may be different with this plugin API compared to other bundlers is that every operation has a filter regular expression. Calling out to a JavaScript plugin from Go has overhead and the filter lets you write a faster plugin by avoiding unnecessary plugin calls if it can be determined using the regular expression in Go that the JavaScript plugin isn't needed. I haven't done any performance testing yet so I'm not sure how much slower this is, but it seemed like a good idea to start things off that way.

One weird thing about writing plugins is dealing with two forms of paths: file system paths and "virtual paths" to automatically-generated code. I struggled with the design of this for a while. One approach is to just use absolute paths for everything and make up non-existent directories to put virtual modules in. That leads to concise code but seems error-prone. Another approach I considered was to make every path into a tuple of a string and a type. That's how paths are represented internally but seemed too heavy for writing short plugins. I'm currently strongly considering marking virtual paths with a single null byte at the front like Rollup convention. Null bytes make the path invalid and the code for manipulating them is more concise than tuple objects.

I thought it'd be a good idea to post an update now even though it's not quite ready to try out, since getting loaders working seemed like an important milestone.

[evanw]

After more thought, I'm no longer thinking of taking the approach Rollup does with virtual paths using a null byte prefix. Instead I'm going back to the "paths are a tuple" model described above. In the current form, each path has an optional namespace field that defaults to file. By default loaders only see paths in the file namespace, but a loader can be configured to load paths from another namespace instead. This should allow for a clean separation between plugins and doesn't seem as verbose as I thought it would in practice.

Also, I just got resolver plugins working! This lets you intercept certain paths and prevent the default resolver from running. Here's an example of a plugin that uses this to load URL imports from the network:

// import value from 'https://www.google.com'
let https = require('https')
let http = require('http')

let httpLoader = plugin => {
  plugin.setName('http')
  plugin.addResolver({ filter: /^https?:\/\// }, args => {
    return { path: args.path, namespace: 'http' }
  })
  plugin.addLoader({ filter: /^https?:\/\//, namespace: 'http' }, async (args) => {
    let contents = await new Promise((resolve, reject) => {
      let lib = args.path.startsWith('https') ? https : http
      lib.get(args.path, res => {
        let chunks = []
        res.on('data', chunk => chunks.push(chunk))
        res.on('end', () => resolve(Buffer.concat(chunks)))
      }).on('error', reject)
    })
    return { contents, loader: 'text' }
  })
}

The resolver moves the paths to the http namespace so the default resolver ignores them. This means they are "virtual modules" because they don't exist on disk.

Plugins can generate arbitrarily many virtual modules by importing new paths and then intercepting them. Here's a plugin I made to test this feature that implements the Fibonacci sequence using modules:

// import value from 'fib(10)'
let fibonacciLoader = plugin => {
  plugin.setName('fibonacci')
  plugin.addResolver({ filter: /^fib\((\d+)\)/ }, args => {
    return { path: args.path, namespace: 'fibonacci' }
  })
  plugin.addLoader({ filter: /^fib\((\d+)\)/, namespace: 'fibonacci' }, args => {
    let match = /^fib\((\d+)\)/.exec(args.path), n = +match[1]
    let contents = n < 2 ? `export default ${n}` : `
      import n1 from 'fib(${n - 1}) ${args.path}'
      import n2 from 'fib(${n - 2}) ${args.path}'
      export default n1 + n2`
    return { contents }
  })
}

Importing from the path fib(N) generates fib(N) modules that are then all bundled into one.

[tooolbox]

The examples are impressive, and the fib(N) is both amusing and a good illustration of the capabilities.

I'm wondering if you could give a little more context regarding usage? From trying to sort through the plugins branch a little, it seems like you're mainly using esbuild's JS API and passing plugins to the transform call, but I'm curious how it would work if you were using direct command-line, or trying to write a plugin directly with Go.

[evanw]

Yes, good point.

The plugin API is intended to be used with the esbuild API. People have already been creating simple JavaScript "build script" files that just call the esbuild API and exit. This is a more convenient way of specifying a lot of arguments to esbuild than a long command line in a package.json script. From there you can use plugins by just passing an additional plugins array. Here's an example:

const { build } = require('esbuild')

let envPlugin = plugin => {
  plugin.setName('env-plugin')
  plugin.addResolver({ filter: /^env$/ }, args => {
    return { path: 'env', namespace: 'env-plugin' }
  })
  plugin.addLoader({ filter: /^env$/, namespace: 'env-plugin' }, args => {
    return { contents: JSON.stringify(process.env), loader: 'json' }
  })
}

build({
  entryPoints: ['entry.js'],
  bundle: true,
  outfile: 'out.js',
  plugins: [
    envPlugin,
  ],
}).catch(() => process.exit(1))

In reality I assume most of these plugins will be in third-party packages maintained by the community, so you would likely import the plugin using require() instead of pasting it inline like this.

The Go API is extremely similar. Here's the same example using the Go API instead:

package main

import (
  "encoding/json"
  "io/ioutil"
  "log"
  "os"
  "path/filepath"
  "strings"

  "github.com/evanw/esbuild/pkg/api"
)

func main() {
  result := api.Build(api.BuildOptions{
    EntryPoints: []string{"entry.js"},
    Bundle:      true,
    Write:       true,
    LogLevel:    api.LogLevelInfo,
    Outfile:     "out.js",
    Plugins: []func(api.Plugin){
      func(plugin api.Plugin) {
        plugin.SetName("env-plugin")
        plugin.AddResolver(api.ResolverOptions{Filter: "^env$"},
          func(args api.ResolverArgs) (api.ResolverResult, error) {
            return api.ResolverResult{Path: "env", Namespace: "env-plugin"}, nil
          })
        plugin.AddLoader(api.LoaderOptions{Filter: "^env$", Namespace: "env-plugin"},
          func(args api.LoaderArgs) (api.LoaderResult, error) {
            mapping := make(map[string]string)
            for _, item := range os.Environ() {
              if equals := strings.IndexByte(item, '='); equals != -1 {
                mapping[item[:equals]] = item[equals+1:]
              }
            }
            bytes, _ := json.Marshal(mappings)
            contents := string(bytes)
            return api.LoaderResult{Contents: &contents, Loader: api.LoaderJSON}, nil
          })
      },
    },
  })

  if len(result.Errors) > 0 {
    os.Exit(1)
  }
}

Plugins aren't designed to be used on the command line. This is the first case of the full API not being available from the command line, but given that plugins are language-specific I think it makes sense to require you to use the language-specific esbuild API to use plugins.

[LarsDenBakker]

It would be useful to have a transform hook where you gain access to the generated AST, so that you can do fast file transformations in Go.

[tooolbox]

Excellent examples, thanks!

Plugins aren't designed to be used on the command line. This is the first case of the full API not being available from the command line, but given that plugins are language-specific I think it makes sense to require you to use the language-specific esbuild API to use plugins.

Okay got it, yeah this seems wise.

[evanw]

It would be useful to have a transform hook where you gain access to the generated AST, so that you can do fast file transformations in Go.

I totally understand why this would be useful, but I don't want to expose the AST in its current form. It's designed for speed, not ease of use, and there are lots of subtle invariants that need to be upheld (e.g. scope tree, symbol use counts, cross-part dependency tracking, import and export maps, ES6 import/export syntax flags, ordering of lowering and mangling operations, etc.). Exposing this internal AST to plugins would be a good way to destabilize esbuild and cause silent and hard-to-debug correctness issues with the generated code.

I'm also trying to keep the quality of esbuild high, both in terms of the user experience and the developer experience. I don't want to expose the internal AST too early and then be stuck with that interface, since I don't think it's the right interface.

Figuring out a good interface for the AST that is easy to use, doesn't slow things down too much, and hard to cause code generation bugs with would be a good project to explore. But this is a big undertaking and I don't think now is the right part in the timeline of this project to do this. It also makes a lot of other upcoming features harder (e.g. code splitting, other file types such as HTML and CSS) because it freezes the AST interface when it might need to change.

For now, it's best to either serialize the AST to a string before passing it to esbuild or use other tools if you need to do AST manipulation.

[chowey]

CSS extraction was actually pretty simple in Go:

package main

import (
    "bytes"
    "io"
    "os"

    "github.com/evanw/esbuild/pkg/api"
)

var cssExport = "export default {};\n"

// CSSExtractor will accumulate CSS into a buffer.
type CSSExtractor struct {
    bytes.Buffer
}

// Plugin can be used in api.BuildOptions.
func (ex *CSSExtractor) Plugin(plugin api.Plugin) {
    plugin.SetName("css-extractor")
    plugin.AddLoader(
        api.LoaderOptions{Filter: `\.css$`},
        func(args api.LoaderArgs) (res api.LoaderResult, err error) {
            f, err := os.Open(args.Path)
            if err != nil {
                return res, err
            }
            defer f.Close()
            if _, err := io.Copy(ex, f); err != nil {
                return res, err
            }

            // CSS is an empty export.
            res.Loader = api.LoaderJS
            res.Contents = &cssExport
            return res, nil
        },
    )
}

This works for me, since I just want to write my CSS to a file.

[jakajancar]

Is my understanding correct that this will then require a “wrapper” around esbuild in either go or node (or another language that implements the protocol node is using), and plugins will have to be written it that language?

I.e. you wont be able to run esbuild --plugin download --plugin somethingelse, and have them be written in whatever?

In other words more than starting an ecosystem of plugins for esbuild, likely a wrapper tool will emerge in both languages and plugin ecosysytems for the wrappers?

[evanw]

I'm expecting all serious usage of esbuild to use the API anyway because specifying a long list of options on the command line isn't very maintainable (e.g. don't get nice diffs or git blame). You can easily do this without a separate "wrapper" package just by calling esbuild's JavaScript API from a file with a few lines of code:

const { build } = require('esbuild')

build({
  entryPoints: ['./src/main.ts'],
  outfile: './dist/main.js',
  minify: true,
  bundle: true,
}).catch(() => process.exit(1))

From that point, adding plugins is just adding another property to the build call. I'm sure some people will create fancy wrappers but a wrapper isn't necessary to use plugins.

I'm also expecting that the large majority of esbuild plugins will be JavaScript plugins. Virtually all of the plugins in the current bundler community are written in JavaScript and people likely won't rewrite them when porting them to esbuild. So my design for plugins is primarily oriented around JavaScript and its ecosystem, not around Go. In that world most people wouldn't need a wrapper.

As far as non-JavaScript languages, that stuff can get extremely custom and I think exposing a general API like the current Go API is better than trying to guess up front what people would want in a native language wrapper and hard-coding that into esbuild. You should be able to use the Go API to do whatever custom native language bindings you want (local sockets, child processes, RPC with a server, etc.) without any performance overhead over what esbuild would have done itself anyway.

[jakajancar]

You can easily do this without a separate "wrapper" package just by calling esbuild's JavaScript API from a file with a few lines of code

Which is what I meant by "wrapper" :)

Consider this: there will be a bunch of "esbuild plugins" built in Node. And if you don't use Node, maybe because you use Deno instead, there will be a separate ecosystem of plugins there that will likely emerge in Deno, distinct from Node's.

It's almost less about plugins for esbuild, than it is about making esbuild pluggable/embeddable itself, with the option of hooks. And it doesn't seem like a JavaScript API, but a language-agnostic (and private?) binary API, and then a _Node_ binding/API for it.

Anyways, makes sense, just the wording got me confused a bit.

[evanw]

Which is what I meant by "wrapper" :)

I see. I thought you meant that it would be complex enough to require an additional wrapper package with a significant amount of code.

And it doesn't seem like a _JavaScript_ API, but a language-agnostic (and private?) binary API, and then a _Node_ binding/API for it.

Yes the binary IPC protocol I'm using is currently private. While it is language-agnostic, it's designed with the JavaScript host in mind. If you need to integrate esbuild with another native binary, I personally think the Go API is much more ergonomic and useful than the binary protocol. For example, the binary protocol serializes everything over a single stream which can limit multi-threaded performance. This isn't a problem for a JavaScript host since JavaScript is single-threaded already, but would be a problem if you're trying to reach maximum performance with another multi-threaded native language. The Go API doesn't have this problem because each plugin invocation is in a separate goroutine.

[baryla]

This is looking really good @evanw! I've been waiting for this feature for a while.

I like the way this going but I have 1 suggestion regarding the filter inside the resolvers and loaders. I understand that regex is super quick and esbuild's aim is to remain lightning fast but have you considered also allowing a function to be passed with args (same as the callback) in case there was a case that required a little bit more logic? Regex by nature is difficult to read and debug so for some people, a function may be an "easier" option.

[evanw]

have you considered also allowing a function to be passed with args (same as the callback) in case there was a case that required a little bit more logic?

I'm not sure what you mean. Both addResolver and addLoader take a function as a callback. The filter regex is just there to speed things up, but you can always make it .* if you need to match everything. The function can return null or undefined to pass control on to the next resolver or loader, so the function can also serve as a filter.

That said, I hope people won't do that when they don't need to. A regex to pre-filter for the relevant file extension is short and (I think) still pretty readable. Using .* instead will slow things down unnecessarily. But sometimes you need to match everything so the ability is there if you want to use it.

[baryla]

@evanw

I'm not sure what you mean. Both addResolver and addLoader take a function as a callback. The filter regex is just there to speed things up, but you can always make it .* if you need to match everything. The function can return null or undefined to pass control on to the next resolver or loader, so the function can also serve as a filter.

That said, I hope people won't do that when they don't need to. A regex to pre-filter for the relevant file extension is short and (I think) still pretty readable. Using .* instead will slow things down unnecessarily. But sometimes you need to match everything so the ability is there if you want to use it.

Maybe I slightly misunderstood the goal of a filter but it does now make sense.
I just thought that the filter could also cater for some more "complicated" filtering if it needed to such as:

const isAllowed = path => {
  if (!path.endsWith('.css)) {
    return false;
  }

  return ['app', 'component'].some(name => path.startsWith(name));
}

const myPlugin = plugin => {
  plugin.resolver({ filter: isAllowed }, args => {
    // logic to resolve file
  })
}

which would still avoid unnecessary plugin calls if I'm not mistaken?

[evanw]

which would still avoid unnecessary plugin calls if I'm not mistaken?

The goal is to minimize the number of calls from Go into JavaScript since crossing this boundary is pretty slow. If you already have to call into JavaScript, you might as well run the entire plugin. It would be even slower to call into JavaScript twice, once for the filter function and once for the actual resolver.

While that plugin would best be written like this:

const myPlugin = plugin => {
  plugin.addResolver({ filter: /^(app|component).*\.css$/ }, args => {
    // logic to resolve file
  })
}

It could also be written like this:

const isAllowed = path => {
  if (!path.endsWith('.css')) {
    return false;
  }

  return ['app', 'component'].some(name => path.startsWith(name));
}

const myPlugin = plugin => {
  plugin.addResolver({ filter: /.*/ }, args => {
    if (!isAllowed(args.path))
      return;

    // logic to resolve file
  })
}

[ggoodman]

@evanw any thoughts on adding this in a stable release (even if it is clearly marked as unstable unstable_plugins: comes to mind)?

What kind of decision decisions are you still evaluating?

[evanw]

What kind of decision decisions are you still evaluating?

I'm also excited about plugins and looking forward to them being released. There are a few reasons why I haven't released the plugin API yet.

One is that code splitting (issue #16) is still very much a work in progress and I feel like I should follow through with that first. It's a pretty foundational feature, people are starting to depend on it, and it deserves to be in a stable spot. I don't want to start too many things at once without finishing what I've started. I also want to make sure I can give the plugin API my full attention when it's released and people start using it.

Another thing I'd like to think more about is how other file types (e.g. HTML and CSS) interact with plugins. I've made progress on thinking through this but have had to put that aside to work on some recent correctness issues around code splitting and source maps. I plan to get back to thinking through alternate file types and plugins after that's in a good spot.

I'm still pushing toward the release of plugins in the meantime. Besides iterating on the API, I have also recently landed the nested source maps feature (issue #211) which is somewhat of a prerequisite for plugins. Source maps for non-JavaScript language plugins won't work without it. I landed it independently because it's a useful feature by itself, but I consider it to have been blocking the plugin API release. This also gives it some time to stabilize first so it's ready when plugins happen.

any thoughts on adding this in a stable release (even if it is clearly marked as unstable unstable_plugins: comes to mind)?

I have considered exposing an unstable plugin API as well but I don't think that solves much. People will still start to depend on it regardless and if the ultimate API ends up undergoing major changes, upgrading to the new plugin API will still be just as much work. So I'd like to keep plugins on a branch for now.

[matthewmueller]

For what it's worth, I've been using the plugins branch the last week or so and it already works great! Two areas that are a bit awkward but totally workable:

1. It would be nice to be able to feed assets back into the pipeline. I think this is what you mean by what to do about HTML and CSS. This would allow additional asset transformations that take advantage of ESBuild's parallelism, cache, etc. I believe this feature is similar to rollup's this.emit functionality inside its plugins. You can workaround this by discovering these assets in the plugin and passing them to a separate asset pipeline.

2. Sometimes the same entrypoints are built for different environments. For example, if you're building pages for both Node and the browser. The various build options like Platform, Format and Plugins will depend on the environment, so you end up needing two instances of ESBuild. Not a big deal, just a possible improvement as you're thinking about the API.

Happy to share additional context or code if helpful!

[stephen]

Hi, just wanted to add another datapoint about our experience trying out esbuild. We experimented with building a dev server for our fairly large frontend app (5k+ files, js/ts/css/graphql). We wrote a .graphql loader plugin and .css loader plugin in the go api and some livereload boilerplate:

• As expected from https://github.com/evanw/esbuild/issues/111#issuecomment-666906970, there wasn't an in-band way to deal with css output.

We ended up doing something similar to https://github.com/evanw/esbuild/issues/111#issuecomment-636408869, emitting a separate fake filesystem for the dev server. The js emit becomes a small stub that adds a new <link> tag to the document head. I imagine if we wanted to emit/bundle css files for a production build later, we'd have to do it ourselves. This wasn't a problem for the .graphql files, since they don't have a separate compiled output.

As a side note, we ended up implementing css compilation (our css assumes compilation with some postcss plugins) by spinning up a small node worker pool and doing http requests against it.

• As @matthewmueller mentioned, importing additional assets from resolved assets would be nice to have.

Our frontend uses webpack-graphql-loader to inline .graphql documents as strings. In our esbuild port, we end up doing additional import resolution, caching and parallelization within the plugin. We didn't end up doing sourcemaps, but it seems like we could with https://github.com/evanw/esbuild/commit/23f0884de58781d04b6300cccec732f4e6ac8eac.

Notably, our css compilation would have suffered the same problem, but postcss is doing @import resolution anyway.

• We didn't end up using the resolver api. For the most part, path resolution looked the same as what the default resolver does.

• The results are really exciting. Our webpack-dev-server startup takes minutes. In esbuild, it's < 10s, and most of that comes from the clunky css compilation.

• _Edit: In the meantime, I started working on a postcss replacement as an experiment in go as well._

[zmitry]

@evanw Any updates on plugins side?

[evanw]

No updates at the moment. I have some stuff going on in my personal life that's taking priority right now (relocating some family members). My focus will be back on esbuild after that is done.

[tinchoz49]

Don't worry :) thank you for working on this.

[intrnl]

Coming from the JSX automatic runtime thing, can the transform API use plugins or is it something that's only for builds?

[littledivy]

Importing from URL was pretty easy using the Go API at plugins branch :smile:

func URLLoader(plugin api.Plugin) {
    plugin.SetName("url-loader")
    plugin.AddResolver(api.ResolverOptions{Filter: "^https?://"},
        func(args api.ResolverArgs) (api.ResolverResult, error) {
            fmt.Println("Downloading ", args.Path)
            // Get the data
            resp, _ := http.Get(args.Path)
            fileName := buildFileName(args.Path)
            defer resp.Body.Close()
            file, err := ioutil.TempFile("", fileName)
            if err != nil {
                log.Fatal(err)
            }
            io.Copy(file, resp.Body)

            defer file.Close()
            fmt.Println("Downloaded ", file.Name())
            return api.ResolverResult{Path: file.Name(), Namespace: "url-loader"}, nil
        })
    plugin.AddLoader(api.LoaderOptions{Filter: "^", Namespace: "url-loader"},
        func(args api.LoaderArgs) (api.LoaderResult, error) {
            fmt.Println("Loading ", args.Path)
            dat, _ := ioutil.ReadFile(args.Path)
            contents := string(dat)
            return api.LoaderResult{Contents: &contents, Loader: api.LoaderTS}, nil
        })
}

Works quite well for a runtime I'm building Done

Cheers @evanw :raised_hands:

[ije]

This is awesome! i just tried, it works fine. thanks @evanw !
My question is whether is possible to rewrite the external module path like:

api.Build(api.BuildOptions{
    EntryPoints: []string{"entry.js"},
    Bundle:      true,
    Write:       true,
    LogLevel:    api.LogLevelInfo,
    Plugins: []func(api.Plugin){
        func(plugin api.Plugin) {
            plugin.SetName("rewrite-external-plugin")
            plugin.AddResolver(
                api.ResolverOptions{Filter: "^(react|react-dom)$"},
                func(args api.ResolverArgs) (api.ResolverResult, error) {
                    newPath := fmt.Sprintf("https://esm.sh/%s", args.Path)
                    return api.ResolverResult{Path: args.Path, AsPath: newPath, External: true, Namespace: "rewrite-external"}, nil
                },
            )
            plugin.AddLoader(
                api.LoaderOptions{Filter: "^(react|react-dom)$", Namespace: "rewrite-external"},
                func(args api.LoaderArgs) (api.LoaderResult, error) {
                    contents := ""
                    return api.LoaderResult{Contents: &contents, Loader: api.LoaderJSON}, nil
                },
            )
        },
    },
})

[evanw]

My problem is whether is possible to rewrite the external module path

Thanks for pointing this out. This was an oversight on my part. I updated the plugins branch and this should now be possible. You would specify Path: newPath and you wouldn't need a loader to do this (you only need a resolver).

[ije]

cool, thanks for the great work!

[zmitry]

I've implemented plugin which allows you to import svgs as react components. It's not battle tested yet.
https://github.com/zmitry/esbuild-svgr

[zmitry]

I've also created setup with create-react-app which allows you to use esbuild with webpack dev server and regular cra setup https://github.com/zmitry/esbuild-cra. It's hacky but it kind of works you just need to install golang and you are ready to go. I do not recommend to use it for production. It's only POC where you can play and decide if you need esbuild and it worth switching.

[st-schneider]

Is there a way to use https://github.com/atlassian-labs/compiled via a plugin for css extraction?

[zmitry]

@evanw I want to add some extra feature requests to plugins api. First use case would be ability to write plugin for html. This requires to have ability to get get name of the processed file inside html. I want to get path to the index js in the output folder or its content in case I want to inline it. Also I want all the subtitunion to work inside html, sometimes I want to replace some variables like in my use case.

<html>
    <meta http-equiv="Content-Security-Policy" content="{CSP_POLICY}" />
    <link rel="icon" type="image/png" sizes="32x32" href="{PUBLIC_URL}/favicon-32x32.png" />
    <link rel="icon" type="image/png" sizes="16x16" href="{PUBLIC_URL}/favicon-16x16.png" />
<script src="./src/indexj.s"></script>
</html>

Another use case which I want is to have ability to add some arbitrary content to the processing pipeline. For example I want to implement simple css in js solution using esbuild. The idea that I want to treat all css("string") as css modules. So what I want to do is to get content of css("") and add it to processing pipeline and replace css call with object with classnames.

const styles = css(`
  .red {
   background: red;
  }
`)
// =>

const styles= { red: 'red-classname' }

[evanw]

I want to get path to the index js in the output folder or it's contents in case I want to inline it.

Thanks for raising this desire. This isn't currently possible outside of hacky nested build approaches. I'll have to think about this. It probably means there needs to be a way for a plugin to add a new entry point during the bundling process, which should be possible.

The idea that I want to treat all css("string") as css modules.

This should be possible already I think. It sounds like the idea is to replace this:

const styles = css(`...code...`)

with this:

import 'some-path.css'
const styles = { red: 'red-classname' }

where some-path.css is whatever path the loader needs to be able to associate the automatically-generated import statement with that particular CSS string. One solution could be to use some prefix that the loader recognizes and then encode the entire contents of the string using URL encoding, but some form of incrementing identifier combined with a mapping in the plugin should also work.

[zmitry]

Could you elaborate on the idea about css strings within js? I didn't quite get how to implement it. For instance if I have the code like following. In this case I need to resolve classes for first chunk and only then parse next chunk.

const stylesA = css("code")
const stylesB = css(`.${stylesA.button}:hover{ background red }`)

[evanw]

Sorry, I'm not sure I understand what you're trying to do anymore. It looks like the CSS can now depend on run-time values because of the use of `${button}`. In that case esbuild won't be able to process the CSS at bundle time. However, it should still be possible to write a function called css that injects <style> elements at run-time to make that code snippet work.

[zmitry]

Sorry, I did mistake. So the idea is that css values can depend on output of another css modules. (see my previous snippet).

[zandaqo]

I'm curious, is anyone looking into a loader for Node.js native addons? This would allow a fast/simple bundler for Node.js , a replacement for ncc, for example, that relies on webpack and is comparatively slow, although admittedly it does a fair bit more than just bundling native addons.

[bep]

Just wanted to chime and say that I have a Hugo branch testing the plugins branch out, and it is a perfect fit, works great. I'm only using the resolver (for now), but given the nature of our module system (a union filesystem, no common root), being able to do custom import resolving has been the missing piece in all of this. Working with this in VS Code in a multi project setup feels almost a little magical given its speed and all. Great job.

[bep]

@evanw a quick yes/no question:

• I understand that the plugins branch is work in progress and that API changes may/will happen.
• But I notice that branch is most of the time in synch with the main branch.
• Is it fair to assume that that branch will _eventually_ land in the main branch with approximately the current feature set intact?

[evanw]

Is it fair to assume that that branch will _eventually_ land in the main branch with approximately the current feature set intact?

Yup. I'm actually working on this right now, it just doesn't look like it. As part of the plugins release I want esbuild to have a real website with comprehensive documentation. It's been a lot of work but it's actually mostly done. However, this work is currently being done in a private repo. I hope to get the esbuild website up and ship plugins in the next week or two.

I'm not totally certain about the current plugin API. It's the first time I'm writing one of these and I'm not a heavy user of other bundler plugin APIs, so I suspect there are things missing that could fundamentally alter the design. For example, plugins don't really compose right now. There's also no way of invoking esbuild's default behavior from within a plugin. So I may need to change the design in a backwards-compatible way before version 1.0.0 of esbuild.

However, the current direction is promising and seems like it's been able to solve most of the problems thrown at it. It has also withstood the first round of people testing it out (thank you very much everyone who tried it!) so I think now is a good point to release it for wider feedback. I also realize that the JavaScript API is significantly harder than the Go API to try out while it's still on a branch, so releasing it is also important for getting wider feedback about integration with the JavaScript ecosystem.

[eigilsagafos]

Thanks @evanw for the update. I have been testing it this week with Yarn 2 (pnp). Looking great, and much more performant than I had expected it to be! I posted in another thread that the biggest issue I'm having currently is a way to ignore packages already marked as external. Now I have to implement that as a part of the plugin. Maybe you could provide a "ignoreExternal" flag as part of the addResolver settings or provide external true/false as part of the args provided to the function?

[evanw]

Maybe you could provide a "ignoreExternal" flag as part of the addResolver settings or provide external true/false as part of the args provided to the function?

Yeah thanks for that feedback. I thought your suggestion on the other thread of an option to _include_ externals was interesting, since then they would be excluded by default. It sounds like excluding them might be the right default behavior. I can't think of a use case for including them off the top of my head.

Actually maybe the API could just switch to always excluding the externals if there's no current use case for including them. This kind of makes sense to me in that the user's configuration should override the plugin's configuration since the user can change their configuration but they can't necessarily change the plugin.

It does mean that esbuild would have to always resolve all paths itself though instead of only resolving them when no plugin resolves the path so there's a performance hit, although I'm guessing it's likely a minimal performance hit.

[lukeed]

For example, plugins don't really compose right now.

Would it be possible to clarify this? Maybe I'm being too short-sighted, but I think the main need here is that the transformed output from one loader _could invoke_ another load. For example, a HTML loader that parses <script> tags for JS entries, which can/may import other .css (css), .svg (baseurl), and .svelte (plugin) files. And the same cycle repeats for the .svelte contents.

But if "composing plugins" means extending a plugin, or calling one directly from another, I'm not sure how necessary that really is.

the JavaScript API is significantly harder than the Go API to try out while it's still on a branch

😅 Do you plan to release esbuild@next (or similar) on npm at some point for testing? Would be a nice, non-binding way to gather additional feedback

[Jarred-Sumner]

@lukeed if you want I have a fork where I manually published the branch with plugins, just add this to your dependencies inside package.json:

"esbuild": "https://github.com/Jarred-Sumner/esbuild/releases/download/pluginbuild/esbuild-0.8.1.tgz",

The only change I made was modifying the install.ts script to not download from NPM/local cache and instead fetch from that github release page (plus version bump so I could be sure it wasn't the non-plugins version)

[evanw]

Getting the initial version of the plugin API out is going to be the focus of my next big release.

One thing I'm worried about is the performance of the JavaScript API. Right now I have primarily optimized the JavaScript API for ease of use. A plugin is just a JavaScript function that can make a few different calls to the plugin API functions. This makes plugins very lightweight because they are small and easy to write inline in the same file:

let examplePlugin = plugin => {
  plugin.setName('example')
  plugin.addLoader({ filter: /.*/ }, async (args) => {
    return { content: await require('fs').promises.readFile(args.path, 'utf8') }
  })
}

await esbuild.build({
  plugins: [examplePlugin],
  ...
})

However, that comes at the expense of some performance opportunity cost because JavaScript is single-threaded. This can potentially be very slow if you are running every input file through a plugin because the build is bottlenecked through a single CPU. There's an argument to be made that the plugin API should focus on ease of use, and also that JavaScript plugins are doomed to be slow anyway. But given the focus of esbuild on pushing for faster tools I think performance should be a strong consideration, and this is arguably especially important for JavaScript-based plugins given the performance handicap.

So I think it's important to investigate the performance impact of a few different API designs for the plugin API during the design phase. Obviously the plugin API can still be changed after releasing it but I think the investigation won't take that long and doing it ahead of time avoids releasing an API only to potentially change it immediately. Sorry about the delay. I know the plugin API is really exciting but I think it's prudent to be careful and deliberate about these design decisions.

My current idea for improving performance is to change the plugin API such that plugins must be in separate files. You would then pass a file name to esbuild and esbuild would spin up several node child processes that handle plugin invocations completely in parallel. Maybe something like this:

// example-plugin.js
module.exports = options => ({
  name: 'example',
  setup(build) {
    plugin.onLoad({ filter: /.*/ }, async (args) => {
      return { content: await require('fs').promises.readFile(args.path, 'utf8') }
    })
  },
})

// build.js
await esbuild.build({
  plugins: {
    './example-plugin.js': { /* options */ },
  },
  ...
})

Plugins that contain synchronized global data structures will be unable to run in parallel. This should be easy to handle by just running them in the host node process instead of in the child node processes. Perhaps they could be specified in a separate serialPlugins property instead of mixing them in with plugins.

I'm going to investigate this approach next and report back with performance numbers.

[intrnl]

I think converting the plugin function to string would work the same way and should avoid the need for esbuild to resolve files on its own, though that could mean the function needs to be async for it to work with ESM.

function sveltePlugin () {
  let { compile } = require('svelte');

  return {
    name: 'svelte',
    // ...
  };
}

[evanw]

I think converting the plugin function to string would work the same way and should avoid the need for esbuild to resolve files on its own

That would break relative imports:

function sveltePlugin () {
  let { compile } = require('./svelte.js');

  return {
    name: 'svelte',
    // ...
  };
}

The code would no longer be able to resolve ./svelte.js when run because esbuild wouldn't know what directory to run it in, since all it has is a function object.

I immediately hit the problem of esbuild needing to resolve files itself when I tried implementing this, so you're totally right that the specific API shape I proposed isn't good. Right now I'm just requiring the caller to run require.resolve() first while I focus on the proof-of-concept demo. This API shape will need more thought if the plugin API ends up going in this direction.

[evanw]

Here's an update. Sorry about the long post.

I'm testing plugin parallelism by running each file in my JavaScript benchmark through a plugin that runs the TypeScript compiler's transpileModule function. This is a no-op because the input files are already JavaScript, but it should be a realistic performance test. After all, there is no difference in speed between esbuild's JavaScript and TypeScript parsers.

The results are disappointing. I was hoping I could get at least a 2x speedup for JavaScript plugins using parallelism because esbuild can do that easily, but I the maximum speedup was 1.5x. I was also hoping that having esbuild create the child processes would be more efficient than having the plugin do that itself because it would mean using completely parallel IPC channels instead of having everything being bottlenecked through the single IPC channel with the host process, but it turns out the IPC channel is so fast that the numbers basically don't matter given how slow the TypeScript compiler is.

Here's the data from my tests:

Count | GOMAXPROCS | Child process of plugin | Child process of esbuild
--: | --: | --: | --:
1 | 1.64s | 18.55s | 19.08s
2 | 0.92s | 12.53s | 13.12s
3 | 0.69s | 12.00s | 12.01s
4 | 0.57s | 12.20s | 13.19s

Each time is the best of three runs and there is probably still a little bit of noise. Here's what each column means:

1. Count is the number of units of parallelism, where 1 means no parallelism (i.e. 1 CPU).
2. "GOMAXPROCS" is a baseline measurement of esbuild doing the TypeScript-to-JavaScript conversion without using any plugins (GOMAXPROCS controls the number of OS threads used by Go).
3. "Child process of plugin" means the plugin itself uses node's child_process.fork() function to create long-lived child node processes that run transpileModule().
4. "Child process of esbuild" means that esbuild uses Go's exec.Command() function to create long-lived child node processes that run transpileModule().

Here is that same data divided by the first row in each column. This represents the relative speedup obtained by using parallelism:

Count | GOMAXPROCS | Child process of plugin | Child process of esbuild
--: | --: | --: | --:
1 | 1.00x | 1.00x | 1.00x
2 | 1.78x | 1.48x | 1.44x
3 | 2.38x | 1.55x | 1.51x
4 | 2.88x | 1.52x | 1.37x

And here is the relative speedup visualized:

Note that it's not possible for these speedup multiples to be exactly equal to the count because some of aspects of bundling cannot be parallelized. Also at some point increasing the count will actually decrease performance because there is a limited number of cores and using more parallel tasks than the number of cores results in wasted effort switching between tasks. My machine has 6 cores so theoretically times should continue to improve up to a count of 6.

While esbuild continues to improve by a significant amount for each unit of parallelism, the TypeScript plugin essentially only benefits from having one additional child process. After that point the results barely improve and then start dropping. This was unintuitive for me and is very unfortunate because it means JavaScript plugins in esbuild are pretty much guaranteed to be slow, especially as compared to Go plugins.

After some investigation I believe I have an explanation: although JavaScript the language is single-threaded, V8 is not and usually uses 200-300% of your CPU (as confirmed by top). I know that at least V8's GC is parallelized, and various aspects of the JIT compiler may be too. I tried to find flags to force V8's GC to run on the main thread but nothing I tried seemed to have an effect.

I think the conclusion is that parallelism of JavaScript plugins isn't a big win and that it probably doesn't make sense to parallelize most plugins. It probably only makes sense to parallelize your primary plugin and even then it's probably not worth creating more than two child processes. Given that parallelism of JavaScript plugins isn't going to be a big focus, right now I'm thinking that it's probably ok for the plugin to just do that itself using child_process instead of building plugin parallelism into esbuild. That means I could move forward with releasing the current plugin API without any fundamental changes.

[Jarred-Sumner]

After some investigation I believe I have an explanation: although JavaScript the language is single-threaded, V8 is not and usually uses 200-300% of your CPU (as confirmed by top). I know that at least V8's GC is parallelized, and various aspects of the JIT compiler may be too. I tried to find flags to force V8's GC to run on the main thread but nothing I tried seemed to have an effect.

This is probably a really dumb idea, but what if you tried using one of the slower, embeddable JavaScript interpretors (without a JIT) instead of V8 for plugins? For example: https://bellard.org/quickjs/ or maybe https://github.com/facebook/hermes. https://github.com/robertkrimen/otto might be easiest to try since its already written in Go.

[evanw]

what if you tried using one of the slower, embeddable JavaScript interpretors (without a JIT) instead of V8 for plugins?

I have some experience with this. While I haven't tried this with esbuild specifically, we did do a performance investigation when we switched to QuickJS for Figma plugins. V8 with a JIT is an order of magnitude faster than non-JIT interpreters which makes switching away from V8 for esbuild a non-starter. You can see some benchmarks here: https://bellard.org/quickjs/bench.html.

There is also the issue of the API exposed to plugins. The point of plugins in esbuild is to integrate with other packages from node's ecosystem, and they expect to use node's full API including all built-in libraries and support for native binary extensions. Replicating all of that on top of another JavaScript interpreter would be a very large (and ongoing) amount of work, so that is also a non-starter.

[Jarred-Sumner]

That makes sense

Another thought, which might be a bad idea: what if a very limited subset
of Node’s internals are replaced by an RPC for the esbuild process. Eg
override fs.readFile and fs.writeFile to be implemented in the Go process
instead of Node.

Another way of asking this question is, what if it’s Node.js internals
that’re worse at parallelism than V8?

On Sat, Oct 31, 2020 at 3:40 AM Evan Wallace [e-mail]
wrote:

what if you tried using one of the slower, embeddable JavaScript
interpretors (without a JIT) instead of V8 for plugins?

I have some experience with this. While I haven't tried this with esbuild
specifically, we did do a performance investigation when we switched to
QuickJS for Figma plugins. V8 with a JIT is an order of magnitude faster
than non-JIT interpreters which makes switching away from V8 for esbuild a
non-starter. You can see some benchmarks here:
https://bellard.org/quickjs/bench.html.

There is also the issue of the API exposed to plugins. The point of
plugins in esbuild is to integrate with other packages from node's
ecosystem, and they expect to use node's full API including all built-in
libraries and support for native binary extensions. Replicating all of that
on top of another JavaScript interpreter would be a very large (and
ongoing) amount of work, so that is also a non-starter.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/evanw/esbuild/issues/111#issuecomment-719916079, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFNGS77YYR3VVHOVMNM3NLSNPSTTANCNFSM4NAM6XYA
.

[foisonocean]

Does the plugins have to be based on JavaScript?
For me, if I need plugins written in JavaScript and I don't care about performance, I would use Babel. After all, babel has a larger ecosystem than esbuild for now.
But I do care about performance. So if JavaScript based plugins would slow down esbuild, I would perfer writing plugins in some native languages and then communicate with esbuild via MessagePack for example.

[bep]

Does the plugins have to be based on JavaScript?

There will be a similar Go API (I'm looking forward to using that in Hugo), but that would not allow you to interact with JS libraries, e.g. the Svelte compiler.

[airhorns]

Not to pile on, but as a user trying to get away from Babel because it's too slow, I'd love it if the esbuild plugin story had a far higher performance ceiling than Babel's. I agree with @foisonocean that that's something special that esbuild could do: offering users a novel point along the investment/performance tradeoff curve. It makes sense that most plugin authors would want to use the same language they're bundling to write plugins, but, esbuild kind of proves that you are destined for a slow build process if you do that, so I would love the option to invest to go fast.

I suggest allowing WASM plugins instead of JS. This would allow plugins to run in a variety of languages including high performance ones, and they could be run server side using a fancy fast golang WASM VM and then potentially natively by the browser when using the WASM compiled version of esbuild there. You don't have to build or worry about an IPC channel, you don't have to deal with child processes, and you can parallelize using Go's plain old concurrency primitives. The WASM spec (mostly) handles the interface definitions such that you wouldn't have to define and evolve a protobuf or messagepack schema or whatever for IPC which is nice too.

It'd be awesome to just go write a little bit of AssemblyScript or golang to implement tiny replacements for the last couple babel plugins I have to run and fully realize the 100x bundling speedup.

[Ventajou]

I went from a Webpack build to a 2 stage setup where esbuild runs first and then Webpack processes the result. I have a handful of tasks that esbuild just can't do without plugins. It's a bit of a complex setup but it's already significantly faster than the original.

Even a single-threaded JS plugin system would be an improvement for me both in speed and complexity. And it would be relatively easy to adopt since plugins are often just glue for other Node libraries. I think it would allow a lot of people to start using esbuild and achieve much faster build times. And the plugin ecosystem would grow really quickly.

If we also have the possibility to write plugins in Go (and use both Go and JS plugins in the same build), it gives people a chance to target the slowest plugins and rebuild them in Go. I'm sure people will upgrade to faster plugins as they become available.

esbuild audience is JS developers and only a subset of them will have the knowledge or time to invest into writing plugins in another language. And there will always be the odd task that's very specific to a team and they'll have to write a plugin for it, JS will be the quick and efficient answer for that.

[lukeed]

@evanw Thanks for the insights – unfortunate, but not super surprising. Spawning processes is expensive.
I wonder if it might make a difference if using worker_threads instead, as opposed to a full process. It'd still require separate plugin files, but since the plugins are known upfront (and their options, presumably), it should still be possible to spawn and track a pool of threads.

(I haven't really done anything in this space, so I can't really help much aside from doc links)

[leeoniya]

I wonder if it might make a difference if using worker_threads instead, as opposed to a full process.

if this strategy allows for keeping the JIT'd code cached rather than being thrown away and re-JIT'd on each process spawn, it could make a significant difference.

[lukeed]

Right, they're significantly cheaper to create, but I believe there's a limit to how many can be spawned.

[evanw]

Does the plugins have to be based on JavaScript?

They don't have to, no. The plugin API is available in both JavaScript and Go. My intuition is that the vast majority of libraries people want to use with esbuild are written in JavaScript, so it seems like a mistake to not have a JavaScript plugin API.

But I do care about performance. So if JavaScript based plugins would slow down esbuild, I would perfer writing plugins in some native languages and then communicate with esbuild via MessagePack for example.

This is possible with the Go API. You can create a Go project which calls esbuild's API with a plugin that communicates with code written in other languages however you want (pipes, child processes, network calls). There wouldn't be any JavaScript executed at all in that scenario.

I suggest allowing WASM plugins instead of JS.

My thought was that since Go is native code and pretty much anything is possible, it should be possible for people to start experimenting with things like this using esbuild's Go API without having to build a whole WASM VM into esbuild itself. There shouldn't be a performance penalty for doing this in a plugin instead of doing this within esbuild.

I wonder if it might make a difference if using worker_threads instead, as opposed to a full process.

I just tried this and the performance numbers are the same as the other two child process approaches.

[leeoniya]

I just tried this and the performance numbers are the same as the other two child process approaches.

that seems odd, is the case here of spawning a process vs a thread and then terminating them and re-spawning later? (https://stackoverflow.com/a/60488780/973988).

if each plugin is a persistent pure/functional worker thread and you're simply feeding stuff to it via .postMessage, the overhead should be on the order of 10ms in extreme cases (certainly an order of magnitude faster than invoking the TS compiler)?

[1] https://wanago.io/2019/05/13/node-js-typescript-13-sending-data-worker-threads/
[2] https://www.jefftk.com/p/overhead-of-messagechannel

[evanw]

that seems odd, is the case here of spawning a process vs a thread and then terminating them and re-spawning later? (https://stackoverflow.com/a/60488780/973988).

In all scenarios, each plugin VM is created once at the start of the build and is then persistent throughout the entire build.

if each plugin is a _persistent_ pure/functional worker thread and you're simply feeding stuff to it via .postMessage, the overhead should be on the order of 10ms (certainly an order of magnitude faster than invoking the TS compiler)?

In all scenarios, the overhead of message passing was pretty insignificant relative to the overall build time. It's possible to have high throughput even with higher latency because JavaScript is slower than Go (so it's the bottleneck) and Go keeps JavaScript's input queue full throughout the build.

[foisonocean]

This is possible with the Go API. You can create a Go project which calls esbuild's API with a plugin that communicates with code written in other languages however you want (pipes, child processes, network calls). There wouldn't be any JavaScript executed at all in that scenario.

From my experience, most users of babel/esbuid don't write plugins, all they do are downloading babel/esbuild and plugins from NPM and writing a configuration file for babel/esbuild, they don't care about which programming language are used in esbuild and its plugins. The current Go API requires users to create a Go project, which is really hard for most users. So I think it would be better if the "Go plugins" can also be used with a simple npm install command and one additional line in the configuration.

More futher, I want to talk about NeoVim's plugin system. NeoVim allows users to write plugin in any languages by using MessagePack. MessagePack is a binary serialization format used for RPC. I haven't look deep into the RPC call overheads, but MessagePack claims it's very efficient and from my experience it's faster than protobuf. Imagine I could write plugins in any language, which means I can write plugins in Go, I can also write plugins in some more performant languesges like c/c++/rust, and I can also write plugins in JavaScript if I have to interact with JS libraries like the Svelte compiler. Yes many RPC solutions also has NodeJS bindings, so esbuild just need to mantain one generic RPC API instead of mantaining two separate JS/Go APIs.

[evanw]

I just released the initial version of the plugin API. It's somewhat different than the version on the plugins branch:

• Plugins are now objects with a name property and a callback instead of just being a function. I made this change because it lets you get the plugin name without invoking the plugin, which could potentially be important in some future scenarios.

• I also renamed "addResolver" and "addLoader" to "onResolve" and "onLoad" to distinguish between the similarly-named built-in concepts, and to make it natural to add more hooks in the future whose names don't lend themselves to verbs.

There is an example of this new format in the release notes. The new plugin API isn't documented yet though. I'm still working on the documentation and I plan to land it sometime in the next few days.

[evanw]

So I think it would be better if the "Go plugins" can also be used with a simple npm install command and one additional line in the configuration.

Yes, point heard. I think the approach you're proposing is interesting. However, I think it can coexist in parallel with the existing API instead of replacing it. I agree that there is a certain elegance to having everything use RPC but there are some drawbacks around ease of use and performance/memory usage. I think the existing API is good to have as an easy entry point and I RPC-based plugins could be left as a more advanced use case.

Right now a JavaScript plugin looks like this:

{ name: 'example', setup(build) { ... } }

With an RPC system, a plugin could look like this instead assuming there's a binary executable called binary in the same directory:

{ name: 'example', executable: path.join(__dirname, 'binary') }

Then esbuild would launch that executable as a child process and communicate with it over stdin/stdout using some RPC protocol. From the user's perspective they wouldn't even know the difference because in both scenarios they would just require('example-esbuild-plugin') which would return one of these objects.

There could also be support for network-based plugins using the same RPC protocol over a TCP stream:

{ name: 'example', network: 'localhost:8080' }

This could help in scenarios where you are doing many builds in quick succession and you want to use a plugin written in an inefficient language with a long start-up time (e.g. Java), so you need to put the plugin in a long-lived process for performance.

You wouldn't want to use JavaScript with either of these options for the reasons described above in this thread. Each node instance uses multiple CPU cores and you'll quickly run out of CPUs if you do that. Instead, JavaScript plugins should be co-located in the host process using esbuild's normal JavaScript plugin API.

One thing I also recently added is the ability for plugins to proxy other plugins. Basically you can now return a different plugin name from a plugin callback if you're proxying for that plugin. That should allow for a single executable or network connection in the above proposal to potentially represent multiple plugins if that's ideal from a performance/memory perspective for your use case.

An interesting thing about this proposal is that it would also make it possible to add the ability to run certain plugins from the CLI directly. I'm imagining flags like --plugin-exec:./path/to/executable and --plugin-net:localhost:8000.

I'm not planning on implementing this proposal immediately because I want to get the initial plugin API to a good place and then fix some other pressing issues. But I do think this direction could be useful for more advanced cases. I'm sure people will want to use it to write plugins in Rust, for example :)

[eigilsagafos]

@evanw Would it make sense to expose the esbuild configuration options to the plugins? I would like to be able to read both external (or have externals ignored by the plugin) and platform to mark node builtins as external.

[evanw]

FYI Plugin documentation is up now: https://esbuild.github.io/plugins/

@evanw Would it make sense to expose the esbuild configuration options to the plugins? I would like to be able to read both external (or have externals ignored by the plugin) and platform to mark node builtins as external.

Yes. That's come up before and is something I'm planning on doing. Right now I'm thinking that the options object could just always be provided to the plugin inside the setup function.

[chowey]

Is there a recommended way to distribute Go plugins? I can't think of anything clever. It seems like I need to re-compile esbuild with my plugin.

[evanw]

Is there a recommended way to distribute Go plugins? I can't think of anything clever. It seems like I need to re-compile esbuild with my plugin.

That's correct. The Go API is intended to be used by Go projects. Go plugins are intended to be distributed the normal way you distribute Go code (e.g. publishing on GitHub is sufficient).

Are you are trying to publish a package written in a native language to npm so it can be called by people who are using esbuild from JavaScript? It's possible to do this but it's a lot of work:

• You'd have to use node's child_process module to run the native code and you'd have to implement communication with it yourself (e.g. RPC over stdin/stdout like esbuild).

• You would have to figure out how to build and publish your native executable for all of the different architectures you want to support (esbuild supports 9 so far, for example).

• If you don't want installing your package to download the binaries for all architectures, you'll need an install script that detects the architecture and downloads the correct one, and you'll need to find some method to host them (I use separate packages on npm for esbuild).

  Getting an install script to work for everyone is tricky because people have lots of custom needs that need special handling (e.g. proxies, alternatives to npm, alternative package managers). It took me a while to get esbuild's install script to a good place.

[chowey]

Mainly I wanted to be able to write a Go plugin and have it usable by other members of the team. I understand now the use case for a Go plugin:

The Go API is intended to be used by Go projects.

This is totally fine.

It seems obvious now that if I want to create a plugin for the NPM ecosystem then I would then have to deal with the NPM ecosystem. It wasn't really what I wanted to do anyway. Go plugins for Go projects will work fine for me.