Esbuild: Any plans to expose the parser and lexer?

[matthewmueller]

Thanks so much for this project!

I was wondering if there's any plans to make the parser and lexer part of the public API? I think that would enable a lot more projects to build off of esbuild.

[tooolbox]

There used to be a disclaimer in the readme:

The Go code in this repo isn't intended to be built upon. Go is just an implementation detail of how I built this tool. The stable interfaces for this project are the command-line API and the JavaScript API, not the internal Go code. I'm may change the internals in a backwards-incompatible way at any time to improve performance or introduce new features.

[evanw]

I removed that disclaimer because it was no longer strictly true after I released the public Go API. But the spirit of it is still true.

I completely understand why exposing the lexer and parser would be useful, but it also adds a lot more additional work for me to develop and maintain a nice stable public API for the AST. This would also get in the way of my main goals for esbuild, which is to implement the MVP I have in mind. Exposing my internal APIs as public APIs comes with an expectation that they are stable, which would slow me down and possibly prevent me from making important changes to the AST.

I think it would be useful to think about an API for the AST when I start thinking about an API for esbuild plugins (#111). Although I was planning to only start thinking about that when esbuild's feature set is more finalized (e.g. code splitting, custom loaders, different output file types such as CSS).

I noticed you also filed https://github.com/golang/go/issues/39889. This is interesting because that worked when I originally set up the project without Go modules. I guess that broke when I introduced Go modules due to community feedback (#66).

Edit: It looks like maybe you found a workaround?

[evanw]

I'm going to close this as I believe I have answered the question about my plans.

See also #23 and #92.

[matthewmueller]

Thanks for your response. I totally understand wanting to keep the scope in check, given this is a side project.

Yep, I was able to get it working! I don't think it's anything broken on your end, it was just me not understanding how go modules work. I still don't quite understand how versioning works, specifically why go get github.com/matthewmueller/esbuild resolves to github.com/evanw/[email protected].

Using go get github.com/matthewmueller/esbuild@master seem to do the trick though. No need to rewrite package files :-)

[evanw]

Do you mind saying what you wanted the parser for? I'm curious about what you're doing with it and why you chose to use esbuild's parser over other JavaScript parsers written in Go with public APIs already.

There are some additional caveats that you should be aware of if you plan to use esbuild as a JavaScript parser:

• The parser is designed primarily with bundling in mind. It does not follow all of the rules in the JavaScript specification because some of them are not important for bundling. For example, esbuild's parser doesn't enforce any of the rules around strict mode.

• There are also some cases where it takes a "garbage in, garbage out" mindset. If the input is valid JavaScript then esbuild should produce valid output, but if the input is invalid JavaScript esbuild may not report an error and may generate invalid output instead. For example, esbuild will compile null = null without an error even though that's an invalid assignment target.

These limitations could be addressed, and I may address them as time goes on, but they aren't important for the bundling use case so I haven't focused on them. If there isn't a strong reason for using esbuild's parser over other parsers, you may be better off using another parser that's intended to be used as a parser to avoid issues like these.

[matthewmueller]

I really appreciate the heads up! I'm currently working on Svelte to Go compiler with the goal of being able to use Go to server-render Svelte components into HTML and have a JS bundle hydrate that HTML in the browser. Essentially giving you a Next.js-like experience without needing to use Node.js in production.

I think those limitations will be totally fine. I don't need too much Javascript intelligence to server-render Svelte components:

For example:

<script>
    let x = 7;
</script>

{#if x > 10}
    <p>{x} is greater than 10</p>
{:else if 5 > x}
    <p>{x} is less than 5</p>
{:else}
    <p>{x} is between 5 and 10</p>
{/if}

Each of the mustache templates can be a JS expression, so I was planning to use esbuild's parser to pull out these expressions. I also don't need much out of the <script> tag above, imports/exports would be nice but not essential.

I'm just prototyping right now, so I'm still not sure if this is a good idea – parsing potentially hundreds of JS fragments with esbuild. I'm planning to test the overhead. I may eventually just handwrite the JS expression parsing. Something like: https://github.com/soney/jsep.

I think I could use the other parsers for this job, but longer-term I'd like to migrate the whole Svelte client-side compiler to the Go stack so I can finally move off of Node.js as a build tool.

I'm also just a fan of this project, its code quality and you. You've managed to single-handedly replace babel & webpack for a lot of people. By better familiarizing myself with the project and its codebase, I hope to be able to do what I can to help push this project forward.

[matthewmueller]

Thinking about this some more, I think a plugin system would solve this problem for me!

I was trying to parse and execute at runtime like html/template, but it's totally fine (and forward-looking) to do these operations at build time.

[zbysir]

Thinking about this some more, I think a plugin system would solve this problem for me!

I was trying to parse and execute at runtime like html/template, but it's totally fine (and forward-looking) to do these operations at build time.

Hi, I feel happy because i do similar things with you, if you are interested, welcome to take a look at my project: https://github.com/zbysir/vpl, maybe it can inspire you.