Eleventy: Modify Eleventy to work with ECMAScript Modules (ESM) by default

[zachleat]

Node 13 projects can switch to ECMAScript Modules using "type": "module" in package.json, using .mjs files, or --input-type https://nodejs.org/docs/latest-v13.x/api/esm.html#esm_enabling

This causes problems with Eleventy, which uses require and CommonJS internally.

Here it is failing on a config file require:

Without a config file, it fails on 11ty.js files too:

---

Explore whether or not this is a possibility. Might need a major version bump? Might want to be prepared for Node 14 stable. We’re currently at Node 8+ right now but it exits maintenance very soon so we’ll need a major version bump to at least do Node 10+: https://nodejs.org/en/about/releases/

[justinfagnani]

I think you may be able to get support without a major version bump, though the support would only work in versions of Node with module support itself, which seems fine.

The first thing to change would be where the JavaScript template engine performs the actual require(): https://github.com/11ty/eleventy/blob/6d7e3a091742c1e57a5b01152cc804f6391c3ee3/src/Engines/JavaScript.js#L52

That will need to become async, but luckily it's internal to the template engine and only called from two already async methods.

Since you can import() a CJS module in Node with JS modules support, to detect and load a module, I think there's only two things that need to be done:

1. Detect if the environment supports modules
2. If so, use import() to load all JS templates, CJS or standard JS

import() only works in >= 13.2, but the syntax is valid from 10 on (not sure the exact version). So, if you support only Node 10+, this should be pretty straightforward. If you want to still support 8+, you'll need the import() expression in a file you only require after detecting module support.

As for that, I'm not sure the best way. you could just key off the Node version, but that would leave off environments in 12 using flags. That might be ok. You could also try to require a file with import(), and if that works, then try to import something, and if that works your'e in an env that supports modules.

That's the basics, but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules, at least without spinning up a new VM context to run the templates in and writing loader and linker functions to make it all go.

[zachleat]

Awesome, this info is very valuable—thank you!

but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules

Whoa, hmm—that would be a huge limitation. We need require/import cache invalidation to get new versions of templates during a --serve or --watch.

Not too much info on the docs either: https://nodejs.org/api/esm.html#esm_no_code_require_cache_code

[justinfagnani]

Yeah, that's the thing to figure out before any of the other work... I wonder how big of a change it would be to spin up a new VM instance for every hot reload?

But... the module support in vm is still marked experimental, and requires a flag, so I think this would be something for a future major version of eleventy.

[georges-gomes]

Hi!

ES module support would be so much nicer (at least for me). I started my own little SSG in ES Module only to achieve that because I thought 11ty couldn't make the switch. As @justinfagnani said maybe there is a possibility...

Here are the things I learn in the process and I would be happy to contribute a few things if I'm good enough :D

(the following are my observation on node 13)

1) There is no require cache for ES Modules like for commonJS.
what has been recommended to me is to use the internal v8 "Debugger.setScriptSource" to replace code live. After few hours of research I came up with this https://gist.github.com/georges-gomes/6dc743addb90d2e7c5739bba00cf95ea
It works most of the time but it fails quite often specially when you start modifying import/exports.
Bugs are open on v8 for better ES modules of this API.
Also, the code is replaced hot so none of the top level side effects are re-executed.
I think this is bad for our purpose here.

2) You can call import multiple times if you change the file name. On HTTP adding query params can do the trick of re-importing the module but it doesn't work on files. Node accept the query params on files but doesn't reimport. I think it's bug again. So you can still copy files and rename them then call import again...

3) You can import commonJS modules from an ES module.

import { createRequire } from "module";
const require = createRequire(import.meta.url);
const es_dev_server = require("a-cjs-module");

I think it would be a good practice to have eleventy boot code in ESModule and load "templates" in cjs with this when required.

Conclusions:
Like @justinfagnani (but it took me days to come to this :)), I think that a spawning a new VM for every hot reload is a better way to go in order to solve this issue and have a single code base for both one-shot generate and serve/watch operations.

I hope this helps
Cheers

[TimvdLippe]

I was able to make ES imports work with the following:

node -r esm node_modules/.bin/eleventy

It uses https://github.com/standard-things/esm to make both require and import {} from work in .11ty.js files.

[lennyanders]

I needed the following command on Windows to make ESM working with 11ty:

node -r esm node_modules/@11ty/eleventy/cmd.js

Also, with the following esm config, I got export default for the 11ty config (.eleventy.js) working:

{
  "esm": {
    "cjs": {
      "dedefault": true
    }
  }
}

[thelucid]

Any news on this? Would be great to be able to use import in Eleventy.

[kuworking]

I was able to make ES imports work with the following:

node -r esm node_modules/.bin/eleventy

It uses https://github.com/standard-things/esm to make both require and import {} from work in .11ty.js files.

Do you have any repo that can be looked at?
Does your setup allow for using something like this in a .11ty.js file?

import { css } from '@linaria/core'

exports.data = {
  title: '',
  date: '',
  templateEngineOverride: '11ty.js,md',
}

exports.render = data =>
...

[thelucid]

@TimvdLippe Does this allow imports in standard ‘.js’ data files?

[TimvdLippe]

@thelucid This is our invocation to Eleventy: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/c12e43d2054d074ac07f8990c4b4be54a172c3cf/package.json#L22 and this is one of our generators: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/master/pages/1-2.11ty.js

[thelucid]

@TimvdLippe Does the .11ty.js do something special over just calling your file 1-2.js? Pardon my ignorance, as I've just been using things.js in the global data directory, and want to use imports within these.

[TimvdLippe]

@thelucid We specify it as input here: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/c12e43d2054d074ac07f8990c4b4be54a172c3cf/.eleventy.js#L11 but other than that we don't do anything special.

[thelucid]

I'm hitting issues with using imports within the data files, in your case _data. So something like _data/things.js doesn't work with imports.

[thelucid]

@TimvdLippe Got it working. Thanks so much. It should be nice to see eleventy sport this by default, so that imports work out of the box.

[justinfagnani]

I'm not sure eleventy should support esm directly, since it's pretty non-standard in its capabilities. I'd like to see Node's VM modules support to stabilize and eleventy can use that to load projects and still support watch mode by creating a fresh context.

[thelucid]

I see, it that why reloads seem to have stopped working with ‘esm’?

[Zearin]

Recent related experience

I recently started trying to port a project that used a deep _data/ directory (yes, the same name! ☻), *.mjs files, along with import and export.

I had to rename a lot of files, and grep my files for uses of import and export, which took a few passes.

It wasn’t particularly awful. But it _was_ tedious to do. Also, I was personally a little disappointed about having to return to CommonJS syntax, as I’ve switched over to ES module syntax everywhere I can.

If not for *.js, at least *.mjs?

Eleventy _already_ makes many decisions based on file extensions. Simply telling Eleventy “If you see any *.mjs files, parse them as ES modules” would leave existing behavior as-is, while still allowing users to opt-in to ES modules by changing a file extension.

Would an extension-based approach address concerns raised above