Coming from this comment: https://github.com/nodejs/modules/issues/151#issuecomment-404437493
It is now possible to create executable, extension-less, files:
#!/usr/bin/env node
console.log(__filename);
But it's not possible to enable ESM as parsing goal.
#!/usr/bin/env node -m
console.log(import.meta.url);
Even if there are OS incapable to parse the whole shebang up to the -m flag (or whatever flag will land in nodejs), there is no way to even define an executable that would like to parse the source file without any extension.
Given the following esm file, reachable through /usr/local/bin or similar OS folder:
#!/usr/bin/env bash
node --module $1
And given the following executable:
#!/usr/bin/env esm
console.log(import.meta.url);
It should be possible to have extensions-less files parsable as ESM.
There is a solution to the single file problem that would still require --module hook to bootstrap the file as ESM parse goal.
There are a variety of solutions we should probably evaluate and see if they are or are not feasible that I can think of.
Right now we can point executables without extensions to other files using symlinks. I'm not curious about what file extension maps to what format, but I am curious about if that disambiguation and/or indirection method is infeasible. I'd be curious about places that cannot support that behavior. Most installations put the .cmd/Symlink as an indirection layer to a folder and the main file for "bin" installations through package.json so this probably falls on if the file extension can be used, whatever it maps to.
In addition the executable could stay CJS and require/import() to another file to change the mode. This also is a method of indirection, but is not relying on file extension and instead relying on w/e disambiguation method is done at runtime, even if it uses the file extension it isn't mandating such.
I also am concerned about hashbang being unusable not just on linux shells but on windows. I agree it is a design space that we can look into but I am not sure how feasible it is. I do not think loaders can solve this problem using a hashbang either because they currently are not runtime configurable, nor am I convinced they should become runtime configurable due to this single issue. If we were to use a loader configuration it would need to be done outside of the file using CLI/ENV/package.json/etc. That makes me think we might want to look at indirection so that we can get that out of band data somehow.
I do have scaling concerns for having multiple executables because things like BinaryAST / WASM / WebPackage are other goals I'd like to support in the future and they would explode the number of executables that node ships with. I think however this format is configured and sent to node it should be planned around adding at least those 2 formats as well.
I do not think wrapping processes are a way to solve this problem either. Notably, Windows does not have an exec capability like unix/shells that can replace the same process ID, and using child_process would create wrappers that might be problematic for various process ID tracking service manager.
Forgot to add, that even though Loaders are not configurable, they could be done in a per package manner for things that are not installed using "bin" from package.json approaches.
Not sure I follow, but I've used Linux in pretty much every dev/production env I know and written dunno how many binary files based on the hashbang #!/usr/bin/env node.
I also am concerned about hashbang being unusable not just on linux shells but on windows.
NodeJS ignores hashbang too since ever because it's been a valid use case for long time so it shouldn't probably be under discussion the ability to create nodejs executable based on ESM, but I hope I've misunderstood your reply.
@WebReflection that is executing the node executable as the shell that invoked the file parsed the hashbang (well env is invoked really). My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable. Even then, the declaration of the executable through the hashbang is picked up by the shell and not node itself.
My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable.
then I misunderstood, and indeed since it doesn't work in the only env I've used it (Linux) I guess we would need a better way.
I don't have ideas for now but I'll keep thinking about it.
If we want to avoid passing parameters in the shebang, we could create an excutable wrapper that passes them (nodem, node-m).
But from my point of view that would be confusing for people used to run node.
This is one of the benefits of the package.json "mode" proposal in that it can allow this scenario to be handled clearly (when executing the "bin", the package.json is loaded and used as source-of-truth of the format).
@xtuc's suggestion is an interesting alternative here too, although agreed being weary of adding to confusion.
Donât forget the case where the extensionless file is outside the package folder, e.g. /usr/local/bin. For example, when CoffeeScript is installed globally, e.g. npm install --global coffeescript, it puts a symlink /usr/local/bin/coffee that points to /usr/local/lib/node_modules/coffeescript/bin/coffee (on Mac/Linux systems, anyway). Maybe in the case of followed symlinks Node can look for a package.json in the folder where the symlink resolves.
node by default uses the extension of the resolved file, not the symlink e.g. file.json -> file.js is seen as cjs not json
FWIW @xtuc suggestion is exact equivalent of my ./esm binary with content:
#!/usr/bin/env bash
node --module $1
This is a pragmatic solution to the shebang gotcha but it feels future hostile needing that in order to have an executable based on modern code so, unfortunately, we might need a better idea than just an indirection đ˘
@WebReflection whats wrong with the symlink indirection that most things do today?
@bmeck if I understand correctly that would require a package.json somewhere else, right? In AUR (or other packaging formats different from npm) that might not be easy/straight forward to implement.
I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.
@WebReflection it would rely on some file that it points to being unambiguous, yea. Not necessarily using package.json, could be a file extension, etc.
I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.
I think this is where all of these are breaking down. Perhaps, the way that things are done today doesn't scale well, even at 2 possibilities we are seeing problems unless we define a different mechanism than today. Wait for BinaryAST and WASM entrypoints and we have 4 possibilities.
@bmeck I don't think anything else different from JS would have the same issue for the simple reason I don't think WASM would allow a shebang on top, right?
Anyway, I forgot I have some hackery to start GJS so that this would be my solution:
#!/usr/bin/env bash
Function=Function//; node -m "$0" "$@"; exit
console.log('ESM');
Function has no meaning but you can assign it to anything, including slashes //; after slashes ends the variable assignment and bootstrap node -m "$0""$@" and the bash program exits (after node exits too so nothing else will be executed / parsed / interpreted from _bash_)Function to itself, or better, there won't be any side effect, and the comment will nullify bash.All it's missing now, is this mechanism to bypass the file extension and enable the --module like parser so that import.meta.url or any other ESM related syntax would be valid.
I don't think anything else different from JS would have the same issue for the simple reason I don't think WASM would allow a shebang on top, right?
I think you are over focusing on a hashbang based solution. Hashbang doesn't work for a variety of cases or at all on some deployment targets like windows. I'd be wary of things that can vary per shell and aren't even in some environments.
windows has numerous way to bring in bash though, but yeah, I'm focusing on developers, those that actually use NodeJS the most, and on Servers, those by stats dominated by Linux.
Focusing on windows without Linux subsystem or chocolate bash, aka focusing on non developers, looks like not so important. Windows has many ways to ship binaries, including ways compatible, or based, on shebangs.
npm would also solve that case via bin, but we miss a way to #!/usr/bin/env node which has been historically both possible and very much used for server side development.
npm would also solve that case via bin
even better, you could use my technique both via npm and as a stand alone file/application, so I don't see any reason to ignore it as use case/solution.
Is this feature documented? can this be closed?
Yes, itâs in the features list in the README: https://github.com/nodejs/modules#existing-nodejs-utility-features
Closing based on above comment. please re-open (or ask me to) if this was a mistake
I don't think this should've been closed until there's a reliable way to do this cross platform without relying on bash hacks.
FWIW, the unambiguous grammar proposal would've easily solved this. :(
FWIW, the unambiguous grammar proposal would've easily solved this.
FWIW, I think that's the only way to solve this (without bash hackery)
FWIW, I think thatâs the only way to solve this (without bash hackery)
Or the shebang suggestion, like starting your file with #!/usr/bin/env node --mode=esm, but people pointed out that that had various issues such as Windows compatibility.
I had suggested elsewhere that unambiguous grammar could be treated as a fallback, like if the mode wasnât explicitly defined via something in package.json or a CLI flag, then Node would try to figure it out by parsing the entrypoint (and then all subsequent imported files would need an explicit marker like an .mjs filename or package.json field). People didnât like this approach, but Iâm still not persuaded why simply halting on an import statement is better, especially if the behavior is limited to extensionless files. I understand the concern where once unambiguous grammar is in the runtime at all, people will take advantage of it as a way to avoid needing to specify the module mode any other way, and then we need to deal with the complexity of adding new syntax in the future, like import(), while avoiding breaking old scripts that depend on some older unambiguous grammar algorithm.
The minimal kernel currently in progress inherits the assumption from --experimental-modules that module mode is determined via metadata in the script or module, e.g. file extension or package.json propertyâa.k.a. author-driven disambiguation. This is at the same time that the minimal kernel (and Node core) just adopted createRequireFromPath, which essentially enables the NPM implementationâs approach of consumer-driven disambiguation where require is used to import CommonJS and import is used to import ESM, and the metadata from file extension or package.json isnât used (yet). I understand the arguments against consumer-driven disambiguation, not least that itâs way more frustrating for users who would then need to figure out what type of module theyâre working with in order to use it. But author-driven disambiguation _could_ be achieved through unambiguous grammar, if we could come up with a bulletproof unambiguous grammar algorithm that worked and we were convinced would be future-proof. Of course, good luck with that.
Even if that could be designed, though, relying on unambiguous grammar for determining module mode has its own issues as listed succinctly in https://github.com/nodejs/modules/pull/150#issuecomment-406515253. Part of me still would love for someone to write up a proposal making the case for author-driven disambiguation via unambiguous grammar; @bmeck did a lot of work on unambiguous grammar years ago, but basically walked away from it as he didnât think it could be made to work. But if someone wants to give it a shot, Iâd love to see another attempt đeven if we come to the same conclusion. If nothing else, it would help persuade the mob that every attempt was made to try to come up with a more developer-friendly UX than whatever we end up shipping with. Or if a workable unambiguous syntax algorithm _could_ be designed, well, that would be very interesting. Weâd still have the other issues to address, perhaps by shipping unambiguous grammar along with metadata flags like file extension and package.json fields, but weâd have more options.
!/usr/bin/env node --mode=esm
Unfortunately the issue there is not just Windows, but Linux too, where that wouldn't work, you need my suggested hacky bootstrap.
But executables with shebangs are extremely common in Linux (servers), so anything that wouldn't work in there makes little sense, in terms of wasted time, IMO.
@GeoffreyBooth itâs not really worth another attempt - it would require TC39 to change the spec, and a number of members of the committee have expressed legitimate opposition such that (unfortunately) it basically can never happen.
itâs not really worth another attempt - it would require TC39 to change the spec
We thought the same thing about named exports, but they appear to have moved on that, havenât they?
Again I donât see how a proposal could resolve all or most of the issues raised in https://github.com/nodejs/modules/pull/150#issuecomment-406515253, so like you Iâm not optimistic that an unambiguous grammar solution is possible. But I _would_ still love to see someone take a crack at persuading me (and perhaps the TC39) otherwise. All it takes is a gist explaining in pseudocode what the proposed unambiguous grammar algorithm would be (perhaps based on @bmeckâs prior work) and how it would be future-proof, and how to address the other issues raised by https://github.com/nodejs/modules/pull/150#issuecomment-406515253.
@GeoffreyBooth i don't think you have an accurate understanding; it's more that the committee has always wanted to help node, but previous attempts weren't tenable. The recent attempt, however, was tenable. As it relates to unambiguous parsing, the committee as a whole actively does not want unambiguous parsing, no matter the solution - many members explicitly want it to be ambiguous, and as such, it's not worth any attempt, because conceptually it's doomed from the start (as dynamic named exports were not).
Why do they want potentially ambiguous parsing?
@GeoffreyBooth specifically, they want users to be able to type live into a <script> or <script type="module"> and not have the parsing goal change on the user. Having unambiguous parsing doesn't inherently conflict with this, but it's impossible to change Scripts to eliminate the ambiguity, and pretty much nobody wants to add a permanent tax to Module code like export {} or "use module";, so there's not really any wiggle room here.
When would a user type live into a <script>?
@GeoffreyBooth it would be any live debug tooling, not just (but including) <script>
Can you be specific? Do you mean like typing into the browser console?
@GeoffreyBooth for example any source in v8 can be modified in place using the inspector (in chromium this is exposed through the devtools).

Iâm not following with how Node supporting unambiguous grammar disqualifies that use case. I see in your example it throwing on ESM code when parsing a file as non-ESM; without unambiguous grammar, how would DevTools know to parse as ESM? Via some external thing, like a checkbox in the UI? Or it could use the MIME type or file extension of the file. Regardless, all of those ways to signify module mode could continue to exist and be used whether or not unambiguous grammar is an additional signifier. DevTools doesnât even need to use unambiguous grammar if it doesnât want to, or it could trigger a prompt like âYou just typed export, do you want to switch to ESM mode?â
Perhaps what happened was that two years ago unambiguous syntax was proposed as _the_ way to signify module mode, and that got rejected. But as _a_ way to signify the mode, among other methods, it could still work. Especially if not everywhere that JS is parsed needs to support unambiguous syntax.
Certainly any file that has import or export in it is a Module, and a file with neither that has with is a Script (and there's probably a few other similar heuristics), but you'd still end up with nonzero possible code patterns that could still be ambiguously a Script and a Module, and that could thus have silently different behavior.
If it doesn't work in 100% of the cases, it doesn't work.
If it doesnât work in 100% of the cases, it doesnât work.
If it doesnât work in 100% of the cases, it canât work _on its own._ But thatâs not the design space weâre working in. Our modules implementation will include .mjs, a --mode or similar flag, and something in package.json. When an explicit signifier such as any of those is present, Node knows what to do and will do it regardless of the syntax in the file. Unambiguous grammar comes into play only when thereâs no foolproof signifier like those available. This extensionless executable file is one such use case.
@ljharb what youâre describing is, in fact, the beginnings of just such an unambiguous grammar algorithm. If it has import or export, else if these other things etc., treat as ESM; else treat as CommonJS or Script. Having such an algorithm would solve the executable file case. Unambiguous grammar clearly has major downsides as compared with those other signifiers, but it also works in some places where those ones donât (executable files, the REPL, etc.). Itâs not going to replace the other signifiers, but itâs also not worth dismissing out of hand.
@GeoffreyBooth regardless of who here says what, node is not in the position to be making decisions about whether or not to use an unambiguous grammar. we are a javascript runtime, and javascript has an ambiguous grammar between scripts and modules. therefore, we as the runtime should not be making decisions based off of the grammar.
Sometimes the proper place for a solution is not core, and the behaviour you want can be put into a userland resolve hook quite easily:
try {
parseAsScriptWithBareReturn(source);
return { format: 'cjs' };
} catch (e) { // hits `export {}` or something
parseAsModule(source);
return { format: 'esm' };
}
we as the runtime should not be making decisions based off of the grammar.
Why not?
It seems to me like this use case, extensionless executable files, has three potential solutions:
'use module'; pragma.Other suggestions are welcome, but I donât consider no. 1 acceptable.
Certainly thatâs a use case we should enable somehow, but that is a very narrow use case that doesnât need to impact - in any way - requires/imports, extensioned files, or maybe node entrypoint.
IMO live changing <script> tags is far more obscure a use case than bin scripts, especially if the desire is to be able to flexibly change between parse goals without reloading the page. That way is madness.
@ljharb You can't say "pretty much nobody wants to add a permanent tax to Module code like export {}" - the UnambiguousJavaScriptGrammar repo has over 400 stars. It was a clean elegant solution with a "tax" that would apply very rarely. Any alternative still has a tax of some kind, it's just an external tax, whether it's the file extension, or a field in the package.json. Of course there's the potential downside of double parsing, but external tags along with smart heuristics would avoid this cost in the majority of cases.
But TPTB have said unambiguous grammar is not an option. Okay we just have to work with what's allowed. But can this issue please be reopened because Function=Function//; node -m "$0" "$@"; exit is not a solution anyone wants.
It'd be better to say "go use standard-things/esm" than this kind of bash wizardry.
@curiousdannii
The common case of bin scripts is by far global installation, where symlinks are used anyway, so this problem doesn't occur.
In the case of directly running a script, I see having .js or .mjs just as crazy as having .sh (e.g. it's not crazy at all!).
P.S. why node -r @std/esm xyz over node --mode=esm xyz?
@curiousdannii by ânobodyâ i mean ânobody who writes the spec or builds a browserâ, and browsers donât care about âexecutablesâ because they donât have that concept. As for stars, i donât think star counts matter, but if they did, thatâs not a persuasive number.
I fully agree that something ergonomic is needed for extensionless files, but unambiguous grammar isnât going to be it. The feature is documented in the readme, so the issue doesnât need to remain open to track it.
The curse of ES in NodeJS is the M of the word module which has nothing to do with this specific issue.
Here the issue is about having an executable file as valid ES6+ syntax, with the ability to eventually use import.meta.url for all the reasons an executable file would need, including the ability to require CommonJS through the mechanism defined in phase one.
ES6+ is not just about writing modules to import or export stuff, is about the environment being different from CommonJS.
Every file since ever has meta informations on their headers, executable, graphics, zipped, text and BOM, you name it, having a "use esm"; on top after the shebang, a practice used for ages that, as ugly as it is worked well for strict and will never be adopted again from TC39 so it's safe to use because has zero side effects and it's not future hostile, would be the easiest way to go for NodeJS disambiguation.
But no ... that won't happen, because without any data, poll, consensus, we have people stating "_nobody wants this, nobody does that_" pretending to represent the whole community.
I'm willing to dig into the meeting documents regarding pragmas and talk to folks at the committee about it again
@WebReflection your decision to continue to engage in a hostile way with a focus specifically on extensions is not helpful
I totally agree and Iâd love to have a required pragma, but it didnât have consensus - ie, at least one person objected to it. We have to make the best decision we can with what we have - which is ambiguous parsing goals, no required in-code disambiguation mechanism, and only the filename from which the author can communicate what kind of file it is absent a package.json.
Agree that @WebReflection can tone it down please, but also statements like âsomething ergonomic is needed . . . but unambiguous grammar isnât going to be itâ come off to me as disrespectful to the group and our process. We as the modules group get to say whether or not we use unambiguous grammar and when, and no individual member gets a veto. Pointing to some TC39 vote years ago when the context was very different (before .mjs, before browsers decided to use MIMEs) feels to me like trying to shut down a legitimate debate. Like @MylesBorins said, we can always go back to the committee with new information and new suggestions.
That said, unambiguous grammar is really just a much more complicated pragma, so if people prefer that solution then that also works just fine. @devsnek Iâd love to see a demo of how symlinks could work, if thatâs possible, including how they would work for Windows. If thatâs another potential solution by all means we should look into it.
Something else we need to figure out is how to do âmulti-modeâ executable files, where e.g. babel or coffee could point to an ESM entry point for ESM-supporting Node and a CommonJS entry point for old Node, or maybe a single entry point that can kick off either code path.
@GeoffreyBooth symlinks already work. ln -s X.mjs X and node X will see it as esm
@MylesBorins I have literally not even mentioned the extension once in this whole thread, just the fact people keep using the word module. Everything I write you read it as hostile. I guess I'm done here, as self-banning myself.
What the heck.
@GeoffreyBooth the context hasnât changed; mjs and mimes were the plan during the entire unambiguous grammar process with TC39 - it wasnât as long ago as you think.
Certainly node can decide to require unambiguous grammar - but weâd be doing that by restricting ourselves to a subset of JavaScript programs, which doesnât seem like a good idea for node dot JS to be doing. As it relates to TC39, thereâs really no further debate to be had. If you want to debate doing our own thing, we can, but i donât see how thatâs legitimate or productive.
@devsnek Apologies for not actually checking that an executable .mjs script would just work. With all the trouble I had for extensionless scripts I'd assumed that at some time I must have also tried it with the extension, but I guess I never had (even though my solution was to have the extension and then make a sh launcher script!) In my defence the docs don't say that it's possible or that this exception to needing --experimental-modules exists :).
Pardon if I'm misunderstanding the documentation, but can't a module-based extensionless script work as ESM by putting itself in a package's bin directory and having "type": "module" in package.json?
Is this a new development since this issue was last commented on, or is that was meant when people mentioned NPM being able to handle it above?
Oh, the problem is that that only applies if --experimental-modules is set, isn't it? And it's not possible to set multiple flags for a shebang?
What's the roadmap for when ESM will be enabled without a flag? Sometime in 2020?
This works now, and youâre right the --experimental-modules makes it awkward:
⌠cd /tmp && mkdir test && cd test
⌠echo '{"type": "module"}' > package.json
⌠echo '#!/usr/bin/env node\nimport { version } from "process";\nconsole.log(version);' \
> executable
⌠chmod +x ./executable
⌠export NODE_OPTIONS='--experimental-modules'
⌠./executable
(node:54196) ExperimentalWarning: The ESM module loader is experimental.
v12.6.0
You can use clever shell scripting to avoid the need for package.json: https://gist.github.com/WebReflection/8840ec29d296f2fa98d8be0102f08590 (via #318).
The plan is to unflag when Node 12 goes LTS in October 2019.
With unflagged modules in 15.5 I'm still getting the error:
node:internal/process/esm_loader:74
internalBinding('errors').triggerUncaughtException(
^
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ""
package.json:
"type": "module",
"module": "index.js",
"bin": "simple-ddos",
@talentlessguy, try the following instead. Node.js does not use the ´"module"´ field and be sure that your paths are relative.
// package.json
{
"name": "mood",
"type": "module",
"exports": "./index.js"
}
You also donât ever need type module; you can name your file with .mjs instead.
Iâm not sure extensionless files work with ESM at all, though, unless theyâre the node entry point and you pass the input-type flag.
@DerekNonGeneric thanks for the tip but it drops the same error :/
@ljharb I tried to do #!/usr/bin/env node --input-type="module" but it gets stuck this way and doesn't run the script:
#!/usr/bin/env node --input-type="module"
the solution I came up with is to create a shell file that launches the js script, e.g. this
âââ bin
â âââ script
âââ cli.js
script's contents:
#!/usr/bin/env sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
node $DIR/cli.js "$@"
you can name your file with .mjs instead
that's half a solution, but also, imagine you had to launch executable files like Chrome.cpp or terminal.gjs and so on đ
I really hope soon there will be a way to have an ESM only executable files, that also force-load everything as ESM without needing a package.json around.
@WebReflection i totally agree there should be some way to make it work without an extension - but i think itâs very important that âtype moduleâ doesnât get thrown around and misunderstood as a solution to anything beyond the one tiny thing it does :-)
@ljharb it's not so tiny though ... and until there is a --default-input-type="module" flag around, people will abuse the package.json hack here and there, so the sooner we can use that flag, the better for anyone willing to have ESM only executable files.
It has nothing to do with extensionless files, though, which is the only relevant part of this thread.
it does! the day that works one can write:
#!/usr/bin/env sh
node --default-input-type="module" anyfile.js "$@"
and call it a day
edit
but yeah, it's not a full solution for extensionless that would like to use node as env, but there are workaround for that, although these need to hack the package.json here and there, which is undesired.
I've already written about solutions for extensionless a while ago, not going to re-iterate there, but mine is a hack, it'd be great to have something native, including node-esm distributed file (these days deno kinda does that)
Right, the goal is avoiding a wrapper like that. type module is utterly unrelated - your wrapper could point to an mjs file just as easily.
Most helpful comment
The curse of ES in NodeJS is the M of the word module which has nothing to do with this specific issue.
Here the issue is about having an executable file as valid ES6+ syntax, with the ability to eventually use
import.meta.urlfor all the reasons an executable file would need, including the ability to require CommonJS through the mechanism defined in phase one.ES6+ is not just about writing modules to import or export stuff, is about the environment being different from CommonJS.
Every file since ever has meta informations on their headers, executable, graphics, zipped, text and BOM, you name it, having a
"use esm";on top after the shebang, a practice used for ages that, as ugly as it is worked well forstrictand will never be adopted again from TC39 so it's safe to use because has zero side effects and it's not future hostile, would be the easiest way to go for NodeJS disambiguation.But no ... that won't happen, because without any data, poll, consensus, we have people stating "_nobody wants this, nobody does that_" pretending to represent the whole community.