Node: Consider something clever for extension-less executables

[haraldrudell]

It would be nice if ECMAScript shebang files with no extension
a. worked :)
b. were assumed to be modules for Node.js current+

Especially for server people that have lots of command-line scripts

And I think we should plan for a future without require

exhibit:

node --print '"#!/usr/bin/env node\n// node --experimental-modules e\nimport {inspect} from QutilQ\nconsole.log(`${inspect(QabcQ)}`)".replace(/Q/g, String.fromCharCode(39))' >e && chmod +x e
./e
(node:38910) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
/opt/foxyboy/sw/pri/subtree/ecmascript2049/packages/e:3
import {inspect} from 'util'
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at wrapSafe (internal/modules/cjs/loader.js:1063:16)
    at Module._compile (internal/modules/cjs/loader.js:1111:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)
    at Module.load (internal/modules/cjs/loader.js:996:32)
    at Function.Module._load (internal/modules/cjs/loader.js:896:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47
$ cat e
#!/usr/bin/env node
// node --experimental-modules e
import {inspect} from 'util'
console.log(`${inspect('abc')}`)

This shebang no work either:

!/usr/local/bin/node --experimental-modules

The solutions at present is to:

1. use .mjs extension
2. symlink to a file that has .mjs extension. Not portable, I suppose, on certain file systems from up north and Applish implementations of WebDAV
3. have a package.json in the same or superior directory. May cause unforeseen troubles
4. symlink to a directory with a package.json. Possibly not portable

node -v && uname -a
v13.11.0
Darwin c87m1.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64

A related trouble is if you happen to have a native-code dependency. requires node_modules and such

[devsnek]

Duplicate of #23868

[devsnek]

We can't pull data out of the hashbang (see the issue I linked), so I'm not sure as this issue is phrased there's anything actionable. In general, extensionless mjs is a place where improvement could be made though.

[guybedford]

This use case was explicitly enabled in early implementations, but disabled by the PRs by @bmeck and @geoffreybooth in https://github.com/nodejs/node/pull/31021 and https://github.com/nodejs/node/pull/31415.

I disagreed with both of these PRs, but let them pass as I did not want to block new work unnecessarily.

I still think the original approach with both of those PRs reverted would be better (31415 followed by 31021), but will not work on it personally.

If anyone wants to put something together along those lines though it would have my full support and argument in the group, although many disagree with the idea that the process main can be used as resolution information. Personally I have no issue with that since it is environment-static, and any environment-static state is fine to assume in the resolver (the definition of what is allowed!).

[bmeck]

I've stated in the past my concerns with being clever about these things.

Per #31021 , it ended up supporting files without extensions and delegated them to the .js extension handling. I would never want to revert the actual change of errors on unknown extensions it contains. Things like a new .wasi file format changing from a .js handler to its own is very concerning to me.

For context, per #31415 , that was done because of https://github.com/nodejs/node/pull/31388 which was made to support loading WASM entry points for WASI being complicated by extension-less mains. Reverting out any extension-less support was seen as preferred for now as some people didn't want to expand the "type" field in package.json.

@haraldrudell an alternative is to use symlinks that point to a file with an extension or have your package.json contain the mapping like: "bin": {"foo": "bin/foo.js"}.

[guybedford]

Per #31021 , it ended up supporting files without extensions and delegated them to the .js extension handling. I would never want to revert the actual change of errors on unknown extensions it contains. Things like a new .wasi file format changing from a .js handler to its own is very concerning to me

Bin users writing bin files ending in .wasi is a very obscure use case to cause the overall feature to be blocked on. While in theory breaking, such a change would have low enough usage to be possible from a practical standpoint.

[bmeck]

@guybedford I don't understand the last comment, the concern is about .wasi files changing interpretation not necessarily about them being the main entry point. These discussions become more complex as things like application runners do wrap the main entry point and would need to account for the variance.

[GeoffreyBooth]

@haraldrudell If you’re looking for something clever, this gist by @WebReflection certainly fits that bill and might fulfill your need:
https://gist.github.com/WebReflection/8840ec29d296f2fa98d8be0102f08590

[WebReflection]

@GeoffreyBooth thanks for pointing at my good old gist! It looks like time is about right to improve that gist and place a temporary {"type":"module"} package.json file as the dirname "$0" path and/or the user $HOME while executing, but I feel like the elephant in the room is that there's no flag to handle also all imports as _ESM_ when a script is being executed as _ESM_, which is the expected behavior, IMHO.

[WebReflection]

Hint: --import-type=module sounds about right to me 👋

_edit_: --default-type=module maybe is a good one too

[guybedford]

I don't understand the last comment, the concern is about .wasi files changing interpretation not necessarily about them being the main entry point

@bmeck the previous code only affected files as the main entry point. So node x.wasi would run it as JS, and yes that would be breaking to change it to wasi in future, but only if it were realistically used by people as a JS entry point with a .wasi extension. The previous approach did not support non-extensions for anything other than the "node main".

[haraldrudell]

This must be done and it is actionable

require is over; it should not be used anymore

note: why I bring it up now, is that I came up with a toolchain where the source, config files, packages and output executable all use import https://github.com/rollup/rollup/issues/3443

I know about that hack suggested in comment. I don't do hacks

the question is what's the best way and who are we breaking. To me personally, I will just retranspile any failing executable using my fancy toolchain: I am good

note that a shebang other than #!/usr/bin/env node is not portable. In particular other shebang, if done for Linux, will not work on macOS and Android

erase the past

[bmeck]

@haraldrudell I have no idea what most of the last comment is trying to say except that you want to break things, but not how nor any response to the other comments in the thread except that gist. Is there are reason none of the alternatives work clearly / what is the clear use case that we are not fulfilling besides a specific solution to the use case?

[haraldrudell]

The use case is single-file deployment of extension-less ECMAScript module executables to bin directories on unix-like operating systems Linux macOS Android

[bmeck]

@haraldrudell I'm not convinced that is common in real world deployments given that even things installed via npm do not do that. Do you have clear examples of this being common enough and reasonable enough to invalidate other guarantees as mentioned above?

[haraldrudell]

I do not know everything

What would someone expect Node.js to do? execute ECMAScript
shebang should provide this expected behavior
require is no longer what the expected behavior is
shebang, and Node.js, should not hold on to 2018 behaviors

Your concern seems to be that this expected behavior will make wasi wasm implementations more complicated. Apparently, pull requests referenced above were written based on a 2018 invariant. Code rot is a bitch

[bmeck]

@haraldrudell CommonJS is not being removed from Node.

[GeoffreyBooth]

@haraldrudell This was discussed here: https://github.com/nodejs/modules/issues/318

[haraldrudell]

CommonJS, though still supported, is going the way of the dinosaurs

ECMAScript modules for shebang and maybe other aspects is a question of when, and I think when is finally here. The discussions referenced are a year old, when that when was not yet here

Looking at the wasi module documentation, there is use strict, require, self-executing function expressions, readFileSync and async functions without catch. That's a bizarre history lesson from ES5 of 2009, from as old as is Node.js. There is not going to be any sympathy from there
https://nodejs.org/api/wasi.html#wasi_webassembly_system_interface_wasi

esm for shebang will only break code that is not in or symlinked from a package because of the ingenious type field in package.json. Maybe it does not break anyone

I have heard of the Node.js Technical Steering Committee from all the scandals. Is this not their decision purview?

[bmeck]

@haraldrudell Modules WG meetings are quite often, and notes are kept of them as well as recordings. I urge you to keep discussion on topic and civil. I was the champion for https://github.com/tc39/proposal-hashbang and am well acquainted with these topics. I can guarantee that all individuals involved in this project are working towards a better tomorrow. I am not sure what scandals you are referring to, but the TSC does get involved and has weighed in on several issues in the past; however, in the governance structure of the project and within their corresponding topics, working groups tend to be the leadership point for decisions.

[GeoffreyBooth]

@haraldrudell If you read https://github.com/nodejs/modules/issues/318 and the resulting proposal, that led to this: https://github.com/nodejs/modules/issues/335#issuecomment-503731771. And the result was that existing options handled all of the discussed use cases except the one you mention, of a desire to have a “loose” (outside of any package/package scope) JavaScript file with no extension and _with_ ESM syntax.

That’s still a gap, because our first priority is to avoid breaking backward compatibility unnecessarily. While CommonJS may be considered legacy by some, there are millions of CommonJS packages out there and it will be many years before it becomes commonplace for folks to build Node apps with no CommonJS dependencies. Therefore we can’t simply deprecate CommonJS or make ESM the default in place of CommonJS, at least not for several years.

So then the question becomes, how to fulfill your use case within these constraints? Currently, the best option is the technique in the gist. It has environment restrictions, though any shebang-based approach will inevitably have some environment concerns. Another option is to put a package.json at some high level, like your drive root /package.json, with {"type": "module"}; though this will break any loose CommonJS files.

One thing we’ve considered off and on for years is a new binary, e.g. node-esm, that is ESM by default. So node-esm --eval 'import "fs"' would work. You could put this in your shebang like #!/usr/bin/env node-esm, and at least that should work for the environments that handle shebangs. But I think adding a second Node binary has its own complexities and is something that I’m not sure Node (the project) wants to consider, since this use case isn’t considered overly common. But a node-esm binary should be something that a userland project could build.

[haraldrudell]

This decision should be based on accurate facts

We're not breaking millions of CommonJS packages and such dependencies will still work

The question is ECMAScript from the file system that has no extension, is not symlinked to or resides inside of a package.json hierarchy. Such files are unlikely to come from any registry or installed package. One thing @bmeck got right is that this is probably rare

shebang ECMAScript modules is apparently the way things were between 4/23/2019 and 12/18/2019. Are there any known complaints from these 8 months of Node.js versions?

from nodejs/modules#318 and https://github.com/nodejs/modules/issues/335#issuecomment-503731771 require can still be used: node --input-type=commonjs --print "require('tty')". Add input-type or NODE_OPTIONS: options and environment variables are already there

The default should be ESM. This is still a gap. It used to work until somebody broke it

[WebReflection]

Another option is to put a package.json at some high level, like your drive root /package.json, with {"type": "module"}; though this will break any loose CommonJS files.

What do you think about having --import-type=module (or --default-type=module) to bypass the need for a package.json file with type module in it? All rules will remain identical but an import "./file.js" would handle that file as ESM by default. It would basically simulate the presence of a package.json file with type module if none is available in the current folder structure, so that nobody needs to put type module in a disk root, 'specially 'cause that easily breaks, hence it's not really a solution.

[bmeck]

@WebReflection the problem is with the hashbang not being a way to pass CLI flags reliably and the desire not to use something like the sh executing approach. So, even with such a flag we would need to use the sh approach.

[WebReflection]

@bmeck the sh hack is just a way to solve this issue, but it simply uses flags usable via node. My question is not directly related to the sh hack, rather to the fact that same as --input-type is needed to specify, and override, the eventual default type, --import-type, or --default-type, would override the current {"type": "anything"} whenever it's needed, as two package.json files cannot coexist in the same folder, and will race on parent folders, while two node within the same folder can be parallelized without any issue, as long as these can also determine how they should import, by default, other files.

[bmeck]

@WebReflection I'm not stating that the feature shouldn't be discussed, but I think that would be a different issue thread than this one.

[GeoffreyBooth]

It’s not clear to me what the ask is here. Is it specifically that shebangs can tell Node to execute a file as ESM, or that a loose extensionless file can be run as ESM?

[bmeck]

@GeoffreyBooth I believe per https://github.com/nodejs/node/issues/32316#issuecomment-600639985 it is that a hashbang containing file can execute as ESM without any other context involved; no symlinks, no package.json, etc.

[GeoffreyBooth]

a hashbang containing file can execute as ESM without any other context involved; no symlinks, no package.json, etc.

Okay. That’s already possible via the solution in the gist:

#!/usr/bin/env sh
J=S//;echo "\n\n$(sed "1,2d" "$0")"|node --input-type=module "$@";exit $?

import { version } from 'process';
console.log(`Running Node ${version} in ESM mode!`);

Is there something about that solution that you find wanting?

[WebReflection]

As author of that gist, I'd like to underline if the J=S part is confusing (it doesn't pass through node interpreter, it's just for sh), or if the IDE complains, it can be any other no-op such as Map=Map//; or Function=Function//; or even global=global//; it's there exclusively to start that line, which is discarded once interpreted, with something JS IDEs wouldn't complain about.

[bmeck]

I do believe the user experience is sub optimal to have to use a non-portable / readable hashbang. I think a compromise might also be having the behavior of the default package be configurable at build time? Then people could build their own form of node-esm easily if they want this feature

[WebReflection]

I do believe the user experience is sub optimal to have to use a non-portable / readable hashbang.

it's a workaround, hence not optimal indeed, but AFAIK there's no env incapable of running it, except Windows PowerShell, but server-side is rarely done on Windows, and most devs are on Linux, macOS, or WSL/Git/Ming shell, even Docker, where it works already, as long as sh is there, which is 99% of the time the case.

node-esm would be ideal, if shipped with node, but it should also automatically import all things as esm unless subfolders specify otherwise or the file extension is .cjs, so that the --import-type flag might be a first step to experiment in that direction with current node.

[GeoffreyBooth]

Is there something about that solution that you find wanting?

This was directed at @haraldrudell. Based on the initial comment:

It would be nice if ECMAScript shebang files with no extension
a. worked :)
b. were assumed to be modules for Node.js current+

It would seem that the gist satisfies both requirements, and it's also clever, so we get bonus points for satisfying the requirement of the thread title. So my question is, can we consider this issue closed or is there something about the gist's solution that is lacking?

node-esm would be ideal, if shipped with node, but it should also automatically import all things as esm unless subfolders specify otherwise

I think what you mean here is that if you had an extensionless JavaScript file with the gist solution, if that file contained import './other-file.js', that that other-file.js would be loaded as ESM? To which I would say, no: if you have at least two files as part of a program, just put them in a folder and throw a package.json in there.

[WebReflection]

To which I would say, no: if you have at least two files as part of a program, just put them in a folder and throw a package.json in there.

My --import-type=module was about this, but I agree it's a less interesting case as one does not simply write an executable with relative dependencies, so I'm OK with your conclusion, yet I think in the near future, forcing the import type would be handy, as in theory we're "all" migrating from CJS to ESM.

[bobjunga]

Excuse me for budding in as a newbie to node dev but I think my different perspective as an experienced linux systems developer might be useful.

You are at a junction where you can easily choose what you want the default standard used by extensionless bin commands to be. It would be crazy if you use that freedom to choose it to be the legacy cjs standard that is being superseded by esm.

Shebang should do what is intuitive and inline with the future unless the burden is too great in breaking legacy stuff which it seems from this conversation that it clearly is not.

JS is a general language now. node does not have to be relegated to just for server side web applications and their supporting scripts. I got here through Atom customization and now am considering electron apps for all sorts of utilities with a UI and writing more cli commands in JS instead of bash, python, or php. (I might even use it for server side web apps too:)

I would include extensionless bin JS cli and GUI commands in my deb and rpm packages, but I am not going to prefer JS if it forces me to use an older standard in stand alone commands and not if If that ugly and cryptic shebang hack is the official way for me to do what I would expect to be the default!

BTW, in linux we would distribute one 'node' executable that has a symlink called 'node-cjs'. The node executable would init the default mode to esm if its invoked as 'node' and cjs if its invoked as 'node-cjs'. Lots of programs do that. I am not sure if you can do that in Win and Mac but as already discussed, you could always build two executables.

--BobG

[GeoffreyBooth]

You are at a junction where you can easily choose what you want the default standard used by extensionless bin commands to be. It would be crazy if you use that freedom to choose it to be the legacy cjs standard that is being superseded by esm.

We can’t actually choose to change this without breaking countless shebang scripts that have been using #!/usr/local/bin/node over the past decade. That said, we can certainly ship a second binary, e.g. node-esm, to allow a shebang like #!/usr/local/bin/node-esm.

[guybedford]

I still think the package.json "type": "module" approach before can work fine here, and am still not sure why this is a problem to be honest.

[GeoffreyBooth]

Here’s how to create your own node-esm:

1. Create a file node-esm somewhere in your PATH, e.g. /usr/local/bin/node-esm:

   #!/usr/bin/env sh
   input_file=$1
   shift
   exec node --input-type=module - $@ <$input_file
   

2. Make it executable:

   chmod +x /usr/local/bin/node-esm
   

Now you can save the following as a file like ./my-script:

#!/usr/bin/env node-esm
import { version } from 'process';
console.log(version);

And once it’s executable, it should work as you want:

$ chmod +x ./my-script
$ ./my-script
v13.12.0

Credit to @jkrems for the first part 😄

[devsnek]

i'd recommend doing #!/usr/bin/env node-esm but otherwise lgtm

[jkrems]

I feel like there's now multiple solutions that don't require changes to node itself. I'd consider this working as intended for the time being. It only affects small scripts that are placed directly in the PATH (so likely aren't loading a complex tree of source files) so the impact seems low given the amount of complexity native support in node would involve. To me the workarounds above look like they are unblocking folks that really want to use modules in those cases without any negative impact on core or the majority of users.

Should we close this issue?

[bobjunga]

_(edit) woops, did not see the other posts before I sent the msg below. I would just say, dont leave it to everyone to make their own node-esm and node-cjs in the long term -- ship them alongside node and document creating standalone shebang scripts._

We can’t actually choose to change this without breaking countless shebang scripts that ...

The statement that the esm default was in place without complaint for months left me with the opposite impression but maybe testing with this version has not been widespread?

In any case it does seem easy to support both. Instead of producing a whole separate executable you could ship two little scripts alongside the node executable.

$ cat node-esm 
#!/usr/bin/env sh
node --input-type=module "$@"
$ 
$ cat node-cjs
#!/usr/bin/env sh
node --input-type=commonjs "$@"
$ 
$ cat mycmd
#!/usr/bin/env node-esm
import { foo } from './foo.mjs'
...

In the meantime, people can create those scripts in an npm package and install them globally (I think either 'bin' property or installing with -g would work)

You could encourage script command authors to be explicit in using node-esm or node-cjs and not rely on the default behavior of 'node' (which may change eventually)

--BobG

[bobjunga]

I just went to write a more robust node-esm script command. That script will be hard to get 100% correct and to maintain if you dont fix the --input-type option.

node-esm should support the same syntax as node.

     node [options] [v8-options] [-e string | script.js | -] [--] [arguments ...]
     node inspect [-e string | script.js | - | <host>:<port>] ...
     node [--v8-options]

The problem is that you can not reliably identify the 'script.js' parameter in that syntax unless the script knows which node and V8 options expect parameters. The script would need to do similar command line processing as node itself.

From the outside, it seems logical that --input-type should affect stdin content, -e string content and the contents of filename specified on the command line. They are all ways to provide the entry point script that node's going to run -- i.e. they all are ways to provide the 'input' that --input-type refers to. Its just confusing otherwise. That option should not affect imports inside the code or anything else -- just the input to node.

If you make that change, any scripts people need to write to add options (not just the case) will be trivial, robust and not need maintenance.

$ cat node-esm 
#!/usr/bin/env sh
node --input-type=module "$@"

--BobG

[GeoffreyBooth]

I just went to write a more robust node-esm script command. That script will be hard to get 100% correct and to maintain

The various shell script solutions listed on this thread are hard to get right and to maintain. Node supports _a lot_ of environments, and I wouldn't want to take on the challenge of ensuring that this is compatible with all of them; and even if we explicitly said that this is only supported for some subset (and people noticed that and didn't report it as a bug for explicitly unsupported platforms), it would be hard to catch all the edge cases like additional arguments or STDIN or process main etc.

The short answer is that this doesn't feel like a high priority use case for Node to support directly, and to implement this robustly and support it would take tremendous effort; probably more effort than is reasonable for the use case/subset of users wishing for this feature. Whereas the various solutions above don't feel terribly burdensome; users who want this functionality can implement whichever of the above solutions works best for their environment and/or scripts.

From the outside, it seems logical that --input-type should affect stdin content, -e string content and the contents of filename specified on the command line.

This was discussed extensively in the design process. See https://github.com/nodejs/modules/issues/300. Basically, there are downsides to every alternative and --input-type is the least problematic, while also being the least useful. If after reading that you'd like to propose a change, and your reasoning isn't already addressed on that thread, please open a new issue.

I think I'm going to close this issue as answered, if others feel otherwise please reopen.