Pm2: Node v10.5 + ESM not working

[damianobarbati]

Repro here: https://github.com/damianobarbati/yarsk/tree/50ms
Just clone, yarn install and yarn serve:dev.

It fires pm2-runtime which just reports the following:

[WARN] PM2 Daemon is already running
(node:4716) ExperimentalWarning: The ESM module loader is experimental.
(node:4734) ExperimentalWarning: The ESM module loader is experimental.
0 application online, retry = 3
(node:4829) ExperimentalWarning: The ESM module loader is experimental.
0 application online, retry = 2
0 application online, retry = 1
0 application online, retry = 0

With pm2 monit I see several weirds things:

• nodejs version is undefined
• flags are properly passed to node interpreter but it dies with SIGINT
• mem: {#aN-fg}

But this does work:

NODE_ENV=development node --experimental-modules index.mjs

Versions:

$ node -v
v10.5.0
$ cat node_modules/pm2/package.json | grep version
  "version": "2.10.4",

[wallet77]

Hi @damianobarbati

I did a lot of tests today, here are my results :

1) Indeed with your code pm2 or pm2-runtime try to restart app and then stop. It seems your app received a SIGINT signal but i don't know why.

2) It seems .mjs files can work with pm2 (note that I use the latest version on dev branch).
Here is my working sample :

import express from 'express'
let app = express()

app.get('/', function(req, res){
  res.send('Home')
});

app.get('/ok', function(req, res){
  res.send('Ok')
});

const server = app.listen(process.env.PORT || 5000, function(){
  console.log('Server express', 'listening on port', server.address().port);
});

Maybe we should simplify your code to find the root cause of SIGINT signal ?

[damianobarbati]

@wallet77 thanks for helping!

It's not the case, I tried (you can try yourself too) to remove everything from index.mjs and just put the following:

console.log('yo');
setInterval(() => console.log('yo'), 1000);

Nothing happens, just crashing.

SIGINT is popping out from nowhere; if it was being fired from the application then node index.mjs would show the same behaviour but it works without any problem.

PM2 is definitely doing something 😅

@Unitech can you investigate on this?

[wallet77]

@damianobarbati

It seems it's related to watch issue. If you remove "watch" property in package.json, it's working well.
Probably an issue related to of https://github.com/Unitech/pm2/issues/3746 or https://github.com/Unitech/pm2/issues/3693

[adamchal]

@wallet77 I don’t think it’s related to the “watch” issue.

Here’s a version of @damianobarbati’s example with watching explicitly disabled:

// ecosystem.config.js
module.exports = {
    apps: [{
        name: 'foo',
        script: 'index.mjs',
        node_args: '--experimental-modules',
        instances: 1,
        exec_mode: "fork",
        wait_ready: false,
        watch: false,
        listen_timeout: 8000,
        kill_timeout: 3000
    }]
}

// index.mjs
console.log('yo');
setInterval(() => console.log('yo'), 1000);

$ pm2 start ecosystem.config.js

$ pm2 log
PM2        | App name:foo id:0 online
0|foo      | (node:60498) ExperimentalWarning: The ESM module loader is experimental.
PM2        | App [foo] with id [0] and pid [60498], exited with code [0] via signal [SIGINT]
PM2        | Starting execution sequence in -fork mode- for app name:foo id:0
PM2        | App name:foo id:0 online
0|foo      | (node:60505) ExperimentalWarning: The ESM module loader is experimental.
PM2        | App [foo] with id [0] and pid [60505], exited with code [0] via signal [SIGINT]
PM2        | Script /srv/foo had too many unstable restarts (16). Stopped. "errored"

$ pm2 show foo
 Describing process with id 0 - name foo
┌───────────────────┬───────────────────────────────────────┐
│ status            │ errored                               │
│ name              │ foo                                   │
│ restarts          │ 15                                    │
│ uptime            │ 0                                     │
│ script path       │ /srv/foo                              │
│ script args       │ N/A                                   │
│ error log path    │ /Users/adam/.pm2/logs/foo-error-0.log │
│ out log path      │ /Users/adam/.pm2/logs/foo-out-0.log   │
│ pid path          │ /Users/adam/.pm2/pids/foo-0.pid       │
│ interpreter       │ node                                  │
│ interpreter args  │ --experimental-modules                │
│ script id         │ 0                                     │
│ exec cwd          │ /srv/foo                              │
│ exec mode         │ fork_mode                             │
│ node.js version   │ N/A                                   │
│ watch & reload    │ ✘                                     │
│ unstable restarts │ 0                                     │
│ created at        │ N/A                                   │
└───────────────────┴───────────────────────────────────────┘

If you simply remove the --experimental-modules flag from ecosystem.config.js, then it runs successfully and if you run the script without PM2, like @damianobarbati described, it also runs correctly.

Just to reiterate, this is only happening with Node v10.5.0. With v10.4.1, ESM scripts run just fine in PM2. There must be some change to 10.5 that is causing PM2 to issue a SIGINT immediately. I’ll do some more digging myself.

[damianobarbati]

@adamchal you're correct, I forgot to mention that until node v10.4.x everything was working fine and removing --experimental-modules flag and converting to require works.
@Unitech @wallet77

[adamchal]

Just tried with pm2 v3.0.0 and node.js v10.6.0 and now I at least see an error reported in pm2 log:

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: …
at Object.Module._extensions..mjs (internal/modules/cjs/loader.js:724:11)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Proxy.Module._load (internal/modules/cjs/loader.js:530:3)
at /usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:323:23
at /usr/local/lib/node_modules/pm2/node_modules/async/dist/async.js:473:16
at next (/usr/local/lib/node_modules/pm2/node_modules/async/dist/async.js:5329:29)
at /usr/local/lib/node_modules/pm2/node_modules/async/dist/async.js:969:16
at WriteStream. (/usr/local/lib/node_modules/pm2/lib/Utility.js:172:13)
at WriteStream.emit (events.js:182:13)

In pm2/lib/ProcessContainer.js:323:23, I do see the require statement. I tried adding @jdalton’s standard-things/esm to the ecosystem.config.js:

        node_args: '-r esm --experimental-modules',

But, this only suppressed the error log, but did not actually resolve the OP.

I tried every combination of pm2 v3.0.0 v2.10.4 and node.js v10.6.0 v10.5.0 and none of them work. The only thing that works is node.js v10.4.1 with both recent versions of pm2 (including v3.0.0).

I’m starting to think that beginning with node.js 10.5.0, they have changed the way that CJS scripts (ex. pm2’s ProcessContainer.js) can load ESM scripts through require(). I’m also surprised that standard-things/esm only suppressed the error messaging, and didn’t actually fix the underlying issue as it has previously before with older versions of pm2.

[jdalton]

Hi @adamchal!

The esm loader was recently updated to also error with

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module

when trying to sideload a .mjs file with require from pm2.

In your esm example you don't need to run Node with --experimental-modules though. Using --experimental-modules will bypass the loader. You just need node_args: '-r esm'.

To avoid the sideload issue _(a limitation of the .mjs extension in Node)_ users of esm can just use .js for their ES module files or have an entry .js file that imports the .mjs file with dynamic import().

The use of .mjs in Node is experimental so users should not expect a 💯 supported experience. I'd also caution users about relying on experimental features in production code.

Update:

Here's an example of using the esm loader to get things to just work:
https://github.com/jdalton/yarsk/commit/7966b2105e1d1adb3b7f43196afbf8f61d665a66

[wallet77]

Thanks @jdalton for the great answer.
I think we can close this issue.

@damianobarbati feel free to reopen if necessary.

[adamchal]

Thanks @jdalton!

[jdalton]

Awesome!

If you all want to create a documentation entry with esm as a workaround for this let me know. I'm happy to review it and provide feedback.

[damianobarbati]

Thanks @jdalton for helping here! However using node_args: '-r esm' results in the following error:

internal/modules/cjs/loader.js:583
    throw err;
    ^

Error: Cannot find module 'esm'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)

Full PM2 config file:

const config = require('../package');

module.exports = {
    name: config.name,
    script: 'index.mjs',
    exec_mode: 'cluster',
    node_args: '-r esm --harmony --trace-deprecation --max_old_space_size=512',
    args: '--color',
    max_memory_restart: '512M',
    max_restarts: 3,
    restart_delay: 3000,
    min_uptime: 3000,
    error_file: 'logs/error.log',
    out_file: 'logs/output.log',
    watch: ['package.json', 'index.mjs', 'config/', 'api/', 'services/', 'queries/', '*.mjs'],
    env_development: {
        NODE_ENV: 'development',
    },
    env_staging: {
        NODE_ENV: 'staging',
    },
    env_production: {
        NODE_ENV: 'production',
    }
};

Or on command-line:

$ node -v
v10.6.0
$ node -r esm index.mjs 
internal/modules/cjs/loader.js:583
    throw err;
    ^

Error: Cannot find module 'esm'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.Module._load (internal/modules/cjs/loader.js:507:25)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at Module._preloadModules (internal/modules/cjs/loader.js:805:12)
    at preloadModules (internal/bootstrap/node.js:539:7)
    at startup (internal/bootstrap/node.js:226:9)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:560:3)

[adamchal]

@damianobarbati It sounds silly, but did you add standard-things/esm to your package.json? Alternatively, you can install it globally npm install -g esm.

If you did, you may try @jdalton’s guidance and rename your index.mjs to index.js. But, based on your invocation, I don’t think this should be a problem. However to avoid issues down the road, I try to only use the .mjs extension when I am running with --experimental-modules. When using standard-things/esm (instead of --experimental-modules), I stick to .js for now.

To avoid the sideload issue (a limitation of the .mjs extension in Node) users of esm can just use .js for their ES module files or have an entry .js file that imports the .mjs file with dynamic import().

[damianobarbati]

@adamchal thanks, I was missing the whole point here: I thought -r esm was some kind of built-in node functionality replacing --experiment-modules and not an external package.

@jdalton the preloaded non-native esm module is indeed very cool and I'll probably move to that even if I'd rather stick to strictly-native solutions even if experimental.
What's the current status over native es6 modules in node? Is there a link to a public discussion going on among the team? :)

@wallet77 the esm module looks good but it's a workaround: are you sure you want to close this without further investigating that unkown SIGINT popping out?

Thank you all guys for the great effort into the community!

[jdalton]

Hi @damianobarbati!

What's the current status over native es6 modules in node?

The implementation side is well known. So the hard part is deciding how to paint the API. There's lots of discussions on if Node's ESM support should be maximally minimal _(ship the most basic form, even less that the current WIP implementation)_ so that we can release and build from that. There's others that want various shades of transparent interop _(CJS and ESM working together)_. And there are some that see non-transparent interop _(ESM/CJS separate; require loads CJS; dynamic/static import loads ESM)_ as the way to go.

Is there a link to a public discussion going on among the team?

There are lots of discussions in the working group repo. The working group is comprised of developers _(some Node core, some from npm, some from the ecosystem)_. Feel free to join the discussion.

[damianobarbati]

@jdalton the solution pm2+esm is now broken with [email protected], reverting back to [email protected] to keep going.
Firing the app from command-line with node command works, just PM2+esm does not.
No error or warning is logged from PM2.

[jdalton]

@damianobarbati Would you please file an issue on our end so I can cover this with a scenario test to help prevent a regression like this in the future.

Update:

No worries, a bug and repro has been filed.

Update:

v3.0.72 is released :tada:

[jim-king-2000]

I don't want esm package, I don't want .js file. It is a regression to disable the parameter "--experimental-modules"!

[jdalton]

@jim-king-2000 Node --experimental-modules are exciting for sure, but they are experimental with no support guarantees. In this case any support you previously had was likely the result of a Node implementation bug which is now resolved. As time goes on there will be other changes. I'd caution against relying on experimental features in production code. However, outside of production code, please keep experimenting and trying things out 🔬

[jim-king-2000]

It is our choice(freedom) to use --experimental-modules in production environment (only this one, no other experimental feature). And we have a reasonable plan to deal with almost any situations node would introduce. So I do think this behavior should be supported even if it could be against by everyone else. Further, from a PM's view point, we should be extremely careful to introduce break changes. Now I update PM2, all my projects fail, I have to spend time investigating & resolving it. That is terrible user experience.

[jdalton]

@jim-king-2000

Now I update PM2, all my projects fail, I have to spend time investigating & resolving it. That is terrible user experience.

I believe the fail is out of PM2's control as it was a change in Node's --experimental-modules implementation. This is why I was stressing the wiggliness of experimental features.

[jim-king-2000]

It is our pm2.json

{
"apps": [
{
"name": ...,
"script": "./src/app.mjs",
"exec_mode": ...,
"instances": ...,
"node_args" : "--experimental-modules",
"env": {
"NODE_ENV": "production",
"NODE_PORT" : ...,
"NODE_CONFIG_DIR": ...
},
"max-memory-restart": ...
}
]
}

And we got the error message below:
"Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: D:projectstenantsrcapp.mjs"

It works with the previous version of PM2.

[jdalton]

@jim-king-2000

Based on the comments here and here have you tried to revert to Node 10.4.1 or Node 10.3.0?

[jim-king-2000]

I read all the threads above. It seems that the break change has been introduced by node 10.5.0. I'm sorry for my misunderstanding. I had ever thought it had been introduced by PM2.
I still don't like "standard-things/esm" and "*.js" file. So, I'm looking forward to the fix which could lead me back to "--experimental-modules".
By the way, I'm using node 10.7.0. I always use the latest one if possible.

[artemxgruden]

09/27/2018 15:55:39: Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /Users/artemxgruden/projects/p1/provider/functions/unbundled/server/main.mjs
    at Object.Module._extensions..mjs (internal/modules/cjs/loader.js:724:11)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Proxy.Module._load (internal/modules/cjs/loader.js:530:3)
    at Proxy.Module_load (/usr/local/lib/node_modules/pm2/node_modules/vxx/src/trace-plugin-loader.js:177:33)
    at Object.<anonymous> (/usr/local/lib/node_modules/pm2/lib/ProcessContainerFork.js:75:21)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)

the same result with node v10.11.0 and pm2 3.1.3
in main.mjs just "console.log( 'OK' )"

[jim-king-2000]

Use "-r esm".

// pm2.json
{
  "apps": [
    {
      "name": "realTimeSender/fireeye",
      "script": "./src/pm2App.js",
      "exec_mode": "fork",
      "instances": 1,
      "node_args" : "-r esm",
      "env": {
          "NODE_ENV": "production",
          "NODE_PORT" : "8091"
      },
      "max-memory-restart": "300M"
    }
  ]
}

// pm2App.js
import "./app.mjs"

[artemxgruden]

@jim-king-2000 thank you, it works, but is it a workaround until 10 LTS?

[jdalton]

@artemxgruden

👋 Node core and Node Module WG member here. There is no set date for when --experimental-modules will be unflagged. There is a goal of having it done before Node 10 maintenance LTS starts in 2020, but nothing is set in stone.

[jim-king-2000]

Hi @jdalton , we do not expect --experimental-modules would be unflagged tomorrow. However, we do expect PM2 can use "node_args" : "--experimental-modules" again tomorrow. We like native things. Transpiler like esm or bable will lead to messy call stack, and it is very hard to map it back to the original line of code.

[jdalton]

@jim-king-2000

However, we do expect PM2 can use "node_args" : "--experimental-modules" again tomorrow.

I hear ya. There's reasons for that not happening.

Transpiler like esm or bable will lead to messy call stack, and it is very hard to map it back to the original line of code

You may dig that esm sanitizes call stacks, provides helpful native-like-or-better errors, transforms code inline _(without pushing it off line)_, and preserves ESM source references when using the Node inspector.

[jim-king-2000]

Hi @jdalton , thanks for the reply. I see the "reasons" and find that it is an old comment. I remember we have discussed this topic several times. So, here I can explain thoroughly why I choose to use --experimental-modules in production.

1. I 100% know --experimental-modules is for experiment and should NOT be used in production.
2. --experimental-modules brings a lot of benefits, such as simple project structure(no babel, no babel config, no target files), easy to deploy(no transpiling, save a lot of time), correct call stack, etc.
3. We have several defenses for the potential bugs or break changes introduced by --experimental-modules. See the next items.
4. We keep our projects fully-decoupled. Every project contains less than 50 source files and every file contains less than 100 lines of source code. That means we can find & fix anything wrong quickly. We simply use "import/export", the possibility of introducing break changes of --experimental-modules is quite low. Even if there would be some huge break changes, we could use -r esm temporarily and change our code immediately.
5. We have regression tests. Even if node (or other components) introduced something wrong tomorrow, we could find it within one day.
6. We have failover mechanism online. If our service were down, it could be restarted immediately.
7. We have monitoring online. It invokes every APIs every minute. It generates warnings if there is something wrong.

In a sentence, we prepare for the worst situation so that we can take the risk of using --experimental-modules online. But no other experimental features would be enabled online. So I really need PM2 support --experimental-modules, if there is no technique issue.
By the way, I never use Node inspector. Since our projects are quite small, I only use logs for debugging.

[jdalton]

@jim-king-2000

So, here I can explain thoroughly why I choose to use --experimental-modules in production.

No need to defend your choice to use --experimental-modules in production.
As long as you understand there's no obligation for Node, or anyone else, to support it we're 👍

So I really need PM2 support --experimental-modules, if there is no technique issue.

The reasons I mentioned above are the blocking technical issue.

[jim-king-2000]

@jdalton , thanks for the replay and your patience. Hope the technical issue be conquered soon. We have waited a very long time.

[jim-king-2000]

Node 13.2.0 drops "--experimental-modules". How about this issue?