Hassio-zigbee2mqtt: Unable to use the external_converters (Cannot find module 'zigbee-herdsman-converters')

Created on 7 Oct 2020 · 27Comments · Source: danielwelch/hassio-zigbee2mqtt

zigbee2mqtt add-on version (if edge, please report commit hash):

Zigbee2mqtt Addon 1.15.0

Operating environment (HassOS, Virtual Machine, Device/platform running Home Assistant):
HassIO 0.115.6

Description of problem:
It's unable to start the z2m if adding the external converter.
The file freepad_ext.js is placed in the folder '/usr/share/hassio/share/zigbee2mqtt
We also tried the original sample freepad_ext.js file and the custom files and both of them did not work.

Your entire configuration from the frontend (with sensitive fields redacted):

// replace this line with your pasted configuration
data_path: /share/zigbee2mqtt
external_converters:
  - freepad_ext.js
devices: devices.yaml
groups: groups.yaml
homeassistant: true
permit_join: false
mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://core-mosquitto'
  user: u1
  password: p1
serial:
  port: /dev/ttyACM0
advanced:
  log_level: info
  pan_id: 6754
  channel: 15
  network_key:
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
    - x
  availability_blocklist: []
  availability_passlist: []
device_options: {}
blocklist: []
passlist: []
queue: {}
frontend:
  port: 8099
experimental:
  new_api: true
socat:
  enabled: false
  master: 'pty,raw,echo=0,link=/dev/ttyZ2M,mode=777'
  slave: 'tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5'
  restartdelay: 1
  initialdelay: 1
  options: '-d -d'
  log: false
zigbee_shepherd_devices: true

Your logs from Home Assistant
https://pastebin.com/ARueGHSF

bug

Source

dinhchinh82

All 27 comments

Could you also provide the content of your freepad_ext.js ?

ciotlosm on 10 Oct 2020

Could you also provide the content of your freepad_ext.js ?

we also tried with default freepad_ext.js sample file from the link below and got error:
https://github.com/Koenkk/zigbee2mqtt.io/blob/master/docs/externalConvertersExample/freepad_ext.js

dinhchinh82 on 11 Oct 2020

👍1

I've tested this and is indeed a problem. I've asked also on z2m project as I'm unsure if this is really a problem with the feature or a path problem on how we handle this in the add-on.

ciotlosm on 11 Oct 2020

👍1

@dinhchinh82 Found the bug. If you want to test out the fix you can try using edge version here: https://github.com/zigbee2mqtt/hassio-zigbee2mqtt as repository.

ciotlosm on 11 Oct 2020

@dinhchinh82 Found the bug. If you want to test out the fix you can try using edge version here: https://github.com/zigbee2mqtt/hassio-zigbee2mqtt as repository.

Great. Let me try the edge version then report you the result.

dinhchinh82 on 13 Oct 2020

@dinhchinh82 make sure you install the new repository. You can use the stable version there as well, as it has the bug corrected (1.15.0-3)

ciotlosm on 13 Oct 2020

@dinhchinh82 make sure you install the new repository. You can use the stable version there as well, as it has the bug corrected (1.15.0-3)

@ciotlosm Just confused which is the new repository? could you help me how to install the stable version?

Thank you.

dinhchinh82 on 16 Oct 2020

The new repository is: https://github.com/zigbee2mqtt/hassio-zigbee2mqtt

ciotlosm on 16 Oct 2020

@ciotlosm I've tested your repo and it works. Can you merge it to the official repo?
Thank you.

dinhchinh82 on 22 Oct 2020

Hello,
I just added this new version of 1.15.0-3 from new repo https://github.com/zigbee2mqtt/hassio-zigbee2mqtt
and having problems with it.
Here is the log of module:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] socat.sh: executing... 
[00:43:20] INFO: Socat not enabled, marking service as down
[cont-init.d] socat.sh: exited 0.
[cont-init.d] zigbee2mqtt.sh: executing... 
[00:43:21] INFO: MQTT service found, fetching server detail ...
[00:43:21] INFO: Previous config file found, checking backup
[00:43:21] INFO: Creating backup config in '/config/zigbee2mqtt/.configuration.yaml.bk'
[00:43:21] INFO: File devices.js found, copying to ./node_modules/zigbee-herdsman-converters/
cp: can't create './node_modules/zigbee-herdsman-converters/devices.js': No such file or directory
[cont-init.d] zigbee2mqtt.sh: exited 1.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.

And here is my configuration:
data_path: /config/zigbee2mqtt external_converters: [] devices: devices.yaml groups: groups.yaml homeassistant: true permit_join: false mqtt: base_topic: zigbee2mqtt server: 'mqtt://core-mosquitto' user: zigbee2mqtt password: __REDACTED___ serial: port: /dev/serial/by-id/usb-Texas_Instruments_CC2538_USB_CDC-if00 advanced: log_level: warn pan_id: 6754 channel: 11 network_key: - 1 - 3 - 5 - 7 - 9 - 11 - 13 - 15 - 0 - 2 - 4 - 6 - 8 - 10 - 12 - 13 availability_blocklist: [] availability_passlist: [] device_options: {} blocklist: [] passlist: [] queue: {} frontend: port: 8099 experimental: new_api: true socat: enabled: false master: 'pty,raw,echo=0,link=/dev/ttyZ2M,mode=777' slave: 'tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5' options: '-d -d' log: false zigbee_shepherd_devices: true
Can someone please help?

P.S. Same happens with edge version...

solevi on 23 Oct 2020

@solevi it is recommended you move away from using devices.js workaround and migrate to use https://github.com/zigbee2mqtt/hassio-zigbee2mqtt/blob/dev/zigbee2mqtt/DOCS.md#using-external_converters

@dinhchinh82 I will merge it at some point. Have to figure out how to merge functionality without breaking CI pipelines.

I will close this issue for now. @solevi feel free to open a new issue for the problem with devices.js as this was for zigbee-herdsman-converters

ciotlosm on 1 Nov 2020

@ciotlosm have you merged this PR to the official 1.16 release yet?
Thank you.

dinhchinh82 on 2 Nov 2020

Not yet, otherwise I would have reflected it in the changelog, but both repos now got the z2m bumped to 1.16.0.
In the long run https://github.com/zigbee2mqtt/hassio-zigbee2mqtt will become the official repo (also in docs) as it has faster pipelines and a much more stable workflow using dev branch for PRs and master only when builds are complete and version is stable.

For now I will support both repos in parallel for new z2m versions until I merge and propose docs update for z2m to point to the new repo. I'm not yet decided on a migration plan, as ideally people should replace their repo in the add-on store, which is harder to achieve.

ciotlosm on 2 Nov 2020

@solevi it is recommended you move away from using devices.js workaround and migrate to use https://github.com/zigbee2mqtt/hassio-zigbee2mqtt/blob/dev/zigbee2mqtt/DOCS.md#using-external_converters

@dinhchinh82 I will merge it at some point. Have to figure out how to merge functionality without breaking CI pipelines.

I will close this issue for now. @solevi feel free to open a new issue for the problem with devices.js as this was for zigbee-herdsman-converters

Hi, Thanks for answering.
My main problem is exactly this Cannot find module 'zigbee-herdsman-converters' error. This is the reason why I found this thread.
The module i wanted to use have zigbee-herdsman-converters as dependency.
When i am adding this module, the plugin is crushing.
here is my config:

data_path: /share/zigbee2mqtt
external_converters: 
  - DIYRuZ_AirSense.js
devices: devices.yaml
groups: groups.yaml
homeassistant: true
permit_join: false
mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://core-mosquitto'
  user: zigbee2mqtt
  password: __redacted__
serial:
  port: /dev/serial/by-id/usb-Texas_Instruments_CC2538_USB_CDC-if00
advanced:
  log_level: info
  pan_id: 6754
  channel: 11
  network_key:
    - 1
    - 3
    - 5
    - 7
    - 9
    - 11
    - 13
    - 15
    - 0
    - 2
    - 4
    - 6
    - 8
    - 10
    - 12
    - 13
  availability_blocklist: []
  availability_passlist: []
device_options: {}
blocklist: []
passlist: []
queue: {}
frontend:
  port: 8099
experimental:
  new_api: true
socat:
  enabled: false
  master: 'pty,raw,echo=0,link=/dev/ttyZ2M,mode=777'
  slave: 'tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5'
  restartdelay: 1
  initialdelay: 1
  options: '-d -d'
  log: false
zigbee_shepherd_devices: true

And here is the log:

npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-11-02T16_46_04_973Z-debug.log
2020-11-02T20:46:05: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
2020-11-02T20:46:05: PM2 log: App [npm:0] starting in -fork mode-
2020-11-02T20:46:05: PM2 log: App [npm:0] online
> [email protected] start /zigbee2mqtt-1.16.0
> node index.js
internal/modules/cjs/loader.js:968
  throw err;
  ^
Error: Cannot find module 'zigbee-herdsman-converters'
Require stack:
- /share/zigbee2mqtt/DIYRuZ_AirSense.js
- /zigbee2mqtt-1.16.0/lib/util/utils.js
- /zigbee2mqtt-1.16.0/lib/util/settings.js
- /zigbee2mqtt-1.16.0/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15)
    at Function.Module._load (internal/modules/cjs/loader.js:841:27)
    at Module.require (internal/modules/cjs/loader.js:1025:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/share/zigbee2mqtt/DIYRuZ_AirSense.js:5:5)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Module.require (internal/modules/cjs/loader.js:1025:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/share/zigbee2mqtt/DIYRuZ_AirSense.js',
    '/zigbee2mqtt-1.16.0/lib/util/utils.js',
    '/zigbee2mqtt-1.16.0/lib/util/settings.js',
    '/zigbee2mqtt-1.16.0/index.js'
  ]
}
npm
 ERR! code ELIFECYCLE
npm ERR! errno 1
npm
 ERR! [email protected] start: `node index.js`
npm ERR! Exit status 1
npm ERR!

Contents of the DIYRuZ_AirSense.js can be found here.

solevi on 2 Nov 2020

@solevi did you try the add-on from this repo: https://github.com/zigbee2mqtt/hassio-zigbee2mqtt ? There is still a bug (until I merge it here as well) with the version here.

ciotlosm on 2 Nov 2020

@ciotlosm Really sorry for noise, for some reason i thought that i installed add-on from that repo...
Now everything is as expected, thank you very much!

solevi on 2 Nov 2020

👍1

hi, tried install the add-on from the new repo but it is unable to start because in doesn't find mosquitto (I'm not using the hassio mosquitto container but a direct installation). I correctly copied my old config (from old repo installation) to the new config-file. I also uninstalled the old installation. but it doesn't seem to recognize the server settings. What am I doing wrong?

log-output:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] socat.sh: executing... 
[17:07:06] INFO: Socat not enabled, marking service as down
[cont-init.d] socat.sh: exited 0.
[cont-init.d] zigbee2mqtt.sh: executing... 
[17:07:07] FATAL: No internal MQTT service found. Please install Mosquitto broker
[cont-init.d] zigbee2mqtt.sh: exited 1.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.

config:

data_path: /config/zigbee2mqtt
external_converters: []
devices: devices.yaml
groups: groups.yaml
homeassistant: true
permit_join: false
mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://raspberrypi.local'
  user: mqtt_user
  password: '__redacted__'
serial:
  port: /dev/ttyACM0
advanced:
  log_level: info
  pan_id: 6754
  channel: 11
  network_key:
    - 1
    - 3
    - 5
    - 7
    - 9
    - 11
    - 13
    - 15
    - 0
    - 2
    - 4
    - 6
    - 8
    - 10
    - 12
    - 13
  availability_blocklist: []
  availability_passlist: []
device_options: {}
blocklist: []
passlist: []
queue: {}
frontend:
  port: 8099
experimental:
  new_api: true
socat:
  enabled: false
  master: 'pty,raw,echo=0,link=/dev/ttyZ2M,mode=777'
  slave: 'tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5'
  options: '-d -d'
  log: false

fabsss on 23 Nov 2020

@fabsss this is a bug caused by https://github.com/zigbee2mqtt/hassio-zigbee2mqtt/blob/752b569062ed4eb5e81b191490955cba36102d93/common/rootfs/etc/cont-init.d/zigbee2mqtt.sh#L7

Thanks for reporting, I'll get it fixed.

ciotlosm on 23 Nov 2020

@fabsss can you test edge version from repo ?

ciotlosm on 23 Nov 2020

I tried the edge-version from the "new" repro with the same results as above. I switched to the edge version of the non-fork repo and this seems to be working.

fabsss on 23 Nov 2020

@fabsss can you post the log of the fork repo?

ciotlosm on 23 Nov 2020

The log of the fork repo is the same as above. There was no difference between fork release (1.16) and edge version.

fabsss on 23 Nov 2020

@fabsss did you remove and reinstall edge from fork? Without reinstall it will not fetch the new container. The startup message should be:

No internal MQTT service found and no MQTT server defined. Please install Mosquitto broker or specify your own.

This is the fix:
Screenshot 2020-11-23 at 21 59 17

ciotlosm on 23 Nov 2020

I'm quite sure that I removed the stable release and installed the edge version. Maybe something went wrong while doing it. I'd rather not uninstall the nonfork-version again since I just restored all my sensor names and associations because the database broke while switching the versions.
I'm just wondering why the original (non fork) edge version has external converter fix as well? Did the change make it into the release?

fabsss on 23 Nov 2020

@fabsss switching between forks and edge/non-edge is seamless as long as you keep the same configuration. It reuses paths and database.

The non-fork has a fix because of https://github.com/Koenkk/zigbee2mqtt/pull/5027 which corrected the need for env variable injection. However we're going to migrate probably to the new fork as it uses better process management, so your feedback is valuable to avoid breaking it before we merge.

ciotlosm on 23 Nov 2020

Very strange. I just uninstalled the non-fork edge and installed the fork edge version and now also the fork-version works.

Log:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] socat.sh: executing... 
[21:28:31] INFO: Socat not enabled, marking service as down
[cont-init.d] socat.sh: exited 0.
[cont-init.d] zigbee2mqtt.sh: executing... 
[21:28:33] INFO: MQTT available, fetching server detail ...
[21:28:33] INFO: Previous config file found, checking backup
[21:28:33] INFO: Creating backup config in '/share/zigbee2mqtt/.configuration.yaml.bk'
[21:28:33] INFO: Adjusting Zigbee2mqtt core yaml config with add-on quirks ...
[cont-init.d] zigbee2mqtt.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[21:28:35] INFO: Handing over control to Zigbee2mqtt Core ...

> [email protected] start /app
> node index.js

Zigbee2MQTT:info  2020-11-23 21:28:43: Logging to console and directory: '/share/zigbee2mqtt/log/2020-11-23.21-28-41' filename: log.txt
Zigbee2MQTT:info  2020-11-23 21:28:43: Starting Zigbee2MQTT version 1.16.1-dev (commit #9b79620)
Zigbee2MQTT:info  2020-11-23 21:28:43: Starting zigbee-herdsman (0.13.31)
Zigbee2MQTT:info  2020-11-23 21:28:46: zigbee-herdsman started
Zigbee2MQTT:info  2020-11-23 21:28:46: Coordinator firmware version: '{"meta":{"maintrel":3,"majorrel":2,"minorrel":6,"product":0,"revision":20190608,"transportrev":2},"type":"zStack12"}'

fabsss on 23 Nov 2020

🎉1

I will release 1.16.1-1 on the fork with the fix.

ciotlosm on 24 Nov 2020

Was this page helpful?

0 / 5 - 0 ratings