I was tracking the fix in #1122 where the config update wasn't working correctly. It seems the update does actually work now, but if I stop/start my docker container (using docker-compose), it will lose the change. I believe this could be due to the package-lock.json file reverting to the original state, as it seems the updates to it are not being persisted. I did add the volume for zwave-config:/usr/src/app/node_modules/@zwave-js/config to my docker-compose, but i am wondering if we also need to persist the package lock file so that the updates aren't lost on a restart?
It makes sense, also package-lock.json should be persisted in that case. If you see available updates it could be that you already have latest version installed but package-lock is reporting the old one shipped with the base image.
CC @AlCalzone
As a test, I tried to bind-mount package-lock.json, as I don't think you can volume-mount single files. Even after re-populating that file and running the update, it still reports an update available, even with the mounted package-lock.json showing the correct version for zwave-config. I'm not entirely sure what's going on under the hood with the update notifier, so I might have to dig deeper, but it seems my initial thought of simply mounting the package-lock.json file is not sufficient, either.
@LXXero can you share me your docker-compose.yml to add this volume persistent ? i'm in trouble.
this is what i'm trying now with the mounted package-lock.json, but you will need to copy the original package-lock.json file into your local directory, or touch an empty file and re-run npm in order to generate package-lock.json again. I don't think there's a way to do docker volumes (which auto-populate) with a single file, so this is a bind mount, which is a bit more annoying, and unfortunately, it doesn't seem to fix the issue, even with the correct zwave-config showing up in there
edit: the codeblock didn't work, it's making me upload a zip file to attach it.
compose.zip
it seems all these files get effected when npm does an update, that said, this might become a bit hackish to try and catch all these individually. I'm not entirely sure what the best solution is here, short of maybe mounting the entire /usr/src/app, but I'd suspect that might create other problems
node_modules/zwave-js/package.json: "@zwave-js/config": "7.3.1-20210503",
node_modules/.package-lock.json: "version": "7.3.1-20210503",
node_modules/.package-lock.json: "resolved": "https://registry.npmjs.org/@zwave-js/config/-/config-7.3.1-20210503.tgz",
node_modules/.package-lock.json: "@zwave-js/config": "7.3.1-20210503",
package-lock.json: "version": "7.3.1-20210503",
package-lock.json: "resolved": "https://registry.npmjs.org/@zwave-js/config/-/config-7.3.1-20210503.tgz",
package-lock.json: "@zwave-js/config": "7.3.1-20210503",
package-lock.json: "version": "7.3.1-20210503",
package-lock.json: "resolved": "https://registry.npmjs.org/@zwave-js/config/-/config-7.3.1-20210503.tgz",
package-lock.json: "@zwave-js/config": "7.3.1-20210503",
I'm not exactly sure how Docker handles changing images on a restart. If they are always reset, you should be able to preserve the changes by mounting the following paths
node_modules/zwave-js/package.json
node_modules/@zwave-js/config
package-lock.json
but make sure when you upgrade the container that these are not preserved.
yep, a docker-compose up/down will reset the image back to stock.
i'd been trying various combinations of mounting things, but i'm STILL not having success, even with all 4 files mounted (those three, plus the .package-lock.json in node_modules folder.) It still wants to do an update after a restart. So, I'm not sure what is missing still for npm to pick it up, but something must be that I haven't caught by grepping around for the version alone.
if this will also add further complexity to the container update process, I'm almost wondering if it makes more sense for docker users to wait for the container to be updated instead of trying to mangle the container up with all these mount points. As it is, even if we figure out whatever it is that's still missing to prevent this from working, i don't think you can volume mount individual files, so you'd have to wipe out local copies, start it up without the mounts, docker cp the files out of the container, then setup the bind-mount again to the newly copied files, etc. It's not as simple as wiping out a docker volume, that's for sure.
@robertsLando I agree with @LXXero here - Docker breaks the entire intent behind the built-in config updates.
Maybe doing a docker commit after the config update would help, without the need for all these mount points.
I think that a docker commit makes more sense here. I could automatically do the commit after the install if I check that the app is running on a docker container? Would this make sense?
I think the commit has to happen outside the container.
is it easily possible to do a commit from within a container? I believe it would need access to the docker socket in order to do this. I'm also not sure it would provide a clean upgrade path, because if you do a commit, it creates a new image, but it doesn't actually update the running container to point to that new image, so a restart will still revert it at that point, until you explicitly relaunch docker with the newly commited image. It's almost no better than just manually rebuilding the image from a dockerfile using the latest npm modules (which is generally the "cleaner" thing to do) and then relaunching with that.
I do sort of wonder if this functionality is just a no-go within docker? I keep going back to this in my head - but it feels like the best solution is just regular updates to the images in docker hub whenever there's changes to the upstream npm modules, rather than trying to rely on self-updating within the container. That's the only thing I can think of that would allow persistence as well as simple container updates. I usually just do "docker-compose down;docker-compose pull;docker-compose up -d", I am not sure you could keep things that simple if trying to rely on a docker commit of a new local image, as you wouldn't really be tracking remote anymore.
@robertsLando We could think about building a new image every time a nightly config version is released. I can add a webhook to the release script to trigger your build.
This is because you’re using up/down which is destroying the container. Use stop/start/restart which won’t recreate the container from the image. The only reason to use down is a change to the docker-compose file, for an upgrade to a new image, or to start over at the base image.
pause/unpause = computer sleep (no application restart)
Stop/start/restart = computer on/off
Up/down = reinstall the OS
@robertsLando We could think about building a new image every time a nightly config version is released. I can add a webhook to the release script to trigger your build.
I would prefer to avoid this if possible
I would prefer to avoid this if possible
See our conversation on Slack. I think there is another way.
This is because you’re using up/down which is destroying the container. Use stop/start/restart which won’t recreate the container from the image. The only reason to use down is a change to the docker-compose file, for an upgrade to a new image, or to start over at the base image.
pause/unpause = computer sleep (no application restart)
Stop/start/restart = computer on/off
Up/down = reinstall the OS
Ultimately it makes no difference, because a commit will not restart with a commited image. and I tested both of those scenarios (restarting docker / the computer versus docker-compose up/down). it makes no difference in practice. I use down/pull/up to update the container, as already mentioned.
It does make a difference because when you upgrade the new image has the most up-to-date device files. This is intended to allow updates to device files between updates. So if you don’t recreate the container in-between, you don’t lose those in-between updates. You shouldn’t need to map in the config folder as that just complicates it. They aren’t designed to persist.
It does make a difference because when you upgrade the new image has the most up-to-date device files. This is intended to allow updates to device files between updates. So if you don’t recreate the container in-between, you don’t lose those in-between updates. You shouldn’t need to map in the config folder as that just complicates it. They aren’t designed to persist.
Yes, I agree mapping that folder doesn't make sense, its just complicating things and likely going to create problems, so the README should likely remove that reference at the very least.
I swore I tested restart vs up/down, and indeed i did, and yes, it does persist some of that data without things mounted and just a restart, however, after said restart, it seems the zwavejs ui goes back to saying it's not up to date again, so that's very strange, especially given "npm ls @zwave-js/config" is reporting @zwave-js/[email protected]. Perhaps this is the true bug here.
I just confirmed again with "docker-compose restart", @zwave-js/[email protected] still listed in package-lock.json and npm ls, but the web UI goes back to saying update available for that same version.
That may very well be a bug. @robertsLando ? How do you detect the config version for comparison?
@zwave-js/[email protected] still listed in package-lock.json and npm ls, but the web UI goes back to saying update available for that same version.
How does node_modules/zwave-js/package.json look like?
What about node_modules/@zwave-js/config/package.json?
@zwave-js/[email protected] still listed in package-lock.json and npm ls, but the web UI goes back to saying update available for that same version.
How does
node_modules/zwave-js/package.jsonlook like?
What aboutnode_modules/@zwave-js/config/package.json?
zwave-js/package.json says 7.3.1-xxxx
however, @zwave-js/config/package.json seems to make no reference to 7.3.1-xxxx, at the top it still says 7.3.0
it's also interesting to note that when you do the npm ls - it seems to show it under the zwave-js module, so I wonder if it's using the metadata from the first file.
# npm ls @zwave-js/config
[email protected] /usr/src/app
`-- [email protected]
`-- @zwave-js/[email protected]
I was also trying to see if clearing the npm cache might help, but i came up empty there on my initial try.
@zwave-js/config/package.json seems to make no reference to 7.3.1-xxxx, at the top it still says 7.3.0
Then you are actually still on 7.3.0. Just to be sure, there a no more volumes for the config folder, right?
it's also interesting to note that when you do the npm ls - it seems to show it under the zwave-js module, so I wonder if it's using the metadata from the first file.
Probably. The way this update is installed via npm is a little bit hacky, but it should work - I have end to end tests.
@zwave-js/config/package.json seems to make no reference to 7.3.1-xxxx, at the top it still says 7.3.0
Then you are actually still on 7.3.0. Just to be sure, there a no more volumes for the config folder, right?
if that is the case, it seems like npm isn't actually finishing the install correctly?
it's also interesting to note that when you do the npm ls - it seems to show it under the zwave-js module, so I wonder if it's using the metadata from the first file.
Probably. The way this update is installed via
npmis a little bit hacky, but it should work - I have end to end tests.
correct, I am no longer mounting any volumes other than the config store directory.
@LXXero can you give me some hints how I can reproduce what exactly happens in your environment?
E.g. by posting the compose file or whatever.
just bring up docker compose with docker-compose pull; docker-compose up -d
then once the web ui is up, do the config upgrade, once done, the red notification will go away.
then docker-compose restart, the red notification will come back
and then poke around with docker-compose exec zwavejs2mqtt /bin/sh
version: '3.7'
services:
zwavejs2mqtt:
container_name: zwavejs2mqtt
image: zwavejs/zwavejs2mqtt:latest
restart: always
tty: true
stop_signal: SIGINT
environment:
- SESSION_SECRET=xxxxx
- TZ=America/Denver
networks:
- zwave
devices:
- '/dev/ttyUSB0:/dev/ttyUSB0'
volumes:
- ./store:/usr/src/app/store
ports:
- '8091:8091' # port for web interface
- '3000:3000' # port for zwave-js websocket server
networks:
zwave:
I'm looking at the pak function being used in build/lib/driver/UpdateConfig.js
it looks like all it's actually doing is updating the dependencies with pak.overrideDependencies() however i don't see any calls to pak.update(), could this be the issue? I imagine we need something like pak.update(`@zwave-js/config@${newVersion}`) or something like that before we update the dependencies. My nodejs syntax is a little rusty though
sync function installConfigUpdate(newVersion) {
// Check which package manager to use for the update
let pak;
try {
pak = await pak_1.detectPackageManager({
cwd: __dirname,
requireLockfile: false,
});
}
catch (_a) {
throw new core_1.ZWaveError(`Config update failed: No package manager detected or package.json not found!`, core_1.ZWaveErrorCodes.Config_Update_PackageManagerNotFound);
}
// And install it
const result = await pak.overrideDependencies({
"@zwave-js/config": newVersion,
});
if (result.success)
return;
throw new core_1.ZWaveError(`Config update failed: Package manager exited with code ${result.exitCode}
${result.stderr}`, core_1.ZWaveErrorCodes.Config_Update_InstallFailed);
}
exports.installConfigUpdate = installConfigUpdate;
overrideDependencies
internally modifies the package-lock and package.json files, then calls npm install to apply these changes.
overrideDependencies
internally modifies the package-lock and package.json files, then calls
npm installto apply these changes.
ahh okay, strange. so it SHOULD be doing an update with just the dependency override then....i just tried to do the update manually and i get this, perhaps a missing dependency of 7.3.1-20210503? and an uncaught error...
# npm update @zwave-js/[email protected]
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@zwave-js%2fmaintenance - Not found
npm ERR! 404
npm ERR! 404 '@zwave-js/[email protected]' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-05-05T09_06_58_155Z-debug.log
Trust me, I've tried about every single combination of commands to even get this working. Just npm update is not enough because zwave-js has a fixed version dependency that's not satisfied by the new one.
@zwave-js/maintenance is a private devDependency that is only relevant for development purposes in the monorepo. It probably shouldn't be listed in the released package though.
Since Docker on Windows does not allow me to mount a device, I tried it without Docker and just executed z2m as a standalone node application. Here it works!
// package-lock.json:
"@zwave-js/config": {
"version": "7.3.1-20210503",
"resolved": "https://registry.npmjs.org/@zwave-js/config/-/config-7.3.1-20210503.tgz",
"integrity": "sha512-JBBAJlQgNkhuCOWtrEPJNuaHeXGmCZs6G5b/vs9NgBeymWM0xWyyZOA0L9L1yuDuwhERNBzG40jGgvhZoCl+Ig==",
"requires": {
"@zwave-js/core": "7.3.0",
"@zwave-js/shared": "7.3.0",
"alcalzone-shared": "^3.0.3",
"ansi-colors": "^4.1.1",
"fs-extra": "^9.0.1",
"json-logic-js": "^2.0.1",
"json5": "^2.2.0",
"semver": "^7.3.2",
"winston": "^3.3.3"
}
},
// node_modules/zwave-js/package.json:
"dependencies": {
"@alcalzone/jsonl-db": "^1.2.4",
"@alcalzone/pak": "^0.3.1",
"@sentry/integrations": "^6.3.0",
"@sentry/node": "^6.3.0",
"@zwave-js/config": "7.3.1-20210503",
// node_modules/@zwave-js/config/package.json:
"version": "7.3.1-20210503"
Can you try the following edit (without the plus sign) so we get some logs?
In node_modules\zwave-js\build\lib\driver\UpdateConfig.js line 75
+ pak.stdall = process.stdout;
const result = await pak.overrideDependencies({
"@zwave-js/config": newVersion,
});
sure, i can try that.
I was reading into npm install and it does mention this:
https://docs.npmjs.com/specifying-dependencies-and-devdependencies-in-a-package-json-file
"When you (or another user) run npm install, npm will download dependencies and devDependencies that are listed in package.json that meet the semantic version requirements listed for each."
could this be related to a version difference of npm you're using in windows, if you were not running it in docker? or do you have that module in your local environment? I understand that its a dev-only dependency, but the npm documentation above seems to indicate it will try to download it. and you can see it listed on the "dependencies" tab here. https://www.npmjs.com/package/@zwave-js/config/v/7.3.1-20210501
I also imagine that you could just remove the store mount for testing with windows. As long as you don't down/up the docker image, and only do a restart, it should persist the data in /usr/src/app/store for our testing purposes.
unfortunately the output after re-running the update with that command doesn't seem super useful.
2021-05-05 03:27:40.242 INFO ZWAVE: Calling api installConfigUpdate with args: [ [length]: 0 ]
added 773 packages, and audited 2363 packages in 13s
found 0 vulnerabilities
npm notice
npm notice New minor version of npm available! 7.6.0 -> 7.11.2
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v7.11.2>
npm notice Run `npm install -g [email protected]` to update!
npm notice
added 14 packages, removed 142 packages, changed 90 packages, and audited 2235 packages in 18s
19 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
npm notice
npm notice New minor version of npm available! 7.6.0 -> 7.11.2
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v7.11.2>
npm notice Run `npm install -g [email protected]` to update!
npm notice
2021-05-05 03:28:11.834 INFO ZWAVE: Success zwave api call installConfigUpdate { success: true, message: 'Success zwave api call', result: true }
I understand that its a dev-only dependency, but the npm documentation above seems to indicate it will try to download it
It will, if you run npm install without the --production flag in the directory of the module that has it as a devDependency. However if you go one level up (zwavejs2mqtt) and run npm install there, only zwavejs2mqtt's devDependencies get installed, but not the ones of the dependencies.
unfortunately the output after re-running the update with that command doesn't seem super useful.
Ok, try adding one more line please:
pak.loglevel = "verbose";
I also imagine that you could just remove the store mount for testing with windows.
The problem is the device, not the store. While that isn't mounted, the driver won't start, so z2m destroys it again.
Hold on - I can reproduce locally with [email protected]
Ok the problem is the following:
The z2m repo contains a package-lock.json that was created with npm@6 (lockfileVersion: 1). The container contains npm@7 which uses lockfileVersion: 2. These have slighly different formats. The npm wrapper library can deal with both and both are tested.
But: When the wrapper library first runs npm install in the container, the lockfile is converted from v1 to v2, which seems to cause npm to ignore the different version that's specified in the lockfile and not update the module on disk.
so i just double checked, and i'm already at lockfileVersion: 2 with a fresh docker image?? I did read that 2 is backwards compatible with 6/7, but version 3 only works with 7, but i didn't see anything with 3, just 2.
I also tried to run an npm i --package-lock-only before doing the update, but this didn't seem to make any difference
v3 isn't used yet in the package root, only hidden somehow.
You're right by the way - I had some old container running. The new one has a v2 file.
Soo... since I cannot do it via the UI, I tried to go entirely through the command line.
Fresh container, opened a shell inside. Before:
/usr/src/app # npm ls @zwave-js/config
[email protected] /usr/src/app
`-- [email protected]
`-- @zwave-js/[email protected]
/usr/src/app # cat node_modules/@zwave-js/config/package.json | grep version
"version": "7.3.0",
Called the config update method by hand:
/usr/src/app # node
Welcome to Node.js v15.11.0.
Type ".help" for more information.
> var upd = require("zwave-js/build/lib/driver/UpdateConfig")
...
> upd.installConfigUpdate("7.3.1-20210503")
Promise { <pending> }
...
Afterwards:
/usr/src/app # cat node_modules/zwave-js/package.json | grep version
"version": "7.3.0",
/usr/src/app # cat node_modules/zwave-js/package.json | grep @zwave-js
"@zwave-js/config": "7.3.1-20210503",
"@zwave-js/core": "7.3.0",
"@zwave-js/serial": "7.3.0",
"@zwave-js/shared": "7.3.0",
"@zwave-js/maintenance": "7.0.0"
/usr/src/app # cat node_modules/@zwave-js/config/package.json | grep version
"version": "7.3.1-20210503",
Which is exactly as expected.
Okay, after docker-compose restart the changes are gone:
/usr/src/app # cat node_modules/zwave-js/package.json | grep @zwave-js
"@zwave-js/config": "7.3.0",
"@zwave-js/core": "7.3.0",
"@zwave-js/serial": "7.3.0",
"@zwave-js/shared": "7.3.0",
"@zwave-js/maintenance": "7.0.0"
hmm that's strange, i don't lose the changes from a docker-compose restart, however, i still can't get the @zwave-js/config/package.json to show the update to 7.3.1-xxx, even with the manual node commands above. it seems only the lock file is getting updated.
/usr/src/app # npm ls @zwave-js/config
[email protected] /usr/src/app
`-- [email protected]
`-- @zwave-js/[email protected]
npm notice
npm notice New minor version of npm available! 7.6.0 -> 7.11.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v7.11.2
npm notice Run npm install -g [email protected] to update!
npm notice
/usr/src/app # cat node_modules/@zwave-js/config/package.json | grep version
"version": "7.3.0",
/usr/src/app # node
Welcome to Node.js v15.11.0.
Type ".help" for more information.
> var upd = require("zwave-js/build/lib/driver/UpdateConfig")
undefined
> upd.installConfigUpdate("7.3.1-20210503")
Promise { <pending> }
/usr/src/app # cat node_modules/zwave-js/package.json | grep version
"version": "7.3.0",
/usr/src/app # cat node_modules/zwave-js/package.json | grep @zwave-js
"@zwave-js/config": "7.3.1-20210503",
"@zwave-js/core": "7.3.0",
"@zwave-js/serial": "7.3.0",
"@zwave-js/shared": "7.3.0",
"@zwave-js/maintenance": "7.0.0"
/usr/src/app # cat node_modules/@zwave-js/config/package.json | grep version
"version": "7.3.0",
and all that remains the same after docker-compose restart, though it does get reset if i do docker-compose down ; docker-compose up -d
Did you start with an entirely fresh container?
Did you start with an entirely fresh container?
yep
this is getting weirder by the minute. I suspect the docker-compose restart not working could be a windows vs linux thing, but what still doesn't make sense is why i keep ending up in this partially updated state.
ok so what's weird is that doing npm install @zwave-js/[email protected] seems to work
but doing npm update @zwave-js/[email protected], does not work right, it ends up in that funky state where the lock file is updated, and the zwave-js/package.json is updated, but not the actual @zwave-js/config/package.json
and of course, the install command doesn't correctly update the dependency files. If i manually run the node command after doing a manual npm install, and/or click the update button in the UI after the manual npm install, it's finally working right. And it's persisting a docker-compose restart now. I don't understand why the update command isn't working right. I am tempted to try updating npm (this made no difference...)
This is really f*ing weird. I just installed Docker in the WSL2 Ubuntu, and now I have the same behavior as you do...
@AlCalzone You should be able to mount the stick into the container on windows. Try this:
https://docs.sevenbridges.com/docs/mount-a-usb-drive-in-a-docker-container
And adding this to the bottom of the docker compose will start it to a command line so you can manually launch it after mapped in.
command: '/bin/sh'
OK i might have figured something out....though i'm not sure what.
if I run
npm update @zwave-js/[email protected]
@zwave-js/config/package.json is still wrong, 7.3.0, however....
if I then re-run update from UI
package.json is now correct all dependencies are now correct, everything is good, and persists after docker-compose restart.
i'm suspecting that initial manual update is updating something that is required for it to work right before the upd.installConfigUpdate is triggered. I'm just not sure what.
@AlCalzone On second thought, the usb thing may only work with block devices. I'm not sure. Can't you startup to the command line and force it to start somehow? Is there a fake driver? Will it start if you just create the dev device?
i'm now suspecting npm update is just updating some random group of modules and something in there is fixing this. I can literally run "npm update blargh" or "npm update @zwave-js/[email protected]" and both will just say "added 773 packages, and audited 2363 packages in 13s", so i'm not sure if i'm just misunderstanding the purpose of specifying something after npm update, although putting any sort of string after it seems to reduce the amount of stuff it updates versus running "npm update" with no string after.
I'm also not sure what, exactly, that its updating that is fixing this, but, after doing this, the upd.installConfigUpdate stuff works again. Perhaps the docker image's npm state needs a refresh? like some old/outdated module is lingering around and breaking things.
npm is doing weird shit. Alyways has and every major update breaks something.
I've gone ahead and switched my test container to yarn. Removed package-lock.json. Ran yarn --production. Called installConfigUpdate and bam, it worked.
I've discussed with @robertsLando to switch the container to yarn and let npm fuck around as it feels like.
yeah, it seems like the npm state is trashed up on that docker image, and pinpointing it seems like some kind of needle-in-a-haystack shit. I suspect there's old modules from a previous version that are lingering around after a node/npm update or something to that extent.
It might have to do with that hidden lockfile node_modules/.package-lock.json, which I'm not patching right now 🤷🏻♂️
Edit: doesn't seem so...
The container should now be using yarn, so this should no longer be an issue.