K3os: Cannot update configuration after initial install

[cwoolum]

Version (k3OS / kernel)

k3os version v0.10.0

5.4.51-v8+ #1327 SMP PREEMPT Thu Jul 23 11:11:34 BST 2020

Architecture

aarch64

Describe the bug

According to the documentation, /var/lib/rancher/k3os/config.yaml is used to define runtime config after initial install. I created the file and added config but it doesn't seem to apply on boot.

To Reproduce

1. Create /var/lib/rancher/k3os/config.yaml
2. Add a configuration
   ```
   k3os:
   modules:
   
   
   
   • nfs
   
   

boot_cmd:
- " /etc/init.d/nfs start"

```

1. Reboot using sudo reboot

Expected behavior

On boot, the nfs service should be started

Actual behavior

The nfs service is not started. Running sudo /etc/init.d/nfs start starts the service correctly.

Additional context

[dweomer]

@cwoolum as per our slack convo on this, changing the boot_cmd start of nfs to a run_cmd was not effectual, is that correct? There is a small fix related to this, I think, that was incorporated into the recently released v0.11.0 that being #547. If you must use v0.10.x the workaround would be to create that directory in a boot_cmd/run_cmd prior to starting the service.

[cwoolum]

I've updated to 0.11.0. I hooked it up to a monitor and don't see the command being run at all. I tried to add an echo to debug but it's not appearing in the logs.

run_cmd:
- "echo 'Starting NFS Service' && /etc/init.d/nfs start"

I'm starting to think that run_cmd is not getting called when defined in /var/lib/rancher/k3os/config.yaml

[dweomer]

@cwoolum AFAIK run_cmd is working, in general, but there may be something that I have overlooked causing your specific case to fail.
Some things to consider:

• To verify that run_cmd / boot_cmd is working you can always mutate the system with something like date > /run/date.
• These are run serially and it is possible that an earlier non-zero exit code is prevent later commands from being executed.

Please verify that sudo k3os config --dump returns without error and if you can please share the output here so that I may attempt to replicate what is blocking success for you.

[cwoolum]

I was able to log a bit more and it is failing when trying to start the service.

Here is my compiled config.

boot_cmd:
- echo "boot_cmd works" >> /tmp/boot_cmd
- mount /dev/sda /mnt/ssd/
- exportfs -r
hostname: k3s-1
k3os:
  dns_nameservers:
  - 192.168.86.1
  - 8.8.8.8
  - 1.1.1.1
  k3s_args:
  - server
  - --cluster-init
  - --disable=traefik,servicelb
  labels:
    plan.upgrade.cattle.io/k3os-latest: yay
  modules:
  - nfs
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
run_cmd:
- echo "before run_cmd works" >> /tmp/run_cmd
- echo 'Starting NFS Service' && sudo service nfs start
- echo "run_cmd works" >> /tmp/run_cmd
ssh_authorized_keys:
- ssh-rsa {redacted}
write_files:
- content: |
    /mnt/ssd 192.168.50.0/16(rw,sync)
  encoding: ""
  owner: ""
  path: /etc/exports
  permissions: ""

boot_cmd is running correctly and the first echo to /tmp/run_cmd is working as well but the second echo is missing so I assume it's failing to start the service.

[cwoolum]

I was able to write the logs from service nfs start to a file and was able to narrow down my issue

 * Mounting nfsd filesystem in /proc ... [ ok ]
 * Exporting NFS directories ... [ ok ]
 * Starting NFS mountd ... [ ok ]
 * Starting NFS daemon ... [ !! ]
 * Starting NFS smnotify ... [ ok ]

Seems to be a problem with the NFS Daemon.

[cwoolum]

Here are the actual logs for the daemon failing to start

Aug 18 18:20:31 k3s-1 authpriv.notice sudo:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service nfs start
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 1, udp): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 1, tcp): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 1, udp6): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 1, tcp6): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 2, udp): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 2, tcp): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 2, udp6): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 2, tcp6): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 3, udp): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 3, tcp): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 3, udp6): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.err rpc.mountd[1648]: Failed to register (mountd, 3, tcp6): svc_reg() err: RPC: Remote system error - Connection refused
Aug 18 18:20:31 k3s-1 daemon.warn rpc.mountd[1648]: mountd: No V2 or V3 listeners created!
Aug 18 18:20:31 k3s-1 daemon.notice rpc.mountd[1649]: Version 2.3.4 starting
Aug 18 18:20:31 k3s-1 daemon.notice sm-notify[1657]: Version 2.3.4 starting
Aug 18 18:20:31 k3s-1 daemon.warn sm-notify[1657]: Failed to open sm: No such file or directory
Aug 18 18:20:31 k3s-1 daemon.err sm-notify[1657]: Failed to open directory sm.bak: No such file or directory
Aug 18 18:20:31 k3s-1 daemon.err /etc/init.d/nfs[1615]: ERROR: nfs failed to start

This is the logs when I am SSH'd in and start the service manually

Aug 18 18:32:34 k3s-1 authpriv.notice sudo:  rancher : TTY=pts/0 ; PWD=/home/rancher ; USER=root ; COMMAND=/etc/init.d/nfs start
Aug 18 18:32:41 k3s-1 daemon.notice rpc.statd[2709]: Version 2.3.4 starting
Aug 18 18:32:41 k3s-1 daemon.notice sm-notify[2710]: Version 2.3.4 starting
Aug 18 18:32:41 k3s-1 daemon.notice sm-notify[2710]: Already notifying clients; Exiting!
Aug 18 18:32:41 k3s-1 daemon.err rpc.statd[2709]: Failed to open directory sm: No such file or directory
Aug 18 18:32:41 k3s-1 daemon.info supervise-daemon[2708]: Child command line: /sbin/k3s server --cluster-init --disable=traefik,servicelb --node-label k3os.io/mode=local --node-label k3os.io/version=v0.11.0 --node-label plan.upgrade.cattle.io/k3os-latest=yay
Aug 18 18:32:41 k3s-1 daemon.warn rpc.statd[2709]: Running as root.  chown /var/lib/nfs to choose different user
Aug 18 18:32:42 k3s-1 daemon.notice rpc.mountd[2761]: Version 2.3.4 starting
Aug 18 18:32:43 k3s-1 daemon.notice sm-notify[2779]: Version 2.3.4 starting
Aug 18 18:32:43 k3s-1 daemon.notice sm-notify[2779]: Already notifying clients; Exiting!

[dweomer]

I booted up a vanilla v0.11.0 with this config.yaml:

boot_cmd:
- rc-update add rpc.statd
- rc-update add nfs

console snippet:

 * Mounting local filesystems ... [ ok ]
 * Starting rpcbind ... [ ok ]
 * Starting NFS statd ... [ ok ]
 * Mounting nfsd filesystem in /proc ... [ ok ]
 * Starting NFS mountd ... [ ok ]
 * Starting NFS daemon ... [ ok ]
 * Starting NFS smnotify ... [ ok ]

Sorry this took so long. Please let me know here and/or Slack if this does not work for you.

---

P.S. using rc-update add ... in the boot_cmd is my preferred way to enable services as early as possible. It doesn't work for everything (ntpd has to be in a run_cmd for some odd reason) but it works for most disabled services,

[cwoolum]

That worked! I've never used rc-update before.

Thanks for the help!