K3os: config.yaml not applied. Virt-install setup

[larssb]

Version (k3OS / kernel)
K3OS version: v0.10.1
Output from uname --kernel-release --kernel-version: 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 (Ubuntu v20.04 server)

Architecture
Output from uname --machine : x86_64

Describe the bug
No errors is thrown when an init_cmd of init_cmd=\"cp /.base/k3os_conf.yaml /config.yaml\" \ via extra_args to virt-install is run.
However, the config seems NOT to be applied. No data is to be seen in the ~/.ssh dir. And the rest of the config data is also not applied. k3os config --dump just have:

k3os-21766 [~]$ sudo k3os config --dump
k3os: {}

As its output.

To Reproduce

The start.sh file:

kernel_args="\
console=ttyS0 \
k3os.mode=install \
k3os.install.silent=true \
k3os.install.power_off=true \
init_cmd=\"cp /.base/k3os_conf.yaml /config.yaml\" \
k3os.install.config_url=/config.yaml \
k3os.install.device=/dev/vda \
k3os.hostname=k3os \
k3os.install.debug=true \
k3os.debug=true \
"

virt-install \
--connect qemu:///system \
--name k3osvm01 \
--ram 2048 \
--vcpus 1 \
--os-type linux \
--os-variant ubuntu20.04 \
--graphics none \
--network bridge=virbr0 \
--disk path=k3os-amd64.iso,device=cdrom \
--disk path=k3osvm01.qcow2,size=8,device=disk \
--install kernel=k3os-vmlinuz-amd64,initrd=k3os-initrd-amd64 \
--initrd-inject k3os_conf.yaml \
--extra-args "${kernel_args}"

The k3os_conf.yaml file:

ssh_authorized_keys:
- github:larssb

k3os:
  modules:
  - kvm
  dns_nameservers:
  - 192.168.1.15
  - 192.168.1.10
  k3s_args:
  - server
  - "--no-deploy"
  - "traefik"

• The k3os_conf.yaml file IS in the same folder as the start.sh script

Expected behavior
That the config.yaml file is applied and persisted to the K3OS system

Actual behavior
The config.yaml file does not seem to be properly applied.

Additional context
The config.yaml file is placed on the local disk, on the same system whereon the K3OS VM is installed via virt-install.

Info from the debug output info, from the install process. In chronological order, as they appear in the debug output.

+ /.base/k3os/system/k3os/current/k3os config --initrd                                                                                                                        
DEBU[0000] running cmd `cp /.base/k3os_conf.yaml /config.yaml`   

 * /run/lock: correcting owner                                                                                                                                                
[    2.602883] pciehp 0000:00:01.5:pcie004: Failed to check link status      

/dev/vda1 contains a ext4 file system labelled 'K3OS_STATE'   

Installation finished. No error reported.                                                                                                                                     
[   13.213563] reboot: Power down  

• Notice that it is a "power off installation". However, the guest is still restarted:

Installing for i386-pc platform.
Installation finished. No error reported.
[   13.213563] reboot: Power down

Domain creation completed.
Restarting guest.
Connected to domain k3osvm01

• I may be misunderstanding something. But it seems odd that the guest is restarted.

• Notice that the init_cmd throws no errors.

---

Looking forward to some help on this. Thank you very much and have a great weekend.

[dweomer]

@larssb regarding the restart, given the output, that is something that virt-install is doing. there is probably a flag to pass to prevent that.

IIRC install takes place after the pivot to /.base which means your /config.yaml is not reachable. if you simply inject it as config.yaml and do nothing else with it, does the install work as expected?

[larssb]

Hi @dweomer,

Thank you for your quick response. I've tried the following based on your input.

FIRST TRY

1. Removed the init_cmd line from the kernel_args in my start.sh file. And the k3os.install.config_url=/config.yaml. Keeping the --initrd-inject k3os_conf.yaml parameter/value
2. Ran start.sh ... no error. But also no data in the ~.ssh folder in K3OS. And

SECOND TRY

1. I then added back the k3os.install.config_url=/config.yaml parameter/value. I then got this error:

k3os/system/kernel/5.0.0-43-generic/
k3os/system/kernel/5.0.0-43-generic/initrd
k3os/system/kernel/5.0.0-43-generic/kernel.squashfs
k3os/system/kernel/current
cp: cannot stat '/config.yaml': No such file or directory
ERRO[0003] exit status 1                                
 [ ok ]
 * Checking Open-iSCSI configuration ... [ ok ]
 * Starting iscsid ... [ ok ]
 * Setting up iSCSI targets ...iscsiadm: No records found
 [ !! ]
 * Starting issue ... [ ok ]

So the init_cmd parameter seems necessary and the config.yaml seems to be available late in the install process when using it. Or am I straying away here?

Also, just to be sure. Seeing this, on K3OS boot, at the login screen: You can configure this system or install to disk using "sudo k3os install". That means that install is ephemeral and has not persisted to disk/completed its setup? I've seen this, so far, in all the ways I've tried to install K3OS.

---

I did find the virt-install parameters to ensure that the K3OS VM is not rebooted immediately.

Thank you.

[larssb]

Just tried going away from the ìnit_cmd<=>--initrd-inject k3os_conf.yamlcombo. And instead defined thek3os.install.config_url=` with a GitHub Gist URI. This one. I can see that the config.yaml is downloaded:

k3os/system/kernel/5.0.0-43-generic/kernel.squashfs
k3os/system/kernel/current
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   170  100   170    0     0    596      0 --:--:-- --:--:-- --:--:--   596
Installing for i386-pc platform.
Installation finished. No error reported.
[   14.539286] reboot: Power down

However, still no data in the .ssh/ folder. So it looks like the config.yaml file is not applied.

[larssb]

Wow just wow 👍. What a hurdle through the woods. So!

I thought, this looks like a case of the installation running again, even with the k3os.install.power_off parameter set to true and using the virt-install -noreboot parameter.

The VM was shutdown:

bingchong@dl380g9:~/k3os$ virsh list --all
 Id   Name       State
---------------------------
 -    k3osvm01   shut off

After executing virsh --connect qemu:///system start k3osvm01 and some waiting, I would see: the You can configure this system or install to disk using "sudo k3os install" message.

So what I did was simply to remove the K3OS ISO file. With: virsh change-media k3osvm01 /home/bingchong/k3os/k3os-amd64.iso --eject. Then I started the VM and now ... drum-roll ...

bingchong@dl380g9:~/k3os$ ssh -i /home/bingchong/.ssh/github_ssh_rsa_nopass -p 22 

----
[email protected]
Welcome to k3OS!

Refer to https://github.com/rancher/k3os for README and issues

By default mode of k3OS is to run a single node cluster. Use "kubectl"
to access it.  The node token in /var/lib/rancher/k3s/server/node-token
can be used to join agents to this server.

----
k3os-27852 [~]$ ls -la .ssh/
total 12
drwx--S--- 2 rancher rancher 4096 May 23 19:19 .
drwxr-sr-x 3 rancher rancher 4096 May 23 19:19 ..
-rw------- 1 rancher rancher  726 May 23 19:19 authorized_keys 

Spank you very much indeedyyyyy! 💯 .... great stuff.

It would be lovely though, if one did NOT have to detach the ISO from the VM. Is that possible somehow? 🥇

Thank you.

[larssb]

For good measure I just tried with the --initrd-inject, init_cmd parameters. And the k3os.install.config_url= set to /config.yaml.

Tadaaaa, that also works! Great. So no need for an external connection. Nice! But again, is there some way to avoid having to detach the ISO? And also, this was kinda hard to grasp, that this was the error. Very benign logical path here. I mean, for someone rather new to K3OS, the root issue here, could be caused by so many things.

What is your take @dweomer ?

Thank you.

[larssb]

For the ones landing on this issue in the future. This is the working start.sh script. Which uses virt-install and auto-setup of K3OS.

kernel_args="\
console=ttyS0 \
k3os.mode=install \
k3os.install.silent=true \
init_cmd=\"cp /.base/k3os_conf.yaml /config.yaml\" \
k3os.install.config_url=/config.yaml \
k3os.install.device=/dev/vda \
k3os.hostname=k3os \
k3os.install.debug=true \
k3os.debug=true \
"

virt-install \
--connect qemu:///system \
--name k3osvm01 \
--ram 2048 \
--vcpus 1 \
--os-type linux \
--os-variant ubuntu20.04 \
--graphics none \
--network bridge=virbr0 \
--disk path=k3osvm01.qcow2,size=8,device=disk \
--disk path=k3os-amd64.iso,device=cdrom \
--install kernel=k3os-vmlinuz-amd64,initrd=k3os-initrd-amd64 \
--initrd-inject k3os_conf.yaml \
--extra-args "${kernel_args}" \
--autostart

So in the end it was a matter of order of the --disk parameters. The cdrom/iso one needs to be after the disk. After this change there was no need to set k3os.install.power_off=true or virt-install's --noreboot.

Alrightyyyyy then . This is a case of Close and comment.

🥇

[mlindhout]

Hi there, when I use the script mentioned above, the install failed with the message
```++++ blkid -L K3OS
+++ K3OS_ISO=/dev/sr0
+++ '[' -n /dev/sr0 ']'
++++ blkid -L K3OS
+++ mount -o ro /dev/sr0 /.base
mount: /.base: unknown filesystem type 'iso9660'.

• rescue
• echo ERROR 'Something went wrong, run with cmdline k3os.debug for more logging'
  ERROR Something went wrong, run with cmdline k3os.debug for more logging
• echo Dropping to shell
  Dropping to shell
• exec bash```

Any ideas?