Aks-engine: [bug] linux-image-4.15.0-1045-azure with grub

[Flask]

Is this a request for help?: YES

---

Is this an ISSUE or FEATURE REQUEST? ISSUE

---

What version of aks-engine?: v0.35.1-aks

---

Kubernetes version: Tried with 1.13.5 and 1.12.8

What happened:
On any newly created cluster there seems a problem with the package linux-image-4.15.0-1045-azure it seems that the grub config is invalid.

/var/log/unattended-upgrades/unattended-upgrades-dpkg.log shows the following entries:

Log started: 2019-05-23  13:08:20
Selecting previously unselected package linux-modules-4.15.0-1045-azure.
(Reading database ... 56882 files and directories currently installed.)
Preparing to unpack .../linux-modules-4.15.0-1045-azure_4.15.0-1045.49_amd64.deb ...
Unpacking linux-modules-4.15.0-1045-azure (4.15.0-1045.49) ...
Selecting previously unselected package linux-image-4.15.0-1045-azure.
Preparing to unpack .../linux-image-4.15.0-1045-azure_4.15.0-1045.49_amd64.deb ...
Unpacking linux-image-4.15.0-1045-azure (4.15.0-1045.49) ...
Preparing to unpack .../linux-azure_4.15.0.1045.49_amd64.deb ...
Unpacking linux-azure (4.15.0.1045.49) over (4.15.0.1042.46) ...
Preparing to unpack .../linux-image-azure_4.15.0.1045.49_amd64.deb ...
Unpacking linux-image-azure (4.15.0.1045.49) over (4.15.0.1042.46) ...
Selecting previously unselected package linux-azure-headers-4.15.0-1045.
Preparing to unpack .../linux-azure-headers-4.15.0-1045_4.15.0-1045.49_all.deb ...
Unpacking linux-azure-headers-4.15.0-1045 (4.15.0-1045.49) ...
Selecting previously unselected package linux-headers-4.15.0-1045-azure.
Preparing to unpack .../linux-headers-4.15.0-1045-azure_4.15.0-1045.49_amd64.deb ...
Unpacking linux-headers-4.15.0-1045-azure (4.15.0-1045.49) ...
Preparing to unpack .../linux-headers-azure_4.15.0.1045.49_amd64.deb ...
Unpacking linux-headers-azure (4.15.0.1045.49) over (4.15.0.1042.46) ...
Selecting previously unselected package linux-azure-tools-4.15.0-1045.
Preparing to unpack .../linux-azure-tools-4.15.0-1045_4.15.0-1045.49_amd64.deb ...
Unpacking linux-azure-tools-4.15.0-1045 (4.15.0-1045.49) ...
Selecting previously unselected package linux-tools-4.15.0-1045-azure.
Preparing to unpack .../linux-tools-4.15.0-1045-azure_4.15.0-1045.49_amd64.deb ...
Unpacking linux-tools-4.15.0-1045-azure (4.15.0-1045.49) ...
Preparing to unpack .../linux-tools-azure_4.15.0.1045.49_amd64.deb ...
Unpacking linux-tools-azure (4.15.0.1045.49) over (4.15.0.1042.46) ...
Selecting previously unselected package linux-azure-cloud-tools-4.15.0-1045.
Preparing to unpack .../linux-azure-cloud-tools-4.15.0-1045_4.15.0-1045.49_amd64.deb ...
Unpacking linux-azure-cloud-tools-4.15.0-1045 (4.15.0-1045.49) ...
Selecting previously unselected package linux-cloud-tools-4.15.0-1045-azure.
Preparing to unpack .../linux-cloud-tools-4.15.0-1045-azure_4.15.0-1045.49_amd64.deb ...
Unpacking linux-cloud-tools-4.15.0-1045-azure (4.15.0-1045.49) ...
Preparing to unpack .../linux-cloud-tools-azure_4.15.0.1045.49_amd64.deb ...
Unpacking linux-cloud-tools-azure (4.15.0.1045.49) over (4.15.0.1042.46) ...
Setting up linux-modules-4.15.0-1045-azure (4.15.0-1045.49) ...
Setting up linux-image-4.15.0-1045-azure (4.15.0-1045.49) ...
I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-1045-azure
I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-1045-azure
Setting up linux-image-azure (4.15.0.1045.49) ...
Setting up linux-azure-headers-4.15.0-1045 (4.15.0-1045.49) ...
Setting up linux-headers-4.15.0-1045-azure (4.15.0-1045.49) ...
Setting up linux-headers-azure (4.15.0.1045.49) ...
Setting up linux-azure-tools-4.15.0-1045 (4.15.0-1045.49) ...
Setting up linux-tools-4.15.0-1045-azure (4.15.0-1045.49) ...
Setting up linux-tools-azure (4.15.0.1045.49) ...
Setting up linux-azure-cloud-tools-4.15.0-1045 (4.15.0-1045.49) ...
Setting up linux-cloud-tools-4.15.0-1045-azure (4.15.0-1045.49) ...
Setting up linux-cloud-tools-azure (4.15.0.1045.49) ...
Setting up linux-azure (4.15.0.1045.49) ...
Processing triggers for linux-image-4.15.0-1045-azure (4.15.0-1045.49) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-1045-azure
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
/etc/kernel/postinst.d/zz-update-grub:
: not foundrub-mkconfig: 5: /etc/default/grub: 
run-parts: /etc/kernel/postinst.d/zz-update-grub exited with return code 127
dpkg: error processing package linux-image-4.15.0-1045-azure (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 linux-image-4.15.0-1045-azure
Log ended: 2019-05-23  13:08:49

uname -a shows that there is the old kernel in place:
Linux aks-nodepool1-40358730-0 4.15.0-1042-azure #46-Ubuntu SMP Thu Apr 4 16:30:23 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

What you expected to happen: dpkg should not be broken

How to reproduce it (as minimally and precisely as possible):
Just connect to a node of a newly installed cluster and run dpkg --configure -a

Anything else we need to know:
Various log files (syslog, auth.log) are empty (zero bytes).

Sudo on the nodes is behaving weird:

sudo su
su: pam_start: error 26

/var/log/messages showing following error.

May 23 13:54:59 aks-nodepool1-40358730-0 su[56034]: PAM unable to dlopen(pam_rootok.so#015): /lib/security/pam_rootok.so#015: cannot open shared object file: No such file or directory
May 23 13:54:59 aks-nodepool1-40358730-0 su[56034]: PAM adding faulty module: pam_rootok.so
May 23 13:54:59 aks-nodepool1-40358730-0 su[56034]: PAM (su) illegal module type:
May 23 13:54:59 aks-nodepool1-40358730-0 su[56034]: PAM (su) no control flag supplied
May 23 13:54:59 aks-nodepool1-40358730-0 su[56034]: PAM (su) no module name supplied

Is there a possibilty to create a cluster with an older image or even with an older aks-engine?

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[CecileRobertMichon]

Hi there, we are aware of this issue and are working on a fix for AKS. If you are deploying using aks-engine, this is fixed in v0.36.0.

Manual workaround:

The grub issue is preventing kernel, grub, initramfs, and similar boot chain updates from succeeding. You can bypass it on a specific VM by installing the the “dos2unix” package and then

$ sudo dos2unix /etc/default/grub
$ sudo apt-get upgrade -y
$ sudo apt-get dist-upgrade -y

[CecileRobertMichon]

The fix is now rolled out to AKS. The latest AKS Engine release (v0.36.1) does not have this issue.