Kubespray: Release Proposal v2.14

[floryut]

Below is a proposal for release notes:

Announcements

We are looking for maintainers, reach out in #5432.

Deprecation / Removal

• Removed support for Fedora 29 and 30 (EOL)
• Remove support for CoreOS Container Linux (EOL)

Major changes:

• Add Oracle Linux 8 support and fixes (#6198)
• Add Ubuntu 20.04 support (#6157)
• Add support for Fedora 32 (#6426)
• Add support for Kata Containers (#6256)
• Switch to Python3 on Debian & Ubuntu (#6157)
• Add Ambassador OSS ingress controller (#6135)
• Add ovn4nfv-k8s-plugin as network plugin (#6381)
• Improve air-gap installation instructions (#6234)
• Add TLS cipher suites support for kubeadm and kubelet (#6024 #6490)
• Update most ETCDCTL_API call to v3 (#5998)
• Upgrade molecule to v3 (#6468)
• Remove-node play will now fail if node can not be drained (#6442)

Applications

• [Azure] Update documentation with az command (#6042)
• [Azure] Add azure_cloud parameter to cloud_config file (#6321)
• [CSI] Update CSI containers to latest versions (#6221)
• [MetalLB] Option to talk BGP (#6383)
• [MetalLB] The deployment becomes one of addons. You can deploy it with a new option metallb_enabled (#6238)
• [Openstack] Support volume type (#6524) _(See Notes)_
• [Openstack] Make it possible to open additional ports on masters (#6547)
• [Openstack] Add support for application credentials (#6534)
• [Openstack] Add snapshot-controller for CSI drivers (#6537)
• [Openstack] Added a default volumesnapshotclass for Cinder CSI (#6537)

Container managers

• Match docker-cli version with docker-engine version (#6163)
• [Docker] Set cgroup driver by default to systemd (#6563)
• [Containerd] Install package is now managed alongside docker (#6218)
• [Containerd] Add support for Fedora (#6094)
• [CRI-O] Use OS packaging default value for apparmor_profile in crio.conf (#6125)
• [CRI-O] Fix kubelet cgroup driver detection (#6331)
• [CRI-O] Align template crio.conf with upstream and set cgroup driver by default to systemd (#6432)
• [CRI-O] Harden downloads with retry (#6374)
• [CRI-O] Add variable to configure unsecure pull (#6568)

Network

• [Weave] Allow Weave DS to support any taint effect (#6159)
• [Calico] Disable bird-check flag for probes of calico-node pods when calico_network_backend is not bird (#6217)
• [Calico] Add FELIX_DEVICEROUTESOURCEADDRESS option (#6508)
• [Kube-Router] Enable portmap CNI plugin with kube-router to allow use of hostPort in container specs (#6204)
• [Kube-Router] Add selectable dns policy (#6586)
• [Cilium] Add a way to deploy cilium alongside another CNI (#6373)
• [Cilium] Add option to configure IPVS timeouts in kube-proxy configration manifest (#6396)
• [Cilium] Support the overwrite of MTU in Cilium agents (#6329)
• [Cilium] Add metrics in Cilium operator and add hubble metrics port in agents (#6513)
• [Cilium] Add hubble server support in cilium (#6575)

Other note worthy changes

• Create custom dashboard namespace if specified (#6107)
• Add support to expose etcd metrics on a custom port (#6092)
• Add additional network configuration options to external Openstack (#6085)
• Fix resolv.conf configuration for Fedora CoreOS (#6138)
• Replace seccomp profile docker/default with runtime/default (#6170)
• Multiples fixes for proxy and no_proxy variables (#6112 #6431 #6558)
• Use connection: local when delegate_to: localhost (#6322)
• Add DNS configuration in NetworkManager for Fedora CoreOS (#6291)
• Allow kubeadm to upgrade etcd (#6345) _(See notes)_
• Add docs for setting up your first cluster (#6544)
• Webhook authorization can now be enabled using inventory variable (#6502)
• Uncordon node that fail to drain (thus failing its upgrade) during upgrade procedure (#6546)
• Added variable kubelet_rotate_server_certificates which enables kubelet server certificate rotation (#6453)
• Add protectKernelDefaults option (default true) to kubelet config file (#6611)

Component versions:

• Kubernetes v1.18.8
• Etcd 3.4.3
• Docker 19.03
• containerd 1.2.13
• Cri-O 1.18
• CNI-plugins v0.8.7
• Calico v3.15.2
• Cilium 1.8.3 _(See Notes)_
• Contiv 1.2.1
• Flannel 0.12.0
• Kube-Router 1.0.1 _(see Notes)_
• Multus 3.6
• kube-ovn 1.3.0 _(see Notes)_
• Weave 2.7.0
• CoreDNS 1.6.7
• nodelocaldns 1.15.13
• Helm 3.2.4
• nginx-ingress 0.35.0
• cert-manager 0.16.1 _(see Notes)_
• Kubernetes Dashboard v2.0.4
• Oracle OCI: v0.7.0

Known issues

TBD

Notes

• etcd will now be upgraded and its certs renewed when using a kubeadm managed etcd (etcd_kubeadm_enabled: true)
• Cilium: Check upgrade guide regarding update to 1.8.0
• Kube-Router: Upgrade to 1.0.0 require an iptable flush
• Kube-ovn is now installed in kube-system namespace, version priori to 1.0.0 should be removed manually
• Cert-Manager: Refer to README.md prior to upgrading in your exisitng Kubernetes cluster
• Openstack: If the nova API is before Stein, Terraform will work but the new volume type feature will not be available. If the entire cloud is upgraded to Stein or later, the new feature can be used. However if the nova versions in the cloud are mixed, with nova server API >= Stein and any nova-compute node < Stein, you will get a HTTP 409 error and VolumeTypeSupportNotYetAvailable exception.

[floryut]

@Miouge1 Let's save you some time when you got back ;)

[floryut]

Pending future PR merged @ 14/08/2020 / https://github.com/kubernetes-sigs/kubespray/commit/ef3e98807ebd38471c91ff5734d4523230f81047

[lukasz-bielinski]

@floryut what is planned release date?

[floryut]

@floryut what is planned release date?

We haven't really discuss that yet (@EppO @LuckySB @mattymo ), but should be in the coming weeks.

[hafe]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

[floryut]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

[bl0m1]

I'm currently working on a PR that will fix snapshots for the Cinder CSI. There is some issue restoring a snapshot to volume when running the old v1alpha1 API. I hope to have the PR up in a few hours. If any other CSI drivers want the same fix I have in mind that everything need to be 100% reusable.

I hope I'm not to late and that we can have this feature added to the v2.14 release

[floryut]

I'm currently working on a PR that will fix snapshots for the Cinder CSI. There is some issue restoring a snapshot to volume when running the old v1alpha1 API. I hope to have the PR up in a few hours. If any other CSI drivers want the same fix I have in mind that everything need to be 100% reusable.

I hope I'm not to late and that we can have this feature added to the v2.14 release

Of course man, only wip release note, 2.14 is not till a few weeks :smiley:

[bmelbourne]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

FYI

I've raised PR https://github.com/kubernetes-sigs/kubespray/pull/6563 to change the default cgroup driver to systemd for Docker.

Maybe we could consider getting this added to v2.14 to align with the CRI-O change?

[floryut]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

FYI

I've raised PR #6563 to change the default cgroup driver to systemd for Docker.

Maybe we could consider getting this added to v2.14 to align with the CRI-O change?

Let's wait for it to be finish (ie: not in WIP status)

[rptaylor]

I'm afraid I've run into an issue with https://github.com/kubernetes-sigs/kubespray/pull/6524 on some clouds, it may be that the Openstack documentation is inaccurate in terms of the microversion requirement, or some other issue related to the use of the Openstack API, looking into it.

[Miouge1]

Thank you so much for this @floryut !

[rptaylor]

I'm pretty sure https://github.com/kubernetes-sigs/kubespray/pull/6524 will be okay for 2.14.
The issue I ran into is described here: https://github.com/terraform-providers/terraform-provider-openstack/issues/1047
I think it is an Openstack deployment problem; if the cloud is only partially upgraded to >= Stein, the Terraform microversion check detects that the Openstack nova API can support the new feature and uses it, but the nova API checks the version of every compute node in the cloud, and if there is any older node it throws an exception. It should be fixed by fully upgrading all the nodes in the cloud to >= Stein.

[floryut]

I'm pretty sure #6524 will be okay for 2.14.
The issue I ran into is described here: terraform-providers/terraform-provider-openstack#1047
I think it is an Openstack deployment problem; if the cloud is only partially upgraded to >= Stein, the Terraform microversion check detects that the Openstack nova API can support the new feature and uses it, but the nova API checks the version of every compute node in the cloud, and if there is any older node it throws an exception. It should be fixed by fully upgrading all the nodes in the cloud to >= Stein.

Ok good to know :)

[floryut]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

FYI

I've raised PR #6563 to change the default cgroup driver to systemd for Docker.

Maybe we could consider getting this added to v2.14 to align with the CRI-O change?

And done ;)

[Miouge1]

Thank you so much @floryut ! v2.14.0 is out!