Rke: Support an option to enable specified feature gates in all components

[yasker]

It's painful to enable feature gates for Kubernetes with RKE. The only way to do it now is the extra_args. And the best way to make sure the feature gate has been enabled is to enable it in all the components. It will result in something like this:

services: 
  etcd: 
    extra_args: 
      election-timeout: "5000"
      heartbeat-interval: "500"
    snapshot: false
  kube-api: 
    extra_args: 
      feature-gates: "DynamicProvisioningScheduling=true"
    pod_security_policy: false
  kube-controller: 
    extra_args: 
      feature-gates: "DynamicProvisioningScheduling=true"
  kubelet: 
    extra_args: 
      feature-gates: "DynamicProvisioningScheduling=true"
    fail_swap_on: false
  scheduler: 
    extra_args: 
      feature-gates: "DynamicProvisioningScheduling=true"
  kubeproxy: 
    extra_args: 
      feature-gates: "DynamicProvisioningScheduling=true"

1. The settings are duplicated.
2. The service names are random. There is no way to tell that a user should use scheduler instead of kube-scheduler, kubeproxy instead of kube-proxy, kube-controller instead of kube-controller-manager unless they look into the code and found this:

type RKEConfigServices struct {
    // Etcd Service
    Etcd ETCDService `yaml:"etcd" json:"etcd,omitempty"`
    // KubeAPI Service
    KubeAPI KubeAPIService `yaml:"kube-api" json:"kubeApi,omitempty"`
    // KubeController Service
    KubeController KubeControllerService `yaml:"kube-controller" json:"kubeController,omitempty"`
    // Scheduler Service
    Scheduler SchedulerService `yaml:"scheduler" json:"scheduler,omitempty"`
    // Kubelet Service
    Kubelet KubeletService `yaml:"kubelet" json:"kubelet,omitempty"`
    // KubeProxy Service
    Kubeproxy KubeproxyService `yaml:"kubeproxy" json:"kubeproxy,omitempty"`
}

Besides of the naming issue, I'd like to have a separate option available to the user to enable feature gate for all the components, e.g.

services: 
  feature-gates:
  - DynamicProvisioningScheduling=true
  - VolumeSnapshotDataSource=true
  - AppArmor=false
  etcd: 
    extra_args: 
      election-timeout: "5000"
      heartbeat-interval: "500"
    snapshot: false
  kube-api: 
    pod_security_policy: false

Then RKE will apply those feature gates to the all the Kubernetes components, which should ensure that the feature gate setting will be in effect.

[cprivite]

The NetApp trident plugin now requires feature gates to be set so this becomes more relevant for customers using NetApp Trident with their Rancher rke deployed instances.
Ultimately, just having a gui menu where you can edit the enabled feature gates would be the best user experience, speaking as a user. :)

[rbkaspr]

I would like this feature to be added to Rancher as well, as the current mechanism for adding feature gates is fiddly and basically undocumented outside of rancher/rancher#14548

@cprivite I took a look over Trident's documentation, it looks like all the features gates mentioned are enabled by default as of 1.15 (and one of them is default as of 1.12)

[cprivite]

@rbkaspr All except for the VolumeSnapshotDataSource gate are on by default in 1.14, true. That one is still set to false by default in 1.15:
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

1.15 is also still experimental in rancher, so not really an option at the moment.

[carysyd]

+1
Rancher v2.3.2 is supporting k8s v1.16.2 officially, but I still find feature gates hard to enable.