Kubeadm: 1.12: missleading errors when passing --config

[neolit123]

NOTE: this page needs an update:
https://kubernetes.io/docs/setup/independent/high-availability/

ref: https://github.com/kubernetes/website/issues/10450

users of kubeadm 1.12.x are reporting that they see errors when they do actions like:

kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml

didn't recognize types with GroupVersionKind: [kubeadm.k8s.io/v1alpha3, Kind=ClusterConfiguration]



md5-a62a13fb141c3bf583c47a76ce27512c



kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha3
kubernetesVersion: v1.12.1
---
kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1alpha3

/kind bug
/priority important-soon
/assign @fabriziopandini @rosti @neolit123
/cc @detiber

[fabriziopandini]

/unassign
Sorry but I don't have cycles to look into this in the short term + alpha phase are going to be removed

[neolit123]

i will see what can be done here in terms of error messages.
otherwise the same ambiguous messages will persist in v1beta1.

[neolit123]

i think the bigger question here is why we need InitConfiguration for phase kubelet write-env-file

[bloodstars]

---

kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1alpha3

worked for me....

[rosti]

Having only ClusterConfiguration should work with kubeadm init. Here is the responsible snippet for 1.12.1:
https://github.com/kubernetes/kubernetes/blob/4ed3216f3ec431b140b1d899130a69fc671678f4/cmd/kubeadm/app/util/config/masterconfig.go#L255

Unfortunately, AnyConfigFileAndDefaultsToInternal does not handle that and this function is used in several spots in the code (kubeadm config migrate and kubeadm alpha phase kubelet).

[rosti]

@neolit123 We need InitConfiguration or JoinConfiguration for phase kubelet write-env-file, because we don't have NodeRegistrationOptions in ClusterConfiguration. See here:
https://github.com/kubernetes/kubernetes/blob/4ed3216f3ec431b140b1d899130a69fc671678f4/cmd/kubeadm/app/cmd/phases/kubelet.go#L152

[Nosmoht]

When using JoinConfiguration with 1.12.2 i get the following error while adding the second node:

[tlsBootstrapToken: Invalid value: "": the bootstrap token is invalid, discovery: Invalid value: "": discoveryToken or discoveryFile must be set]

It only works with InitConfiguration.

[neolit123]

added comment in op that this page needs an update:
https://kubernetes.io/docs/setup/independent/high-availability/

[tannh]

When kubernetes v1.13 is release ?

[neolit123]

@tannh 3rd December.

[tannh]

@neolit123 https://kubernetes.io/docs/setup/independent/high-availability/
so with v1.12.2 stable, cannot creating HA cluster with kubadm.

[neolit123]

you can create HA with 1.12.

[tannh]

@neolit123 flowing on that page, i get error

didn't recognize types with GroupVersionKind: [kubeadm.k8s.io/v1alpha3, Kind=ClusterConfiguration]

apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration

kubernetesVersion: stable

My configure for all kubeadm-config.yaml on all node master.

[neolit123]

try using a config like that:

apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration

[neolit123]

docs for v1alpha3:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3

[tannh]

This config below:
apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
is for all nodes master, right?

[neolit123]

is for all nodes master, right?
yes

it should not be giving you this error.
try combining the two configs in the same file like so:

apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration

[tannh]

@neolit123
Thanks. My issue is solved. 👍

[tannh]

@neolit123 with kubeadm HA. If I want to be able to schedule pods on the master, it's okay ?. Thanks you.

[neolit123]

@tannh
try this on your worker pods:
https://docs.platform9.com/support/disallowing-workloads-on-kubernetes-master-nodes/

[neolit123]

@rosti do you think w can fix anything in here for 1.14?

[rosti]

That's on my hit list starting from tomorrow.

[yagonobre]

/lifecycle active

[rosti]

The original issue was fixed in kubernetes/kubernetes#71315.
@tannh's issues seem fixed too.

I did a few tests with 3 invalid YAML files and init/join. I think, that the resulting error messages are desent and easy to understand at this point.
My test is not a complete one (I did not test all possible sub-commands and invalid configs), but I checked a few other commands and got the same errors.

rosti@ubuntu:~/go/src/k8s.io/kubernetes$ cat ~/empty.yaml 
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ cat ~/unknown.yaml 
apiVersion: foo.bar.com/v1
kind: UnknownConfiguration
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ cat cmd/kubeadm/app/util/config/testdata/validation/invalid_nodecfg.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: NodeConfiguration
caCertPath: relativepath
discovery:
  timeout: not-a-time
  bootstrapToken:
    token: invalidtoken
    apiServerEndpoints:
    - INVALID_URL
    unsafeSkipCAVerification: false
  file:
    kubeConfigPath: relativepath
nodeRegistration:
  criSocket: relativepath
  name: NODE-1
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ ./_output/bin/kubeadm init --config ~/empty.yaml 
no InitConfiguration or ClusterConfiguration kind was found in the YAML file
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ ./_output/bin/kubeadm init --config ~/unknown.yaml 
W0215 12:56:57.018520   64236 strict.go:47] unknown configuration schema.GroupVersionKind{Group:"foo.bar.com", Version:"v1", Kind:"UnknownConfiguration"} for scheme definitions in "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go:31" and "k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs/scheme.go:28"
[config] WARNING: Ignored YAML document with GroupVersionKind foo.bar.com/v1, Kind=UnknownConfiguration
no InitConfiguration or ClusterConfiguration kind was found in the YAML file
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ ./_output/bin/kubeadm init --config cmd/kubeadm/app/util/config/testdata/validation/invalid_nodecfg.yaml
W0215 12:57:07.015278   64244 strict.go:47] unknown configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta1", Kind:"NodeConfiguration"} for scheme definitions in "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go:31" and "k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs/scheme.go:28"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta1, Kind=NodeConfiguration
no InitConfiguration or ClusterConfiguration kind was found in the YAML file
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ ./_output/bin/kubeadm join --config ~/empty.yaml 
no JoinConfiguration found in config file "/home/rosti/empty.yaml"
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ ./_output/bin/kubeadm join --config ~/unknown.yaml 
no JoinConfiguration found in config file "/home/rosti/unknown.yaml"
rosti@ubuntu:~/go/src/k8s.io/kubernetes$ ./_output/bin/kubeadm join --config cmd/kubeadm/app/util/config/testdata/validation/invalid_nodecfg.yaml
no JoinConfiguration found in config file "cmd/kubeadm/app/util/config/testdata/validation/invalid_nodecfg.yaml"

[rosti]

This issue is has evolved into having a much larger scope, than it was originally intended. Also, the original issue is long solved now.

Thus, I'll close this one and ask users who spot bad and not easy to understand error messages to open new issues with the exact command and config (if any).

/close

[k8s-ci-robot]

@rosti: Closing this issue.

In response to this:

This issue is has evolved into having a much larger scope, than it was originally intended. Also, the original issue is long solved now.

Thus, I'll close this one and ask users who spot bad and not easy to understand error messages to open new issues with the exact command and config (if any).

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

thanks for testing @rosti !

[273539918]

3Q , it's work for me @neolit123