Kubeadm: Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring

[mamer67]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version): {Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): {Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
• Kernel (e.g. uname -a): Linux 4.9.137-0.1.ac.218.74.329.metal1.x86_64 #1 SMP Wed Dec 5 15:11:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

What happened?

On doing kubeadm join, everything works successfully, however logs on worker nodes keeps showing these errors Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring

E0108 09:42:01.655993   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:02.656241   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:03.656508   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:04.656777   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:05.657077   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:06.657331   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:07.657615   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:08.657855   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:09.658086   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0108 09:42:10.658327   26056 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
....

This doesn't affect node health. However its noise in logs. After I manually create the missing folder /etc/kubernetes/manifests, this error stops.

What you expected to happen?

I expected, kubeadm during join process to create that folder.

How to reproduce it (as minimally and precisely as possible)?

1. Install kubeadm/kubelet/kubectl of same version 1.13.1. Doesn't matter if its installed from binaries or from yum repos. Both have same behavior.
2. Make sure you run kubeadm reset on all master/worker nodes. Also remove /etc/kubernetes folder, to test kubeadm bootstrap process.
3. Run kubeadm init on master and kubeadm join on worker node.
4. Check logs on worker node and you should see same errors as I posted above.

[timothysc]

How did you install kubeadm?
.deb/.rpm ?

[mamer67]

I tried two different setup methods, both with same results

• Followed RHEL instructions at this page https://kubernetes.io/docs/setup/independent/install-kubeadm/
• Installed v1.13.1 from here https://github.com/kubernetes/kubernetes/releases and downloaded kubectl/kubelet/kubeadm binaries using ./kuberenetes/cluster/get-kube-binaries.sh script.

[RA489]

I can take a look.

[fabriziopandini]

Thanks @RA489
/lifecycle active

[RA489]

@mamer67 I investigated this issue on CentOS but i didn't find any missing config path "/etc/kubernetes/manifests"

[mamer67]

@RA489 Did u use kubeadm on that host before? I mean, have you made sure before running kubeadm that /etc/kubernetes folder doesn't exist?.

[RA489]

@mamer67 yea i didn't exist before

[mamer67]

@RA489 Strange!. This almost showed up on my side on every worker node, while running on AWS/EC2 AmazonLinux hosts. I will try again on my side, and provide more snapshots.

Could you also post kubeadm version you are using?

[RA489]

@mamer67 yea you can recheck from your side if problem still exists

[mamer67]

Oh I see you are using kubeadm v1.13.2, I was reporting the issue on kubeadm v1.13.1. Let me update kubeadm on my box and try again, and see. Its probably fixed already with version change.

[RA489]

@mamer67 I don't think there will be much difference between v1.13.1 and v1.13.2 but yeah you can upgrade your kubeadm box.

[mamer67]

@RA489 Changing the version doesn't help as you said. I am still experiencing same issues on worker node.

E0124 09:56:43.089609   25648 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0124 09:56:44.089888   25648 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0124 09:56:45.090173   25648 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0124 09:56:46.090459   25648 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0124 09:56:47.090712   25648 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
E0124 09:56:48.090983   25648 file_linux.go:61] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring

In worker node I only get this,

ls /etc/kubernetes
bootstrap-kubelet.conf  kubelet.conf  pki

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:33:30Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

kubelet --version
Kubernetes v1.13.2

Can I also ask, how did u install k8s tools?

[RA489]

@mamer67 let me also try with kubeadm version v1.13.1 also so that we can see if problem exists.

[RA489]

@mamer67 I checked on both versions v1.13.1 and v1.13.2 but i couldn't find any missing config path "/etc/kubernetes/manifests".

[chuckha]

@mamer67 When you install kubelet with deb/rpm it will automatically create the /etc/kubernetes/manifests directory. https://github.com/kubernetes/release/blob/master/debian/xenial/kubelet/debian/postinst#L35

This means that kubeadm expects that directory to exist before it is run. kubeadm will not take the liberty of creating the directory for you and so you must do it manually if you have deleted it.

Please run mkdir -p /etc/kubernetes/manifests and rerun kubeadm and let us know if the issue still persists.

[RA489]

@chuckha manually creating the missing folder /etc/kubernetes/manifests, this error stops.

[mamer67]

@chuckha I see. Let me then test again by getting the tools using RPM. I honestly thought that folder is getting created as part of the join/bootstrap process done by kubeadm. Otherwise anyone running kubeadm from binaries will have to create that folder manually now.

@RA489 Thanks for your work above. I will try using RPM now and see.

[RA489]

@mamer67 let me know if you need any help while installation.

[mamer67]

@chuckha @RA489 I gave it a try using RPM and it does indeed create the missing directory. I didn't notice that before.

Before closing this, I am thinking that I should create a feature request, asking for having those directories created by kubeadm itself and not deb/rpm installers. The only reason I believe it should be part of kubeadm, incase anyone is getting the tools using binaries. Thoughts?

[chuckha]

That's out of scope for kubeadm. Kubeadm assumes configuration management/something will be taking care of system dependencies such as file system structure. However, I think we could do a better job enumerating what kubeadm expects a system to look like before it will work.

[mamer67]

OK. Thanks all for your help on this. Closing this now.

[fOO223Fr]

I am getting the same warming and its dumping my journald in v1.13.4... used kubeadm to setup the cluster. Initially i had my staticpodmanifest variable in my kubeletConfiguration but even after removing it i keep getting the dump!

ALso manual creation of the /etc/kubernetes/manifests stops the warning

[baifuwa]

I created "manifests" ,then reloaded the k8s servers manually ,then error stoped.