Rke: error while creating mount source path '/etc/kubernetes': mkdir /etc/kubernetes: permission denied

[jazoom]

I'm just checking out RKE and ran into this problem.

I suspect this is due to having installed Docker via snap, which I did as a way to install 17.03.2-ce on Ubuntu 18.04.

RKE version: v0.1.8

Docker version: (docker version,docker info preferred)

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 17.03.2-ce
Storage Driver: aufs
 Root Dir: /var/snap/docker/common/var-lib-docker/aufs
 Backing Filesystem: extfs
 Dirs: 15
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version:  (expected: 4ab9917febca54791c5f071a9d1f404867857fcc)
runc version: N/A (expected: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe)
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-29-generic
Operating System: Ubuntu Core 16
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 985.4 MiB
Name: kube1
ID: TYVR:WN6J:5UL5:E3O7:4EH4:HUJZ:KG5N:GZ7U:GAFY:YL5R:H7EX:SPST
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 15
 Goroutines: 23
 System Time: 2018-08-03T06:19:10.408234813Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

uname -r: 4.15.0-29-generic

NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Vultr

cluster.yml file:

# https://rancher.com/docs/rke/v0.1.x/en/example-yamls
nodes:
  - address: *address*
    port: *address*
    internal_address: *address*
    role:
      - controlplane
      - etcd
    hostname_override: kube1
    user: root
    docker_socket: /var/run/docker.sock
    ssh_key: ""
    ssh_key_path: ""
    labels: {}
  - address: *address*
    port: *address*
    internal_address: *address*
    role:
      - controlplane
      - worker
      - etcd
    hostname_override: kube2
    user: root
    docker_socket: /var/run/docker.sock
    ssh_key: ""
    ssh_key_path: ""
    labels: {}
  - address: *address*
    port: *address*
    internal_address: *address*
    role:
      - controlplane
      - worker
      - etcd
    hostname_override: kube3
    user: root
    docker_socket: /var/run/docker.sock
    ssh_key: ""
    ssh_key_path: ""
    labels: {}
  - address: *address*
    port: *address*
    internal_address: *address*
    role:
      - worker
    hostname_override: kube4
    user: root
    docker_socket: /var/run/docker.sock
    ssh_key: ""
    ssh_key_path: ""
    labels: {}
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    snapshot: false
    retention: ""
    creation: ""
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: true
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
authentication:
  strategy: x509
  options: {}
  sans: []
addons: ""
addons_include: []
system_images:
  etcd: rancher/coreos-etcd:v3.1.12
  alpine: rancher/rke-tools:v0.1.10
  nginx_proxy: rancher/rke-tools:v0.1.10
  cert_downloader: rancher/rke-tools:v0.1.10
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.10
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.8
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.8
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.8
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: rancher/hyperkube:v1.10.5-rancher1
  flannel: rancher/coreos-flannel:v0.9.1
  flannel_cni: rancher/coreos-flannel-cni:v0.2.0
  calico_node: rancher/calico-node:v3.1.1
  calico_cni: rancher/calico-cni:v3.1.1
  calico_controllers: ""
  calico_ctl: rancher/calico-ctl:v2.0.0
  canal_node: rancher/calico-node:v3.1.1
  canal_cni: rancher/calico-cni:v3.1.1
  canal_flannel: rancher/coreos-flannel:v0.9.1
  wave_node: weaveworks/weave-kube:2.1.2
  weave_cni: weaveworks/weave-npc:2.1.2
  pod_infra_container: rancher/pause-amd64:3.1
  ingress: rancher/nginx-ingress-controller:0.10.2-rancher3
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
ssh_key_path: *path-to-key*
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""

Steps to Reproduce:
rke up

Results:
Seems to work okay until ...

...INFO[0014] [network] Port listener containers removed successfully
INFO[0014] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts
FATA[0014] Failed to copy file [/etc/kubernetes/.tmp/kube-proxy.pem] from container [cert-fetcher] on host [*address*]: Error response from daemon: error while
creating mount source path '/etc/kubernetes': mkdir /etc/kubernetes: permission denied

Will RKE running the setup containers as privileged get around this issue?

[superseb]

There has been a reported issue on this here: https://github.com/rancher/rke/issues/582

Also, from https://github.com/docker/docker-snap:

NOTE: The Docker Snap is no longer maintained by Docker, Inc. The last release version was 17.06.2 and will not be updated. The future of the Docker Snap is unclear, but it won't be developed here going forward. This repository will redirect to its new home if one will exist.

You can install Docker using apt and it should work without any issue. Let me know if it doesn't work.

[jazoom]

There has been a reported issue on this here:

I did search the issues but that didn't come up for me.

I wish they put that on their snap page. Regardless, it doesn't really matter if it's maintained for my current purposes. I just wanted to get the older version installed on Ubuntu 18.04.

You can install Docker using apt and it should work without any issue. Let me know if it doesn't work.

I tried with your Docker install script and it failed.apt-get doesn't report those Docker versions as being available on Ubuntu 18.04. I didn't try with apt. I'll see if it's any different.

[superseb]

Docker 17.03 is not a released version for Ubuntu 18.04 (https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/).

We have this open to validate Ubuntu 18.04: https://github.com/rancher/rancher/issues/13888

[jazoom]

Yes. That's why I used snap.

[jonaskello]

I got around this issue by using --classic when installing snap as mentioned here.

sudo snap install docker --channel=17.03/stable --classic

# Create and join the docker group.

sudo addgroup --system docker
sudo adduser $USER docker
newgrp docker

# You will also need to disable and re-enable the docker snap if you added the group while it was running.

sudo snap disable docker
sudo snap enable docker

[leoyer]

thank you! I reinstall docker resovle this problem
but when reboot machine the problem reproduce !

https://github.com/moby/moby/issues/33060

Finally, I found a solution.

sudo rm -rf /var/lib/snapd/apparmor/profiles/snap.docker.*
sudo apt-get autoremove docker-ce
sudo apt-get install docker-ce