Minikube: snap-installed docker: could not read CA certificate ca.pem permission denied

[brylor]

docker could not read CA certificate ca.pem permission denied

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG

What happened:

I'm on ubuntu (sid) and using minikube v0.28.2, docker 17.06.2-ce (from snap)
I can start minikube with the kvm2 driver, I run eval $(minikube docker-env), but when I docker ps. I get:

could not read CA certificate "/home/hero/.minikube/certs/ca.pem": open /home/hero/.minikube/certs/ca.pem: permission denied. 

The perms look ok all the way down:

hero@hv1:~$ ls -lhad .minikube/
drwxrwxr-x 10 hero hero 4.0K Aug 17 14:46 .minikube/

hero@hv1:~$ ls -lhad .minikube/certs/
drwxrwxr-x 2 hero hero 4.0K Aug 17 14:44 .minikube/certs/

hero@hv1:~$ ls -lha .minikube/certs/ca.pem
-rw-rw-r-- 1 hero hero 1.1K Aug 17 14:44 .minikube/certs/ca.pem

What you expected to happen:

docker to connect to the minikube instance of docker

How to reproduce it (as minimally and precisely as possible):

1. Install ubuntu
2. install minikube via curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.2/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
3. install kvm2 driver via: curl -Lo docker-machine-driver-kvm2 https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2
4. minikube start --vm-driver kvm2
5. eval $(minikube docker-env)
6. docker ps
   could not read CA certificate "/home/hero/.minikube/certs/ca.pem": open /home/hero/.minikube/certs/ca.pem: permission denied.

[tvansteenburgh]

I believe the problem here is that the docker snap is strictly confined, which means it can't read arbitrary locations on the file system. It can read the home directory, but only non-hidden files, which excludes dot-files, such as the .minikube directory in your example.

[brylor]

Hi @tvansteenburgh. Yes you are right, snap is being blocked by AppArmor.

I was able to resolve this by adding:

owner @{HOME}/.minikube/certs/* to: /var/lib/snapd/apparmor/profiles/snap.docker.docker, and then run: apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.docker.docker

docker ps is now happy:

hero@hv1:~$ docker ps
CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS              PORTS               NAMES
39b186b346f5        k8s.gcr.io/k8s-dns-sidecar-amd64           "/sidecar --v=2 --..."   44 minutes ago      Up 44 minutes                           k8s_sidecar_kube-dns-86f4d74b45-q74tw_kube-system_c708749e-a3da-11e8-a19f-e452b0928c7b_0
30280d9c5deb        k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64     "/dnsmasq-nanny -v..."   44 minutes ago      Up 44 minutes                           k8s_dnsmasq_kube-dns-86f4d74b45-q74tw_kube-system_c708749e-a3da-11e8-a19f-e452b0928c7b_0
10be502d9c0c        gcr.io/k8s-minikube/storage-provisioner    "/storage-provisioner"   44 minutes ago      Up 44 minutes                           k8s_storage-provisioner_storage-provisioner_kube-system_c8863e32-a3da-11e8-a19f-e452b0928c7b_0
09bb155312a5        k8s.gcr.io/kubernetes-dashboard-amd64      "/dashboard --inse..."   44 minutes ago      Up 44 minutes                           k8s_kubernetes-dashboard_kubernetes-dashboard-5498ccf677-w5vcb_kube-system_c868c565-a3da-11e8-a19f-e452b0928c7b_0
...

If I did this to best practice though is to be determined, I will try to funnel this to some Ubuntu/Snap folks. For now I will close this.

Thanks.

[dlbradford]

Hi @brylor,

I tried to apply your fix above, and now Apparmor complains to me:
'AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap.docker.docker in /var/lib/snapd/apparmor/profiles/snap.docker.docker at line 291: syntax error, unexpected TOK_END_OF_RULE, expecting TOK_MODE'

Maybe it is where I placed it? or some formatting I am not entering? I am new to this and any insight you could provide would be appreciated.

file snippet for context below:
# Miscellaneous accesses
/dev/{,u}random w,
/etc/machine-id r,
/etc/mime.types r,
@{PROC}/ r,
@{PROC}/version r,
@{PROC}/version_signature r,
/etc/{,writable/}hostname r,
/etc/{,writable/}localtime r,
/etc/{,writable/}mailname r,
/etc/{,writable/}timezone r,
owner @{PROC}/@{pid}/cgroup r,
@{PROC}/@{pid}/io r,
owner @{PROC}/@{pid}/limits r,
owner @{HOME}/.minikube/certs/*,
owner @{PROC}/@{pid}/loginuid r,
@{PROC}/@{pid}/smaps r,
@{PROC}/@{pid}/stat r,
@{PROC}/@{pid}/statm r,
@{PROC}/@{pid}/status r,
@{PROC}/@{pid}/task/ r,

[dlbradford]

Hi @brylor,

Disregard report above please. I don't really know why, but removing the line and restoring the file to the original state seems to have moved past this issue.

Sorry for the report.

[eric-burel]

For absolute beginners / googlers like me the line to add is precisely:

owner @{HOME}/your-folder-that-fails r,

Eg in my case:

owner @{HOME}/.docker/machine/machines/** r,

** means any subfolder, * means any file. I met this issue after trying Docker Machine for the first time.

[staticdev]

It seems like minikube doesn't work quite well with snap installed docker. If reparsing apparmor still doesn't work, you can follow the steps:
1- Uninstall from snap: snap remove docker
2- Install from apt-get

sudo apt update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

3- Minikube delete and start (from a new terminal!)

[afbjorklund]

Actually you only need the CLI, not the daemon nor containerd (since minikube provides those).

Could even use these static binaries: https://download.docker.com/linux/static/stable/x86_64/

Not that you _need_ to use docker for minikube (or kubernetes).

Installing kubectl should be enough for interacting with cluster.

[tstromberg]

Closing because there isn't much we can do about this situation. The workarounds listed in here are however quite useful.

My recommendation is to not use the version of docker installed by snap, or
configure /var/lib/snapd/apparmor/profiles/snap.docker.docker so that it has access to $HOME/.minikube.