Minikube: kubeadm w/ corp proxy: x509: certificate signed by unknown authority

[sj0509]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:

Environment: Windows 10 Pro

Minikube version (use minikube version): v0.33.1

• OS (e.g. from /etc/os-release): Windows 10 Pro
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): Hyper-V
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): "Boot2DockerURL": "file://N:/.minikube/cache/iso/minikube-v0.33.1.iso",
• Install tools: minikube start --vm-driver "hyperv" --hyperv-virtual-switch "SJ Virtual Switch" --docker-env HTTP_PROXY=http://host:port \ --docker-env HTTPS_PROXY=http://host:port --v 9999
• Others:

What happened: Minikube failed to start (Does create the VM though)

What you expected to happen: For minikube to start successfully and completely

How to reproduce it (as minimally and precisely as possible):
minikube start --vm-driver "hyperv" --hyperv-virtual-switch "SJ Virtual Switch" --docker-env HTTP_PROXY=http://host:port \ --docker-env HTTPS_PROXY=http://host:port --v 9999

Output of minikube logs (if applicable):

error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.2: output: v1.13.2: Pulling from kube-apiserver
73e3e9d78c61: Pulling fs layer
503f459b2f97: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:177db4b8e93a6a74ab19435edf17111d3ad18a8a4efef728712ea067ea8047c1: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.2: output: v1.13.2: Pulling from kube-controller-manager
73e3e9d78c61: Pulling fs layer
ef3ba03ba5d4: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:b9027a78d94c15a4aba54d45476c6f295c0db8f9dcb6fca34c8beff67d90a374: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.2: output: v1.13.2: Pulling from kube-scheduler
73e3e9d78c61: Pulling fs layer
9346ad146311: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:3193be46e0b3e215877b122052c0c7d3ef0902cf1dd6efaf3db95f37cf697002: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.2: output: v1.13.2: Pulling from kube-proxy
73e3e9d78c61: Pulling fs layer
0c440f353724: Pulling fs layer
9f11bf6a2d3d: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:01cfa56edcfc350d36cea9c2fc857949b36bc69bf69df6901e0fd9be3c826617: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: 3.1: Pulling from pause
67ddbfb20a22: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: 3.2.24: Pulling from etcd
8c5a7da1afbc: Pulling fs layer
0d363128e48e: Pulling fs layer
1ba5e77f0f6e: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:3cab8e1b9802cbe23a2703c2750ac4baa90b049b65e2a9e0a83e9e2c29f0724f: x509: certificate signed by unknown authority
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: 1.2.6: Pulling from coredns
2796eccf0de2: Pulling fs layer
6ad5128a7d32: Pulling fs layer
error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:f59dcacceff45b5474d1385cd5f500d0c019ed9ca50ed5b814ac0c5fcec8699e: x509: certificate signed by unknown authority
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

Anything else do we need to know:
I'm able to pull the same images via docker pull command without an issue.
Except these two:
PS C:> docker pull k8s.gcr.io/kube-scheduler:v1.13.2
Error response from daemon: Get https://k8s.gcr.io/v2/: Proxy Authentication Required

PS C:> docker pull k8s.gcr.io/etcd:3.2.24
Error response from daemon: Get https://k8s.gcr.io/v2/: Proxy Authentication Required

[sj0509]

Weirdly, if I try the docker pull multiple times for the last 2 failing ones, (Proxy Authentication Required), they seem to work, and I'm able to get the images.

[sefroberg]

I think to simplify the issue all you have to do is go onto the vm and issue a docker pull for one of the images. In my case I can reproduce this with this command:
'docker pull k8s.gcr.io/kube-apiserver:v1.13.2'

[sefroberg]

It also happens on Centos 7.3 and 7.6

[tstromberg]

I'm pretty sure there is corporate SSL interception happening here, similar to #2739 - but apparently mostly a problem within the VM.

[sefroberg]

I was able to get minikube up and running yesterday afternoon by following the instructions for first creating a folder structure in the 'files' folder inside the ~/.minikube folder.
.minikube/files/etc/ssl/certs
Then I placed the .pem files I needed for my IT's compliance into this folder. Then rebuilt the vm (need to issue minikube delete and then minikube start). I confirmed the newly created vm has the certs in the correct place.

[tstromberg]

@sefroberg - Excellent. That you needed to run minikube delete is definitely a bug though -- you should just be able to start it again.

[tstromberg]

Thank you @sefroberg for mentioning a solution. I've added it to our official documentation here:

https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md

I think this will really help future minikube users.

[mrdulin]

@tstromberg The link is broken. New link: https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/#x509-certificate-signed-by-unknown-authority

Besides, is there any solution if I can't ask the IT department for the appropriate PEM file?

Some command flag like: --skip-verify-certificate?