Microk8s: How do I use proxy in microk8s.docker

[joye-wang]

I'm in china, so I can't access k8s.gcr.io directly, and I use 'lantern' as a http/socks proxy to access it.
So when I try to exec command microk8s.enable istio, the status of istio pod is always ContainerCreating
the output of command kubectl describe is

Warning  FailedScheduling        64s (x3 over 65s)  default-scheduler                0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled               59s                default-scheduler                Successfully assigned istio-system/grafana-6b99c54645-pghj4 to tianyu-virtual-machine
  Warning  FailedCreatePodSandBox  16s (x2 over 43s)  kubelet, tianyu-virtual-machine  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I followed the article: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
And now I can pull images via default docker(not microk8s.docker):

tianyu@tianyu-virtual-machine:~$ docker pull k8s.gcr.io/kube-apiserver:v1.13.3
v1.13.3: Pulling from kube-apiserver
Digest: sha256:cdb80dc78f3c25267229012a33800b8b203e8e8b9fa59f9fe93e156cc859f89c
Status: Image is up to date for k8s.gcr.io/kube-apiserver:v1.13.3

But when I try to pull image via microk8s.docker, the output is:

tianyu@tianyu-virtual-machine:~$ microk8s.docker pull k8s.gcr.io/kube-apiserver:v1.13.3 
[sudo] password for tianyu: 
Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

How do I use proxy in microk8s.docker? or Can microk8s use default docker(not microk8s.docker)? or can I share the images between docker and microk8s.docker?

[ktsakalozos]

Hi @15050050972

Did you follow the proxy setup instructions in https://github.com/ubuntu/microk8s#deploy-behind-a-proxy ?

You will need to edit /var/snap/microks8/current/args/dockerd-env and restart MicroK8s (or just the dockerd shipped with MicroK8s).

[joye-wang]

Yes, It worked! I can pull images via microk8s.docker, I execute microk8s.disable istio and microk8s.enable istio, but the status of pod is still ContainerCreating,
I execute kubectl describe, but I didn't know what's wrong, can you give me some help? Thanks very much!
I am on a virutal machine of ubuntu 18, here is output

tianyu@tianyu-virtual-machine:~$ kubectl describe pod istio-citadel-68c48d4fc8-5bjwb -n istio-system
Name:               istio-citadel-68c48d4fc8-5bjwb
Namespace:          istio-system
Priority:           0
PriorityClassName:  <none>
Node:               tianyu-virtual-machine/172.16.109.130
Start Time:         Mon, 04 Mar 2019 16:07:26 +0800
Labels:             istio=citadel
                    pod-template-hash=68c48d4fc8
Annotations:        scheduler.alpha.kubernetes.io/critical-pod: 
                    sidecar.istio.io/inject: false
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/istio-citadel-68c48d4fc8
Containers:
  citadel:
    Container ID:  
    Image:         gcr.io/istio-release/citadel:1.0.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --append-dns-names=true
      --grpc-port=8060
      --grpc-hostname=citadel
      --citadel-storage-namespace=istio-system
      --self-signed-ca=true
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        10m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from istio-citadel-service-account-token-w68cs (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  istio-citadel-service-account-token-w68cs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-citadel-service-account-token-w68cs
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age   From                             Message
  ----     ------            ----  ----                             -------
  Warning  FailedScheduling  119s  default-scheduler                0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled         113s  default-scheduler                Successfully assigned istio-system/istio-citadel-68c48d4fc8-5bjwb to tianyu-virtual-machine
  Warning  FailedMount       112s  kubelet, tianyu-virtual-machine  MountVolume.SetUp failed for volume "istio-citadel-service-account-token-w68cs" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulling           109s  kubelet, tianyu-virtual-machine  pulling image "gcr.io/istio-release/citadel:1.0.0"

[ktsakalozos]

Is it possible you have to give it a bit more time to grab the images?

[placidchat]

How do you set the timeouts for waiting ?

[ktsakalozos]

How do you set the timeouts for waiting ?

I am not aware of any good way of monitoring the progress of pulling images (but I haven't looked at it either). When on a machine with limited network bandwidth I just look at the network traffic.

[joye-wang]

I waited for a moment, now It worked. thank you

[jesinity]

what is the format of the dockerd-env ?
Can you provide an example?

[Hubro]

Hi @15050050972

Did you follow the proxy setup instructions in https://github.com/ubuntu/microk8s#deploy-behind-a-proxy ?

You will need to edit /var/snap/microks8/current/args/dockerd-env and restart MicroK8s (or just the dockerd shipped with MicroK8s).

I'm guessing this answer might be outdated? I just installed the newest version of microk8s today, and I can't find the command microk8s.docker and I can't find the file /var/snap/microks8/current/args/dockerd-env. Also, your link is broken (there is no deploy-behind-a-proxy header).

I need a HTTP proxy to reach the internet, so all my pods are stuck in ContainerCreating.

[Hubro]

Ah I see, microk8s uses containerd on my system. The right file to edit is /var/snap/microk8s/current/args/containerd-env.

[galo]

I am using v1.17.2 with Eoan. even after setting up /var/snap/microk8s/current/args/containerd-env with my corp proxy I still see this errors get getting containers.

Yype Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 4m37s (x322 over 74m) kubelet, 15.41.130.97 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "k8s.gcr.io/pause:3.1": failed to pull image "k8s.gcr.io/pause:3.1": failed to resolve image "k8s.gcr.io/pause:3.1": no available registry endpoint: failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.1: proxyconnect tcp: tls: first record does not look like a TLS handshake

How is the proxy conf passed to containerd? Can i change this directly?

[galo]

Just an update. Setting proxies on /etc/environment on top of containerd-env fixed my issue