Microk8s: cannot work with insecure gitlab container
Thanks for the awesome project.
I am trying to setup gitlab autodevops over microk8s, autobuild seem stuck at build due to some HTTP/HTTPs problem
Logging to GitLab Container Registry with CI credentials...
Error response from daemon: Get https://172.22.0.92:5050/v2/: http: server gave HTTP response to HTTPS client
I tried to edit /var/snap/microk8s/current/args/containerd-template.toml
and restart microk8s but seems it does not effect
# 'plugins."io.containerd.grpc.v1.cri".registry.mirrors' are namespace to mirror mapping for all>
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io", ]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."http://172.22.0.92:5050"]
endpoint = ["http://172.22.0.92:5050"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.22.0.92:5050"]
endpoint = ["http://172.22.0.92:5050"]
muhammadnasr
All 26 comments
Sadly this is a bug in containerd https://github.com/containerd/cri/issues/1433
Weโll have to wait until 1.4 comes out to bundle into microk8s.
joedborg
on 28 Jul 2020
Thank for the prompt reply.
Is there any workarounds? For ex: using self-signed crt or skip crt verification
muhammadnasr
on 28 Jul 2020
just tried skip tls verify:
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io", ]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."http://172.22.0.92:5050"]
endpoint = ["http://172.22.0.92:5050"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.22.0.92:5050"]
endpoint = ["http://172.22.0.92:5050"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.22.0.92"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.22.0.92".tls]
insecure_skip_verify = true
but it give falcom@node1:~$ microk8s.ctr image pull 172.22.0.92:5050/my-busybox:latest
ctr: failed to resolve reference "172.22.0.92:5050/my-busybox:latest": failed to do request: Head "https://172.22.0.92:5050/v2/my-busybox/manifests/latest": x509: certificate signed by unknown authority
muhammadnasr
on 29 Jul 2020
What i normally do is to add that certificate to your system's trusted certificate.
balchua
on 29 Jul 2020
Ah thanks @balchua. Would you happen to have 5 mins to list the steps? I can then add that to our docs until containerd 1.4 is released.
joedborg
on 29 Jul 2020
Sure. I will have to recall the exact steps coz its not something i do all the time hahaha.
I will post the steps here.
balchua
on 29 Jul 2020
That's awesome, thanks so much.
joedborg
on 29 Jul 2020
Here's roughly how i add a self signed cert in ubuntu.
- Go to
/usr/local/share/ca-certificates/ - Create a new directory (ex. "my-ca") to identify the self signed CA.
- Copy the CA's certificate in
.crtformat into the directory you just created. - Run
sudo update-ca-certificates
Any other self signed cert signed by this CA is now host trusted.
Make sure you do this to all the nodes in the kubernetes cluster.
balchua
on 30 Jul 2020
Thank you for your support.
Are you sure these will work with microk8s? it uses DinD and containerd so
we will have to do that to all new pods which is technically not practical.
Correct me if I am wrong.
On Thu, Jul 30, 2020 at 1:02 AM balchua notifications@github.com wrote:
Here's roughly how i add a self signed cert in ubuntu.
- Go to /usr/local/share/ca-certificates/
- Create a new directory (ex. "my-ca") to identify the self signed CA.
- Copy the CA's certificate in .crt format into the directory you just
created.- Run sudo update-ca-certificates
Any other self signed cert signed by this CA is now host trusted.Make sure you do this to all the nodes in the kubernetes cluster.
โ
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ubuntu/microk8s/issues/1444#issuecomment-665953025,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABIMCCPLXYFKBVXQR3LEGTR6CL75ANCNFSM4PK6JLTA
.
muhammadnasr
on 30 Jul 2020
MicroK8s is not using DinD. All cluster components are in fact managed by systemd, including containerd.
balchua
on 30 Jul 2020
Sadly this is a bug in containerd containerd/cri#1433
Weโll have to wait until 1.4 comes out to bundle into microk8s.
Which version of microk8s has this issue? We used revision 1515. It doesn't have this issue. Now it automatically update to 1550, and it stops working. We updated to latest revision 1551 which is using microk8s.ctr 1.3.4, the issue still preserves. Wasn't sure why it wasn't an issue in 1515. It's kind of a big roadblock for us now.
decewei
on 30 Jul 2020
@decewei which channel / track are you following?
joedborg
on 30 Jul 2020
@decewei which channel / track are you following?
Was 1.18/stable. For revision 1551, we switched to the stable/ channel.
decewei
on 30 Jul 2020
@decewei I would suggest reverting to 1.18/stable until this is fixed upstream. From what I understand, it's been a issue in multiple (and not necessarily concurrent) versions of containerd as there are multiple bugs and PRs to address it.
joedborg
on 30 Jul 2020
@decewei I would suggest reverting to 1.18/stable until this is fixed upstream. From what I understand, it's been a issue in multiple (and not necessarily concurrent) versions of containerd as there are multiple bugs and PRs to address it.
Thanks for the prompt reply. The thing is 1.18/stable which the team updated over the weekend was also broken... we didn't have this last week....
decewei
on 30 Jul 2020
The 1.18 branch [1] has been frozen for some time now. From revision 1515 to 1550 nothing went into that branch. Containerd is pinned to specific commit.
Where can I read more on your setup (setup gitlab autodevops with microk8s)? Is it something I can do easily in order to reproduce the error? I have not used gitlab in the past.
ktsakalozos
on 31 Jul 2020
The 1.18 branch [1] has been frozen for some time now. From revision 1515 to 1550 nothing went into that branch. Containerd is pinned to specific commit.
Where can I read more on your setup (setup gitlab autodevops with microk8s)? Is it something I can do easily in order to reproduce the error? I have not used gitlab in the past.
We didn't use gitlab autodevops with microk8s. Our setup is only a snap install microk8s --channel=1.18/stable.
We removed the last release install, that's no where to find. But it is definitely revision 1515.
The steps to reproduce this issue in revision 1550 are: (I reproduced this on my home vm machine too):
snap install microk8s --channel=1.18/stable(install the revision 1550)- Start an insecure private registry with docker or with microk8s built-in registry
- Add the address to /etc/docker/daemon.json
- Docker build and push to the registry
- Setup the endpoint if you are using private registry in /var/snap/microk8s/current/args/containerd-template.toml
- Try
microk8s.ctr image pull registry_ip:port/yourimage
Without --plain-http, this generates ctr: failed to resolve reference "ip/yourimage": failed to do request: Head "https://ip/v2/yourimage/manifests/v1": x509 certificate singed by unknown authority.. It seems for insecure registry, it does NOT check the endpoint in the template, bu automatically add https. This wasn't the behaviour in revision 1515, we could pull from the registry with no problem using http.
decewei
on 31 Jul 2020
Some thoughts, I've looked through some code in containerd/containerd and containerd/cri.
It seems like the error message we get is coming from containerd. E.g. Here in containerd/containerd pull.go, Pull calls fetch which causes the failed to resolve reference error indicating above. There is no endpoint checking from containerd-template.toml.
Whereas in code of cri here checks the endpoint when it gets resolver.
I guess the logic should be cri check endpoint resolves -> containerd do its logic. Seems like cri fails to resolve the host and protocol before it passes to containerd.
decewei
on 31 Jul 2020
I have the same problem with the gitlab private repository that is
connected to microK8s cluster for autodevoos
I tried configuration using insecure and using secured self signed
certificate but both does not work.
I am think of trying k3s but first will try the mention solution of
installing certificate on my ubuntu nodes
ูู ุฌู ุนุฉุ 31 ูููููุ 2020 ูู 6:03 ู ุ ูุชุจ decewei notifications@github.com:
Some thoughts, I've looked through some code in containerd/containerd and
container/cri.It seems like the error message we get is coming from containerd. E.g. Here
in containerd/containerd pull.go
https://github.com/containerd/containerd/blob/d184a0a3430dc4a17a47cce37fb36126ac0c699a/pull.go,
Pull calls fetch which causes the failed to reference error indicating
above. There is no endpoint checking.Whereas in code of cri here
https://github.com/containerd/cri/blob/release/1.2/pkg/server/image_pull.go
checks the endpoint when it gets resolver.It seems like microk8s.ctr is using the former logic rather than the
latter logic. I'm not sure where the invoke of these functions happening.
Could this be the problem?โ
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ubuntu/microk8s/issues/1444#issuecomment-667167048,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABIMCCVMDDTSIZMNIS7AVDR6LMMRANCNFSM4PK6JLTA
.
muhammadnasr
on 31 Jul 2020
As suggested by @balchua
ูู ุฌู
ุนุฉุ 31 ูููููุ 2020 ูู 6:25 ู
ุ ูุชุจ Muhammad Hassan Nasr <
[email protected]>:
I have the same problem with the gitlab private repository that is
connected to microK8s cluster for autodevoos
I tried configuration using insecure and using secured self signed
certificate but both does not work.I am think of trying k3s but first will try the mention solution of
installing certificate on my ubuntu nodesูู ุฌู ุนุฉุ 31 ูููููุ 2020 ูู 6:03 ู ุ ูุชุจ decewei notifications@github.com:
Some thoughts, I've looked through some code in containerd/containerd and
container/cri.It seems like the error message we get is coming from containerd. E.g. Here
in containerd/containerd pull.go
https://github.com/containerd/containerd/blob/d184a0a3430dc4a17a47cce37fb36126ac0c699a/pull.go,
Pull calls fetch which causes the failed to reference error indicating
above. There is no endpoint checking.Whereas in code of cri here
https://github.com/containerd/cri/blob/release/1.2/pkg/server/image_pull.go
checks the endpoint when it gets resolver.It seems like microk8s.ctr is using the former logic rather than the
latter logic. I'm not sure where the invoke of these functions happening.
Could this be the problem?โ
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ubuntu/microk8s/issues/1444#issuecomment-667167048,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABIMCCVMDDTSIZMNIS7AVDR6LMMRANCNFSM4PK6JLTA
.
muhammadnasr
on 31 Jul 2020
As suggested by @balchua ูู ุฌู ุนุฉุ 31 ูููููุ 2020 ูู 6:25 ู ุ ูุชุจ Muhammad Hassan Nasr < [email protected]>:
โฆ
I have the same problem with the gitlab private repository that is connected to microK8s cluster for autodevoos I tried configuration using insecure and using secured self signed certificate but both does not work. I am think of trying k3s but first will try the mention solution of installing certificate on my ubuntu nodes ูู ุฌู ุนุฉุ 31 ูููููุ 2020 ูู 6:03 ู ุ ูุชุจ decewei @.*>: > Some thoughts, I've looked through some code in containerd/containerd and > container/cri. > > It seems like the error message we get is coming from containerd. E.g. Here > in containerd/containerd pull.go > https://github.com/containerd/containerd/blob/d184a0a3430dc4a17a47cce37fb36126ac0c699a/pull.go, > Pull calls fetch which causes the failed to reference error indicating > above. There is no endpoint checking. > > Whereas in code of cri here > https://github.com/containerd/cri/blob/release/1.2/pkg/server/image_pull.go > checks the endpoint when it gets resolver. > > It seems like microk8s.ctr is using the former logic rather than the > latter logic. I'm not sure where the invoke of these functions happening. > Could this be the problem? > > โ > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#1444 (comment)>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/AABIMCCVMDDTSIZMNIS7AVDR6LMMRANCNFSM4PK6JLTA > . >
Thanks. We've tried adding certificates on RHEL7. But we are still getting x509 issue when it falls back to https. Which channel and revision of microk8s are you using? What about the ctr version?
decewei
on 31 Jul 2020
The steps to reproduce this issue in revision 1550 are: (I reproduced this on my home vm machine too):
@decewei I must be missing something. I am following the instructions on https://microk8s.io/docs/registry-built-in with no problems on an AWS machine. Here is a full run:
ubuntu@ip-172-31-40-63:~/microk8s$ snap install microk8s --channel=1.18/stable --classic
microk8s (1.18/stable) v1.18.6 from Canonicalโ installed
ubuntu@ip-172-31-40-63:~/microk8s$ microk8s enable registry:size=40Gi
Enabling the private registry
Enabling default storage class
deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
Applying registry manifest
namespace/container-registry created
persistentvolumeclaim/registry-claim created
deployment.apps/registry created
service/registry created
The registry is enabled
ubuntu@ip-172-31-40-63:~/microk8s$ cd ../tmp/
ubuntu@ip-172-31-40-63:~/tmp$ cat > Dockerfile
FROM nginx
ubuntu@ip-172-31-40-63:~/tmp$ sudo docker build . -t localhost:32000/mynginx:registry
Sending build context to Docker daemon 12.29kB
Step 1/1 : FROM nginx
latest: Pulling from library/nginx
6ec8c9369e08: Pull complete
d3cb09a117e5: Pull complete
7ef2f1459687: Pull complete
e4d1bf8c9482: Pull complete
795301d236d7: Pull complete
Digest: sha256:0e188877aa60537d1a1c6484b8c3929cfe09988145327ee47e8e91ddf6f76f5c
Status: Downloaded newer image for nginx:latest
---> 8cf1bfb43ff5
Successfully built 8cf1bfb43ff5
Successfully tagged localhost:32000/mynginx:registry
ubuntu@ip-172-31-40-63:~/tmp$ docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied
ubuntu@ip-172-31-40-63:~/tmp$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
busybox latest 018c9d7b792b 3 days ago 1.22MB
localhost:32000/my-busybox latest 018c9d7b792b 3 days ago 1.22MB
nginx latest 8cf1bfb43ff5 9 days ago 132MB
localhost:32000/mynginx registry 8cf1bfb43ff5 9 days ago 132MB
busybox <none> c7c37e472d31 4 weeks ago 1.22MB
localhost:32000/my-busybox <none> c7c37e472d31 4 weeks ago 1.22MB
ubuntu@ip-172-31-40-63:~/tmp$ sudo docker push localhost:32000/mynginx
The push refers to repository [localhost:32000/mynginx]
98b4c818e603: Pushed
1698c1b7e3e6: Pushed
227442bb48dc: Pushed
d899691659b0: Pushed
95ef25a32043: Pushed
registry: digest: sha256:deb724a427ea79face617392a5a471fdcb4cdb57f971ee6b7e492b90fecb199f size: 1362
ubuntu@ip-172-31-40-63:~/tmp$
cat > d.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: localhost:32000/mynginx:registry
ports:
- containerPort: 80
ubuntu@ip-172-31-40-63:~/tmp$ microk8s.kubectl apply -f ./d.yaml
deployment.apps/nginx-deployment created
ubuntu@ip-172-31-40-63:~/tmp$ microk8s.kubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
container-registry pod/registry-7cf58dcdcc-l8p52 1/1 Running 0 9m43s
default pod/nginx-deployment-7b77749f6-s52vl 1/1 Running 0 12s
kube-system pod/hostpath-provisioner-75fdc8fccd-rns59 1/1 Running 0 9m43s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
container-registry service/registry NodePort 10.152.183.11 <none> 5000:32000/TCP 9m47s
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 9m50s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
container-registry deployment.apps/registry 1/1 1 1 9m48s
default deployment.apps/nginx-deployment 1/1 1 1 12s
kube-system deployment.apps/hostpath-provisioner 1/1 1 1 9m48s
NAMESPACE NAME DESIRED CURRENT READY AGE
container-registry replicaset.apps/registry-7cf58dcdcc 1 1 1 9m44s
default replicaset.apps/nginx-deployment-7b77749f6 1 1 1 12s
kube-system replicaset.apps/hostpath-provisioner-75fdc8fccd 1 1 1 9m44s
ubuntu@ip-172-31-40-63:~/tmp$ microk8s.ctr image pull localhost:32000/mynginx:registry
localhost:32000/mynginx:registry: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:deb724a427ea79face617392a5a471fdcb4cdb57f971ee6b7e492b90fecb199f: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:795301d236d7c3cd7c21f28faa8e9dc7f6381c980a0241cae40614986ee070b5: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d3cb09a117e500ee7466b6d21351c35321c9443442d21404267bc9e338bf86b6: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ef2f145968791b3e117e32ead3685173095d01e8dd887225f857d7fea64cfc8: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e4d1bf8c948256cc69d7121bf623603039d70e616a90ac92eb690aed97918e58: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:8cf1bfb43ff5d9b05af9b6b63983440f137c6a08320fa7592197c1474ef30241: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6ec8c9369e08152361a01729f2c8a1e7aae898426c6e67267f41894bf9524827: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:deb724a427ea79face617392a5a471fdcb4cdb57f971ee6b7e492b90fecb199f...
done
What am I doing differently?
ktsakalozos
on 31 Jul 2020
Might be worth taking a look at https://github.com/ubuntu/microk8s/issues/196#issuecomment-667160883
ktsakalozos
on 31 Jul 2020
Might be worth taking a look at #196 (comment)
Let me perform this on a clean install. I'll report the result back. The thing is we have multiple machines having the same problem. There must be something missing here.
In the meantime, for people who steps on this issue, and need quick solution. For RHEL, uses the following command to install the CA you used to sign your registry certficate on your microk8s machine:
echo "Installing registry CA... "
mv mk8s-registry.ca /etc/pki/ca-trust/source/anchors
# Check if it is moved to the folder:
ls /etc/pki/ca-trust/source/anchors
# Update host machien certificate
sudo update-ca-trust extract
# Check if your certificate is in /etc/ssl/certs/ca-bundle.crt
cat /etc/ssl/certs/ca-bundle.crt | grep SOME_SUB_STRING_OF_YOUR_CA
@joedborg Thank you for your efforts.
Is this pushed to snap or not yet?
I am guessing it is included in those snap channel:
1.16/stable: v1.16.14 2020-08-23 (1606)
and
latest/candidate: v1.18.8 2020-08-19 (1626)
muhammadnasr
on 23 Aug 2020
Hi @muhammadnasr
The containerd upgrade to v1.3.7 should be on latest/candidate and will be on the v1.19 release (1.19/* channels) but will not be backported to 1.16 channels (so it is not on 1.16/stable).
ktsakalozos
on 24 Aug 2020
Related issues
khteh
ยท
5Comments
avarf
ยท
5Comments
carmine
ยท
3Comments
xrpn
ยท
4Comments
linxuyalun
ยท
3Comments