rancher 🚀 - Integrate Rancher 2.0 with gitlab kubernetes function

Also very interested in this guide/feature! +1

stefanvangastel on 28 May 2018

The CA certificate needs to be in PEM format. So you need to take the certificate-authority-data from kubeconfig and base64 decode it, like

echo "LS0tLS1CRUdJTiBDRV ..." | base64 -d

wich results in

-----BEGIN CERTIFICATE-----
MIIC7jCCAdag....
-----END CERTIFICATE-----

tkuther on 28 May 2018

👍5

Tried with the CA certificate in PEM format and I get another error :

Kubernetes error: Namespace in version "v1" cannot be handled as a Namespace: no kind "Namespace" is registered for version "clusters/v1"

Any idea ?

drskullster on 30 May 2018

😕2

For some reasons github wasn't sending a notification that this issue has been replied.

Anyway, thanks for @tkuther though I am also getting a new Error now

FYI I am using an external Load Balancer (nginx) using Certificate Signed By A Recognized Certificate Authority (3-node-externalssl-recognizedca.yml)

Is there an additional step I need to do ?

dmastag on 30 May 2018

I checked the Documentation on Installing RKE again and noticed there is something i haven't done.

https://rancher.com/docs/rancher/v2.x/en/installation/ha-server-install-external-lb/#part-12-remove-default-certificates

But everytime I try to delete and save the cacert, it comes back for some reason.
Is this a separate bug that I need to submit as an issue?

dmastag on 30 May 2018

I get this error:
bildschirmfoto 2018-05-31 um 14 31 13

Kubernetes error: Namespace in version "v1" cannot be handled as a Namespace: no kind "Namespace" is registered for version "clusters/v1"

I installed rancher with Let's encrypt from the beginning, so I don't have the problem with the valid certificates. Did anybody of you get it running yet?

JannikZed on 31 May 2018

See https://github.com/rancher/rancher/issues/13058 & https://github.com/abonas/kubeclient/issues/318

superseb on 31 May 2018

👍2

thank you @superseb. So we have to wait I assume.

JannikZed on 31 May 2018

Hi, I'm stuck in the same problem and being new to Kubernetes, I'm not much sure how to make it work. Could someone please guide me how to add an reverse-proxy server in the existing Kubernetes cluster?

KartikeyaRokde on 13 Jun 2018

👍4

@KartikeyaRokde I tried using multiple reverse proxy setups on my nginx external load balancer but none are working
My last attempt was https://github.com/rancher/rancher/issues/13058#issuecomment-399004765

dmastag on 21 Jun 2018

👍1

So I finally got it working by combining a couple of Articles

I used the internal IP address at :6443, but since its https I had to extract the PEM certificate.

So I based on this Answer on the gitlab forum got the PEM
https://forum.gitlab.com/t/kubernetes-integration-something-went-wrong-while-installing-helm-tiller/15627/6

I had a problem with the Token and also got the following problem
"Kubernetes error: namespaces "gitlab-managed-apps" is forbidden: User "system:serviceaccount:gitlab-managed-apps:gitlab-sa" cannot get namespaces in the namespace "gitlab-managed-apps"

Luckily I was able to find the following article.

http://centosquestions.com/kubernetes-error-namespaces-gitlab-managed-apps-forbidden-user-systemserviceaccountgitlab-managed-appsgitlab-sa-cannot-get-namespaces-namespace-gitlab-managed-apps/

dmastag on 26 Jun 2018

@dmastag would you mind writing up all the steps you took to get it working in some sort of tutorial?

mKeRix on 26 Jun 2018

👍3

Also, doesn't connecting to the master api server directly bypass the rancher rbac header handling, thus the access rights? With the service account it moves to cluster security, but that's a big selling point of rancher?

krishofmans on 26 Jun 2018

Actually really nice, that you got it running with a lot of hacking, but I wouldn’t close this issue. This really can’t be the right way of connecting these two services. Bypassing rancher is in my eyes a bad idea and there might be many downsides of it in a later step ..

From: Kris notifications@github.com
Sent: Tuesday, June 26, 2018 4:53:52 PM
To: rancher/rancher
Cc: JZinkl; Comment
Subject: Re: [rancher/rancher] Integrate Rancher 2.0 with gitlab kubernetes function (#13736)

Also, doesn't connecting to the master api server directly bypass the rancher rbac header handling, thus the access rights? With the service account it moves to cluster security, but that's a big selling point of rancher?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/rancher/rancher/issues/13736#issuecomment-400339032, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHLXHsO_OUoYSDPgv7a-w3mgiBJmBKesks5uAksAgaJpZM4UOxhn.

JannikZed on 26 Jun 2018

👍8

I am having the same issues with my Rancher setup, I am not able to register the cluster... I only find outdated documentation. :(

tamasgal on 27 Jun 2018

I got it to the point where I receive the Kubernetes error: Namespace in version "v1" cannot be handled as a Namespace: no kind "Namespace" is registered for version "clusters/v1" and now I probably have to give up, unless I bypass Rancher with a handful of hacks...

tamasgal on 27 Jun 2018

😕2

@JannikZed True
So I am Reopening this

dmastag on 27 Jun 2018

👍5

Managed to get it working following the same hacks as @dmastag
Had to shot RBAC down...
If anyone try, there's currently a bug with v11 of Gitlab, with SSL and the current version of Alpine (3.6).
If you have an error like can't fetch helm charts from one of google website, here's the trick:
On your gitlab host

perl -i -pe s/alpine:3.6/alpine:3.7/ /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/pod.rb

perl -i -pe "s/ca-certificates openssl/ca-certificates openssl wget/" /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/base_command.rb

Then gitlab-ctl restart may have you go good until they merge this into 11.1

More infos here : https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20471#note_86687238

r-mach on 10 Jul 2018

I get the following error when trying to install Helm Tiller (gitlab.com):

Something went wrong while installing Helm Tiller

ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/main: DNS lookup error WARNING: Ignoring APKINDEX.84815163.tar.gz: No such file or directory ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/community: DNS lookup error WARNING: Ignoring APKINDEX.24d64ab1.tar.gz: No such file or directory ERROR: http://mirror.clarkson.edu/alpine/v3.6/main: DNS lookup error WARNING: Ignoring APKINDEX.490b721e.tar.gz: No such file or directory ERROR: http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.6/main: DNS lookup error WARNING: Ignoring APKINDEX.44c98ace.tar.gz: No such file or directory ERROR: unsatisfiable constraints:

EDIT: Resolved this. Was a bad domain in the search list of my /etc/resolv.conf file on my cluster host. Caused dns lookups to fail from the alpine container Helm uses when installing.

colinwilson on 21 Jul 2018

Any updates on this?

WoLfulus on 18 Aug 2018

👍14

If you have DNS issue or some thing like rancher ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/main: temporary error

In Rancher go to Projects/Namespaces and just move namespace gitlab-managed-apps to Default Project.

ru-it-specialist on 25 Aug 2018

Take a look at this guide: https://rancher.com/blog/2018/2018-08-07-cicd-pipeline-k8s-autodevops-rancher-and-gitlab/

PhilippHeuer on 10 Sep 2018

👍7

Thanks for the report.
Please check https://github.com/rancher/rancher/issues/13736#issuecomment-419906942 and https://github.com/rancher/rancher/issues/13736#issuecomment-393515935

You can find the guild via https://rancher.com/blog/2018/2018-08-07-cicd-pipeline-k8s-autodevops-rancher-and-gitlab/

Please let us know in a comment if you can still reproduce the issue, and we'll reopen it

loganhz on 21 Oct 2018

👍1

Hey, I'd like to thanks Chris Urwin for writing this guide on rancher.com blog. This solution is quite ok. I was able to deploy my application to environments like review, staging and production. But it has few "places for improvement".

Author suggest to put deployments inside gitlab-managed-apps namespace. I think it should be reserved only for real GItlab Managed Apps like Tiller, Prometheus and Gitlab Runner.

Also this solution doesn't work with those real Gitlab Managed Apps. When I added configuration from http://x.co/rm082018 I wasn't able to install them from Gitlab Cluster settings page. I contacted gitlab support, and they send me this link - https://docs.gitlab.com/ee/user/project/clusters/#role-based-access-control-rbac . I found out that when GKE is created there are created three things - ServiceAccount, ClusterRoleBinding and Secret. I created my own configuration with three of those as it's in table from Gitlab Docs.

apiVersion: v1 
kind: ServiceAccount 
metadata: 
  name: gitlab 
  namespace: default 
--- 
kind: ClusterRoleBinding 
apiVersion: rbac.authorization.k8s.io/v1beta1 
metadata: 
  name: gitlab-admin 
  namespace: default 
subjects: 
  - kind: ServiceAccount 
    name: gitlab 
    namespace: default 
roleRef: 
  kind: ClusterRole 
  name: cluster-admin 
  apiGroup: rbac.authorization.k8s.io 
--- 
apiVersion: v1 
kind: Secret 
metadata: 
  name: gitlab-token 
  namespace: default 
  annotations: 
    kubernetes.io/service-account.name: gitlab 
type: kubernetes.io/service-account-token

Thanks to this configuration I was able to install Gitlab Managed Apps. Unfortunately I wasn't able to deploy application to environments. I'm not a Kubernetes genius so I applied configuration from Rancher blog guide, but changed namespace to my-project-name. Thanks to that (without changing already created token used for Gitlab Managed Apps ) I was able to deploy applications.

Conclusion - I think that someone with better knowledge of Kubernetes should merge those two configurations and it should work like a charm :sunglasses:

There is also some Ingress issue, but I think it's not related to Rancher. By default clusters can have L7 Nginx Ingress and Gitlab is trying to install L4 and it's not supported by Rancher.

You can also check my comment here - https://gitlab.com/gitlab-org/gitlab-ce/issues/45939#note_110193977 It's also connected with Gitlab <-> Rancher issues. There is even a link to configuration from Rancher Blog guide.

marcinmaruszewski on 22 Oct 2018

👍4

I am trying to get this running with our self hosted GitLab installation but I get a general error and no idea where to get better logs. I followed the instructions from https://rancher.com/blog/2018/2018-08-07-cicd-pipeline-k8s-autodevops-rancher-and-gitlab/ but I am not sure about the API key. I got one from the Kubernets cluster config and tried a new API key which I generated over the Rancher 2.0 web interface but I always get this:

screenshot 2018-11-02 at 20 25 28

Any ideas?

tamasgal on 2 Nov 2018

👍5

@tamasgal , instructions you mentioned are working only for application deployment using Auto DevOps.

Gitlab Managed Apps like Helm Tiller, Ingress, Gitlab Runner work differently. When you are trying to install Helm Tiller Gitlab is trying to create two resources - tiller ServiceAccount and tiller-admin ClusterRoleBinding. ServiceAccount created from configuration that is mentioned on blog post doesn't have enough permission to do it so there is an error.

Since my last comment I was trying to check different approach. and I created this configuration:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab
  namespace: default
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: gitlab-admin
  namespace: default
subjects:
  - kind: Group
    name: system:serviceaccounts
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-token
  namespace: default
  annotations:
    kubernetes.io/service-account.name: gitlab
type: kubernetes.io/service-account-token

When I used it I was able to install Gitlab Managed Apps and deploy application. You can also check this issue on Gitlab - https://gitlab.com/gitlab-org/gitlab-ce/issues/45939

I also think that API keys from Rancher UI won't work in this case. They are intended to use when you are trying to manage your cluster through Rancher API, not created by Rancher Kubernetes Cluster.

marcinmaruszewski on 2 Nov 2018

Thanks @marcinmaruszewski, I also tried that config but no success.

Well, AutoDevOps... to be honest, I thought it’s the same. The final goal is using GitLab and Rancher to do continuous integration.
Currently we use Jenkins but the community edition of GitLab (self hosted) is very limited wrt to Jenkins.

tamasgal on 2 Nov 2018

@tamasgal, that's a pity. I tried this configuration many times and it worked for me. Please try to reproduce my steps:

Create Kubernetes Cluster inside Rancher.
Use bash script from blog post to get kubeconfig (get_kubeconfig_custom_cluster_rancher2.sh).
Don't apply configuration from http://x.co/rm082018 but that I mentioned above. You can find online version here https://gitlab.com/snippets/1767299/raw
Get PEM certificate and gitlab-token.
Create Cluster configuration inside Gitlab.
5.1 Provide url to Cluster, not Rancher API.
5.2 Be sure that you are using Gitlab 11.4.x because you need to enable RBAC.
5.3 Leave blank namespace field. Gitlab will create namespace only for your app like my-github-project-123 (where my-github-project is name, and 123 is project ID).
Try to install Helm Tiller, Runner, Prometheus (you will need to create persistent volume) and skip Ingress (Gitlab L4 Ingress doesn't work on Clusters created with Rancher).

Good luck!

marcinmaruszewski on 2 Nov 2018

👍1

Many thanks @marcinmaruszewski I am now trying.

In step 3, do I need to get rid of the old configurations? I am a bit confused... does this only apply for a single cluster? (Sry for the dumb question, it’s a bit voodoo for me, still)

tamasgal on 2 Nov 2018

No problem @tamasgal . I hope that someone smarter than us will gather it together and make it work with one click ;)

My configuration create different resources, tokens, service account etc. so I think It's gonna be ok. But I'd rather create a new cluster and do it from scratch.

marcinmaruszewski on 2 Nov 2018

Ok i did, many thanks so far ;)

Well, it was the first time the “Install” Button turned into “installing...” (before it just turned grey for like 10 sec and then error), however, it still shows “Kubernetes error” 😔

tamasgal on 2 Nov 2018

Maybe I am doing something wrong during the configuration application. Just to be sure: I am running that command (with your config @marcinmaruszewski ) on the rancher/rancher docker instance... that’s nothing cluster specific 🤨 I think there is my mistake? But how to apply that to the specific cluster?

tamasgal on 2 Nov 2018

@tamasgal , this configuration should be applied to specific cluster. In API URL there is link to bash script.

Download the content of get_kubeconfig_custom_cluster_rancher2.sh from https://gist.github.com/superseb/f6cd637a7ad556124132ca39961789a4

You have to run it on machine where rancher/rancher is running. When you run it there will be created kubeconfig file. In order to apply my configuration you need to run kubectl --kubeconfig kubeconfig -f https://gitlab.com/snippets/1767299/raw.

There is also shorter way. If you don't have kubectl, then you can go to your Rancher, select your cluster, click Lunch kubectl and run this command kubectl apply -f https://gitlab.com/snippets/1767299/raw.

marcinmaruszewski on 2 Nov 2018

OK, still fighting...

For the sake of completeness:

I created a new cluster called testcluster2
Executed the bash script to create the kubeconfig file.
On the Rancher UI, I entered the kubectl of testcluster2 and launched kubectl apply -f https://gitlab.com/snippets/1767299/raw (this showed ...secret gitlab-token created)
Copied the token value from kubectl describe secrets/gitlab-token
Create the PEM string (converted with base64)
Got the URL (not the API URL, but the one with the 6443 port)

Pasted everything into GitLab and then Kubernetes error. again. :(

I added a persistent storage, but not sure if it's working. Is this needed for this?

tamasgal on 3 Nov 2018

👍1

@tamasgal , in order to get token created from my configuration you have to run
kubectl describe secret gitlab-token

marcinmaruszewski on 3 Nov 2018

👍1

Yep, I did kubectl describe secrets/gitlab-token, which also showed a token (sorry, I edited afterwards). I am now trying again from scratch... Btw. it's a bit weird that when I create a new cluster, the old one does not work anymore (This cluster is currently Unavailable; areas that interact directly with it will not be available until the API is ready.)

Anyways, still no success yet, so I guess I am yet not replicating your workflow somehow, but I don't know where :-/

tamasgal on 3 Nov 2018

I am wondering if this is a firewall issue... However, I can access telnet 131.x.x.x 6443 from outside :-?

tamasgal on 3 Nov 2018

I am really puzzled. I went through the process several times, checked the firewall configuration but I could not find any suspicious setting. It’s really annoying that the error message is so generic...

One last question @marcinmaruszewski is there maybe something special about the cluster creation? I just create a new cluster in Rancher (default member nodes and cluster options), enable all three node roles and execute the docker command on the server. After that I just wait until the cluster is active (it displays one node). Is there maybe something I forgot to set up?

tamasgal on 3 Nov 2018

@tamasgal I did nothing special when I created my cluster. I followed instructions from Rancher blog except applying their configuration as I mentioned earlier.

Maybe try to add Kubernetes integration in Gitlab and then try to deploy simple application to Cluster using Gitlab Pipelines (you need to have some Gitlab Runner to do it). If deploy job will fail there will be more information about issues.

marcinmaruszewski on 3 Nov 2018

Thanks @marcinmaruszewski I will try to set that up. Have to search for a good tutorial though ;)

tamasgal on 5 Nov 2018