Gitpod: Can't install selfhosted

[quasardroide]

Describe the bug

I'm not able to install gitpod-selfhosted. To do this I followed the selfhosted guide.
First I tried the selfhosted repository which is depreached according the readme.md. There I ran into a go error (loader.go line 126) after fixing the minio-deployment bug. I tried to fix this error by using another helm 3 version but that error remained.
I also tried installing from the helm directory in this repository and the chart directly directory.
In the helm diretctory i experienced the same error like with the selfhosted repository.
With the chart directory directly helm starts all pods but ws-proxy cant be pulled. It seem helm is loocking for it on the wrong location. It tries to pull it from here but I only found it here. I tried to fix this but helm always tried to pull if from the wrong location. Also the version is wrong. I couldn't fix that eighter.

Steps to reproduce

See description.

Expected behavior

Having a running gitpod deployment after installing.

Additional information

My cluster is a vanilla k3s cluster (based on containerd so docker-compose does not work). Tried all helm 3 versions that downgrade allowed me to install. My systems are Manjaro and Arch.

[hm2075]

hi

if using helm charts

use base values.yaml and modify https://github.com/gitpod-io/gitpod/blob/master/chart/values.yaml

values.yaml
set -- version: 0.5.0

imagePrefix: eu.gcr.io/gitpod-io/self-hosted/

helm repo add gitpod https://charts.gitpod.io
helm repo update
helm install gitpod gitpod/gitpod --timeout 60m --values values.yaml

[quasardroide]

hi,

thanks for your help hm2075.

I was able to pull all images wich was my issue in the beginning. I'm now stuck at the same point like daimor in issue #2141 .
My ws-proxy and my registry-facade pods are failing due to missing certificates ({"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"open /mnt/certificates/fullchain.pem: no such file or directory","message":"cannot serve registry","serviceContext":{"service":"registry-facade","version":""},"severity":"fatal","time":"2020-11-07T15:56:02Z"}).
I can confirm that my secret is correctly mounted in registry-facade. My secret (auto generated via cert-manager) only contains *.crt & *.key. ws-proxy is looking for fullchain.pem and privkey.pem.
I had no luck adding the missing files manually -> files do not contain pem data.

I also have a problem with poxy which throws an error for not being able to open port on node ("0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports"). But both ports seem to be opened at node. When changing ports the error remains.

[hm2075]

#My secret (auto generated via cert-manager) only contains *.crt

use openssl to convert crt to pem reference https://zzz.buzz/2020/03/11/openssl-commands-for-certificate-management/

in some cases your crt might already be in pem format, if the file's content begins with -----BEGIN

https://stackoverflow.com/questions/991758/how-to-get-pem-file-from-key-and-crt-files
second post

rest of the instructions

https://www.gitpod.io/docs/self-hosted/latest/install/https-certs//

[quasardroide]

hello,

thanks again, it was a bit more complicated than it appeard but i it seems like i was able to fix the problem with my certs.

I was not able to convert my certificate and i was not able to use certbot cause it didn't like my api-tokens. But with acme.sh i finally hat succes.
I converted the key with this and renamed the other so they the requrements of the docs.

I used kubectl to create a ceret with the certificates:
kubectl create secret generic gitpod-cert --from-file=path/to/https-certificates/

But gitpod is still not running (or I'm not able to access it)
according to kubectl everything is fine:

kubectl get deployment -n gitpod
NAME READY UP-TO-DATE AVAILABLE AGE
minio 1/1 1 1 8d
theia-server 1/1 1 1 10m
ws-scheduler 1/1 1 1 10m
ws-manager 1/1 1 1 10m
dashboard 1/1 1 1 10m
registry 1/1 1 1 10m
registry-facade 1/1 1 1 10m
ws-proxy 1/1 1 1 10m
proxy 1/1 1 1 10m
image-builder 1/1 1 1 10m
mysql 1/1 1 1 10m
messagebus 1/1 1 1 10m
ws-manager-bridge 1/1 1 1 10m
server 1/1 1 1 10m

But one error remains:

• svclb-proxy:
  > 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

I don't understand this error cause the ports requested by the proxy-pod are free and opened when I deploy gitpod. I don't see any other ports that are intefreing with the node itself.

Also I had this error:

• node-daemon
  > node/k3os-9939 not patched
  > Theia (version 0.5.0) became available BUT we've failed to mark the node (attempt 10/10)
  > will not retry - failing
  But i was able to fix it by removing the lables from my node

[hm2075]

yes, remove label each time you upgrade/install because of the label bug

I have to patch my proxy after installing gitpod

kubectl patch svc proxy -p '{"spec": {"type": "LoadBalancer", "externalIPs":["xx.xx.xx.xx"]}}'

where xx.xx.xx.xx is ip address

you may need to do something similar to svclb-proxy:

[quasardroide]

hi,

it seems like i can't do that. It does not have a service. But it seems this pod tries to connect to port 80 and port 443 on my node derectly. I cant change that cause i can't find it's definition.

But I Found another problem:

My mysql Pod throws an error:

2020-11-16T20:14:04.715597Z 10 [Note] Aborted connection 10 to db: 'gitpod' user: 'gitpod' host: 'X.X.X.X' (Got an error reading communication packets)

Also ome of my pods reported being unhealthy with an error message:

• dashboard
• proxy
• registy
• ws-proxy

Readiness probe failed: Get http://X.X.X.X:Y/schemas/gitpod-schema.json: dial tcp X.X.X.X:80: connect: connection refused

I'm confused... they are not able to connect to themself?

Their logs seem to be OK.

[hm2075]

hi

1. Do you use the included mysql pod or external mysql? if external use the ip address
   db:
   host: xx.xx.xx.xx
   port: 3306
   password: xxx

and disable

mysql:
enabled: false

1. please output kubectl get services

[quasardroide]

hi,

i only use external minio since I was not able to use fix the deplyoment bug when installing directly. So my mysql is internal. I forgot to mention that the IP which mysql tries to connect to does not exist in my gitpod workspace.

$ kubectl get services -n gitpod
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                 AGE
minio             ClusterIP      10.43.159.57    <none>           9000/TCP                                10d
registry          ClusterIP      10.43.244.104   <none>           443/TCP                                 22h
dashboard         ClusterIP      10.43.98.137    <none>           3001/TCP                                22h
db                ClusterIP      10.43.37.141    <none>           3306/TCP                                22h
messagebus        ClusterIP      10.43.110.44    <none>           5672/TCP,25672/TCP,4369/TCP,15672/TCP   22h
registry-facade   ClusterIP      10.43.131.23    <none>           3000/TCP                                22h
image-builder     ClusterIP      10.43.81.253    <none>           8080/TCP                                22h
theia-server      ClusterIP      10.43.213.135   <none>           80/TCP                                  22h
ws-manager        ClusterIP      10.43.240.223   <none>           8080/TCP                                22h
server            ClusterIP      10.43.255.93    <none>           3000/TCP,9500/TCP                       22h
ws-proxy          ClusterIP      10.43.200.151   <none>           8080/TCP                                22h
mysql             ClusterIP      10.43.163.140   <none>           3306/TCP                                22h
blobserve         ClusterIP      10.43.214.149   <none>           4000/TCP                                22h
proxy             LoadBalancer   10.43.251.11    <Extern IP>   80:32080/TCP,443:32443/TCP              22h

I have news:

While I was typing this comment something changed. Before I was able to access my Gitlab and my nextcloud instance but now every domain whose IP points to my cluster is pointing to something which appears to be gitpod. The tab title says gitpod and the footer is Copyright © 2020 TypeFox | All Rights Reserved.
I can access settings and admin (when i try to open my gitlab settings or admin via history). When I'm trying to access gitpod directly (gitpod.my.domain) i'm directed to the workspaces. But when I try to log in I get

Sorry, this page can't be found 😳 404 .

--- some mins later ---

OK while i was doing futher investigations erveythings seems to be normal. Except gitlab is still directed to gitpod. Perhaps because is is using a subsubdomain?
On the other side gitpod stopped working.
My domain is directed to gitpod workspaces but it seems everything except header and footer cant be loaded.

--- some mins later again ---

I was not able to find any changes regarding pods restarting or something like that. Bun now i cant even acces gitpod anymore due to NET::ERR_CERT_AUTHORITY_INVALID.

[hm2075]

hi,

for certs issue, first thing I would do is not use certs in values.yaml

2nd - have you setup domain correctly in values.yaml?

also go to

http(s)://url/workspaces/

if you go to url directly without /workspaces or /login then i think it redirects to gitpod

[quasardroide]

hello,

today i tried both several times but the result was always the same: 404 page not found.
Regarding the mysql-error I have to add that the IP, which is listed as host in the error message, is not used in my gitpod-namespace.

2020-11-18T18:16:45.895877Z 3 [Note] Aborted connection 3 to db: 'gitpod' user: 'gitpod' host: 'x.x.x.x' (Got an error reading communication packets)

have you setup domain correctly in values.yaml?

Host ist gitpod.my.domain
External IP is my cluster's IP