Jx: Support for Kubernetes 1.16

Created on 2 Oct 2019 · 19Comments · Source: jenkins-x/jx

Summary

jx install is not supporting Kubernetes 1.16
extensions/v1beta1 is obsolete/deperecated:
https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/docker-registry/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/jenkins/templates/part0-jenkins-master-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/nexus/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"'

Steps to reproduce the behavior

install kube v1.16.0
install jx version: 2.0.809
run: jx install --provider=kubernetes --on-premise

Expected behavior

Jenkins X installs successfully.

Actual behavior

Install fails:
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/docker-registry/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/jenkins/templates/part0-jenkins-master-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "/tmp/helm-template-workdir-812017120/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/nexus/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"'

Jx version

2.0.809

The output of jx version is:
2.0.809

COPY OUTPUT HERE
WARNING: Failed to find helm installs: running helm list --all --namespace jx: failed to run 'helm list --all --namespace jx' command in directory '', output: 'Error: could not find tiller'
WARNING: Failed to get helm version: failed to run 'helm version --short' command in directory '', output: 'Client: v2.14.3+g0e7f3b6
Error: could not find tiller'
NAME               VERSION
jx                 2.0.809
Kubernetes cluster v1.16.0
kubectl            v1.16.0
git                2.17.1
Operating System   Ubuntu 18.04.2 LTS

Jenkins type

[ ] Serverless Jenkins X Pipelines (Tekton + Prow)
[ X] Classic Jenkins

Kubernetes cluster

Operating system / Environment

Ubuntu 18.04

arehelm arekubernetes kinbug prioritcritical

Source

schizofreindly

👍16

Most helpful comment

I created a dirty solution, which worked for me. That is a python script which should be started before jx boot --verbose and it looks for these deployment files and trying to fix them programmatically if found.

import inotify.adapters
import yaml
import os.path
import time

def _main():

    i = inotify.adapters.InotifyTree('/tmp')

    for event in i.event_gen(yield_nones=False):

        (_, type_names, path, filename) = event

        if 'IN_ISDIR' not in type_names:
            continue

        if 'IN_CREATE' not in type_names:
            continue

        fpath = '/'.join([path, filename])

        if 'helm-template-workdir' not in fpath:
            continue

        print("MATCH / PATH=[{}] EVENT_TYPES={}".format(fpath, type_names))

        files = [
          'jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml',
          'jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml'
        ]

        for f in files:
            filepath = '/'.join([fpath,f])
            slept = 0
            while not os.path.exists(filepath):
                time.sleep(0.005)
                slept += 0.005
                if slept > 1:
                    print("Cannot find file %s after 1 sec of waiting" % filepath)
                    break
            if os.path.isfile(filepath):
                fix_file(filepath)
                print("FIXED FILE: PATH=[{}] EVENT_TYPES={}".format(fpath, type_names))


def fix_file(fpath):

    print(fpath)

    with open(fpath, 'r') as file:

        doc = yaml.load(file, Loader=yaml.FullLoader)

        doc['apiVersion'] = 'apps/v1'

        labels = doc['spec']['template']['metadata']['labels']

        doc['spec']['selector'] = {}
        doc['spec']['selector']['matchLabels'] = labels.copy()

        with open(fpath, 'w') as file:
            yaml.dump(doc, file)


if __name__ == '__main__':
    _main()

inotify and pyyaml pip modules required.

In a working case it looks like:

MATCH / PATH=[/tmp/helm-template-workdir-952010232] EVENT_TYPES=['IN_CREATE', 'IN_ISDIR']
/tmp/helm-template-workdir-952010232/jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml
FIXED FILE: PATH=[/tmp/helm-template-workdir-952010232] EVENT_TYPES=['IN_CREATE', 'IN_ISDIR']
/tmp/helm-template-workdir-952010232/jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml
FIXED FILE: PATH=[/tmp/helm-template-workdir-952010232] EVENT_TYPES=['IN_CREATE', 'IN_ISDIR']

If this does not work for you, try to play with timings then...
// updated sleeping time to 0.005 sec and waiting time to 1 sec.

update: see my comment https://github.com/jenkins-x/jx/issues/6957#issuecomment-635528161 with links to the gists

stepanselyuk on 24 May 2020

👍4

All 19 comments

I bumped into the same problem.

For anyone else having this issue. As a workaround you can temporary enable Apps and Extensions v1beta1 by editing the kube-apiserver.yaml on your master node.

vi /etc/kubernetes/manifests/kube-apiserver.yaml
Add - --runtime-config=apps/v1beta1=true,extensions/v1beta1/deployments=true command at:
spec: -> containers: -> - command:

After changing the kube-apiserver.yaml verify that the kube-apiserver pod restarted. In my case K8S restarted it automatically after I changed the file.

Warning
The helm templates need to be fixed though during the fact that this workaround will be deprecated in the upcoming v1.18 of kubernetes https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#v1160

WinnThePooh on 3 Oct 2019

👍3

The workaround doesn't work for kube version 1.16.2 worked well for 1.16.1

My settings:

--runtime-config=extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/replicasets=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicies=true

schizofreindly on 22 Oct 2019

Hello, can we tell to jx to use minikube kubectl instead of oldest kubectl installed locally and used for other version of kube cluster?

malaoui79 on 11 Nov 2019

I bumped into the same problem.

For anyone else having this issue. As a workaround you can temporary enable Apps and Extensions v1beta1 by editing the kube-apiserver.yaml on your master node.

vi /etc/kubernetes/manifests/kube-apiserver.yaml
Add - --runtime-config=apps/v1beta1=true,extensions/v1beta1/deployments=true command at:
spec: -> containers: -> - command:

After changing the kube-apiserver.yaml verify that the kube-apiserver pod restarted. In my case K8S restarted it automatically after I changed the file.

Warning
The helm templates need to be fixed though during the fact that this workaround will be deprecated in the upcoming v1.18 of kubernetes https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#v1160

this workaroud does work.
envrionment:
minikube
k8s 1.16.2
helm 2.16

zhangxpp on 22 Nov 2019

Has this been fixed by this commit?
https://github.com/jenkins-x/jx/commit/ffa64e1f372220218a19dda7e02d0de04767979b

faroguy on 6 Jan 2020

Has this been fixed by this commit?
ffa64e1

As far as my Kubernetes knowledge goes I would say yes. But I havent tried this fix.

WinnThePooh on 6 Jan 2020

attempted jx boot right after upgrading from jx version 2.0.1103 to 2.0.1104

NAME VERSION
jx 2.0.1104
Kubernetes cluster v1.16.2
kubectl v1.14.8
helm client Client: v2.16.1+gbbdfe5e
git 2.17.2 (Apple Git-113)
Operating System Mac OS X 10.13.6 build 17G6030

STEP: install-jenkins-x command: /bin/sh -c jx step helm apply --boot --remote --name jenkins-x --provider-values-dir ../kubeProviders in dir: /Users/user/jenkins-x-boot-config/env

Modified file /Users/user/jenkins-x-boot-config/env/Chart.yaml to set the chart to version 1
Ignoring templates/.gitignore
Applying the kubernetes overrides at ../kubeProviders/kubernetes/values.tmpl.yaml
Verifying the helm requirements versions in dir: . using version stream URL: https://github.com/jenkins-x/jenkins-x-versions.git and git ref: v1.0.285
error: upgrading helm chart '.': failed to run 'kubectl apply --recursive -f /var/folders/gj/3xx_fr756011yxwdvzg847m00000gr/T/helm-template-workdir-340704090/jenkins-x/output/namespaces/jx -l jenkins.io/chart-release=jenkins-x --namespace jx --wait --validate=false' command in directory '/var/folders/gj/3xx_fr756011yxwdvzg847m00000gr/T/jx-helm-apply-645702902/env', output: 'persistentvolumeclaim/jenkins-x-chartmuseum unchanged

... ... ...
deployment.apps/tekton-pipelines-controller unchanged
unable to recognize "/var/folders/gj/3xx_fr756011yxwdvzg847m00000gr/T/helm-template-workdir-989583902/jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

Deleting the jenkins-x-boot-config folder and re-running jx boot from the start returns the same result

tpl-eugenerebong on 7 Jan 2020

Ah you are right, my bad. They have to change the yaml files in the chart. @tpl-eugenerebong Have you tried my work around?

WinnThePooh on 7 Jan 2020

Same issue. How can this be done in a managed cluster where we don't have access to api server?

keshavprasadms on 13 Feb 2020

Wanted to try out jenkins-x and also stumbled upon this.

toonsevrin on 22 Feb 2020

So while some parts of jx have been fixed to be compatible with K8s 1.16, there are still some charts used by jenkins-x which are not: jenkins, chartmuseum, docker-registry, heapster

Are these charts public somewhere?

- condition: jenkins.enabled
  name: jenkins
  repository: http://chartmuseum.jenkins-x.io
  version: 0.10.38
- condition: chartmuseum.enabled
  name: chartmuseum
  repository: http://chartmuseum.jenkins-x.io
  version: 1.1.5

I've been trying to update the versions of the charts for jenkins, chartmuseum, docker-registry, heapster, but I cannot find anywhere the charts referenced above in the jenkins-x-platform requirements.

cc @jstrachan @deanesmith

alexchiri on 7 Mar 2020

I am facing the same issues with version 1.18.2. My cluster is setup using kubeadm on ubuntu 16.04 LTS. I used the following command "jx install --provider=kubernetes --on-premise" I am trying for a default configuration and also configured my github account. I think the command failed at the last step while creating deployment. I have attached the logs here in this comment

`unable to recognize "/tmp/helm-template-workdir-887089309/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml":

unable to recognize "/tmp/helm-template-workdir-887089309/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/docker-registry/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

unable to recognize "/tmp/helm-template-workdir-887089309/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

unable to recognize "/tmp/helm-template-workdir-887089309/jenkins-x/output/namespaces/jx/jenkins-x-platform/charts/jenkins/templates/part0-jenkins-master-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"'`

vpoptani on 6 May 2020

I wanted to try it out and i'm also facing the same issue.

jx version:
NAME VERSION
jx 2.1.37
Kubernetes cluster v1.16.7+vmware.1
kubectl v1.13.2
helm client 2.12.2
git 2.24.2 (Apple Git-127)
Operating System Mac OS X 10.15.4 build 19E287

unable to recognize "/var/folders/xq/krygb4_d3cjb1k17bxv1vgfh0000gn/T/helm-template-workdir-343620994/jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml": no matches for kind "D
eployment" in version "extensions/v1beta1"
unable to recognize "/var/folders/xq/krygb4_d3cjb1k17bxv1vgfh0000gn/T/helm-template-workdir-343620994/jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml": no matches for kind "Depl
oyment" in version "extensions/v1beta1"'
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step helm apply --boot --remote --name jenkins-x --provider-values-dir ../kubeProviders' command in directory 'env', output: ''

MarcoDima on 15 May 2020

import inotify.adapters
import yaml
import os.path
import time

def _main():

    i = inotify.adapters.InotifyTree('/tmp')

    for event in i.event_gen(yield_nones=False):

        (_, type_names, path, filename) = event

        if 'IN_ISDIR' not in type_names:
            continue

        if 'IN_CREATE' not in type_names:
            continue

        fpath = '/'.join([path, filename])

        if 'helm-template-workdir' not in fpath:
            continue

        print("MATCH / PATH=[{}] EVENT_TYPES={}".format(fpath, type_names))

        files = [
          'jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml',
          'jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml'
        ]

        for f in files:
            filepath = '/'.join([fpath,f])
            slept = 0
            while not os.path.exists(filepath):
                time.sleep(0.005)
                slept += 0.005
                if slept > 1:
                    print("Cannot find file %s after 1 sec of waiting" % filepath)
                    break
            if os.path.isfile(filepath):
                fix_file(filepath)
                print("FIXED FILE: PATH=[{}] EVENT_TYPES={}".format(fpath, type_names))


def fix_file(fpath):

    print(fpath)

    with open(fpath, 'r') as file:

        doc = yaml.load(file, Loader=yaml.FullLoader)

        doc['apiVersion'] = 'apps/v1'

        labels = doc['spec']['template']['metadata']['labels']

        doc['spec']['selector'] = {}
        doc['spec']['selector']['matchLabels'] = labels.copy()

        with open(fpath, 'w') as file:
            yaml.dump(doc, file)


if __name__ == '__main__':
    _main()

inotify and pyyaml pip modules required.

In a working case it looks like:

MATCH / PATH=[/tmp/helm-template-workdir-952010232] EVENT_TYPES=['IN_CREATE', 'IN_ISDIR']
/tmp/helm-template-workdir-952010232/jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/chartmuseum/templates/part0-deployment.yaml
FIXED FILE: PATH=[/tmp/helm-template-workdir-952010232] EVENT_TYPES=['IN_CREATE', 'IN_ISDIR']
/tmp/helm-template-workdir-952010232/jenkins-x/output/namespaces/jx/env/charts/jenkins-x-platform/charts/heapster/templates/part0-deployment.yaml
FIXED FILE: PATH=[/tmp/helm-template-workdir-952010232] EVENT_TYPES=['IN_CREATE', 'IN_ISDIR']

If this does not work for you, try to play with timings then...
// updated sleeping time to 0.005 sec and waiting time to 1 sec.

update: see my comment https://github.com/jenkins-x/jx/issues/6957#issuecomment-635528161 with links to the gists

stepanselyuk on 24 May 2020

👍4

jxui (not open source) also seems to use extensions/v1beta1 for Deployment. @deanesmith should I create a separate issue for that?

gazal-k on 5 Jun 2020

@gazal-k, no need. We can take care of it since it was already on our radar for the UI chart. FYI @mgoltzsche - can you handle this?

deanesmith on 5 Jun 2020

For JXUI this was just fixed by @abayer.

UPDATE: Actually not everything is fixed yet but we're taking care of it.

mgoltzsche on 5 Jun 2020

Thank you, I just checked jxui-0.0.1273 and it does seem to use apps/v1 for Deployment. Ingress still uses the old extensions/v1beta1, but that can be addressed later of course.

And since we are here, can someone please address #7076 in jxui chart? That would allow adding workload specific permissions for jxui to access cloud buckets for log storage. It should be a trivial change. I would've added it myself if jxui was open source

gazal-k on 6 Jun 2020

This should be done. If there are things that aren’t updated, please open new tickets.