Project: kubedb/redis does not work with k8s 1.18

[fjammes]

I have this error message when installing redis on k8s 1.18 with kubedb:

kubectl apply -f manifest/redis-cluster.yaml
The Redis "redis-cluster" is invalid: metadata.managedFields.fieldsType: Invalid value: "": must be `FieldsV1`

Here is my yaml:

apiVersion: kubedb.com/v1alpha1
kind: Redis
metadata:
  name: redis-cluster
  namespace: demo
spec:
  version: 4.0-v2
  mode: Cluster
  cluster:
    master: 3
    replicas: 1
  storageType: Durable
  storage:
    resources:
      requests:
        storage: 1Gi
    storageClassName: "standard"
    accessModes:
    - ReadWriteOnce
  terminationPolicy: Terminate 
  updateStrategy:
    type: RollingUpdate

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

$ kubedb version
Version = v0.13.0-rc.0
VersionStrategy = tag
GitTag = v0.13.0-rc.0
GitBranch = release-0.13
CommitHash = 9023600306c9c2fe67a2c1e6ac933bc43c24b3ff
CommitTimestamp = 2019-08-22T10:41:59
GoVersion = go1.12.9
Compiler = gcc
Platform = linux/amd64

$ kubectl version                 
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T20:56:08Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

[fjammes]

It seems the error is solved when disabling kubedb apiserver extension:

$ ./kubedb.sh --enable-mutating-webhook=false --enable-validating-webhook=false
$ kubectl apply -f ./manifest/redis-cluster.yaml
$ kubectl get pods -n demo
NAME                     READY   STATUS    RESTARTS   AGE
redis-cluster-shard0-0   1/1     Running   0          4m11s
redis-cluster-shard0-1   1/1     Running   0          4m1s
redis-cluster-shard1-0   1/1     Running   0          3m56s
redis-cluster-shard1-1   1/1     Running   0          3m51s
redis-cluster-shard2-0   1/1     Running   0          3m46s
redis-cluster-shard2-1   1/1     Running   0          3m41s

The error could be caused by k8s-1.18 new server side apply feature: https://kubernetes.io/docs/reference/using-api/api-concepts/#server-side-apply

[tamalsaha]

This seems to affect other projects, too.

• https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/issues/206
• https://github.com/Kong/kuma/issues/676

This makes me think, we have to update to k8s.io/client-go to v0.18.x. But this is an non-trivial change, since they updated every client-go method to add context. :hankey:

[tamalsaha]

I have looked into this further: https://gist.github.com/tamalsaha/328451a77aa95835f4004682c06a100d

The issue seems to be that our mutating webhook is causing problem as the ObjectMeta structure has changed. Based on twitter conversation, we need to update to k8s.io/client-go v1.16 or later.

https://twitter.com/tsaha/status/1250800701006950403

[jazibjamil]

Hi, I am facing a similar issue when deploying seldoncore apps to my kubernetes cluster.

kubectl apply -f - << END
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: iris-model
  namespace: testseldon
spec:
  name: iris
  predictors:
  - graph:
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/iris
      name: classifier
    name: default
    replicas: 1
END

The SeldonDeployment "iris-model" is invalid: metadata.managedFields.fieldsType: Invalid value: "": must be FieldsV1

[toonsevrin]

I also cannot use kubedb on my 1.18 cluster due to this.

This makes me think, we have to update to k8s.io/client-go to v0.18.x. But this is an non-trivial change, since they updated every client-go method to add context. hankey

Gofix will do you wonders for this!

[tamalsaha]

Further conversation on this topic here: https://github.com/kubedb/installer/pull/57#issuecomment-632186541

[tamalsaha]

It seems that updating to client-go v0.18.3 does not solve the issue. There is something more broken.

https://github.com/kubedb/redis/pull/161/checks?check_run_id=698166945

[toonsevrin]

@tamalsaha Now the apiservices don't start properly timed out waiting for the condition on apiservices/v1alpha1.mutators.kubedb.com, that's a different issue then the one everyone was reporting wrt managedFields.

[tamalsaha]

1.18 support is stuck until we find a solution for https://github.com/kubernetes/kubernetes/issues/91395 .