K8s-config-connector: ContainerCluster setting labels appears not to work

[Jonpez2]

I am creating a gke cluster using a ContainerCluster resource, and I am setting labels on it like so:

apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
annotations:
cnrm.cloud.google.com/remove-default-node-pool: "true"
name: cluster-xxx
labels:
builduser: xxx
spec:
location: us-central1
initialNodeCount: 1
loggingService: logging.googleapis.com/kubernetes
monitoringService: monitoring.googleapis.com/kubernetes

However I get the following error. Removing the 'labels' section gets rid of the error (and, obvs, the labels!).

Warning UpdateFailed 13m (x2 over 13m) containercluster-controller Update call failed: error applying desired state: googleapi: Error 400: Labels could not be set due to fingerprint mismatch; try again with the proper fingerprint: incorrect fingerprint supplied was "", the expected one is "a826bcc1"., failedPrecondition

[Scorpiion]

Hi, I think it would be helpful if you could provide a formatted version of your yaml so that it's easier to reproduce (it's a bit extra work to copy your text and try to fix the indentation, if you put three ` signs before and after the yaml the indentation is preserved).

I have not tested your yaml myself but have a feeling labels should "just work" as it's in the example here:

https://cloud.google.com/config-connector/docs/reference/resources#containercluster

Can you try the example cluster yaml above and see if it works for you?

[tonybenchsci]

I think this is a real issue. Exact same as https://github.com/terraform-providers/terraform-provider-google/issues/5608

[Scorpiion]

I just tried this sample and it works for me:

apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
  annotations:
    cnrm.cloud.google.com/remove-default-node-pool: "false"
  labels:
    label-one: "value-one"
  name: cluster-test
spec:
  location: us-central1-a
  initialNodeCount: 1
  loggingService: logging.googleapis.com/kubernetes
  monitoringService: monitoring.googleapis.com/kubernetes
  masterAuth:
    clientCertificateConfig:
      issueClientCertificate: false

After creation:

I'm running this version: gcr.io/cnrm-eap/controller:24eb6ff

[Scorpiion]

Update, I also get an error message in the controller logs:

{"level":"error","ts":1583957955.6496632,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"containercluster-controller","request":"cluster-test","error":"Update call failed: error applying desired state: googleapi: Error 400: Labels could not be set due to fingerprint mismatch; try again with the proper fingerprint: incorrect fingerprint supplied was \"\", the expected one is \"ed1aab07\"., failedPrecondition","stacktrace":"cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr/zapr.go:128\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

[Jonpez2]

here's a repo with my yamls https://github.com/Jonpez2/k8s-configconnector-problems

if you change to your GCP project, then you should be able to uncomment the 'labels' chunk in cluster.yaml, and that should repro the problem.

[Jonpez2]

@Scorpiion I think that error message corresponds to my WARNING in the output of kubectl describe. And I think that means that my cluster remains not ready, which means that my nodepool never gets hooked up to the cluster.

How do I find out what version of the configconnector stuff I'm using? I installed latest as instructed here - https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall, but that means I don't really know what I'm running...

[Jonpez2]

Oh sorry, I just read your linked issue, and you got there way before me. Thanks!

[Scorpiion]

Hi @Jonpez2 , great with the repo example, it makes it a lot easier.

Can you test this one:

apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
  annotations:
    cnrm.cloud.google.com/remove-default-node-pool: "false"
  name: cluster-xxx
spec:
  location: us-central1
  initialNodeCount: 1
  loggingService: logging.googleapis.com/kubernetes
  monitoringService: monitoring.googleapis.com/kubernetes
  releaseChannel:
    channel: "REGULAR"

I only change this:

    cnrm.cloud.google.com/remove-default-node-pool: "true"
=>
    cnrm.cloud.google.com/remove-default-node-pool: "false"

I'm thinking that you are removing the default node pole without specifying any node pool, maybe that is the issue?

I'm a bit too busy to try out on my own, but thought with a little luck this might be the issue.

[Scorpiion]

Regarding getting the version, you can get it like this (the docker tag is the version):

kubectl describe pods --namespace cnrm-system cnrm-controller-manager-0 | grep "Image:"

=>

    Image:         gcr.io/cnrm-eap/controller:24eb6ff

Version: 24eb6ff

[spew]

Thanks for reporting this issue and thanks for all the great work @Scorpiion on reproducing this issue.

Am I to understand that your comment where you said "I just tried this sample and it works for me:" reproduces the problem?

[Scorpiion]

Hi @spew, no problem, happy to help out where I can. I have reproduced the issue and can confirm that I get the same error.

At first I thought that it worked (as in "no bug", it looked good with kubectl and in the UI) but then I realized that the controller logs was giving errors all the time.

[spew]

The thing to look at with kubectl is the status.conditions and find the ReadyCondition. It will likely contain an error as well.

[Scorpiion]

I got this on one of my cluster now, I think it's connected (not the same cluster as previously, I create/recreate clusters quite a lot at the moment):

status:
  conditions:
  - lastTransitionTime: "2020-03-13T16:51:58Z"
    message: 'Update call failed: error applying desired state: googleapi: Error 400:
      Labels could not be set due to fingerprint mismatch; try again with the proper
      fingerprint: incorrect fingerprint supplied was "", the expected one is "a5234742".,
      failedPrecondition'
    reason: UpdateFailed
    status: "False"
    type: Ready

In the logs:

{"level":"error","ts":1584346313.5409987,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"containercluster-controller","request":"xxxxxxxxxxxxxxx/yyyyyyyyyyyyy","error":"Update call failed: error applying desired state: googleapi: Error 400: Labels could not be set due to fingerprint mismatch; try again with the proper fingerprint: incorrect fingerprint supplied was \"\", the expected one is \"a5234742\"., failedPrecondition","stacktrace":"cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr/zapr.go:128\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

[Scorpiion]

Hi, this issue is really blocking me at the moment. Every time I try to create a cluster with a separate node pool it just get stuck (@Jonpez2 has the same problem higher up in this issue). I am able to create a "simple" cluster with node pool, but mine with a bigger config keeps failing, it when using only the default node pool.

Do you know of any workaround for this @spew?

[Scorpiion]

Note that both clusters end up with the same error after a while, but the "simpler cluster" gets the node pool setup before it starts failing the label check I think. Now both gives this:

"Simple setup" (see full example below)

status:
  conditions:
  - lastTransitionTime: "2020-03-16T14:04:00Z"
    message: 'Update call failed: error applying desired state: googleapi: Error 400:
      Labels could not be set due to fingerprint mismatch; try again with the proper
      fingerprint: incorrect fingerprint supplied was "", the expected one is "4804bcbf".,
      failedPrecondition'
    reason: UpdateFailed
    status: "False"
    type: Ready

"Complex cluster (more settings)"

status:
  conditions:
  - lastTransitionTime: "2020-03-16T13:46:47Z"
    message: 'Update call failed: error applying desired state: googleapi: Error 400:
      Labels could not be set due to fingerprint mismatch; try again with the proper
      fingerprint: incorrect fingerprint supplied was "", the expected one is "802f4159".,
      failedPrecondition'
    reason: UpdateFailed
    status: "False"
    type: Ready

Full config for "simple cluster" (based on the examples in the docs):

apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
  annotations:
    cnrm.cloud.google.com/remove-default-node-pool: "false"
  name: containernodepool-dep
spec:
  location: us-east1-c
  initialNodeCount: 1
  loggingService: logging.googleapis.com/kubernetes
  monitoringService: monitoring.googleapis.com/kubernetes
  masterAuth:
    clientCertificateConfig:
      issueClientCertificate: false

---

apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerNodePool
metadata:
  name: containernodepool-sample
spec:
  location: us-east1-c
  initialNodeCount: 1
  autoscaling:
    minNodeCount: 1
    maxNodeCount: 3
  nodeConfig:
    diskSizeGb: 100
    diskType: pd-standard
    tags:
      - tagone
      - tagtwo
    preemptible: false
    minCpuPlatform: "Intel Haswell"
    oauthScopes:
      - "https://www.googleapis.com/auth/logging.write"
      - "https://www.googleapis.com/auth/monitoring"
    guestAccelerator:
      - type: "nvidia-tesla-k80"
        count: 1
    metadata:
      disable-legacy-endpoints: "true"
  management:
    autoRepair: true
    autoUpgrade: true
  clusterRef:
    name: containernodepool-dep

If you apply this yaml above and just wait you should get the same error as well.

I'm running this version:

Image:         gcr.io/cnrm-eap/controller:0079060

[spew]

Hi @Scorpiion, I do not know of a workaround for this issue yet, I will report back when I know more.

[Scorpiion]

Okay, thanks for the quick reply! Let me know if I can help debug this further.

[janwillies]

running into the same issue

[tonybenchsci]

Just bumping this a little more, since it blocks creation of new nodepools with DependencyNotReady because the cluster is constantly in Updating or FailedToUpdate state.

[kibbles-n-bytes]

Hey all; I am actively working on getting a fix out. I'll update you later today with more info.

[kibbles-n-bytes]

We have a fix merged, and are planning to do a patch release of Config Connector tomorrow with it included.

[Jonpez2]

That’s great thank you!

On Thu, 19 Mar 2020 at 22:07, Michael K. [e-mail] wrote:

We have a fix merged, and are planning to do a patch release of Config
Connector tomorrow with it included.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/110#issuecomment-601439536,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN425M7DNWJAREAI6LK5LTRIKJSDANCNFSM4LE5VVXA
.

[tonybenchsci]

We have a fix merged, and are planning to do a patch release of Config Connector tomorrow with it included.

Hoping for Monday Mar 23 :)

[Jonpez2]

Good luck!

On Sun, 22 Mar 2020 at 19:43, tonybenchsci [e-mail] wrote:

We have a fix merged, and are planning to do a patch release of Config
Connector tomorrow with it included.

Hoping for Monday Mar 23 :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/110#issuecomment-602261859,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN425KPN4DUJW4KA4MDP7TRIZS75ANCNFSM4LE5VVXA
.

[kibbles-n-bytes]

Hey all, thanks for your patience. Release 1.5.2 is now available. Please upgrade via the new steps in our documentation. Please let us know if you have any further issues with this.

[Jonpez2]

Looking good so far. Also excellent that you cleaned up the upgrade
process. Thanks all round!

On Tue, Mar 24, 2020 at 3:43 AM Michael K. [e-mail] wrote:

Closed #110
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/110.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/110#event-3157955455,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN425LDWQKGXZ6CYT3ZQXDRJAT4ZANCNFSM4LE5VVXA
.