Kubernetes-client: How to do a rolling update

[jsimao71]

Hi!

What is the fabric8 java client equivalent of:

kubectl -n=myns rollout restart deploy myapp

I've tried this

client.apps().deployments().inNamespace(myns).withName("myapp")
            .rolling().updateImage("registry/someimage:0.1");

But get error:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://2c82a54d-405e-44f3-b59c-108b3378232.xxxx.com/apis/apps/v1/namespaces/einnovator/deployments/notifications. Message: Deployment.apps "myapp" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"myapp", "deployment":"851bc111aa0d2f431512f1bfb0fc3f1c"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable.

Lost here..any help will my valuable...could this be a bug on the fabrix8 java-client implementation?!?

J.

[Etone]

Instead of using .rolling(), have you tried just editing the deployment? Kubernetes triggeres a rolling update when editing the image by itself (at least thats what we have ssen in the past)

kubernetesClient.apps() .deployments() .withName(deploymentName) .edit() .editSpec() .editContainer(0) .withNewImage(imageName) .endContainer() .endSpec() .endTemplate() .endSpec() .done();

This works just fine for us

[jsimao71]

Thanks for the reply Etone!
Sure, that works.. except..when the image name (including tag) is exactly the same K8 does not replace the pod, even if the image policy is pull always.
That is the reason I believe kubectl in a very recent version added the rollout restart. This was a highly request features as per kubectl issue tracking on github. And it works perfect with kubectl.
Thus I was happy to see that the fabric8 api does have similar api. Unfortunately gives error.
Thus two possibilities: either I'm missing something in the using the java api, or there is some "bug" in implementation.
Would be great if one of the developer of fabric8 or somebody that run into the same issue (you?!), could say what is going wrong here -- i.e. which of the two possibilities: wrong usage of api or some "bug", and how to get around/fix.
This is really important if you want make lots of micro-update to an app/image -- say multiple times a day, and you don't want to keep different tags for each snapshoot. The bottom line is: if kubectl allow it to do it, it should be true as well from fabric8 java api.
Thanks,
J.

[hanzhc]

I met the same question.How can I rolling update a deployment that has been created by using kubernetes client api ? Do anyone has an example for it? Appreciate!

[pradasouvanlasy]

Same question as the original poster there, deployments rolling updates from the fabric8 Java API will always fail since the API itself tries to change the label selectors that are immutable:
https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/RollingUpdater.java#L113
https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/DeploymentRollingUpdater.java#L88

The workaround suggested by @Etone can allow to emulate rollout restart deploy by adding a dummy annotation on the deployment itself:

client
        .apps()
        .deployments()
        .inNamespace(namespace)
        .withName(deploymentName)
        .edit()
        .editSpec()
        .editTemplate()
        .editMetadata()
        .addToAnnotations("dummy", "du_"+System.currentTimeMillis())
        .endMetadata()
        .endTemplate()
        .endSpec()
        .done()

Hope it can help

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

[wilx]

Stupid bots. Is this really stale? It seems like this is just unsolved problem.

[rohanKanojia]

@wilx : Thanks for bringing it to our notice. I think it's very important use case and we should prioritize it in next sprint.

[jsimao71]

Hi! Guys! The trick suggested kind of works..but obviously, the team behind should the library should look into it.

[ashwiniramesha]

I am facing the same exception when trying to patch an existing deployment with a patch YAML as follows:

Deployment deploymentAfterPatch = getK8sClient().apps().deployments().inNamespace(namespace).withName(deploymentName).rolling().patch(deploymentSnippet);

Where my complete deployment yaml and patch yaml are taken from official k8s document:
https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/

Deployment.yaml:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: patch-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: patch-demo-ctr
        image: nginx
      tolerations:
      - effect: NoSchedule
        key: dedicated
        value: test-team

patch.yaml:

spec:
  template:
    spec:
      containers:
      - name: patch-demo-ctr-2
        image: redis

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.44.4.126:6443/apis/apps/v1/namespaces/default/deployments/nginx. Message: Deployment.apps "nginx" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"nginx", "deployment":"1e9be354160b96aa2b7738239c432c6c"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable.

If this needs to be submitted as a separate issue, please let me know.

[jsimao71]

The solution we are using currently is based on update a "dummy" annotation
hash with new value -- and do a createOrReplace().
As of our understanding, labels can not be updated, but annotations can. .
For most part, it basically end up in the same end result -- ie. trigger a
rolling update that works even if the image tag is exactly the same.

Cheers,
J.

PS: BTW,
This is what is implemented in the current public version of einnovator
cloud manager -- an effective UI for K8s.
If curious, you can check it here: cloud.einnovator.org. or simply
einnovator.org

On Mon, 25 May 2020 at 22:38, Ashwini [e-mail] wrote:

I am facing the same exception when trying to patch an existing deployment
with a patch YAML as follows:

Deployment deploymentAfterPatch =
getK8sClient().apps().deployments().inNamespace(namespace).withName(deploymentName).rolling().patch(deploymentSnippet);

Where my complete deployment yaml and patch yaml are taken from official
k8s document:

https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/

Deployment.yaml:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: patch-demo
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: patch-demo-ctr
image: nginx
tolerations:
- effect: NoSchedule
key: dedicated
value: test-team

patch.yaml:

spec:
template:
spec:
containers:
- name: patch-demo-ctr-2
image: redis

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
PATCH at:
https://10.44.4.126:6443/apis/apps/v1/namespaces/default/deployments/nginx.
Message: Deployment.apps "nginx" is invalid: spec.selector: Invalid value:
v1.LabelSelector{MatchLabels:map[string]string{"app":"nginx",
"deployment":"1e9be354160b96aa2b7738239c432c6c"},
MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable.

If this needs to be submitted as a separate issue, please let me know.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/fabric8io/kubernetes-client/issues/1868#issuecomment-633723178,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJHSVEZJ65RTDAF7TZGDNTRTLQONANCNFSM4JNQZKYA
.

--
Jorge Simão | Software Engineer, IT Trainer, Consultant |
Ph.D, M.Sc., B.Sc., Computer Science and Engineering |
Trainer, Author, and Director (Education&Consultancy) at EInnovator |
Pivotal/VMWare/SpringSource Partner Trainer (VCI) |
M: ([phone] ; Web: www.einnovator.org

[ashwiniramesha]

The solution we are using currently is based on update a "dummy" annotation hash with new value -- and do a createOrReplace(). As of our understanding, labels can not be updated, but annotations can. . For most part, it basically end up in the same end result -- ie. trigger a rolling update that works even if the image tag is exactly the same.

This workaround does not seem to be working for updating a deployment with patch yaml.
Here are the steps I'm doing:
Updating a dummy annotation to existing deployment:

 Deployment existingDeployment = getK8sClient().apps().deployments().inNamespace(namespace).withName(deploymentName)
        .edit().editSpec().editTemplate().editMetadata()
        .addToAnnotations("dummy", "du_"+System.currentTimeMillis())
        .endMetadata().endTemplate().endSpec()
        .done();

Do createOrReplace of the patch yaml snippet with last modified deployment.
This is how I call createOrReplace() with patch YAML.

Deployment DeploymentAfterPatch = getK8sClient().apps().deployments().inNamespace(namespace).withName(deploymentName).createOrReplace(deploymentSnippet);

Am I doing something wrong?

I get the following exception:

java.lang.NullPointerException
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:84)
HasMetadataOperation.java:84
at io.fabric8.kubernetes.api.model.apps.DoneableDeployment.done(DoneableDeployment.java:27)
DoneableDeployment.java:27
at io.fabric8.kubernetes.api.model.apps.DoneableDeployment.done(DoneableDeployment.java:6)
DoneableDeployment.java:6
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:91)
HasMetadataOperation.java:91
at io.fabric8.kubernetes.client.dsl.internal.apps.v1.RollableScalableResourceOperation.replace(RollableScalableResourceOperation.java:170)
RollableScalableResourceOperation.java:170
at io.fabric8.kubernetes.client.dsl.internal.apps.v1.DeploymentOperationsImpl.replace(DeploymentOperationsImpl.java:113)
DeploymentOperationsImpl.java:113
at io.fabric8.kubernetes.client.dsl.internal.apps.v1.DeploymentOperationsImpl.replace(DeploymentOperationsImpl.java:55)
DeploymentOperationsImpl.java:55
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:431)

My deployment object is created out of the following patch.yaml:

kind: Deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          imagePullPolicy: Always

Full Deployment yaml:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
      tolerations:
      - effect: NoSchedule
        key: dedicated
        value: test-team

[jsimao71]

Oh! Yes...my understanding, and of other people on this thread I believe,
is that's really just an internal logical error of the frabric8 library. So
to continue using the library, one needs to "navigate" around it.
Not clear why it is not yet fixed. Strangely, I don't see any
developer/maintainer of the library posting on this thread.
Cheers,
J.

On Tue, 26 May 2020 at 00:28, Ashwini [e-mail] wrote:

@jsimao71 https://github.com/jsimao71 Thanks. I'm trying to understand
this better. My patch yaml does not have labels or selectors to update.
They are kept the same,I don't intend on updating them. I'm only updating
the container image.

Is it something internal within fabric8 APIs which is thinking the
labels/selectors need to be updated?

Running kubectl patch deployment patch-demo --patch "$(cat patch.yaml)"
works well with the above patch.yaml.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fabric8io/kubernetes-client/issues/1868#issuecomment-633741624,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJHSVH36DO5EJGR7LZEHL3RTL5JLANCNFSM4JNQZKYA
.

--
Jorge Simão | Software Engineer, IT Trainer, Consultant |
Ph.D, M.Sc., B.Sc., Computer Science and Engineering |
Trainer, Author, and Director (Education&Consultancy) at EInnovator |
Pivotal/VMWare/SpringSource Partner Trainer (VCI) |
M: ([phone] ; Web: www.einnovator.org

[rohanKanojia]

Umm, thanks for bringing this to our notice, we will prioritize this issue for upcoming sprint.

[ashwiniramesha]

I found that doing a createOrReplace() with a modified deployment object seems to be doing a rolling upgrade(confirmed by checking kubectl rollout history)..
Is this intended and equal to what the kubectl patch does?

[jsimao71]

For the purpose in discussion, should be about the same in the end result.
createOrReplace is a fabric8 api method that does what the name says:
creates if non existing, replace/updates if not.
kubectl patch should be about the same, but assume that the resource
already exists.

However, kubectl latest version also as a rolling update command that
retrigger the rolling mechanism again. This likely has direct support on
the K8s REST API. I'm not currently familiar with all the details of the
REST API, but I'm guessing this could be the case.
Fabric8 has a corresponding method for that: .rolling().updateImage(deploy)
but it does not work. Which is the reason I raised this issue several
month ago.
To get around this, people update some dummy field (hash annotation) to
force a rolling update and emulate the same with createOrReplace. But the
K8s REST API endpoints are likely different, has no patch/update to the
spec is required.

Rohan Kumar early message suggests that the fabric8 team will possibly look
into the issue and fix and/or clarify usage in upcoming versions.

Cheers,
J.

On Wed, 27 May 2020 at 16:13, Ashwini [e-mail] wrote:

I found that doing a createOrReplace() with a modified deployment object
seems to be doing a rolling upgrade(confirmed by checking kubectl rollout
history)..
Is this intended and equal to what the kubectl patch does?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fabric8io/kubernetes-client/issues/1868#issuecomment-634730004,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJHSVFQFSGDQHYSXFV2MELRTUU2ZANCNFSM4JNQZKYA
.

--
Jorge Simão | Software Engineer, IT Trainer, Consultant |
Ph.D, M.Sc., B.Sc., Computer Science and Engineering |
Trainer, Author, and Director (Education&Consultancy) at EInnovator |
Pivotal/VMWare/SpringSource Partner Trainer (VCI) |
M: ([phone] ; Web: www.einnovator.org