Kubernetes-client: It appears that createOrReplace() is broken for deployments

[JGulbronson]

The following code yields this error:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://104.154.224.241/apis/extensions/v1beta1/namespaces/default/deployments. Message: resourceVersion should not be set on objects to be created. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=resourceVersion should not be set on objects to be created, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).

    val currentDeployment = client.apps().deployments()
        .inNamespace(namespace)
        .withName(deploymentName)
        .get()

    val newDeployment = currentDeployment.apply {
      metadata {
        resourceVersion = null
      }
      spec {
        replicas = deploymentGoal.spec.replicas ?: replicas
      }
    }

    logger.warn { newDeployment.toString() }

    return client.apps().deployments().createOrReplace(newDeployment)

I'm simply trying to update the number of replicas for an existing deployment, using createOrReplace(). It seems that instead of calling update internally, it's calling create, so Kubernetes (rightfully) throws an exception. #948 looks related, but client.extensions().deployments() is deprecated.

Any workarounds in the meantime? Is there a way to force replace instead of create?

[JGulbronson]

@rohanKanojia are you able to confirm this is unexpected?

[rohanKanojia]

Hi, I just checked but was unable to reproduce this. Could you please provide a test case in Java which is causing this problem? Here is the implementation of createOrReplace: https://github.com/fabric8io/kubernetes-client/blob/cee8550053e671bb0b3489e37196924f4c949a3f/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/base/BaseOperation.java#L402-L422

[JGulbronson]

Hey @rohanKanojia sorry for the delay. Unfortunately no test case is able to trigger it, it only happens against a real K8 cluster. Again, the (Kotlin) code looks like this:

    val currentDeployment = client.extensions().deployments()
        .inNamespace(namespace)
        .withName(deploymentName)
        .get()

    val newDeployment = currentDeployment.apply {
      metadata {
        resourceVersion = null
      }
      spec {
        replicas = deploymentGoal.spec.replicas ?: replicas
      }
    }

    logger.warn { newDeployment.toString() }

    return client.extensions().deployments().createOrReplace(newDeployment)

That warn is printed correctly, and contains the state of the new deployment (image, replicas, pod template, etc). However, when I call createOrReplace this is the exception:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.100.0.1/apis/apps/v1/namespaces/default/deployments. Message: deployments.apps "deployer" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=apps, kind=deployments, name=deployer, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=deployments.apps "deployer" already exists, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:411)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:780)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:349)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:418)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:415)
    at com.faire.deployer.actions.UpdateDeploymentAction.updateDeployment(UpdateDeploymentAction.kt:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at kotlin.reflect.jvm.internal.FunctionCaller$Method.callMethod(FunctionCaller.kt:97)
    at kotlin.reflect.jvm.internal.FunctionCaller$InstanceMethod.call(FunctionCaller.kt:114)
    at kotlin.reflect.jvm.internal.KCallableImpl.call(KCallableImpl.kt:106)
    at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod(KCallableImpl.kt:148)
    at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:110)
    at misk.web.actions.WebActionsKt$asChain$1.intercept(WebActions.kt:39)
    at misk.web.RealChain.proceed(RealChain.kt:18)
    at misk.web.RequestBridgeInterceptor.intercept(BoundAction.kt:242)
    at misk.web.RealNetworkChain.proceed(RealNetworkChain.kt:31)
    at misk.web.interceptors.MarshallerInterceptor.intercept(MarshallerInterceptor.kt:21)
    at misk.web.RealNetworkChain.proceed(RealNetworkChain.kt:31)
    at misk.web.exceptions.ExceptionHandlingInterceptor.intercept(ExceptionHandlingInterceptor.kt:33)
    at misk.web.RealNetworkChain.proceed(RealNetworkChain.kt:31)
    at misk.web.interceptors.MetricsInterceptor$intercept$1.invoke(MetricsInterceptor.kt:21)
    at misk.web.interceptors.MetricsInterceptor$intercept$1.invoke(MetricsInterceptor.kt:15)
    at misk.time.TimedKt.timed(Timed.kt:11)
    at misk.time.TimedKt.timed(Timed.kt:7)
    at misk.web.interceptors.MetricsInterceptor.intercept(MetricsInterceptor.kt:21)
    at misk.web.RealNetworkChain.proceed(RealNetworkChain.kt:31)
    at misk.web.interceptors.RequestLoggingInterceptor.intercept(RequestLoggingInterceptor.kt:28)
    at misk.web.RealNetworkChain.proceed(RealNetworkChain.kt:31)
    at misk.web.interceptors.RequestLogContextInterceptor.intercept(RequestLogContextInterceptor.kt:32)
    at misk.web.RealNetworkChain.proceed(RealNetworkChain.kt:31)
    at 

My guess is fromServer().get() == null is returning true, so it's trying to create a new deployment instead of modifying the existing one. But again, it only happens on a real cluster.

[JGulbronson]

Hey @rohanKanojia I think I found the issue. It seems that the test Kubernetes server is more liberal than the real one. I had to update my code to this:

client.apps().deployments()
        .inNamespace(namespace)
        .withName(deploymentName)
        .createOrReplace(newDeployment)

To get it to work with a real server. It seems the test one doesn't require the deployment name or namespace, and lets you replace it anyway. This seems to be a bug in the test server, as it doesn't mimic the real deal, thoughts?

[rohanKanojia]

Hey, I tried reproducing your case but unfortunately could not. For me it was working on both MockWebServer as well as on minishift. I tried this simple code to test update replicas:

    deployment1 = new DeploymentBuilder()
      .withNewMetadata()
        .withName("deployment1")
        .addToLabels("test", "deployment")
      .endMetadata()
      .withNewSpec()
        .withReplicas(1)
        .withNewTemplate()
          .withNewMetadata()
          .addToLabels("app", "httpd")
          .endMetadata()
          .withNewSpec()
            .addNewContainer()
              .withName("busybox")
              .withImage("busybox")
              .withCommand("sleep","36000")
            .endContainer()
          .endSpec()
        .endTemplate()
        .withNewSelector()
          .addToMatchLabels("app","httpd")
        .endSelector()
      .endSpec()
      .build();

    client.apps().deployments().inNamespace(currentNamespace).create(deployment1);
    DeploymentSpec spec = deployment1.getSpec();
    spec.setReplicas(2);  
    client.apps().deployments().inNamespace(currentNamespace)
      .withName(deployment1.getMetadata().getName()).createOrReplace(deployment1);

[JGulbronson]

Got it, will post my test case when I get the chance

[chamhaw]

Hey @rohanKanojia I think I found the issue. It seems that the test Kubernetes server is more liberal than the real one. I had to update my code to this:

client.apps().deployments()
        .inNamespace(namespace)
        .withName(deploymentName)
        .createOrReplace(newDeployment)

To get it to work with a real server. It seems the test one doesn't require the deployment name or namespace, and lets you replace it anyway. This seems to be a bug in the test server, as it doesn't mimic the real deal, thoughts?

You save my day!! Thanks.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!