Skaffold: Error from server (Invalid): error when applying patch: Job.batch "migration-elasticsearch" is invalid:

[giladsh1]

Information

Skaffold latest version fails to apply a job.
The previous version (v0.40.0) works with no issue.
When deploying to GKE (v1.13.11-gke.5), I get the following error -

- Error from server (Invalid): error when applying patch:
 - {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"},\"name\":\"migration-elasticsearch\",\"namespace\":\"com-test\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"job\":\"migration-elasticsearch\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"cloud.google.com/gke-nodepool\",\"operator\":\"In\",\"values\":[\"default-pool\"]}]}]}}},\"containers\":[{\"envFrom\":[{\"configMapRef\":{\"name\":\"common-config\"}},{\"configMapRef\":{\"name\":\"elasticsearch-config\"}}],\"image\":\"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"migration-elasticsearch\",\"resources\":{\"limits\":{\"cpu\":\"150m\",\"memory\":\"100Mi\"},\"requests\":{\"cpu\":\"75m\",\"memory\":\"50Mi\"}}}],\"initContainers\":[{\"command\":[\"sh\",\"-c\",\"while ! nc -zv -w 1 elasticsearch 9200; do echo \\\"waiting for elasticsearch to be available\\\"; sleep 2; done;\"],\"image\":\"busybox:1.30.0\",\"name\":\"check-postgres\"}],\"restartPolicy\":\"Never\"}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"migration-elasticsearch"}],"containers":[{"image":"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1","name":"migration-elasticsearch"}]}}}}
 - to:
 - Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
 - Name: "migration-elasticsearch", Namespace: "com-test"
 - Object: &{map["apiVersion":"batch/v1" "kind":"Job" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"},\"name\":\"migration-elasticsearch\",\"namespace\":\"com-test\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"job\":\"migration-elasticsearch\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"cloud.google.com/gke-nodepool\",\"operator\":\"In\",\"values\":[\"default-pool\"]}]}]}}},\"containers\":[{\"envFrom\":[{\"configMapRef\":{\"name\":\"common-config\"}},{\"configMapRef\":{\"name\":\"elasticsearch-config\"}}],\"image\":\"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-433-gd7d9c5ab@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"migration-elasticsearch\",\"resources\":{\"limits\":{\"cpu\":\"150m\",\"memory\":\"100Mi\"},\"requests\":{\"cpu\":\"75m\",\"memory\":\"50Mi\"}}}],\"initContainers\":[{\"command\":[\"sh\",\"-c\",\"while ! nc -zv -w 1 elasticsearch 9200; do echo \\\"waiting for elasticsearch to be available\\\"; sleep 2; done;\"],\"image\":\"busybox:1.30.0\",\"name\":\"check-postgres\"}],\"restartPolicy\":\"Never\"}}}}\n"] "creationTimestamp":"2019-11-04T15:26:34Z" "labels":map["app.kubernetes.io/managed-by":"skaffold-v0.41.0" "skaffold.dev/builder":"google-cloud-build" "skaffold.dev/cleanup":"true" "skaffold.dev/deployer":"kubectl" "skaffold.dev/profiles":"com-test" "skaffold.dev/run-id":"com" "skaffold.dev/tag-policy":"git-commit" "skaffold.dev/tail":"true"] "name":"migration-elasticsearch" "namespace":"com-test" "resourceVersion":"32010771" "selfLink":"/apis/batch/v1/namespaces/com-test/jobs/migration-elasticsearch" "uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f"] "spec":map["backoffLimit":'\x06' "completions":'\x01' "parallelism":'\x01' "selector":map["matchLabels":map["controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f"]] "template":map["metadata":map["creationTimestamp":<nil> "labels":map["app.kubernetes.io/managed-by":"skaffold-v0.41.0" "controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f" "job":"migration-elasticsearch" "job-name":"migration-elasticsearch" "skaffold.dev/builder":"google-cloud-build" "skaffold.dev/cleanup":"true" "skaffold.dev/deployer":"kubectl" "skaffold.dev/profiles":"com-test" "skaffold.dev/run-id":"com" "skaffold.dev/tag-policy":"git-commit" "skaffold.dev/tail":"true"]] "spec":map["affinity":map["nodeAffinity":map["requiredDuringSchedulingIgnoredDuringExecution":map["nodeSelectorTerms":[map["matchExpressions":[map["key":"cloud.google.com/gke-nodepool" "operator":"In" "values":["default-pool"]]]]]]]] "containers":[map["envFrom":[map["configMapRef":map["name":"common-config"]] map["configMapRef":map["name":"elasticsearch-config"]]] "image":"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-433-gd7d9c5ab@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1" "imagePullPolicy":"IfNotPresent" "name":"migration-elasticsearch" "resources":map["limits":map["cpu":"150m" "memory":"100Mi"] "requests":map["cpu":"75m" "memory":"50Mi"]] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File"]] "dnsPolicy":"ClusterFirst" "initContainers":[map["command":["sh" "-c" "while ! nc -zv -w 1 elasticsearch 9200; do echo \"waiting for elasticsearch to be available\"; sleep 2; done;"] "image":"busybox:1.30.0" "imagePullPolicy":"IfNotPresent" "name":"check-postgres" "resources":map[] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File"]] "restartPolicy":"Never" "schedulerName":"default-scheduler" "securityContext":map[] "terminationGracePeriodSeconds":'\x1e']]] "status":map["completionTime":"2019-11-04T15:26:39Z" "conditions":[map["lastProbeTime":"2019-11-04T15:26:39Z" "lastTransitionTime":"2019-11-04T15:26:39Z" "status":"True" "type":"Complete"]] "startTime":"2019-11-04T15:26:34Z" "succeeded":'\x01']]}
 - for: "STDIN": Job.batch "migration-elasticsearch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"skaffold.dev/tail":"true", "skaffold.dev/run-id":"com", "app.kubernetes.io/managed-by":"skaffold-v0.41.0", "job":"migration-elasticsearch", "skaffold.dev/profiles":"com-test", "skaffold.dev/builder":"google-cloud-build", "skaffold.dev/deployer":"kubectl", "skaffold.dev/tag-policy":"git-commit", "skaffold.dev/cleanup":"true", "controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f", "job-name":"migration-elasticsearch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container{core.Container{Name:"check-postgres", Image:"busybox:1.30.0", Command:[]string{"sh", "-c", "while ! nc -zv -w 1 elasticsearch 9200; do echo \"waiting for elasticsearch to be available\"; sleep 2; done;"}, Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]core.Container{core.Container{Name:"migration-elasticsearch", Image:"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc01d569ee0), SecretRef:(*core.SecretEnvSource)(nil)}, core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc01d569f00), SecretRef:(*core.SecretEnvSource)(nil)}}, Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{"memory":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Mi", Format:"BinarySI"}, "cpu":resource.Quantity{i:resource.int64Amount{value:150, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"150m", Format:"DecimalSI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:75, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"75m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:52428800, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"50Mi", Format:"BinarySI"}}}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc00ef2a8e8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc001ba89a0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(0xc01d569fe0), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil)}}: field is immutable

• Skaffold version: v0.41.0
• Operating system: MacOS
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta17
kind: Config
build:
  local:
    push: false
  artifacts:

    # db migrations
    - image: eu.gcr.io/com/migration-postgres
      context: db/postgres
    - image: eu.gcr.io/com/migration-elasticsearch
      context: db/elasticsearch

profiles:
  - name: local-dev
    build:
      local:
        push: false
    activation:
      - kubeContext: minikube
    deploy:
      kubectl:
        manifests:
          - k8s/local/**
  - name: com-test
    build:
      googleCloudBuild:
        projectId: com
        timeout: "900s"
        machineType: N1_HIGHCPU_8
    deploy:
      kubectl:
        manifests:
          - k8s/gke/com-test/**

Steps to reproduce the behavior

skaffold run -l skaffold.dev/run-id=com -p com-test -v debug

[chinmaya-n]

I just experienced a similar error when deploying a job. This only happens if the resource already exists in the K8 cluster and the current run is trying to override the existing ones. If you run this after removing the existing objects using skaffold delete, it deploys fine.

[chinmaya-n]

Here is a repo with command output in README and source files to demonstrate the bug. This works in v0.40.0 but not in v0.41.0.

[doman412]

Not sure if related but I've encountered an issue with -p not being respected and instead using the local kubectx. Using the same names for both profiles and contexts.

[chinmaya-n]

Hi @tejal29 , This is not a feature request. This is actually a bug. I tested the same code with v0.40.0 it works fine but v0.41.0 breaks it. I didn't try it with v1.0.0 yet.

[giladsh1]

I can confirm this error still exists in version v1.0.0

[giladsh1]

@balopat any chance you can promote the fix to this bug?
we're essentially stuck on version 0.40 since we can't deploy jobs without deleting them first.

[nkubala]

hey all, sorry for the delay on this one. I was able to reproduce with the repo that @chinmaya-n provided (thank you!), and it looks like this was broken by https://github.com/GoogleContainerTools/skaffold/pull/3086. we'll need to add back force applying to make this work again for jobs.

[lienharc]

Can confirm this issue persists on v1.1.0

[tejal29]

@dgageot Can i revert your PR #3086 which caused a regression?

[giladsh1]

@tejal29 the problem is that I think that reverting this PR will cause #2687 and #2752 to reappear, which means that deployments will forced to be configured each run, and they won't be idempotent.

[dgageot]

Let me try to summarise my understanding because I'm not sure I've it all right:

• Before #3086, we had --force flag on by default on skaffold dev|run|debug. For skaffold deploy, it was supposed to be off by default but was in fact always on.
• After #3086, we have --force flag on by default only on skaffold dev and off for skaffold deploy|run|debug.

@giladsh1 @lienharc @doman412 @chinmaya-n Can you confirm that running skaffold with --force fixes your issue?

[giladsh1]

@dgageot I can confirm that adding the --force flag fixes the issue.

[dgageot]

Good! Now we need to define sensible defaults.
The idea of using --force only on skaffold dev was that it's a bit too destructive for the other commands that could run against pre-prod or prod. Wdyt?

[giladsh1]

IMHO adding force by default doesn't make sense for commands other than dev.

The question is why does it seldom break the deployment for cronjobs only?
Am I missing something related to that specific object in k8s?

[paymog]

I'm seeing this happen with deployments too on v1.3.1

[paymog]

@dgageot I can confirm the the force flag fixes this for me with run on v1.3.1

[tejal29]

Looks like from last comment, this issue is fixed on v1.3.1.
Please re-open if not

[paymog]

I don't think this is fixed. If I don't use force, I get an error:

`selector` does not match template `labels`

because the newly issued skaffold run creates a new run-id. Maybe I'm conflating issues here.

[paultiplady]

I believe the underlying issue on this ticket is that skaffold is setting the client version in the k8s labels;

                        app.kubernetes.io/managed-by=skaffold-v1.9.1

And updating labels on an existing k8s Deployment is not allowed (https://github.com/kubernetes/kubernetes/issues/26202).

If you successfully deployed your Deployment with skaffold deploy on v0.40.0, then upgrading your client will mean that the skaffold deploy (or any kubectl apply) will fail, as it's trying to modify the labels.

You can work around the issue using --force to delete the Deployment and recreate, but running --force for your full deploy seems drastic, and deleting Deployment history is undesirable (for example, you can't reverse the rollout using deployment history if you deleted the Deployment).

You could also presumably use #2752 to override the kustomize client version too, something like

kustomize -l app.kubernetes.io/managed-by=skaffold ...

However this seems like a suboptimal workaround.

[tstromberg]

There is a lot going on in this issue - so if folks are still running into problems similar to this, please open a new issue containing a reproduction case and logs from the latest version of skaffold.

Thank you for sharing your experience!