Argo: PodSpecPatch cannot patch metadata

[sudermanjr]

Checklist:

• [x] I've included the version.
• [x] I've included reproduction steps.
• [x] I've included the workflow YAML.
• [x] I've included the logs.

What happened:
Using podSpecPatch, I attempted to add metadata to the argo pods of a workflow. The metadata was not patched, but other patches worked fine.

What you expected to happen:
I expected to be able to add labels and/or annotations to the pods.

How to reproduce it (as minimally and precisely as possible):

This is a slightly modified version of the podSpecPatch example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: pod-spec-patch-
spec:
  entrypoint: whalesay
  arguments:
    parameters:
      - name: mem-limit
        value: 50Mi
  podSpecPatch: |
    metadata:
      labels:
        foo: bar
    containers:
      - name: main
        resources:
          limits:
            memory: "{{workflow.parameters.mem-limit}}"
          requests:
            memory: 25Mi
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]

Anything else we need to know?:

Here is the resulting pod yaml.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 100.108.53.14/32
    workflows.argoproj.io/node-name: pod-spec-patch-qb4kx
    workflows.argoproj.io/template: '{"name":"whalesay","arguments":{},"inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["cowsay"],"args":["hello
      world"],"resources":{}}}'
  creationTimestamp: "2020-04-03T17:26:28Z"
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/workflow: pod-spec-patch-qb4kx
  name: pod-spec-patch-qb4kx
  namespace: kepler-feature-params
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Workflow
    name: pod-spec-patch-qb4kx
    uid: c2c75f1a-485e-480c-8668-7d20ed87444c
  resourceVersion: "8727829"
  selfLink: /api/v1/namespaces/kepler-feature-params/pods/pod-spec-patch-qb4kx
  uid: 2f9ac64c-1fd2-40f3-9fcb-5e3b504baa68
spec:
  containers:
  - command:
    - argoexec
    - wait
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: argoproj/argoexec:v2.7.0
    imagePullPolicy: IfNotPresent
    name: wait
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /argo/podmetadata
      name: podmetadata
    - mountPath: /var/run/docker.sock
      name: docker-sock
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: argo-token-8lwg5
      readOnly: true
  - args:
    - hello world
    command:
    - cowsay
    image: docker/whalesay:latest
    imagePullPolicy: Always
    name: main
    resources:
      limits:
        memory: 50Mi
      requests:
        memory: 25Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: argo-token-8lwg5
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-172-20-17-179.us-east-2.compute.internal
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: argo
  serviceAccountName: argo
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podmetadata
  - hostPath:
      path: /var/run/docker.sock
      type: Socket
    name: docker-sock
  - name: argo-token-8lwg5
    secret:
      defaultMode: 420
      secretName: argo-token-8lwg5
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-04-03T17:26:28Z"
    reason: PodCompleted
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-04-03T17:26:28Z"
    reason: PodCompleted
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-04-03T17:26:28Z"
    reason: PodCompleted
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-04-03T17:26:28Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://8969c998e2914518a1c91172c799ae7c4f526eabf11093fa71d6b7454bd9c51a
    image: docker/whalesay:latest
    imageID: docker-pullable://docker/whalesay@sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
    lastState: {}
    name: main
    ready: false
    restartCount: 0
    state:
      terminated:
        containerID: docker://8969c998e2914518a1c91172c799ae7c4f526eabf11093fa71d6b7454bd9c51a
        exitCode: 0
        finishedAt: "2020-04-03T17:26:30Z"
        reason: Completed
        startedAt: "2020-04-03T17:26:30Z"
  - containerID: docker://754e1c47f94ea7ed989769e3d7738e70d88f3edb5de74cfa38f023e4ca2d9ad5
    image: argoproj/argoexec:v2.7.0
    imageID: docker-pullable://argoproj/argoexec@sha256:d2cbb99e8271715d315341cd2fd3edc1c0a0ae5e2f515b4c9ac34461d2bdaea2
    lastState: {}
    name: wait
    ready: false
    restartCount: 0
    state:
      terminated:
        containerID: docker://754e1c47f94ea7ed989769e3d7738e70d88f3edb5de74cfa38f023e4ca2d9ad5
        exitCode: 0
        finishedAt: "2020-04-03T17:26:30Z"
        reason: Completed
        startedAt: "2020-04-03T17:26:29Z"
  hostIP: 172.20.17.179
  phase: Succeeded
  podIP: 100.108.53.14
  qosClass: Burstable
  startTime: "2020-04-03T17:26:28Z"

Environment:

• Argo version:

$ argo version
argo: v2.7.0
  BuildDate: 2020-03-31T23:35:43Z
  GitCommit: 4d1175eb68f6578ed5d599f877be9b4855d33ce9
  GitTreeState: clean
  GitTag: v2.7.0
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

Workflow controller is also at v2.7.0

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-12-11T12:42:56Z"
  compiler: gc
  gitCommit: 6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4
  gitTreeState: clean
  gitVersion: v1.15.7
  goVersion: go1.12.12
  major: "1"
  minor: "15"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-02-11T20:05:26Z"
  compiler: gc
  gitCommit: 1bea6c00a7055edef03f1d4bb58b773fa8917f11
  gitTreeState: clean
  gitVersion: v1.15.10
  goVersion: go1.12.12
  major: "1"
  minor: "15"
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

argo get <workflowname>
Name:                pod-spec-patch-qb4kx
Namespace:           kepler-feature-params
ServiceAccount:      argo
Status:              Succeeded
Created:             Fri Apr 03 11:26:28 -0600 (4 minutes ago)
Started:             Fri Apr 03 11:26:28 -0600 (4 minutes ago)
Finished:            Fri Apr 03 11:26:31 -0600 (4 minutes ago)
Duration:            3 seconds
Parameters:
  mem-limit:         50Mi

STEP                     TEMPLATE  PODNAME               DURATION  MESSAGE
 ✔ pod-spec-patch-qb4kx  whalesay  pod-spec-patch-qb4kx  2s

• workflow-controller logs:

kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)
time="2020-04-03T17:26:28Z" level=info msg="Processing workflow" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:28Z" level=info msg="Updated phase  -> Running" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:28Z" level=info msg="Pod node {pod-spec-patch-qb4kx pod-spec-patch-qb4kx pod-spec-patch-qb4kx Pod whalesay nil    Pending   2020-04-03 17:26:28.634455846 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil [] []} initialized Pending" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:28Z" level=info msg="Skipped pod pod-spec-patch-qb4kx (pod-spec-patch-qb4kx) creation: already exists" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:28Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"pod-spec-patch-qb4kx\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:28Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:28Z" level=info msg="Update retry attempt 1 successful" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:28Z" level=info msg="Workflow update successful" namespace=kepler-feature-params phase=Running resourceVersion=8727801 workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:29Z" level=info msg="Processing workflow" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:30Z" level=info msg="Processing workflow" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:30Z" level=info msg="Updating node &NodeStatus{ID:pod-spec-patch-qb4kx,Name:pod-spec-patch-qb4kx,DisplayName:pod-spec-patch-qb4kx,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Pending,BoundaryID:,Message:ContainerCreating,StartedAt:2020-04-03 17:26:28 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} status Pending -> Running"
time="2020-04-03T17:26:30Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"pod-spec-patch-qb4kx\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:30Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:30Z" level=info msg="Update retry attempt 1 successful" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:30Z" level=info msg="Workflow update successful" namespace=kepler-feature-params phase=Running resourceVersion=8727816 workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:31Z" level=info msg="Processing workflow" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:31Z" level=info msg="Processing workflow" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:31Z" level=info msg="Updating node &NodeStatus{ID:pod-spec-patch-qb4kx,Name:pod-spec-patch-qb4kx,DisplayName:pod-spec-patch-qb4kx,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Running,BoundaryID:,Message:,StartedAt:2020-04-03 17:26:28 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} status Running -> Succeeded"
time="2020-04-03T17:26:31Z" level=info msg="Updated phase Running -> Succeeded" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:31Z" level=info msg="Marking workflow completed" namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:31Z" level=info msg="Checking daemoned children of " namespace=kepler-feature-params workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:31Z" level=info msg="Workflow update successful" namespace=kepler-feature-params phase=Succeeded resourceVersion=8727822 workflow=pod-spec-patch-qb4kx
time="2020-04-03T17:26:32Z" level=info msg="Labeled pod kepler-feature-params/pod-spec-patch-qb4kx completed"
time="2020-04-03T17:27:03Z" level=info msg="Alloc=29960 TotalAlloc=7155286 Sys=140418 NumGC=1516 Goroutines=109"
W0403 17:30:37.858441       1 reflector.go:302] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: watch of *v1.ConfigMap ended with: too old resource version: 8726509 (8728488)
time="2020-04-03T17:32:03Z" level=info msg="Alloc=27279 TotalAlloc=7155535 Sys=140418 NumGC=1519 Goroutines=109"

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[simster7]

@sarabala1979 this seems to be in your area of expertise

[sarabala1979]

PodSpecPatch will only support to patch all elements under the spec section in Pod resource. you can't patch pod metadata. For your case, you can add your label on workflow label section which will be propagated to all pods in that workflow

[sudermanjr]

Thanks!

It makes sense now that you say it, but that wasn’t entirely clear.

Is this documented somewhere?

[sarabala1979]

@simster7 is working on documentation

[sarabala1979]

2549

[sudermanjr]

Sorry to keep beating this, but I cannot figure out how to do what I need. I asked in Slack and got no response.

Basically, I would like to templatize pod annotations on every pod in the workflow.

I have tried to add an annotation to every workflow template, but the templating doesn't seem to work. slack question.

The podSpecPatch doesn't work (as discussed previously).

Is there any way to do what I'm trying to do, or do I need to find another strategy?

[valiantljk]

waiting for the solution...