Argo: PVC's in WorkflowTemplates are not carried over to Workflows

[arbakker]

The following Workflow based of a WorkflowTemplate fails with the message volume 'workdir' not found in workflow spec :

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test-pvc
  generateName: test-pvc-wft-
spec:
  entrypoint: test-pvc
  volumeClaimTemplates:
    - metadata:
        name: workdir
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: standard
        resources:
          requests:
            storage: 1Gi
  templates:
    - name: test-pvc
      container:
        image: ubuntu
        volumeMounts:
          - name: workdir
            mountPath: /data
        command: [sh, -c]
        args: ["echo test > /data/test.txt"]

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-pvc-wft-wf-
spec:
  entrypoint: test-pvc
  templates:
    - name: test-pvc
      steps:
        - - name: call-test-pvc
            templateRef:
              name: test-pvc
              template: test-pvc

Using a PVC works for regular workflows, see the following example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-pvc-wf-
spec:
  entrypoint: test-pvc
  volumeClaimTemplates:
    - metadata:
        name: workdir                   
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: standard
        resources:
          requests:
            storage: 1Gi
  templates:
  - name: test-pvc
    container:
      image: ubuntu
      volumeMounts:
        - name: workdir
          mountPath: /data
      command: [sh, -c]
      args: ["echo test > /data/test.txt"]

See related issue. In the comments @foobarbecue remarked that PVC do not work for WorkflowTemplates, but I could not find an existing open issue.

Diagnostics

Version

v2.10.1

Full workflow

Output kubectl get wf -o yaml test-pvc-wft-wf-mbzmk:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2020-09-10T18:09:04Z"
  generateName: test-pvc-wft-wf-
  generation: 2
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Failed
  name: test-pvc-wft-wf-mbzmk
  namespace: delivery
  resourceVersion: "6932"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/delivery/workflows/test-pvc-wft-wf-mbzmk
  uid: 79b68f6f-429f-4395-aa30-5091888e96b9
spec:
  arguments: {}
  entrypoint: test-pvc
  serviceAccountName: workflow
  templates:
  - arguments: {}
    inputs: {}
    metadata: {}
    name: test-pvc
    outputs: {}
    steps:
    - - arguments: {}
        name: call-test-pvc
        templateRef:
          name: test-pvc
          template: test-pvc
status:
  conditions:
  - status: "True"
    type: Completed
  finishedAt: "2020-09-10T18:09:04Z"
  message: child 'test-pvc-wft-wf-mbzmk[0].call-test-pvc' errored
  nodes:
    test-pvc-wft-wf-mbzmk:
      children:
      - test-pvc-wft-wf-mbzmk-3564086167
      displayName: test-pvc-wft-wf-mbzmk
      finishedAt: "2020-09-10T18:09:04Z"
      id: test-pvc-wft-wf-mbzmk
      message: child 'test-pvc-wft-wf-mbzmk[0].call-test-pvc' errored
      name: test-pvc-wft-wf-mbzmk
      outboundNodes:
      - test-pvc-wft-wf-mbzmk-1965749888
      phase: Failed
      startedAt: "2020-09-10T18:09:04Z"
      templateName: test-pvc
      templateScope: local/test-pvc-wft-wf-mbzmk
      type: Steps
    test-pvc-wft-wf-mbzmk-1965749888:
      boundaryID: test-pvc-wft-wf-mbzmk
      displayName: call-test-pvc
      finishedAt: "2020-09-10T18:09:04Z"
      id: test-pvc-wft-wf-mbzmk-1965749888
      message: volume 'workdir' not found in workflow spec
      name: test-pvc-wft-wf-mbzmk[0].call-test-pvc
      phase: Error
      startedAt: "2020-09-10T18:09:04Z"
      templateRef:
        name: test-pvc
        template: test-pvc
      templateScope: local/test-pvc-wft-wf-mbzmk
      type: Pod
    test-pvc-wft-wf-mbzmk-3564086167:
      boundaryID: test-pvc-wft-wf-mbzmk
      children:
      - test-pvc-wft-wf-mbzmk-1965749888
      displayName: '[0]'
      finishedAt: "2020-09-10T18:09:04Z"
      id: test-pvc-wft-wf-mbzmk-3564086167
      message: child 'test-pvc-wft-wf-mbzmk[0].call-test-pvc' errored
      name: test-pvc-wft-wf-mbzmk[0]
      phase: Error
      startedAt: "2020-09-10T18:09:04Z"
      templateName: test-pvc
      templateScope: local/test-pvc-wft-wf-mbzmk
      type: StepGroup
  phase: Failed
  startedAt: "2020-09-10T18:09:04Z"
  storedTemplates:
    namespaced/test-pvc/test-pvc:
      arguments: {}
      container:
        args:
        - echo test > /data/test.txt
        command:
        - sh
        - -c
        image: ubuntu
        name: ""
        resources: {}
        volumeMounts:
        - mountPath: /data
          name: workdir
      inputs: {}
      metadata: {}
      name: test-pvc
      outputs: {}

Logs workflow controller

time="2020-09-10T18:09:04Z" level=info msg="Processing workflow" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Updated phase  -> Running" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Steps node test-pvc-wft-wf-mbzmk initialized Running" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="StepGroup node test-pvc-wft-wf-mbzmk-3564086167 initialized Running" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Pod node test-pvc-wft-wf-mbzmk-1965749888 initialized Pending" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=error msg="Mark error node" error="volume 'workdir' not found in workflow spec" namespace=delivery nodeName="test-pvc-wft-wf-mbzmk[0].call-test-pvc" workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk-1965749888 phase Pending -> Error" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk-1965749888 message: volume 'workdir' not found in workflow spec" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk-1965749888 finished: 2020-09-10 18:09:04.063364902 +0000 UTC" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Step group node test-pvc-wft-wf-mbzmk-3564086167 deemed errored due to child test-pvc-wft-wf-mbzmk[0].call-test-pvc error: volume 'workdir' not found in workflow spec" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk-3564086167 phase Running -> Error" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk-3564086167 message: child 'test-pvc-wft-wf-mbzmk[0].call-test-pvc' errored" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk-3564086167 finished: 2020-09-10 18:09:04.063520948 +0000 UTC" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="step group test-pvc-wft-wf-mbzmk-3564086167 was unsuccessful: child 'test-pvc-wft-wf-mbzmk[0].call-test-pvc' errored" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Outbound nodes of test-pvc-wft-wf-mbzmk-1965749888 is [test-pvc-wft-wf-mbzmk-1965749888]" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Outbound nodes of test-pvc-wft-wf-mbzmk is [test-pvc-wft-wf-mbzmk-1965749888]" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk phase Running -> Failed" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk message: child 'test-pvc-wft-wf-mbzmk[0].call-test-pvc' errored" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="node test-pvc-wft-wf-mbzmk finished: 2020-09-10 18:09:04.063737437 +0000 UTC" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Checking daemoned children of test-pvc-wft-wf-mbzmk" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Updated phase Running -> Failed" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Updated message  -> child 'test-pvc-wft-wf-mbzmk[0].call-test-pvc' errored" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Marking workflow completed" namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Checking daemoned children of " namespace=delivery workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:04Z" level=info msg="Workflow update successful" namespace=delivery phase=Failed resourceVersion=6932 workflow=test-pvc-wft-wf-mbzmk
time="2020-09-10T18:09:05Z" level=info msg="test-pvc-wft-wf-mbzmk released all acquired locks"
time="2020-09-10T18:09:05Z" level=info msg="Workflow update successful" namespace=delivery phase=Failed resourceVersion=6932 workflow=test-pvc-wft-wf-mbzmk

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[foobarbecue]

Note that the previous issue referred to above was about the use of volume, whereas this one is talking about volumeClaimTemplate. The problem was fixed for volume, but sounds like it still exists for volumeClaimTemplate.

[sarabala1979]

It is by design. Spec level entities in WorkflowTemplate cannot be carried over to workflow when workflow refers only template from workflowtemplate. you have to define PVC in workflow which will be pass to workflowtemplate in templateref scenario.

[arbakker]

Thanks for the quick response @foobarbecue and @sarabala1979 !

You mention _Spec level entities in WorkflowTemplate cannot be carried over to workflow when workflow refers only template from workflowtemplate_. Is there a scenario where spec level entities can be carried over to workflows from workflowtemplates?

To be clear, you mean to define the PVC in the workflow like this?

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: test-pvc-wft-wf-
spec:
volumeClaimTemplates:
- metadata:
name: workdir
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: standard
resources:
requests:
storage: 1Gi
entrypoint: test-pvc
templates:
- name: test-pvc
steps:
- - name: call-test-pvc
templateRef:
name: test-pvc
template: test-pvc

[sarabala1979]

yes

[foobarbecue]

This seems like a valid feature request to me.

People expect WorkflowTemplates to be able to do everything Workflows can do. Is there any reason, other than not wanting to write more code, that they can't?

(And of course I can't help coming back to what I always say which is that in my uninformed opinion WorkflowTemplates and Workflows should never have been two separate things -- we should have just allowed for uninstantiated Workflows.)