Argo: `templateRef` rules are confusing and not clear

[simster7]

Currently we only allow using templateRef to trigger WorkflowTemplates from steps or dags. We do not allow this directly from a template. This has caused confusion in #2046 and #2305, among others.

We should either:

• Explicitly forbid using templateRef in templates
• Allow it
• Redesign our approach to this, perhaps in conjunction with the discussions in #2007

[delwaterman]

@simster7 I totally agree. Here's my thinking:

As part of my mental model for Argo I had to drill into my head that a WorfklowTemplate (an item defined under the templates YAML section) is equivalent to a function definition. And with dealing with either steps or dag the tasks under the steps and tasks were function invocations. This was a tricky model to get into my head (and my teams) but the fact that templates had inputs which were defining the parameters of the function and steps/dags used arguments which defined what arguments to pass into the function invocation.

I would argue for not allowing templates to directly reference a templateRef, and in the short term submit single templates using steps or dag.

I would also advocate for then using #2007, since the only use case for directly using a single workflow template as a workflow seems equivalent to just invoking the workflow template directly.

[seaneagan]

I don't understand, isn't the purpose of WorkflowTemplate to be able to define workflow definitions that can be reused (called) from higher level workflow definitions (Workflow or WorkflowTemplate)? And isn't #2447 deprecating that ability? Is there a proposal to add that back in some other way? Our team's usage of Argo is highly dependent on this ability.

I also do not understand why templateRef is confusing, and than the overloading of the term "template" between WorkflowTemplate (Workflow template) and templates (task templates). I think documentation updates for WorkflowTemplate would go a long way.

[simster7]

I don't understand, isn't the purpose of WorkflowTemplate to be able to define workflow definitions that can be reused (called) from higher level workflow definitions (Workflow or WorkflowTemplate)? And isn't #2447 deprecating that ability? Is there a proposal to add that back in some other way? Our team's usage of Argo is highly dependent on this ability.

Hi @seaneagan. To be clear, the proposal in #2447 does not deprecate the ability to define workflow definitions that can be reused from higher level definitions. It just deprecates one specific location you can do so.

In short, the change will deprecate calling other templates on a template directly and will encourage users to call other templates in template.steps or template.dag.

For example. The following would be deprecated (notice how templateRef is directly below template).

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: example-workflow-template
spec:
  entrypoint: on-workflow-template
  templates:
  - name: on-workflow-template
    templateRef:    # Proposal would deprecate `templateRef` here
      name: workflow-template-whalesay-template
      template: whalesay-template
    inputs:
      parameters:
        - name: message
          value: "Hello, world"

However, the following would still be valid (notice how templateRef is below template.steps).

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: example-workflow-template
spec:
  entrypoint: on-workflow-template
  templates:
    - name: on-workflow-template
      steps:
        - - name: call-whalesay-template
            templateRef:    # Proposal does NOT deprecate `templateRef` here
              name: workflow-template-whalesay-template
              template: whalesay-template
            arguments:
              parameters:
              - name: message
                value: "Hello, world"

The purpose of template.steps or template.dag is to call other templates (either in the same Workflow or externally on other WorkflowTemplates), so calling WorkflowTemplate definitions there will certainly still be allowed. template.steps.templateRef and template.dag.tasks.templateRef will not be deprecated.

The change simply encourages users to call their reusable definitions under template.steps.templateRef and template.dag.tasks.templateRef, instead of under template.templateRef directly. (For the reasoning behind this see #2447).

I also do not understand why templateRef is confusing, and than the overloading of the term "template" between WorkflowTemplate (Workflow template) and templates (task templates). I think documentation updates for WorkflowTemplate would go a long way.

Agree with you on both counts. I think documentation on this would be in order.

[seaneagan]

Ah, in that case I totally agree. Thanks for the explanation!

[jpambrun]

It may be late to chime in, but those two are not equivalent. I have a dag with tasks based on templates and I use the conditional 'when' with {{tasks.prev-step.outputs.result}}. This works with local-ref as still documented here, but not with the proposed embedded steps.

[simster7]

@jpambrun Not sure I fully understood, could you paste a sample Workflow of the case that you think would be affected by this change?

To be clear, you can still use both templateRef and template under a dag template just the same as a steps template.

[jpambrun]

This is what I am doing. I am trying to avoid copy paste as much as possible. Doing it under the dag as you proposed I would have to copy paste the command parameters. I also have scripts where the whole code would have to be copy pasted multiple times in multiple dags.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: asdf
spec:
  entrypoint: training-workflow
  templates:
  - name: training-workflow
    dag:
      tasks:
      - name: needs-tfrecords
        template: needs-tfrecords
      - name: tfrecords
        template: tfrecords
        when: '{{tasks.needs-tfrecords.outputs.result}}'
  - # more dags using the same template library

  # this templats is calles in many dags
  - name: needs-tfrecords
    template: run-on-32cpu
    inputs:
      parameters:
      - name: command
        value: 'some long command that I dont want to copy paste'

  # this template is reuse many times like needs-tfrecords
  - name: run-on-32cpu
    inputs:
      parameters:
      - name: script
      artifacts:
      - name: code
        path: /workdir
        gcs:
          bucket: somebucket
          key: src.tar.gz
    tolerations:
    - key: dedicated/highcpu
      operator: Exists
    container:
      env: # lots of env vars..
      resources:
        requests:
          cpu: 31
        limits:
          cpu: 31
      image: someImage
      command: [sh, '-c', '{{inputs.parameters.command}}']

[simster7]

I see. Why can't you have this instead? Everything else on your workflow can stay the same and it should work correctly.

  - name: needs-tfrecords
    steps:
      - - name: call-run-on-32cpu
          template: run-on-32cpu
          arguments:
            parameters:
              - name: command
                value: 'some long command that I dont want to copy paste'

You said that this wouldn't work this way. Why?

[simster7]

Oh I see, the when statement is causing trouble here. Let me take a look

[simster7]

@jpambrun Shouldn't you be using dependencies instead of when?

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: asdf
spec:
  entrypoint: training-workflow
  templates:
  - name: training-workflow
    dag:
      tasks:
      - name: needs-tfrecords
        template: needs-tfrecords
      - name: tfrecords
        template: tfrecords
        dependencies: ["needs-tfrecords"]
  # more dags using the same template library

  # this templats is calles in many dags
  - name: needs-tfrecords
    steps:
      - - name: call-run-on-32cpu
          template: run-on-32cpu
          arguments:
            parameters:
              - name: command
                value: 'some long command that I dont want to copy paste'

  # this template is reuse many times like needs-tfrecords
  - name: run-on-32cpu
    inputs:
      parameters:
      - name: script
      artifacts:
      - name: code
        path: /workdir
        gcs:
          bucket: somebucket
          key: src.tar.gz
    tolerations:
    - key: dedicated/highcpu
      operator: Exists
    container:
      env: # lots of env vars..
      resources:
        requests:
          cpu: 31
        limits:
          cpu: 31
      image: someImage
      command: [sh, '-c', '{{inputs.parameters.command}}']

[jpambrun]

I have mistakenly removed the dependency statement when creating this minimum working example, both the when and dependencies are in my true files.

The needs-tfrecords scripts actually prints true or false to stdout. That is what the when statement is using.

[simster7]

I see @jpambrun. Then in that case you would need to surface the result through the step template into your dag template. I just created the following example and it worked with both echo true and echo false:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: asdf
spec:
  entrypoint: training-workflow
  templates:
  - name: training-workflow
    dag:
      tasks:
      - name: needs-tfrecords
        template: needs-tfrecords
      - name: tfrecords
        dependencies: ["needs-tfrecords"]
        template: tfrecords
        when: '{{tasks.needs-tfrecords.outputs.parameters.result}}'

  - name: needs-tfrecords
    steps:
      - - name: call-run-on-32cpu
          template: run-on-32cpu
          arguments:
            parameters:
            - name: command
              value: "echo true"
    outputs:
      parameters:
        - name: result
          valueFrom:
            parameter: "{{steps.call-run-on-32cpu.outputs.parameters.result}}"

  - name: run-on-32cpu
    inputs:
      parameters:
        - name: command
    container:
      image: "alpine:latest"
      command: [sh, '-c', '{{inputs.parameters.command}} > /tmp/res.txt']
    outputs:
      parameters:
        - name: result
          valueFrom:
            path: "/tmp/res.txt"

  - name: tfrecords
    container:
      image: "alpine:latest"
      command: [sh, '-c', 'echo hello']

I realize this is slightly more cumbersome, but due to design and consistency considerations this is the correct way to do this.

[jpambrun]

yikes, going through a file feels a bit hacky. The previous method of composing templates was far more versatile and this depreciation makes quite a bit harder to reuse code.

I really appreciate the workaround. I could not have come up with that on my own.

[simster7]

yikes, going through a file feels a bit hacky. The previous method of composing templates was far more versatile and this depreciation makes quite a bit harder to reuse code.

I understand the frustration 🙁. Unfortunately, that method of composing templates should have not been supported since it has some major design flaws and could produce larger bugs. (If you are interested you can read more about the thought process here: https://github.com/argoproj/argo/issues/2007#issuecomment-589388470)

However, there is an issue currently open for being able to do this correctly and without a file: https://github.com/argoproj/argo/issues/2530.

I really appreciate the workaround. I could not have come up with that on my own.

Glad to help!