Argo: Failed to submit workflow when a nested template is chosen via input parameter.

[shadiramadan]

Checklist:

• [x] I've included the version.
• [x] I've included reproduction steps.
• [x] I've included the workflow YAML.
• [x] I've included the logs.

What happened:

argo submit test.yaml
2020/04/30 04:24:17 Failed to submit workflow: templates.test-param.tasks.test-param templates.nested-dag.tasks.test-nested-task template name 'placeholder-3' undefined

What you expected to happen:
I expected the workflow to submit and for the the template echo-hello to run.
It does not seem to be resolving the input parameter.

How to reproduce it (as minimally and precisely as possible):
test.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-param-
spec:
  entrypoint: test-param

  # arguments:
  #   parameters:
  #     - name: template-name

  templates:
    - name: echo-hello
      container:
        image: alpine:latest
        command: [echo]
        args: ["Called template echo-hello"]

    - name: test-param
      dag:
        tasks:
          - name: test-param
            template: nested-dag
            arguments:
              parameters:
                - name: template-name
                  value: echo-hello

    - name: nested-dag
      inputs:
        parameters:
        - name: template-name
      dag:
        tasks:
          - name: test-nested-task
            template: "{{inputs.parameters.template-name}}"

argo submit test.yaml



md5-a90043576a8fe767dce4886fd4857772



$ argo version
argo: v2.7.6+70facdb.dirty
  BuildDate: 2020-04-28T17:09:06Z
  GitCommit: 70facdb67207dbe115a9029e365f8e974e6156bc
  GitTreeState: dirty
  GitTag: v2.7.6
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64



md5-dfe12b978fbf375786df832f77f70c6c



$ kubectl version -o yaml
clientVersion:
  buildDate: "2020-02-13T18:08:14Z"
  compiler: gc
  gitCommit: 06ad960bfd03b39c8310aaf92d1e7c12ce618213
  gitTreeState: clean
  gitVersion: v1.17.3
  goVersion: go1.13.8
  major: "1"
  minor: "17"
  platform: darwin/amd64
serverVersion:
  buildDate: "2019-10-15T12:02:12Z"
  compiler: gc
  gitCommit: 211047e9a1922595eaa3a1127ed365e9299a6c23
  gitTreeState: clean
  gitVersion: v1.14.8
  goVersion: go1.12.10
  major: "1"
  minor: "14"
  platform: linux/amd64

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[simster7]

Hmm, I don't think we officially support parametrizing template names... although I don't see why we wouldn't. Let me explore this

[leoluz]

We are having a similar issue.

A brief explanation of our use-case:

We want to run validation steps in parallel and after each validation we need to run a generic post processing step. Validations will be provided by different teams, so it should be easy to add new validations in future. Ideally, to provide a new validation a team would have to add a new container template in the workflow and add its name in the validation list argument.

This is a minimalistic example of what Im trying to do:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: validation-
  namespace: workflow
spec:
  entrypoint: validations
  arguments:
    parameters:
      - name: validation
        value: opa-validation
      - name: validations
        value: |
          [
            "opa-validation",
            "config-validation"
          ]

  templates:
  - name: validations
    steps:
      - - name: validate
          template: invoke-validation
          arguments:
            parameters:
            - name: validation-name
              value: "{{item}}"
          withParam: "{{workflow.parameters.validations}}"

  - name: invoke-validation
    inputs:
      parameters:
        - name: validation-name
    steps:
      - - name: invoke-validate
          template: "{{inputs.parameters.validation-name}}"
      - - name: post-step
          template: do-post-step

  - name: opa-validation
    script:
      image: alpine
      command: [sh]
      source: |
        echo "opa validation"

  - name: config-validation
    script:
      image: alpine
      command: [sh]
      source: |
        echo "config validation"

  - name: do-post-step
    script:
      image: alpine
      command: [sh]
      source: |
        echo "post step"

if I try to submit this workflow I get:

Failed to submit workflow: templates.validations.steps[0].validate templates.invoke-validation.steps[0].invoke-validate template name 'placeholder' undefined

However parametrizing the template name doesn't seem to be an issue because if I replace the line:

template: "{{inputs.parameters.validation-name}}"

with:

template: "{{workflow.parameters.validation}}"

it works.

Having the ability to provide the step names as input parameters would not only reduce the size of the workflow but it also makes it much simpler for different teams to extend it.

[simster7]

Hi @leoluz, thanks for adding your use case. I agree that if we support template: "{{workflow.parameters.validation}}" we should certainly support template: "{{inputs.parameters.validation-name}}". I'll get fix going.

[simster7]

I've looked into this more and it seems the difficulty with input parameter template naming is validating subsequent fields that reference the template.

As an example, if we call a template with an input parameter:

  - name: invoke-validation
    inputs:
      parameters:
        - name: validation-name
    steps:
      - - name: invoke-validate
          template: "{{inputs.parameters.validation-name}}"

We won't know (for instance) what the step's outputs may or may not be since we don't even know which template will run. This means that we won't be able to validate if, say, {{steps.invoke-validate.outputs.parameter.foo-bar}} is valid or not and therefore we won't be able to validate any references that depend on that reference down the line. This is also true for template-specific variables such as {{steps.invoke-validate.ip}} for Container templates.

Note that this is different for {{workflow.parameters.*}} parameters since those are always resolved.

The way we currently implement validation makes it difficult for us to "pass down" the ambiguity down the line of references without disabling parameter name validation altogether, which is not something I'm sure we'd want to do. I'll keep trying to explore this and see if I come up with a solution. Feel free to add your thoughts.

[simster7]

After a discussion with the team, we decided that we don't want to go down this slippery slope for the reasons outlined above, so I'll be closing this issue.

Feel free to add your thoughts and continue the discussion if you have new ideas.

[leoluz]

yeah.. I understand your reasoning.

This could to be implemented if we had some sort of generic template that would only define the contract of inputs and outputs. This way it could be validated statically and still provide different implementations during runtime.

This is one possible solution but not sure about the complexity level it would introduce in the codebase.