Pipeline: Add support for Task Step looping

[n3wscott]

It would be interesting if I could create [0, n] steps inside a Task based on parameters:

Something like this Task spec:

inputs:
  params:
    foos: 
      - foo
      - bar
      - baz
steps:
  - image: ubuntu
    command: [echo]
    args: ["start"]
  - forEach: $(param.foos) 
    image: ubuntu
    command: [echo]
    args: ["at $(step)"]
  - image: ubuntu
    command: [echo]
    args: ["end"]

would result in a Task run with output like:

at foo
at bar
at baz
end

[vdemeester]

It's a little bit linked to #2050

/kind feature

[n3wscott]

The real use case here is I wrote a tool that does a single thing (mutates a file) and I wrote a task to use that tool. But I found no way to call that tool for each usage I need inside of Tekton directly. I would have to either 1- create a task with a known # of steps, or 2- wrap it in a bash script.

I avoid the bash script because I am not trying to write bash here, I can use tekton and other tools directly. It would be backwards for me to take a dep on a bash script. And I would have to rethink all of my images... (cli tools built in go, shipped with ko)

[ImJasonH]

This might also be an opportunity to invest in a DSL that supports syntactic sugar for loops and compiles down to unrolled steps/tasks. Nobody wants to be programming in Turing complete YAML. Even Bash is better than that.

[bobcatfish]

This is an interesting idea @n3wscott ! One thing that might help is to ground us in a use case, do you have an example of a Task that you'd create if you had the ability to loop with steps?

[booninite]

Something like this example would represent what we would like to do with the ability: https://github.com/argoproj/argo/blob/master/examples/loops-param-argument.yaml

There are other looping examples that are more/less real world related than that in the same repository.

[bobcatfish]

hmmm that's an interesting example @booninite ! I know this is gonna disagree with @ImJasonH but I really could see that making sense for Tekton Pipelines 🤔 - I'm not sure I'd want it to be quite as automagical as the argo example, where the looping is implied.

@booninite @n3wscott are you sure you absolutely want this at the level of the steps in the Task? As mentioned in https://github.com/tektoncd/pipeline/issues/2050 we've discussed adding a list comprehension type syntax at the pipeline level, making the above example something like:

kind: Pipeline
spec:
    params:
    - name: os-list
      type: #hand-waving here
       value:  
        [
          { "image": "debian", "tag": "9.1" },
          { "image": "debian", "tag": "8.9" },
          { "image": "alpine", "tag": "3.6" },
          { "image": "ubuntu", "tag": "17.10" }
        ]
  tasks:
  - name: loop-param-arg-example
    taskRef: cat-os-release
    forEach: # i like the idea of making this explicit, but we keep going back and forth on that wiht our `from` syntax so i guess it could be implicit like argo is too
      param: os-list
      params:
      - name: image
         value: os-list[*].image
      - name: tag
         value: os-list[*].tag

(Or literally any other syntax that looks better than that, just trying to get the basic idea across)

Looking at https://github.com/argoproj/argo/blob/master/examples/loops-param-argument.yaml it looks like the steps themselves have a template, which in Tekton feels somewhat analogous to Tasks.

[booninite]

I think ultimately the snippet that you have shared is pretty much in line with the functionality in the Argo example I linked -- the main difference (as you noted) is that templates are in-lined instead of having a clear boundary with parameters passed in explicitly. For this usage scenario, we can think of a template in Argo as a Task in Tekton. Our usages today map closer to your example than the Argo example per se, we use withParam on a step which is itself a template containing multiple steps (or a sub-DAG). All of that is a long-winded way of saying "yes, I agree, this isn't necessary at the step level, only the task level" :).

I also favor making behavior like forEach explicit, FWIW.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.