Argo: Proposal: Rename & generify output.parameters.path to valueFrom and make it context aware

[jessesuen]

Today, we have a little known feature to support output parameters for containers/scripts, like the following spec, where /some/path is a path in the container:

outputs:
  parameters:
  - name: myparam
    path: /some/path

However, there are now more use cases of a template having output parameters from different sources. Some examples:

1. from a file in the container (container/script template case)
2. from a field in a resource (resource template case)
3. from the outputs of a step (steps template case)
4. from the outputs of a dag task (dag template case)

This proposal is to rename path to valueFrom to make it a generic field that can be used for all our template types. Using the above cases, below are examples of how valueFrom could work:

1. from a file in the container (container/script template case)
   valueFrom: /some/path

2. from a field in a resource (resource template case)
   valueFrom: status.completions

3. from the outputs of a step (steps template case)
   valueFrom: steps.stepone.outputs.parameters.foo

4. from the outputs of a dag task (dag template case)
   valueFrom: tasks.mytask.outputs.parameters.foo

As you can see, the value in valueFrom is context dependent on the type of template.
@edlee2121 thoughts?

[edlee2121]

Being able to "return" various types of values from templates is a great idea.
In addition to conditionals and loops, they make it much simpler to produce and consume small values in workflows.

Have you run into a specific use case where this would be particularly valuable?

Perhaps we should also consider allowing a command for valueFrom.
E.g.
valueFrom: ['cat', '/some/path']
valueFrom: ['echo', '{{steps.stepone.outputs.parameters.foo}}']

[jessesuen]

The best use case I can think of is for resource templates (e.g. create a service, wait for it to have a external IP, and return the external IP as an output parameter).

Regarding a command for valueFrom, although I like this idea, this is not feasible without touching the user's container entrypoint/command (our v1 implementation). Basically, we have no knowledge or ability to inject some kind of an exit hook when the container completes. So everything we capture must be after the main container exits. This essentially means all we have to go off of are files (which we have access to through docker.sock).

[jessesuen]

Rather than a simple string field which is context aware, I think valueFrom should be made a struct with different fields for the appropriate context. Revised examples:

container:

valueFrom:
  path: /some/path

resource:

valueFrom:
  jsonPath: status.completions

steps:

valueFrom:
  parameter: "{{steps.stepone.outputs.parameters.foo}}"

tasks:

valueFrom:
  parameter: "{{tasks.mytask.outputs.parameters.foo}}"

[jessesuen]

Fixed