Argo: Need a way to support each step's spec in a modular way

[dvavili]

Currently, the entire workflow needs to be specified in a single file. It'll be good to have a way to specify each pod spec individually and just stitch the workflow referencing these pod spec location.

[dvavili]

This may be similar to how helm charts are organized and handled

[jessesuen]

This is actually something that been discussed internally off and on for some time now. The crux of the problem is that in order to support the ability to reference other templates, you need context. Context could be something like a relative or absolute filepath. Or it could be a globally unique name. With filepaths, the problem is that YAML can have multiple specs contained in a single file, so filepath alone is not a good identifier. Furthermore, we believe that adding understanding of context into the workflow manifest itself is outside the purview of the workflow YAML.

Being able to reference/inline other pieces of YAML into a "main" yaml is a problem that is not unique to Argo. At the moment this is something we are hoping that higher level tooling like ksonnet can solve for us.

[bhack]

See also https://github.com/kubernetes/helm/issues/2577

[jessesuen]

Just to update where we are at on this issue. I'm working on generating an argo workflow.libsonnet to see what the end user experience is like to use ksonnet + argo libsonnet to generate and possibly submit their workflows. This requires some OpenAPI work/tooling which is mostly done.

[prarabdh9909]

Thanks @jessesuen ! Looking forward to it.

[prarabdh9909]

Don't know if it helps but tektoncd tried to solve the same thing by differentiating by decoupling invocation with functionality using Task and TaskRun. https://github.com/tektoncd/pipeline/tree/master/docs