Argo: Workflow templates should be submittable

[jessesuen]

Summary

Currently, WorkflowTemplates only acts as a reusable component library whose spec.templates can be referenced from Workflow definitions.

By themselves, WorkflowTemplates are not submittable. This is because WorkflowTemplates are missing key information necessary to submit it (namely entrypoint, but also other relevant items such as serviceAccount). A large and common use case (which WorkflowTemplates does not solve), is to have a workflow definition that exist in the cluster (not on local disk, or git, or s3), which can be referenced and submitted from the CLI or UI, or even other CRDs such as argo-events or CronWorkflow.

Proposal

In order to make WorkflowTemplates more usable (i.e. submittable), WorkflowTemplates could essentially be a verbatim definition of a workflow, defined as follows:

type WorkflowTemplate struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
    Spec              WorkflowSpec `json:"spec" protobuf:"bytes,2,opt,name=spec"` // <- this is WorkflowSpec and not WorkflowTemplateSpec
}

My original thought was to simply add entrypoint to the WorkflowTemplateSpec, but this discounted all of the other spec fields would be needed to be brought in as well (such as serviceAccount, volumes, etc...). So in the end, in order to support all of the spec options of a workflow, we would need to either duplicate or reuse the WorkflowSpec as the WorkflowTemplate's spec.

With this proposal, the only difference between Workflow and WorkflowTemplates would be that a WorkflowTemplate is a definition of a workflow that exists in the cluster, whereas a Workflow would be an instantiation, possible originating (but not necessarily) from a WorkflowTemplate. Also, WorkflowTemplates could still act in the capacity of a component library and be a partial definition of a Workflow (i.e. the current design), missing entrypoint, serviceAccount, etc....

---

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[jessesuen]

/cc @dtaniwaki

[danxmoran]

@jessesuen it sounds like you picture an either/or scenario where the template is either a full workflow OR a reference-able component. Do you think the implementation of this would support the and/or scenario of templates which could be runnable on their own, but which also make sense to embed as a component in another workflow?

1747 is a concrete use-case for what I’m asking about. If this feature was implemented, could workflow templates w/ volume definitions be used as a component? Or would the presence of a volumes field prevent it?

[dtaniwaki]

Thank you for raising new idea.

A large and common use case (which WorkflowTemplates does not solve), is to have a workflow definition that exist in the cluster (not on local disk, or git, or s3)

I agree with this, but I'm still not sure about the expected behavior. Do you think workflow-template-global settings should be propagated? In other words, does it affect outside of a workflow template?

Another consideration is, if a WorkflowTemplate has the same spec as Workflow, There's no need to have WorkflowTemplate but we can just make Workflow referenceable from Workflow.

[amarrella]

Hi, I think this is a good idea and would be very useful!

Will with this proposal Workflow be able to reference a WorkflowTemplateby name instead of just inlining (same for argo events eventually)? If yes this would be great!

[ecurtin2]

I think this would be very nice for development/testing of workflow components in isolation

[samath117]

Why not just make Workflows savable to the cloud in the same way as WorkflowTemplates? I'm envisioning an argo workflow ... API analogous to the current argo template ... API.

This way, you wouldn't need to change the file specs or have two different notions of WorkflowTemplates.

[jessesuen]

Another consideration is, if a WorkflowTemplate has the same spec as Workflow, There's no need to have WorkflowTemplate but we can just make Workflow referenceable from Workflow.

Why not just make Workflows savable to the cloud in the same way as WorkflowTemplates?

@sarabala1979 had the same suggestion and here were my counter points:

1. Workflows vs. WorkflowTemplates require different RBAC. The former causes resources to be used in the cluster. Furthermore workflows may contain sensitive information in the actual payloads themselves (e.g. locations of s3 artifacts, IDs, etc...).

By having different resource kinds, it supports the ability to grant different rbac: e.g. everyone could have read or even create privileges on WorkflowTemplates (which would have no consequence in the cluster since it is just metadata), but limit the creation and reading of Workflows.

1. WorkflowTemplates in the future may have different information in the .status field. For example, we could someday track metrics/statistics about WorkflowTemplates (invocation counts, success/fail stats), and even export template information to prometheus.

2. WorkflowTemplates might one day have additional spec fields not present in workflows. I would actually revise my original proposal for the WorkflowTemplate to something like following:

type WorkflowTemplate struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
    Spec              WorkflowTemplateSpec `json:"spec" protobuf:"bytes,2,opt,name=spec"`
}

type WorkflowTemplateSpec struct {
    WorkflowSpec   `json:",inline"`

        // Future options go below:
        // Global indicates that this template can be referenced from any namespace
    Global              bool `json:"global"`
}

1. If we were to overload WorkflowTemplate and Workflow into a single object, there would need to be a new Workflow spec field to indicate when something is a template and not a runnable which needs garbage collection, archiving, etc.... At this point, it's better to introduce a different API resource type because you cannot easily query the API server for templates vs. workflows, nor leverage the informer framework effectively. Not to mention the two objects are fundamentally different in purpose (definition vs. instantiation)

2. There is precedent for this. Most people don't know this, but there is such thing as a PodTemplate resource object.

$ kubectl api-resources | grep PodTemplate
podtemplates                                                                        true         PodTemplate

PodTemplate are simply pod definitions which live in the cluster. It's not used by native kubernetes types, but there are users who do use it.

[jessesuen]

@jessesuen it sounds like you picture an either/or scenario where the template is either a full workflow OR a reference-able component. Do you think the implementation of this would support the and/or scenario of templates which could be runnable on their own, but which also make sense to embed as a component in another workflow?

Yes, in this proposal, WorkflowTemplates is not an either/or in terms of wether or not they are referenceable or not. It can be both. Even if a WorkflowTemplate is fully defined and submittable, it could still have its templates referenceable as a component from another Workflow or WorkflowTemplate.

1747 is a concrete use-case for what I’m asking about. If this feature was implemented, could workflow templates w/ volume definitions be used as a component? Or would the presence of a volumes field prevent it?

I think your use-case is achievable today. I posted a solution: https://github.com/argoproj/argo/issues/1747#issuecomment-575858810.

[jessesuen]

Will with this proposal Workflow be able to reference a WorkflowTemplateby name instead of just inlining (same for argo events eventually)? If yes this would be great!

Actually, I think what we have today is close to what you are looking for:
https://github.com/argoproj/argo/blob/master/examples/workflow-template/hello-world.yaml

[samath117]

I think my main objection is just to the naming. I would like to have Workflows whose definition is on the cluster, with their own WorkflowTemplate-like API and distinct RBAC, but I just think they should be a third type of object, not a redefinition of WorkflowTemplate.

Could we, at least in discussing the idea, call them something like RemoteWorkflows instead?

If we were to overload WorkflowTemplate and Workflow into a single object, there would need to be a new Workflow spec field to indicate when something is a template and not a runnable which needs garbage collection, archiving, etc.... At this point, it's better to introduce a different API resource type because you cannot easily query the API server for templates vs. workflows, nor leverage the informer framework effectively. Not to mention the two objects are fundamentally different in purpose (definition vs. instantiation)

This is exactly why WorkflowTemplates and RemoteWorkflows should be kept separate. WorkflowTemplates are definitions and (Remote)Workflows are instantiations. This distinction is pretty fundamental to Argo, and is something that all Argo users have to familiarize themselves with in order to know whether to use inputs or arguments and to always write steps.<instantiation>.outputs, not steps.<template>.outputs. Conflating Workflows with WorkflowTemplates would just be more confusing, and would also require a disambiguation at some point anyways.

Regarding RBAC, I would think that while RemoteWorkflows should have different access rights to locally-created Workflows, they should also have different access rights to WorkflowTemplates. WorkflowTemplates may technically just be metadata, but they can and should be affecting the resources that regular Workflows utilize, whereas nothing would depend on RemoteWorkflows in the same fashion.

(This naming system would be a bit more consistent if WorkflowTemplates were also renamed to RemoteWorkflowTemplates. Then regular WorkflowTemplates would essentially be the local WorkflowTemplate files before we run argo template create but with some API that allows them to be specified along with the Workflow when it is run...)

[alexec]

@dtaniwaki , we'd like to include this change in v2.6. That's 30 days away (yes - it is soon). @simster7 and I wanted to check if you wanted to work on it, or if we should?

[dtaniwaki]

@alexec I’d love to but I think I won’t have enough time to implement the feature by myself. Could you include me as a reviewer instead?

[simster7]

@alexec I’d love to but I think I won’t have enough time to implement the feature by myself. Could you include me as a reviewer instead?

I'll take it over and assign you as a reviewer. Thanks, @dtaniwaki

[alexec]

@dtaniwaki thank you for volunteering to be reviewer on this!

[alexec]

One key difference between WorkflowSpec and WorkflowTemplateSpec is that entrypoint is optional in the latter and mandatory in the former.

Options:

1. Make entrypoint optional for workflows - and throw errors at run time.
2. Make entrypoint optional for workflows - and if absent choose the first value.
3. Make entrypoint mandatory for workflow templates - a breaking change.

@dtaniwaki & @jessesuen - I like option 2

Bootnote: these are all breaking changes to the GRPC APIs that would change v2.6 to be v3.0 (semver)

[alexec]

I'm taking over from @simster7 to free his time up.

[alexec]

I envisage a new endpoint and CLI command to submit a workflow template. This would require you to enter some variables:

https://github.com/argoproj/argo/blob/master/docs/variables.md#global

So when we submit a template I think we need these options:

• template name
• namespace (optional - defaulting to the template's namespace)
• workflow.outputs.parameters - not sure
• workflow.annotations - copy from the temple and allow override
• workflow.labels - copy from the temple and allow override
• workflow.creationTimestamp - obvs
• workflow.priority - from user input?

[samath117]

The problem is that spec.entrypoint is far from the only thing that needs to be changed to essentially turn WorkflowTemplates into Workflows. You also need spec.arguments, metadata.generateName, and support for all of those workflow.parameters, workflow.outputs.parameters, workflow.outputs.artifacts, workflow.namespace, and so on, which currently aren't allowed to be inside WorkflowTemplates (except through the backdoor of lazily evaluated default input parameters; see https://github.com/argoproj/argo/issues/2057).

Even after all of these changes, if I as a user have a Workflow that I would like to save into the cloud for future use by myself or others, I would have to rename its kind to WorkflowTemplate before saving it. I still maintain (as I argued previously) that it would be better to keep the Specs different and just allow me to save my Workflow as-is to be submitted later, through an API like argo workflow create/submit/delete.

[Ark-kun]

@jessesuen

I really liked most aspects of your original design of Argo's structures.

With this proposal, the only difference between Workflow and WorkflowTemplates would be that a WorkflowTemplate is a definition of a workflow that exists in the cluster, whereas a Workflow would be an instantiation, possible originating (but not necessarily) from a WorkflowTemplate.

The way we've handled this in the component system used by Kubeflow Pipelines is to have a clear distinction between component objects and task objects. Components is like a function, class or template. Task is component+argument, a function call or class instance.
Argo had this distinction too (Template vs. DagTask). One design quirk was that the WorkflowSpec itself did not derive from DagTask even though it was essentially DagTask with some additional attributes.

In KF Pipelines, the tasks can reference their components by URI (or by name, or just inline the spec):

task1:
  componenRef:
    uri: http://....
  arguments:
    param1: 42

Unfortunately, the way the WorkflowTemplates were introduced in Argo, broke the nice design Argo had. It blurred the separation of tasks and templates and gave rise to monsters like templates referring to other templates referring to other templates. See my critique here: https://github.com/argoproj/argo/pull/1312#issuecomment-542960392

This issue/PR might be moving the structures further to the same unfortunate direction.
The correct way would have been unification of WorkflowSpec with DagTask, making the tasks submittable.

This way you can have submittable tasks that either reference components/templates or contain them inlined:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTask
spec:
  componenRef:
    uri: http://..../component.yaml
  arguments:
    message: "Hello"

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTask
spec:
  componenRef:
    spec:
      inputs:
        parameters:
        - name: message
      container:
        image: alpine
        command: [cowsay, '{{inputs.parameters.message}}'
  arguments:
    message: "Hello"

[jessesuen]

Components is like a function, class or template. Task is component+argument, a function call or class instance.
Argo had this distinction too (Template vs. DagTask). One design quirk was that the WorkflowSpec itself did not derive from DagTask even though it was essentially DagTask with some additional attributes.

The correct way would have been unification of WorkflowSpec with DagTask, making the tasks submittable. This way you can have submittable tasks that either reference components/templates or contain them inlined:

To be honest, I don't really think there is as a big of difference as you might believe, and I actually think we are heading in a direction which is closer to what you are looking for, by making WorkflowTemplates as a component definition which can also be submittable.

Taking your example WorkflowTask, this is the equivalent WorkflowTemplate:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTask
spec:
  componentRef:
    spec:
      inputs:
        parameters:
        - name: message
      container:
        image: alpine
        command: [cowsay, '{{inputs.parameters.message}}'
  arguments:
    message: "Hello"

vs.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
  entrypoint: cowsay
  arguments:
    parameters:
    - name: message
      value: "Hello"
  templates:
  - name: cowsay
    inputs:
      parameters:
      - name: message
      container:
        image: alpine
        command: [cowsay, '{{inputs.parameters.message}}'

Comparing the above, I don't see much of a fundamental difference between these two. Really, the major differences which I see are:

1. WorkflowTask componentRef allows for references to definitions outside the cluster (via HTTP URIs) whereas Workflow templateRef reference kubernetes definitions inside the cluster. I would argue that the latter is preferred since it is kubernetes native, and available to every user. Another way to say this is: why HTTP and not git or S3? If HTTP, then shouldn't it also support HTTP auth headers? I believe that component definitions should live in cluster, especially when you have other CRDs which also benefit from being able to reference these in-cluster objects. Aside from Workflows themselves, two other examples of this are: Argo Events, and CronWorkflows.

2. WorkflowTemplates allow for define multiple, related "components" that work together, to be defined inside the same definition (i.e. it is a list of templates), whereas WorkflowTask is essentially acting as a single, executable container template. I feel the former is more flexible. WorkflowTemplates allows encapsulation of a set of components working in conjunction (i.e. a DAG + multiple container templates), abstracted into a single, consumable, referencable unit. I view the WorkflowTemplate is a superset of the capabilities of the WorkflowTask since it can be used both in the capacity of a single container component as well as a consolidated set. My issue with WorkflowTask, is that it does not allow for this encapsulation of multiple components working together.

If we are using programming analogies: WorkflowTemplates are equivalent to a golang package/library. It exposes multiple functions in that package to be consumed by a caller (a Workflow, or other WorkflowTemplates). What we are enhancing with this proposal, is the ability to add a main.go to the package, so that the WorkflowTemplate itself is a definition something which can execute, versus its current form as just a stand-alone component library which is only useful to be referenced.

[samath117]

WorkflowTemplates allow for define multiple, related "components" that work together, to be defined inside the same definition (i.e. it is a list of templates), whereas WorkflowTask is essentially acting as a single, executable container template. I feel the former is more flexible. WorkflowTemplates allows encapsulation of a set of components working in conjunction (i.e. a DAG + multiple container templates), abstracted into a single, consumable, referencable unit. I view the WorkflowTemplate is a superset of the capabilities of the WorkflowTask since it can be used both in the capacity of a single container component as well as a consolidated set. My issue with WorkflowTask, is that it does not allow for this encapsulation of multiple components working together.

This seems to be a misrepresentation of the suggestion for WorkflowTask to derive from DagTask. After all, DagTask itself can consist of a DAG and multiple container templates.

If we are using programming analogies: WorkflowTemplates are equivalent to a golang package/library. It exposes multiple functions in that package to be consumed by a caller (a Workflow, or other WorkflowTemplates). What we are enhancing with this proposal, is the ability to add a main.go to the package, so that the WorkflowTemplate itself is a definition something which can execute, versus its current form as just a stand-alone component library which is only useful to be referenced.

Workflows themselves are already that executable package, and don't need to be reinvented from WorkflowTemplates. They just need to be saved to the cloud.

[jessesuen]

After all, DagTask itself can consist of a DAG and multiple container templates.

I think I missed that from the example which was given, but if true, then I feel there is even less of a difference between what's being proposed in this issue versus what you desire from WorkflowTask/DagTask. Perhaps I don't fully understand the DagTask. Also why is it only DAGTask not also Workflow Steps? There are many workflows which do not use DAGs and only use Steps templates.

They just need to be saved to the cloud.

I agree, which is the primary purpose of WorkflowTemplate -- it's the definition and not the instantiation. But when you say, "saved to the cloud," I strongly feel it it should be a CRD in the cluster and not a URI, since the CRD is neutral and an available option to everyone, including the controller which needs to look up the definition when referenced in a workflow. To rely on other medium which exists outside the cluster (HTTP URLS, Git, S3) requires a much more complex configuration to get around clusters which restrict egress, URIs which might require authentication, or even air-gapped clusters, not to mention it would require users to stand up specific infrastructure (an HTTP server) for the sole purpose of serving component definitions.

BTW, Kubernetes has precedence for this XXXTemplate concept. There is a PodTemplate kind (https://github.com/kubernetes/kubernetes/blob/5268f69405251a4a74130fa903e055a59071179a/pkg/apis/core/types.go#L2914) which is nothing but a resource containing a PodTemplateSpec.

The WorkflowTemplate kind is equivalent to Kubernetes PodTemplate kind, and Workflow kind is equivalent to Pod kind. By inlining the WorkflowSpec into the WorkflowTemplate, we are actually leveraging the exact same technique which was used for PodTemplate.

[jessesuen]

One key difference between WorkflowSpec and WorkflowTemplateSpec is that entrypoint is optional in the latter and mandatory in the former.

Options:

1. Make entrypoint optional for workflows - and throw errors at run time.
2. Make entrypoint optional for workflows - and if absent choose the first value.
3. Make entrypoint mandatory for workflow templates - a breaking change.

@dtaniwaki & @jessesuen - I like option 2

Option 4: optional for workflows. a WorkflowTemplate which has no entrypoint is not submittable, and is only useful as a component definition. Picking the first item of the template list is not explicit enough. Either it has entrypoint and is submittable, or it doesn't and is not.

Bootnote: these are all breaking changes to the GRPC APIs that would change v2.6 to be v3.0 (semver)

I don't think this is the case. This is why I changed the proto numbers in WorkflowTemplate to align with Worklows a few weeks back (before the v2.5 release of the gRPC server), in order to prepare for this change.

[samath117]

But when you say, "saved to the cloud," I strongly feel it it should be a CRD in the cluster and not a URI, since the CRD is neutral and an available option to everyone, including the controller which needs to look up the definition when referenced in a workflow. To rely on other medium which exists outside the cluster (HTTP URLS, Git, S3) requires a much more complex configuration to get around clusters which restrict egress, URIs which might require authentication, or even air-gapped clusters, not to mention it would require users to stand up specific infrastructure (an HTTP server) for the sole purpose of serving component definitions.

Personally, I'm all for making it a CRD, but I just think it should be a separate CRD, very similar to / derived from WorkflowSpec and not a mangled version of WorkflowTemplateSpec. I proposed the term RemoteWorkflows earlier in this thread. In that way, WorkflowTemplates would be definitions saved to the cloud, and RemoteWorkflows would be instantiations saved to the cloud.

In the ideal world to me as an Argo user, these two would have the same effect:

argo submit my-workflow.yaml
argo workflow create my-workflow.yaml && argo workflow submit my-workflow

(I'm imagining an argo workflow... CLI similar to the current argo template... but with submit.)

To make this work as a user, the only change is that my-workflow.yaml would need that name in its metadata, something that could look like this:

metadata:
  generateName: my-workflow-
  remoteName: my-workflow

[Ark-kun]

@jessesuen
Thanks for the answer. I think I might have chosen bad examples and they focused the attention on the wrong parts.

componentRef allows for references to definitions outside the cluster (via HTTP URIs) whereas Workflow templateRef reference kubernetes definitions inside the cluster.

That's not my point. Let's say that componentRef is exactly the same as templateRef - it can point to a template inside a CRD.

WorkflowTemplates allow for define multiple, related "components" that work together, to be defined inside the same definition (i.e. it is a list of templates), whereas WorkflowTask is essentially acting as a single, executable container template.

WorkflowTask is not a template (single or not). It has a template [reference]. The differece between to be something vs. to have something.

acting as a single, executable container template

The template can still be a DAG or Steps template. That's how Argo always did it.

//I'll try to add better examples in another comment.

Here is how I see the distinction between "templates" and "tasks":

Template:

• Declares inputs and outputs
• Has "implementation" - container, dag, script, resource etc
• Does not have arguments
• Cannot be submitted (since it lacks arguments)

Task:

• Does not declare inputs and outputs
• Does not have "implementation", but refers to some "Template" that has the implementation.
• Has arguments
• Can be submitted

In Argo versions <=2.3.0, the Template struct was the perfect realization of the "Template" entify described above and Workflow, DagTask and WorkflowSteps structs were perfect realization of the "Task" entity described above.

The PR that introduced WorkflowTemplate broke this design. It added arguments and template references to Template. So now you can write schematically-correct monsters like a template that declares inputs A and B, but refers to another template that declares inputs C and D and refers to another template... etc.

What I'd like to see is a system with clear distinction between the "Template"-like and "Task"-like objects, kike Argo 2.3.0 had.
Argo's "Template" struct should not have arguments or template references. WorkflowTemplate should not have arguments or template references.

Taking your example WorkflowTask, this is the equivalent WorkflowTemplate:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTask
spec:
  componentRef:
    spec:
      inputs:
        parameters:
        - name: message
      container:
        image: alpine
        command: [cowsay, '{{inputs.parameters.message}}'
  arguments:
    message: "Hello"

vs.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
  entrypoint: cowsay
  arguments:
    parameters:
    - name: message
      value: "Hello"
  templates:
  - name: cowsay
    inputs:
      parameters:
      - name: message
      container:
        image: alpine
        command: [cowsay, '{{inputs.parameters.message}}'

Comparing the above, I don't see much of a fundamental difference between these two.

The major difference is that the my example features "kind: WorkflowTask", and has a natural composition of a Task - a component/template [reference] plus arguments. The second snippet on the other hand shows an object of type "WorkflowTemplate", but violates the idea of Template by having arguments. When someone tries to use that "WorkflowTemplate", they suddenly have two sets of arguments.

Returning to the topic of this issue: I agree that there should be an easy way to submit run based on a template.

Here is how I'd propose to use CLI to submit template tasks for execution:

argo submit -f template1.yaml --argument param1=arg1 --argument param2=arg2

argo submit template-cr-1/template-name-1 --argument param1=arg1 --argument param2=arg2

Here, neither template in template1.yaml, nor template-cr-1/template-name-1 have arguments, but arguments are provided through the command-line. CLI combined the Template and the Arguments, creating a Task and submits it for execution.
Submitting template without specifying arguments will only work if the template has no inputs defined or all of them have default values.

[Ark-kun]

Here is an example of how I'd design the WorkflowTemplate structure and how I'd handle the submission of those templates:

WorkflowTemplate has inputs, outputs and implementation (container, dag, steps, resource, etc), but no arguments and no templateRef.

Container template (also script, resource, etc):

kind: WorkflowTemplate
metadata:
  name: container1-template
spec:
  inputs: ...
  outputs: ...
  container: ...

Dag template (also Steps, etc):

kind: WorkflowTemplate
metadata:
  name: dag1-template
spec:
  inputs: ...
  outputs: ...
  dag:
    tasks:
      task-1:
        templateRef: ...
        arguments: ...
      task-2:
        templateRef: ...
        arguments: ...

Optional. Template group:

kind: WorkflowTemplateGroup
metadata:
  name: templates1-group
spec:
  templates:
    template1: 
      inputs: ...
      outputs: ...
      container: ...
    template2: 
      inputs: ...
      outputs: ...
      dag:
        tasks:
          task-1:
            templateRef: ... # Can easily reference templates in the group or outside the group
            arguments: ...

Task object. WorkflowTask has templateRef, arguments, but no inputs, etc (although it can be possible to have template spec inlined in templateRef).

Can be submitted using kubectl to trigger a new run.

kind: WorkflowTask
metadata:
  generateName: dag1-task-1
spec:
  templateRef:
    kubernetesResource:
      name: dag1-template
  arguments: ...

argo submit templates1-group/template-1 --argument param1=arg1 --argument param2=arg2

argo submit dag1-template --argument param1=arg1 --argument param2=arg2

kubectl create -f dag1-task-1.yaml

Optional: Adding template group to the task object to make it standalone (like Argo's Workflow object):

kind: WorkflowTask
metadata:
  generateName: container1-task-1
spec:
  templateRef:
    name: template1
  arguments: ...
  templates:
    template1: 
      inputs: ...
      outputs: ...
      container: ...
    template2: 
      inputs: ...
      outputs: ...
      dag:

[sarabala1979]

My understanding from the above example, v2.6 workflow-template space splits into three different CRDs WorkflowTemplate, WorkflowTemplateGroup and WorkflowTask. I didn't see any benefit of splitting to separate CRDs . v2.6 Workflow-template can be defined as WorkflowTemplate, WorkflowTemplateGroup and WorkflowTask.
The current v2.6 submittable workflow-template spec will support the above three examples. v2.6 template will support the argument to pass into templates.

WorkflowTemplate:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
spec:
  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]

Template group:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-nested-template
spec:
  templates:

  - name: whalesay-inner-template
    templateRef:
      name:  workflow-template-submittable
      template: whalesay-template
    inputs:
      parameters:
      - name: message

WorkflowTask :

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
spec:
  entryPoint: whalesay-template.  --> this is optional, It can be passed through CLI
  arguments:
    parameters:
      - name: message
        value: hello world
  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]

argo submit workflowtemplate/workflow-template-submittable  -p message=hello --entrypoint whalesay-template

Am I missing any other points ?
@Ark-kun

[Ark-kun]

Am I missing any other points ?

Argo's current Template and WorkflowTemplate structs certainly make what I described possible. The desired and correct usage is possible.
Unfortunately, the schema also allows for many ambiguous and incorrect specs. Usually, ambiguous schemas lead to user and implementor errors.
The core reason for that is mixing the "Template"-like and "Task"-like entities together. This enables ambiguous and broken usage - templates that have multiple input/output sets and multiple argument sets.
I consider strict type schemas to have benefits since they prevent creation of invalid structures.
Imagine Argo just used unstructured dictionaries without any clear schema. Without schema, all correct behavior is still possible, but the problem is with incorrect behaviors.

Let me demonstrate the difference between the strict schema that has distinction between "Template" and "Task" and the one that does not:

This is the correct case that works with or without strict schema:

# Function definition ~= Template
def my_func(param1: str, param2: int = 42):
    return other_func(param1, param2, 7)

# Function call ~= Task
result = my_func(param1="Hello")

These are broken cases that are enabled by weak schemas that mix "Template" and "Task"

# Broken case 1: Function def/template that also has arguments (analogous to WorflowTemplate with arguments):
def my_func(param1: str, param2: int = 42) (param1="Hello"):
    return other_func(param1, param2, 7)

```python

Broken case 2: Function def that references other function (analogous to WorflowTemplate with templateRef):

def my_func(param1: str, param2: int = 42) xxx_func(foo: str, bar: str, baz: int):
# body of my_func
return other_func(param1, param2, 7)
# body of xxx_func
return param1 + param2

```python
# Broken case 3: Function call that gives arguments to something that already has argument: references other function (analogous to WorflowTemplate with arguments that uses template/templateRef pointing to template that also has arguments)
result = my_func(param1="Hello")(param1="ZZZZ")

```python

Broken case 4: Combination of the above of any size:

def func1(param1: str, param2: int = 42)(param1="hello") func2(foo: str, bar: str, baz: int)(foo="XXX") func3(p: str, q: str, r: int)(p="ppp"):
# body of func1
return ...1
# body of func2
return ...2
# body of func3
return ...3

Here is how the last case looks in Argo 2.4.0:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: func1
spec:
  entryPoint: func1
  arguments:
    parameters:
      - name: param1
        value: Hello
  templates:
    - name: func1
      template: func2 # ???
      inputs:
        parameters:
          - name: param1
          - name: param2
            value: 42
      arguments: # ???
        parameters:
          - name: param1
            value: Hello
      container:
        image: func1_body

    - name: func2
      template: func3 # ???
      inputs:
        parameters:
          - name: foo
          - name: bar
            value: bar_def
          - name: baz
            value: baz_def
      arguments: # ???
        parameters:
          - name: foo
            value: XXX
      container:
        image: func2_body

    - name: func3
      template: func1 # ???
      inputs:
        parameters:
          - name: p
          - name: q
            value: q_def
          - name: r
            value: r_def
      arguments: # ???
        parameters:
          - name: foo
            value: XXX
      container:
        image: func3_body

So, my point is that the abominations above should not be enabled by the schema.
Argo 2.3.0 schema did not allow those. Template structure did not have template, arguments or templateRef.
Argo 2.4.0 schema enabled those broken cases by adding Template.template (what?), Template.arguments and Template.templateRef. I think it should not have been done. It was possible to implement WorkflowTemplate without changing the schema of Template (it was enough to only add DagTask.templateRef and WorkflowSteps.templateRef).

I wish that Template.template, Template.arguments or Template.templateRef are removed from Template. Those attributes should only be available on DagTask, WorkflowSteps, Workflow.

[Ark-kun]

Regarding the current issue:
To submit a template the CLI could just create and submit a Workflow object with templateRef:

argo submit template-group1/template-1 -p message=hello

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: template-group1-template-1-
spec:
  templateRef:
    name:  template-group1
    template: template-1
  arguments:
    parameters:
      - name: message
        value: hello world

Is there any reason why Workflow with templateRef cannot be used for submitting templates?

[dtaniwaki]

Is there any reason why Workflow with templateRef cannot be used for submitting templates?

We can make it with a very small change.
https://github.com/argoproj/argo/blob/cdbc61945e09ae4dab8a56a085d050a0c358b896/workflow/controller/operator.go#L241

Just change the type of entrypoint from string to TemplateRef.

[sarabala1979]

We need arguments ,entrypoints and some case volumemount when we submit the workflow template. To consider above scenarios, we replaced workflowtemplatespce to workflowspec

[seaneagan]

Just wanted to add my use case. I would like to define a reusable workflow (workflow template) which can mount a secret (or config map) volume into a task. And ideally i would like to be able to pass the secret name as a parameter to the workflow template:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: secret-message
  namespace: test
spec:
  entrypoint: entrypoint
  arguments:
    parameters:
      - name: message_secret
      - value: my_secret_message
  templates:
  - name: entrypoint
      container:
        image: alpine
        command: [cat, /tmp/message]
      volumeMounts:
        - name: message
          mountPath: /tmp/message
          readOnly: true
  volumes:
    - name: message
      secret:
        secretName: {{workflow.parameters.message_secret}}

Would that be supported by this proposal?

[delwaterman]

Could we simply this approach by just modifying a Workflow definition, to allow entrypoint and exitHandler to take templateRef arguments:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-hello-world-
spec:
  entrypoint: 
    templateRef:
      name: workflow-template-whalesay-template
      template: whalesay-template
    arguments:
      parameters:
      - name: message
        value: "hello world"

[seaneagan]

@delwaterman that would be convenient, but it's already fairly easy to wrap workflow templates in a workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: call-foo
spec:
  entrypoint: entrypoint
  templates:
    - name: entrypoint
      steps:
        - - name: call-foo
            templateRef:
              name: foo
              template: entrypoint

my use case here though is being able to define other Workflow fields besides templates e..g arguments and volumes in a WorkflowTemplate ( inert Workflow ) which can be called from a higher level WorkflowTemplate or Workflow.

[seaneagan]

I think the cleanest way to support reusable workflow definitions would be to fully separate workflow definition from workflow instantiation. Currently Workflow is definition+instantiation, while WorkflowTemplate is only a partial definition. We can't remove the definition aspect from Workflow at this point (it's convenient anyway), but adding a full workflow definition ability to WorkflowTemplate (as proposed here) at least completes the concept of composable workflow definitions.

[seaneagan]

One thing that's unclear with the proposal, do global metadata variables (workflow.) referenced in WorkflowTemplates still represent the entrypoint Workflow? I believe that is the current behavior, and remains in #2222? Should they instead reference the current WorkflowTemplate? That would have much better data locality, and then any data from calling Workflows/WorkflowTemplates can be passed down the call stack as arguments.

However, if we do want to retain direct access to the entrypoint Workflow data under workflow., perhaps we can keep that and just add a new self. or similar prefix to refer to the immediately containing Workflow/WorkflowTemplate. But I'm still leary of WorkflowTemplates knowing about their callers.

[simster7]

For those interested in the problematic template-to-template referencing (@Ark-kun, @seaneagan, @delwaterman), I have opened #2447 to address this. Please feel free to add your comments there.

[simster7]

Hello everyone. Please be advised that we will be deprecating calling templates directly on a template object and will now only support calling them under template.steps or template.dag.

Please see this doc for more information: https://github.com/argoproj/argo/blob/master/docs/workflow-templates.md#referencing-other-workflowtemplates

The docs on WorkflowTemplates have also been updated to help with potential confusion: https://github.com/argoproj/argo/blob/master/docs/workflow-templates.md

Let me know if you have any questions or feedback.