Argo: Workflow Template Catalog

[alexec]

With the introduction of workflow templates, we could not create a sharable library of templates that people could download and install.

E.g.

$ argo hub search ci
NAME         URL
ci-example https://argoproj.github.io/argo/templates/ci.yaml

$ argo template create -f https://argoproj.github.io/argo/templates/ci.yaml

[ericandrewmeadows]

My thought behind this was to enable Airflow-like behavior with Hooks and Operators. We have a lot of code that we could share template-wise for CI/CD, but I would imagine that you could create templates as well for model training and saving, in order to standardize some data I/O interfaces to steps.

Other places where this could be useful: Slack notifications, email notifications, etc. as templates.

[dcherman]

Absolutely! Similar to the catalog that Tekton has, I think it would be great to have a set of WorkflowTemplates/ClusterWorkflowTemplates for common tasks, that way everyone isn't re-inventing the wheel.

[alexec]

https://github.com/tektoncd/catalog

[alexec]

Some additional thoughts on this:

The TektonCD catalog appears to have about 50 items in it. I'd imagine we'd want to start with at least 20 ourselves. Our catalog will succeed or fail based on content.

Helm already provides a way to package manifests.

Rather than storing templates, we could store packages of resources, e.g. if the workflow depends on other Kuberenetes resources.

Let us call these template packages.

We'll want to GitOps them, so they need to work with kubctl apply -f. I.e. each

kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/argocd-task-sync-and-wait/0.1/argocd-task-sync-and-wait.yaml

Each package is going to be similar to a Helm chart.

We need to store metadata - in fact very similar metadata to the TektonCD catalog.

annotaions:
  # a semantic version range to indicate with versions of Argo Workflow the template is compatible with
  workflows.argoproj.io/version: v2.9.2
  # a description of the template
  workflows.argoproj.io/description: ...
  # comma-separated tags to enable searching and categorization
  workflows.argoproj.io/tags: ...

We do not need versions of each template package, that is just a Git tag.

[alexec]

Notes on Helm charts:

Meta-data is in Chart.yaml.
A chart is a zip file.
We can't use Helm due to {{.
If template packages can be installed using kubectl apply then we can support older versions.

[alexec]

Maybe:

kubectl apply -f https://raw.githubusercontent.com/argoproj/catalog/master/templates/argocd-task-sync-and-wait/manifests.yaml

[alexec]

Argo Helm charts: https://github.com/argoproj/argo-helm

https://argoproj.github.io/argo-helm/

[alexec]

Argo Helm charts are automatically published as soon as merged to master. We could do something similar using Github Actions. Creating an index and maybe a small searchable website?

https://github.com/argoproj/argo-helm/blob/gh-pages/index.yaml

[alexec]

@ericandrewmeadows, @dcherman, @maguowei, @dcherman @ramanNarasimhan77 - you are all interested.

I think there is some discussion around formatting of templates and the like, but what I think we need more than anything else is a number of seed templates for the catalog.

Would you each be willing to provide some, and perhaps to help with reviewing future submissions?

I think this is critical to the the success.

[ericandrewmeadows]

@alexec - definitely would love to provide some on the CI/CD side of things, and then willing to review future submissions. I am currently waiting on legal to blanket support our OSS contributions - that should be easy; I will check on the progress with that tomorrow.

[alexec]

This is a PoC for a online catalog (took about 1hr):

https://github.com/argoproj-labs/argo-workflows-catalog

To browse the catalog:

https://argoproj-labs.github.io/argo-workflows-catalog/

[dcherman]

Sure, I'm happy to contribute a few templates + participate in reviews

[ramanNarasimhan77]

@alexec
I'm using Argo mainly for CI process and I might be able to contribute templates for Kaniko build etc. I am also open to participate in reviews.

[alexec]

@ericandrewmeadows @dcherman @ramanNarasimhan77 I challenge you to put your money where you mouth is ;)

If I set-up a repository, would you contribute your first one or two templates? I'll look to the community to get more help.

[dcherman]

Sure. I think one of the first tasks to make will be an argo-cd sync/wait task, similar to what exists in the Tekton catalog which I'll be using for inspiration. Seems to make sense given the project :)

[dcherman]

Rather than storing templates, we could store packages of resources, e.g. if the workflow depends on other Kuberenetes resources.

Let us call these template packages.

We'll want to GitOps them, so they need to work with kubctl apply -f. I.e. each

kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/argocd-task-sync-and-wait/0.1/argocd-task-sync-and-wait.yaml

I 100% agree with this and would hope to avoid something more complex like Helm. Off the top of my head, I would not expect us to provide access to every single option that Argo can support, but instead a common set of defaults. For things like resource requests/limits, affinities, etc, it would be fantastic if we could support a Kustomize based workflow so that consumers can still use our manifests as a base while adding in the options they need for their specific use case

[alexec]

OK. I've added instruction to create templates:

https://github.com/argoproj-labs/argo-workflows-catalog/blob/master/README.md

Let me know how you get on.

[dcherman]

First draft PR created! I did find a couple questions/issues that I either filed issues for or put a note in the PR about

[alexec]

[ericandrewmeadows]

Got clearance - whether the work will be associated with my company or not, we can make the OSS contributions :).

[alexec]

@ericandrewmeadows great news!

[ramanNarasimhan77]

@alexec I need some guidance. I currently have a workflow that does the kaniko build by reading source files from a pvc. I'm thinking of modifying this to make the WorkflowTemplate

But I am not sure how pvc is shared between the Workflow where the pvc is defined and the WorkflowTemplate where it is referred. Is there an example available for this?

[ericandrewmeadows]

@ramanNarasimhan77 - maybe we can collaborate on https://github.com/argoproj-labs/argo-workflows-catalog/pull/4 - I'm working on understanding how I can abstract the volumes as inputs here.

[ramanNarasimhan77]

@ramanNarasimhan77 - maybe we can collaborate on argoproj-labs/argo-workflows-catalog#4 - I'm working on understanding how I can abstract the volumes as inputs here.

@ericandrewmeadows
Sorry for the delayed response.
Sure, I have reviewed the PR and the draft template looks good to me. I think using artifacts instead of PV for source code is a good idea.
I am looking forward to upgrade to argo 2.9.3 and try this out on my dev cluster.

[alexec]

I'm justing bumping this issue. For this to be a success, we need to seed the catalog with some templates. We currently have 3, and I think we should try and get at least 10 more. Would anyone be willing to contribute any?

[dcherman]

Definitely. I also spent a couple hours tonight and hacked up a basic integration tests runner so that authors can begin to submit tests alongside the catalogs.

[alexec]

This is now available.