Pipelines: Logically grouping/packaging steps?

[velascoluis]

/area frontend
/kind feature

As pipelines gets more complex, is it possible to group/package steps in logical units?
This way would it be easier to declare dependencies, have pan-group shared variables, access to PVCs, .. define success criteria for the group ..
It would also lets each component maintain its versioning, base images independently ..

[Ark-kun]

Is this a UX issue or an SDK issue?

On DSL-side it's possible to create and then use graph components ("pipeline-as-component"): https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.create_graph_component_from_pipeline_func

However loaded graph components are currently fully inlined at compilation time, so there is not grouping (dues to UX not showing it anyways).

Another possible to introduce a group is to use something like

with dsl.Condition(true):
   op1()
   op2()

For the UX, I think we could show only a single graph at a time with sub-graph nodes having a link that navigates to them.

[Bobgy]

I personally feel feature support for this will be very important in the long run.
Allowing extra layers of abstraction is something engineers always want for systems.

[rmgogogo]

(@rmgogogo to keep eyes on IR changes to cover this one)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[Bobgy]

/lifecycle frozen
/reopen

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/lifecycle frozen
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[google-oss-robot]

@Bobgy: Reopened this issue.

In response to this:

/lifecycle frozen
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[maorzalt]

This one is essential when running many (300+) tasks in the same graph.
The visualization freezes in this case!