Pipelines: How to access run info within a run

[santiagoolivar2017]

Once a run is executed how can I retrieve information like the experiment name, run id, pipeline name, etc. I need this information to manage where the artifacts will be stored and woud like to organize the paths based on this information.

[santiagoolivar2017]

I figured it out.
I am using kfp.dsl.EXECUTION_ID_PLACEHOLDER and kfp.dsl.RUN_ID_PLACEHOLDER within the pipeline specification.

[Bobgy]

I'm glad you figured out by yourself

[santiagoolivar2017]

@Bobgy
Those placeholders are very useful and I have been using them to uniquely identify a run. Nevertheless, I am still not able to get the run name and the experiment name (assuming that the run is part of one experiment). Any idea how this can be done?

[Bobgy]

/reopen

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bobgy]

I think we need to expose it

[Bobgy]

/kind feature

[Bobgy]

/cc @hongye-sun
Have we thought about placeholder syntax for exposing run uuid?

The existing placeholders have [[RunUUID]] format as other placeholders supported in scheduled workflow controller, but in IR we are now standardizing on
https://github.com/kubeflow/pipelines/blob/7ed110e9845237115591d04d82085d957e343795/api/v2alpha1/pipeline_spec.proto#L595-L613

I think it makes more sense to stick with IR format.

Context: @StefanoFioravanzo has already implemented PRs to add the placeholder. The major remaining concern is aligning on a preferred placeholder format.

[hongye-sun]

How about we introduce the following placeholders:

$.job returns the json payload of the job metadata which includes ID, name and pipeline name, etc.
$.job.id returns the unique ID of the job.