Pipelines: Active/Archive management on Runs for Archived Experiment

[zijianjoy]

What happened:

During the PR #5040, which allows user to see Active/Archive run lists of an experiment, regardless of whether that experiment is active or archived. There is discussion on whether a user can restore a Run from an Archived Experiment. See https://github.com/kubeflow/pipelines/pull/5040#issuecomment-773952554 for the initial question. And https://github.com/kubeflow/pipelines/pull/5040#issuecomment-774061658 for the argument. Currently this behavior is allowed on the latest code.

Decision Making

The question: Can a user restore a Run from an Archived Experiment? If so, what is the use case? If not, why it is discouraged?

Follow-up items

• If this behavior is allowed, should we update UI to call out the state of Experiment in ExperimentDetail.tsx file?
• If this behavior is not allowed, what is the error message we should show to users?

/kind discussion
/area frontend

[StefanoFioravanzo]

I think the question is: "Should we allow Active Runs in Archived Experiments"?

If the answer was no, I can think of the following implications:

• Archiving an Experiment would imply archiving of ALL the Runs belonging to the Experiment. We could prompt users of course, saying that the Experiment archival will have this effect.
• Restoring an Experiment will restore ALL Runs belonging to the Experiment.

It is natural to me thinking about Experiments as direct parents (containers) of Runs, thus if I see an archived Experiment I would expect all of its runs to be Archived as well.

It's clear that KFP follows a more loosely coupled approach. @Bobgy Was this a specific decision, taken at some point in time, grounded on some specific rationale? Maybe there is some old issue where this had been discussed.

[Bobgy]

There were no formal discussion when it was implemented, and we sort of took the shortest path. Feel free to argue about whatever option makes the most sense to you.

[Bobgy]

One weird case would be:

1. Archive some runs, but not all
2. Archive the experiment, so all runs are archived
3. Restore the experiment, should all runs get brought back? What if archiving the experiment was a mistake?

Just to mean this is sth we can discuss, we may not need to support this corner case so well.

[Bobgy]

I agree with @StefanoFioravanzo 's proposal and when experiment is archived, runs should fail to unarchiv

[StefanoFioravanzo]

@Bobgy This is a fair point!

One weird case would be:

1. Archive some runs, but not all
2. Archive the experiment, so all runs are archived
3. Restore the experiment, should all runs get brought back? What if archiving the experiment was a mistake?

Just to mean this is sth we can discuss, we may not need to support this corner case so well.

I think the following approach addresses it:

• Archive Experiment: All Runs belonging to the Experiment get archived as well (upon user confirmation)
• UnArchive Experiment: All Runs REMAIN archived; if the user wants to unarchive them all then they can go to the Experiment Details page, Archived Runs tab, and perform a bulk restore

[StefanoFioravanzo]

In this way we enforce this first implication

Archived Experiment -> Archived Runs

But not the opposite, and it's completely the users' choice whether to restore Runs, all of them or just in part. And if they want to restore an Experiment and keep it empty of Active Runs, to start creating new ones, they can do so.

[Bobgy]

SGTM

[Bobgy]

if the user wants to unarchive them all then they can go to the Experiment Details page, Archived Runs tab, and perform a bulk restore

In fact, the argument seems to apply reversely too, if all runs are restored by default, users can bulk archive them again.

I think we should think about which operation is more common as user intent? I feel like restoring all runs when restoring the experiment would be more Intuitive for me. WDYT?

[StefanoFioravanzo]

I think we should just make this configurable. When you restore an experiment we can ask: "Do you want to restore all archived Runs as well?" -> "yes"/"no". So we definitely cover all potential use cases. WDYT?

[Bobgy]

Agreed, that's a good point!

[zijianjoy]

I agree with the statement that Archived Experiment should not contain Active Runs.

In terms of state management of archiving/restoring an Experiment, I think the original intent of Archived Runs is to avoid accidental deletion of active runs, because active runs cannot be deleted, it has be archived first. And the concept of Archived Experiment is to group a list of runs which are ready to be deleted. See https://github.com/kubeflow/pipelines/issues/3283#issuecomment-626094965.

Therefore, I agree with the interactive approach to ask users whether to restore Runs while restoring Experiment.

To summarize the list of changes required as a result of this decision:

1. frontend: Hide Active runs list in ExperimentDetail.tsx when Experiment is Archived. Because no run should be active if Experiment is archived.
2. frontend: When restoring an experiment, popup should shows 3 options for user to choose: Cancel, Restore Experiment and Restore Experiment and All Runs, and highlight Restore Experiment because that is the default behavior.
3. backend: Check whether a run is under archived experiment when Unarchive API is called. If so, fail with error 400 (FAILED_PRECONDITION).

Feel free to comment if I am missing anything or need adjustment. Thank you!

[Bobgy]

I agree with the statement that Archived Experiment should not contain Active Runs.

In terms of state management of archiving/restoring an Experiment, I think the original intent of Archived Runs is to avoid accidental deletion of active runs, because active runs cannot be deleted, it has be archived first. And the concept of Archived Experiment is to group a list of runs which are ready to be deleted. See https://github.com/kubeflow/pipelines/issues/3283#issuecomment-626094965.

Therefore, I agree with the interactive approach to ask users whether to restore Runs while restoring Experiment.

To summarize the list of changes required as a result of this decision:

1. frontend: Hide Active runs list in ExperimentDetail.tsx when Experiment is Archived. Because no run should be active if Experiment is archived.

Personal opinion, UI should be stable, when a button doesn't apply in a context, we can disable it and add a hover tooltip to tell why. Removing it might leave people confused -- where is the tab?

1. frontend: When restoring an experiment, popup should shows 3 options for user to choose: Cancel, Restore Experiment and Restore Experiment and All Runs, and highlight Restore Experiment because that is the default behavior.

Note, we need to implent this feature in backend too.

1. backend: Check whether a run is under archived experiment when (Unarchive API)[https://www.kubeflow.org/docs/pipelines/reference/api/kubeflow-pipeline-api-spec/#operation--apis-v1beta1-runs--id-:unarchive-post] is called. If so, fail with error 400 (FAILED_PRECONDITION).

Feel free to comment if I am missing anything or need adjustment. Thank you!

Thanks! Per priority, sounds like 3 should be done first, 1 and 2 are good to haves. Users wouldn't be very distracted with current implementation too.

[StefanoFioravanzo]

Personal opinion, UI should be stable, when a button doesn't apply in a context, we can disable it and add a hover tooltip to tell why. Removing it might leave people confused -- where is the tab?

Yes! I completely agree on this!

[Bobgy]

I think

backend: Check whether a run is under archived experiment when (Unarchive API)[https://www.kubeflow.org/docs/pipelines/reference/api/kubeflow-pipeline-api-spec/#operation--apis-v1beta1-runs--id-:unarchive-post] is called. If so, fail with error 400 (FAILED_PRECONDITION).

can be a good first issue.
Welcome contribution!