Pipelines: VolumeOp doesn't support GC after workflow deletion

[hongye-sun]

The PVC created from VolumeOp is not automatically recycled after workflow is deleted.

I wonder if owner reference can work with PVC when it is set with workflow's reference. If it works, we can provide an option in volumeop to set it when creating the PVC.

If it doesn't work, we should provide a way to either use Argo's volume API or support a delete PVC op in exit handler.

[prcastro]

What is the practical consequence of this? I'm having an issue with bound PVCs from pipelines that ran long ago.

[elikatsis]

Hi all,
I'm working on a solution using owner references that seems to be working just fine!
I'll follow up with a PR soon.

/assign elikatsis

[elikatsis]

As I mentioned in the previous comment, I've been working on solving this issue using OwnerReferences.

An OwnerReference requires two basic fields that refer to the actual Kubernetes resource: name and uid.
Argo enables access to those using Argo (v2.3.0) variables: {{workflow.name}}, {{workflow.uid}}.

PR kubeflow/pipelines#1720, however, changed {{workflow.uid}} so that it points to runID instead, making it impossible to add OwnerReferences.

I'm not sure I understand the rationale behind this. [ cc @IronPan ]
Why don't we use {{workflow.labels.pipeline/runid}}?
[ cc @Ark-kun since it's used in the DSL in RUN_ID_PLACEHOLDER ]
It feels very odd having {{workflow.uid}} modified and not function as it's supposed to (i.e., be replaced with workflow's UUID), especially when there's a valid way to do what we desire.

[karlschriek]

I would be very interested in this. We have workflows where we sometimes need to dump out fairly large files to disk, but they are only relevant while the component is executing. Currently we have to always manually delete the PVCs afterwards

[elikatsis]

@Ark-kun @IronPan friendly ping :smile:

[PatrickXYS]

Willing to help, and will send a PR for adding support a delete PVC op.

[elikatsis]

@Ark-kun @IronPan

Hi there! Pinging again for an answer to this comment: https://github.com/kubeflow/pipelines/issues/1779#issuecomment-572001702

Since it's been some time (KFP has had many releases, KF ships with a KFP version containing #1720), can we move to using a different _Argo magic string_ and not override the existing {{workflow.uid}}?
Is it OK if I submit a PR fixing that behavior?

If we go with {{workflow.labels.pipeline/runid}} rather than a new {{kfp.run.id}}, there will also be no problem with incompatibilities between SDK and apiserver.

[Ark-kun]

I generally agree with the suggestion.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[PatrickXYS]

I don't think it's fixed now, please keep open until we fix.

[marrrcin]

Is there any workaround to remove the actual volume (disk) after the pipeline finishes?
Using

my_volume = dsl.VolumeOp(
        name="create-volume",
        resource_name="asd",
        size="10Gi",
        modes=dsl.VOLUME_MODE_RWO,
    )
dsl.ResourceOp(
        name="delete-volume",
        k8s_resource=my_volume.k8s_resource,
        action="delete"
    )

still leaves the unused disk in GCE.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jackwhelpton]

This is still a sorely-needed feature for us, would be good to keep the ticket open until it is resolved.

[Gsantomaggio]

Agree with @jackwhelpton
I would reopen it.
Thank you

[talhairfanbentley]

Hey, did anyone found any workaround for this?

[PatrickXYS]

@kubeflow/wg-pipeline-leads I think this is a well-known issue and all the cloud-provider should get the same issue.

As I can see, we met similar issue here. Can you put some efforts on this issue and I can help as well?

/cc @Bobgy

[Bobgy]

/assign @chensun
Can you take a look?
Looks like this is requested by a lot of people

[elikatsis]

Since we have now upgraded Argo client version used by the backend (apart from the deployment) we can take advantage of the setOwnerReference field of the ResourceTemplate (source).

Using this we will be able to set an owner reference of the workflow to PVCs created via the SDK, without having to deal with the [odd] design described above (https://github.com/kubeflow/pipelines/issues/1779#issuecomment-617887202).

@Bobgy I'm preparing a PR for this and will open it pretty soon :)

[PatrickXYS]

@elikatsis Feel free to cc me the PR

Would like to help as well

[RoyerRamirez]

Hi, I'm following along. Have we found a fix for this issue? We currently run many jobs in parallel, and having to go back to delete the disks after they are used can be a pain. Not to mention, it can get costly over time if users don't clean up their environment.

[elikatsis]

@RoyerRamirez we've got #4831 targeting this but it's not merged yet.