Upload-artifact: Support Retention Policies

[jahed]

Now that GitHub has a strict quota and charges on storage usage, the ability to remove artifacts is an essential feature. This is covered in #5, however, since Actions are automated, manual removal is too tedious.

I'd like to suggest supporting retention policies instead where after a period of time (specified in the job step), artifacts automatically clear themselves out to avoid wasting storage space.

For example, when sharing artifacts between jobs, they're only needed for an hour maximum, and for debugging tests they may not be needed for more than a day.

Here is a configuration example where expires is the retention policy given in seconds.

- uses: actions/upload-artifact@v1
  with:
    name: my-artifact-for-one-hour
    path: path/to/artifact
    expires: 3600
- uses: actions/upload-artifact@v1
  with:
    name: my-artifact-for-one-day
    path: path/to/another/artifact
    expires: 86400

Without this, the entire idea of "Shared Storage" in GitHub Actions is unusable for Teams and Individuals with fixed budgets as the costs will just keep growing.

5 will still be needed to remove artifacts that don't have a retention policy.

[JCMais]

This would be really great. It is currently not viable to use this action when storing big, short-lived artifacts, it will just increase costs indefinitely.

[splitt3r]

The way GitLab CI solves this is pretty nice (https://docs.gitlab.com/ee/ci/yaml/#artifactsexpire_in). That would be a great addition.

[thomasboyt]

Also looking for this. My current project I'm adopting happens to be a public one so it's not a big deal, but this would stop me from ever using GH Actions artifacts in a private project.

Considering this is the only way to share state between jobs, an option to delete artifacts after the workflow completes would also be great. This would make it possible to not store artifacts that are only used for sharing state at all.

[le0m]

@thomasboyt you can use the cache action to share state between jobs during the same workflow as a workaround. That's what we're currently doing, waiting for this issue to be resolved.

[konradpabjan]

Thanks for the suggestion @jahed!

Retention policies is something that would be great to have, lots of uses cases for things such as storage quotas.

Nearly identical (keeping both open): https://github.com/actions/upload-artifact/issues/49

An expiration time that is just tied to seconds seems a little limited though. Certain users might want 30 days for example (vs the current 90 default) so having to type 2592000‬ seems pretty ugly and hard to understand if there are no comments.

I like what GitLab does with their expiration times, (days, hours and even years are supported). There is also this time format that i've come across that is defined here that uses d, s, and m for things like days, seconds and minutes (brief and easy to understand).

Combining ideas, how about something like this?

30 days

- uses: actions/upload-artifact@v2-preview
  with:
    name: my-artifact-for-30-days
    path: artifact/path
    expiration: 30d # '30days' or '30 days' would also work

1.5 hours

    expiration 1.5h #'1.5hrs', '1.5hours' also works

20 minutes

    expiration: 20m # '20min' or '20 min' also works

90 seconds

    expiration: 90s #'90sec' or '90 sec' also works

It could also be possible to combine types

    expiration: 5d 6h 10m 30s

[jahed]

I mainly used seconds as an example as time units can be controversial and I didn't want to get into it. :sweat_smile:

But since we're on the subject, a lot of tech has their own approach and people will be used to different approaches. Without type-safety, it's easy to get it wrong.

I'm personally used to Nginx's approach, but I'm sure others have their own preferences. Nginx uses ms, s, m, h d, w, M, y. The disambiguation for Minute/Month isn't ideal but I'm used to it.

The Azure Docs you (@konradpabjan) linked mentions most of those units except weeks, months and years. I don't know if that's needed (or even supported) for this specific use case but it doesn't sound too unreasonable to expect them to be available.

Any system is good with me, as long as each possible unit is exhaustively documented like the Azure Docs. The GitLab Docs @splitt3r linked IMO is an example of how not to document it as it's not easily clear what is and isn't available (e.g. there's no mention of seconds but it's possibly implied in the last example).

[MLDMoritz]

Can we PLEASE have this feature? I thought I wasn't looking right, repositories with 50gb storage because the artifacts are still there, blowing up the bill.

[jahed]

I'm unsubscribing from this issue as I don't think there's much left for me to add. (And I want to avoid the +1 spam :grimacing:). If you still have questions for me, please @ mention me.

Thanks.

[williamdes]

Here is a delete script for those who have reached quota
I also need this feature so artifacts are not kept for to long
[Fixed version]

#!/bin/bash

owner="orgname"
repo="repo"
auth="token xxxxx"
# See: https://developer.github.com/v3/actions/artifacts/#list-workflow-run-artifacts
ARTIFACTS=$(curl -X GET -H "Authorization: $auth" https://api.github.com/repos/$owner/$repo/actions/artifacts)
# More than 1 MB
URLS=$(echo $ARTIFACTS |jq '.artifacts[] | select( .size_in_bytes > 10000000 )' |jq '.url')
# Yesterday from now
#URLS=$(echo $ARTIFACTS |jq ".artifacts[] | select(.created_at|fromdateiso8601 < $(date -d "yesterday" +%s))" |jq '.url')

echo $URLS | xargs echo -n | xargs --max-lines=1  --null -d " " -I'{}' curl -X DELETE -H "Authorization: $auth" "{}"

echo "Done."

[dschinkel]

@williamdes how are you getting your auth token from that script? also what is jq

[williamdes]

@williamdes how are you getting your auth token from that script?

Hi @dschinkel
Here: https://github.com/settings/tokens

jq is a binary to manipulate json, see: https://stedolan.github.io/jq/

[dschinkel]

thanks, I installed it with brew.

But for some reason after filling in the values and running it I get:

jq: error (at <stdin>:1): Cannot iterate over null (null)
xargs: illegal option -- -
usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
             [-L number] [-n number [-x]] [-P maxprocs] [-s size]
             [utility [argument ...]]

I guess my $ARTIFACTS is null or something. Let me play with it more....

[williamdes]

Do you have curl installed ?

[dschinkel]

yea I think I just have the url wrong. I keep getting a not found response on my curl. I'll figure it out, thanks! I am running this from OS X btw.

[williamdes]

yea I think I just have the url wrong. I keep getting a not found response on my curl. I'll figure it out, thanks! I am running this from OS X btw.

Are you using the right repo (the one where artifacts are still stored) for the script ?

[dschinkel]

the curl command works now...I'm past that

[ahdbilal]

You can now change artifact and log retention duration either through Actions Settings tab or via YAML. You can learn more about this feature here. Feedback is welcome!