Harbor: Epic: Tag Retention Policies

[nlowe]

This is an epic tracking the implementation of the proposal outlined in goharbor/community#4 and #5882

Summary

Not many people have the luxury of unlimited storage. For images and tags that are effectively ephemeral (like those built and tagged for use only by an intermediate process in a CI pipeline) it does not make sense to keep these tags around for an extended period of time, consuming precious resources. Other tags, like those correlating to released or deployed software may need to be kept for an extended period of time or even forever for various legal or compliance reasons. While this is possible today via Harbor's API, Harbor Administrators could greatly benefit from having this functionality built-in to harbor itself.

Progress

As this is a large feature and I can really only work on it in my free time, I'm splitting it up into multiple smaller issues. If you see something you're interested in working on feel free to contribute!

I envision this being implemented in roughly this order:

• [ ] Groundwork
  
  
  
  • [x] [Common types / interfaces / boilerplate](https://github.com/goharbor/harbor/issues/6655)
  
  
  • [x] [Filter chain builder](https://github.com/goharbor/harbor/issues/6656)
  
  
  • [x] [Database Persistence of Retention Policies](https://github.com/goharbor/harbor/issues/79333)
  
  
• [ ] Filter Implementations
  
  
  
  • [x] [Keep Everything](https://github.com/goharbor/harbor/issues/6657)
  
  
  • [x] [Always Keep tag x](https://github.com/goharbor/harbor/issues/6658)
  
  
  • [x] [Keep if downloaded at least n days ago](https://github.com/goharbor/harbor/issues/6659)
  
  
  • [x] [Keep most recent n tags](https://github.com/goharbor/harbor/issues/6660)
  
  
• [ ] API
  
  
  
  • [x] [CRUD for policies](https://github.com/goharbor/harbor/issues/6662)
  
  
  • [x] [Get logs for execution](https://github.com/goharbor/harbor/issues/6663)
  
  
  • [x] [Trigger tag processing](https://github.com/goharbor/harbor/issues/6664)
  
  
• [ ] Misc
  
  
  
  • [x] [Run tag processing on a schedule](https://github.com/goharbor/harbor/issues/6665)
  
  
  • [x] UI Proposal drafted
  
  

The UI proposal draft can start at any stage. The actual scheduling / execution of tag processing will be disabled until the feature is fully implemented. This way we can work on this in small, reviewable chunks.

[jeffweilee]

Really needs purge policy feature in real practice because dev images grow so fast...

• Keep most recent n tags
• Delete tags older than n days

[runzexia]

How about adding regex Filter?
In my company, we use jenkins running pipeline to build image。
There are two forms of images, one is the image during the test and release stages.The other is the image after the release of the product release.
The first type image name is xxx/$BRACH_NAME-$BUILD_NUMBER-SNAPSHOT.
The second type image name is v*.
The image we want to clean up is builded during the development test phase. For the image that has been officially launched, we want to save it all the time.

[nlowe]

@runzexia So essentially, "Keep/Delete tag x if it matches the regular expression xyz"?

[runzexia]

@nlowe yes

[supereagle]

How about adding regex Filter?

I think this makes sense.

[thatsk]

Also want to clean up old helm charts.

[nlowe]

I'm going to be iterating on this in the feat/retention/tracking branch on my fork until the merge window opens.

[steven-zou]

All the work are done! Close this epic.