Brigade: Secrets\ConfigMaps not cleared up on removal of Brigade

[TimJones]

I've been doing some test of Brigade that required lots of installation & removal and noticed that upon removal, project & worker secrets\configmaps are not cleared up.

I noticed that the controller is not setting metadata.ownerReferences (https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/), which I think should be required for all resources that brigade creates on-the-fly.

[mumoshu]

@TimJones Thanks for the good catch! Yeah, I believe so too.
Would you by any chance to contribute a fix?
You may already know but the similar improvement in prometheus-operator seems to be an useful reference.

[TimJones]

@mumoshu Sure, I'd be happy to help!

[TimJones]

@mumoshu In this case the 'real' owner should be the Controller Deployment rather than the Pod itself right? If a controller Pod gets rescheduled we wouldn't want to kill off any running workers...

[technosophos]

We should probably not clean up project ConfigMaps automatically, since there are a number of use cases where you would delete Brigade but keep projects. (For example, I destroy and re-install brigade regularly, but re-use the same projects).

But we should definitely delete:

• cache PVCs
• Worker secrets/configmaps
• old job pods (maybe?)

[TimJones]

@technosophos Great points. I'll focus on just those resources for now, with an eye on making it easy to add to in the future should we want to extend to cleaning up other resources.

[radu-matei]

Related: https://github.com/Azure/brigade/issues/724#issue-391230685