Flux: Be able to force apply a resource

[hiddeco]

Kubernetes has some immutable resources (PodDisruptionBudget being one of them).

Flux is able to create these but unable to apply any updates made to them in git, making it either impossible to manage them through git or a pain to roll them out; as you have to remove them by hand so Flux can re-apply them during the next sync loop.

An idea may be to support a new annotation (flux.weave.works/force: "true") which instructs the daemon to apply resources with this annotation set with --force.

User interest: [[1][1]] [[2][2]]

[2opremio]

I think that, rather than adding a new annotation, a good solution would be to simply force-apply immutable resources.

I wonder if there is a programatic way to know whether a resource is immutable.

[alanjcastonguay]

I don't think the resource field immutability is exposed in the API contract (eg OpenAPI spec) anywhere. Instead it's baked into tests in the code like https://github.com/kubernetes/kubernetes/blob/83ff0f6c64485ffe491d5116596072f466be1bd5/pkg/apis/policy/validation/validation.go#L52 and https://github.com/kubernetes/kubernetes/blob/56735065400825d03f8e8f1a546c7a7b5ba9c5fd/pkg/apis/apps/validation/validation.go#L157 and https://github.com/kubernetes/kubernetes/blob/6c31101257bfcd47fa53702cea07fe2eedf2ad92/pkg/apis/storage/validation/validation.go#L64 and others.

Instead of trying to know ahead of time (LBYL) whether a (field of a) resource is immutable (consider CRDs, future k8s versions, etc) it may make more sense to catch the HTTP/422 and deal with the immutable resource/field (EAFP) following some policy (one of klaxons, revert, force, ignore).

[hiddeco]

a good solution would be to simply force-apply immutable resources.

I am opposed to this idea as we then start making opinionated rules, giving the user less control over what happens and no way to block it.

it may make more sense to catch the HTTP/422 and deal with the immutable resource/field (EAFP) following some policy (one of klaxons, revert, force, ignore).

If we just follow the user's command and blindly apply the --force when we are instructed to do so we do not need to create any HTTP/422 fallback logic, making the solution easier.

[2opremio]

I am opposed to this idea as we then start making opinionated rules, giving the user less control over what happens and no way to block it.

I would normally agree with you. However, I don't think it makes much sense to track immutable resources in git unless they are force-applied when updated. Thus, I think force-applying should be the default.

In other words, your annotation proposal makes sense but since I think everybody (or at least most people) will be tagging immutable resources with that annotation, it would make sense to simply make it the default. I think the opposite would be confusing for the end user.

This tangentially reminds me of why maps are not declared as pointers in Golang:

In the very early days what we call maps now were written as pointers, so you wrote *map[int]int. We moved away from that when we realized that no one ever wrote map without writing *map.

Phrasing it differently, Do you see a scenario in which the user wouldn't want to force-apply?

If you don't think that's the majority of cases, I would go for an annotation/flag disabling the force-apply, but I would still force-apply by default.

[alanjcastonguay]

I am trying to think of a concrete situation where force-applying (or delete+create) is a very bad thing. My gut is wary about a PV or LoadBalancer or similar with an outside-cluster dynamically-provisioned resource involved.

[rocktavious]

Personally - i think having an annotation which you opt-in to is better then by default because of the unexpected consequences of force applying something you don't want to force apply - such as with an immutable field in a loadbalancer (AWS ELB) and that could create a cascading failure because the load balancer was scaled up and now the new one you got is not pre-warmed.

If I as and end user was able to pick and choose what is force applied i can safely take the risks that come with force apply on a case by case basis.

[2opremio]

After thinking about it, I am starting to like the force annotation since it is also applicable (and in some cases desirable) to deletions during garbage collection. Maybe we should have a different variant for application and creation.

[hiddeco]

Maybe we should have a different variant for application and creation.

I would then propose to make it one annotation that accepts a string list, e.g. flux.weave.works/force: "apply,delete".

Can you elaborate some more on the cases where it would be desirable during gc?

[2opremio]

Off the top of my head, orphan workloads or workloads which are stuck terminating.

e.g. https://stackoverflow.com/q/35453792/1914440

The workflow would be a bit odd though, since you normally try to delete stuff forcibly once attempting to delete normally, which in the case of flux implies deleting the resource from flux. So it probably would imply re-adding the resource with the force annotation.

Maybe we should find a better solution for it.

[rocktavious]

Seems like in that case flux should know that the resource wasn't deleted and automatically do the force for you. But that probably means implementing the HTTP/422 fallback logic AND the annotation for force apply. FWIW I've pinged my co-workers on the idea of flux auto force apply all resources and most all of them say they'd rather have it be something we can explicitly enable on a file by file thing so we know the risks we are opting into. Seems like it would also reduce any unexpected "bugs" end users might claim when a resource was accidentally deleted and re created because of auto force apply.

[2opremio]

I think the annotation should be called something like sync indicating wether we should recreate (i.e. delete and apply) force, ignore validation ( see https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html ) etc when syncing

[wsilva]

I faced the same issue with cert-manager (https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html), so an annotation on the resource like
flux.weave.works/validation: "false|false" or like flux.weave.works/force: "true"
would be great.

If possible to fluxctl validate before running kubectl apply --dry-run -f underneath, so we can figure out which resources needs a special approach would be even better.

[setonez]

I have the same issue with kubernetes Jobs, making almost impossible to use GitOps approach with them.

[phillebaba]

I am personally in favor of having an annotation fluxcd.io/recreate: "true" that would delete + create the resource. I could have a crack at implementing it, if that is the direction we want to go.

[jacobsin]

I have the same issue with kubernetes Ingress, host being immutable, making almost impossible to use GitOps approach with them.

[phillebaba]

I have added a policy that allows disabling of validation for individual resources. All it does is adds the --validate=false flag to the apply command for those resources.
https://github.com/phillebaba/flux/tree/sync-settings

The recreate logic is a bit trickier. Adding a sync: recreate|force policy is easy, but the question is when it should be used. I agree with @alanjcastonguay that the policy should only be in play when there is an error. Having it any other way would make it difficult to avoid unnecessarily forcing or recreating resources.

Does anyone else have a different opinion?

[jacobsin]

I have added a policy that allows disabling of validation for individual resources. All it does is adds the --validate=false flag to the apply command for those resources.
https://github.com/phillebaba/flux/tree/sync-settings

The recreate logic is a bit trickier. Adding a sync: recreate|force policy is easy, but the question is when it should be used. I agree with @alanjcastonguay that the policy should only be in play when there is an error. Having it any other way would make it difficult to avoid unnecessarily forcing or recreating resources.

Does anyone else have a different opinion?

Agreed with force apply only when error (via --dryrun?), that will reduce chance of unnecessary service disruption due to resource recreation (e.g. ingress with updated hostname, service with updated selector)

Hopefully k8s will also review some of these unsupported mutations with real use case in near future, to make force apply an exception rather than a norm, and to make gitops a complete and completely declarative experience.

[dignajar]

Hi,
there is some workaround for this issue? because I'm getting warnings about field is immutable with a big output of a template for the jobs I'm keeping in Git.

method=Sync cmd="kubectl apply -f -" err="running kubectl: exit status 1 ....

[phillebaba]

The workaround is to remove any of the immutable resources that you have updated with kubectl. Flux cant on its own deal with immutable updating fields.

[dignajar]

Ok, I going to use this https://docs.fluxcd.io/en/latest/faq/#can-i-temporarily-make-flux-ignore-a-manifest

Is better than remove the job from my git.

[rocktavious]

I'm wondering now if really what we need is an annotation to beable to specify extra commandline arguments to kubectl apply command? This will then be future proof and allow other edgecase usages like --validate=false and --overwrite=true as well as --force=true

Just a thought.

[marcelser]

Any news on this? And is the flux.weave.works/validation: "false|false" or flux.weave.works/validation: "false" now available? I actually face this issue when I try to update a k8s job. It seems atm flux cannot update jobs at all and we can't use "generateName" with kubectl apply -f either it only works with kubectl create