Flux: Flux redeploying Deployments deleted by Flagger

[eddiewang]

Hi there!

Big fan of FluxCD. We've been using it in production for quite awhile and it works great.

I'm pretty sure this is already a solved problem, but I wasn't able to find any documentation referencing this issue: essentially, our pods are getting redeployed after Flagger successfully promotes a canary. This is because we have the deployment yamls in the Flux repo, and when Flux decides it's time to sync-up, it will redeploy the deleted pods.

Is there a best-practices approach to handling this issue so that we don't have useless pods consuming cluster resources?

[stefanprodan]

You can remove the replicas field from the Deployment spec, create a HPA with minReplicas= maxReplicas and add the HPA to Flagger canary under autoscalerRef.

[eddiewang]

@stefanprodan thanks for the quick feedback. just got around to testing this. I tried setting the deploy with the command k autoscale deploy $DEPLOY_NAME--min=1 --max=1 which setup the HPA. Problem is the deploy name is split in flagger between deploy and deploy-primary so not sure that's being caught and resolved.

Do you have a YAML I could grok at? Maybe I'm configuring it wrong.

[eddiewang]

Ok, setting the autoscalerRef and deleting / reapplying the canary seems to have done the trick.
Just to better understand how HPA-syncs, when I edit the HPA settings on the deployment, does it get automatically updated on the -primary HPA? or am I expected to reinitialize the Flagger workload everytime HPA changes?

[stefanprodan]

As with secrets, configmaps the HPA is being promoted after a successful canary analysis.

More details on this here https://github.com/weaveworks/flagger/issues/591