Flux: It takes a long time for HelmReleases to be applied

[jan-schumacher]

Describe the feature
It would be really handy when fluxctl has a command to force new creation of missing releases. I.e.: When I do kubectl delete hr redis it takes a lot of time to recreate the missing releases.

What would the new user story look like?
As a user I would love to force create missing releases. So that when I delete an existing release I don't have to wait a long time till it's recreated.
A force option on fluxctl sync command would be enough to fit the need.
fluxtctl sync --force or fluxtctl sync --force-create

Expected behavior
When I use fluxtctl sync --force or fluxtctl sync --force-create all missing releases will be created immediately.

[2opremio]

What do you mean by _missing release_?

[jan-schumacher]

What do you mean by _missing release_?

Well, when I delete a release manually with kubectl delete hr it is "missing" from there on for reasons. And in most of our cases we would love to immediately force flux to run the recreation process because the release is missing then.

[2opremio]

Ahh, you mean a helm release. Sorry.

[hiddeco]

What extra functionality would the --force flag on fluxctl sync offer? If you would do a fluxctl sync at the moment it would already create those resources instantly (and the Helm operator would act on the creation shortly after).

[jan-schumacher]

What extra functionality would the --force flag on fluxctl sync offer? If you would do a fluxctl sync at the moment it would already create those resources instantly (and the Helm operator would act on the creation shortly after).

No, it does not. We tried that several times. On our observation fluxctl sync does only repository syncing.

[hiddeco]

fluxctl sync will not say the sync was successful before it has applied all resources to the cluster, so the HelmReleases must be created (or we have bigger problems).

I have a suspicion the Helm operator does not react as fast as you'd like. Can you share the logs from the operator while you're doing a fluxctl sync?

[jan-schumacher]

I will share them later, I have a meeting, but what I can say is that it takes up to 10 minutes till recreation. Even when I do fluxctl sync

[2opremio]

Related #1128

[semyonslepov]

I think the initial problem here is that there is no way to force the Helm operator. You can force flux to sync the HRs but can't force Helm operator to start its "watch changes and apply if any" right now. I also hit this on our setup (not critical yet but it would be nice to have a similar functionality on the operator side).

[JVMartin]

I can confirm, this is an issue not just for Helm releases, but for any Kubernetes resources.

Say that you have a secret managed with Flux. If you delete it, Flux won't put it back where it belongs.

Example:

> kubectl delete secret mysecret
secret "mysecret" deleted
> fluxctl sync
Synchronizing with [email protected]:trend-capital/flux.git
Revision of master to apply is a1e0c48
Waiting for a1e0c48 to be applied ...
Done.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> fluxctl sync
Synchronizing with [email protected]:trend-capital/flux.git
Revision of master to apply is a1e0c48
Waiting for a1e0c48 to be applied ...
Done.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.
> kubectl get secret
No resources found in default namespace.

It can take hours for the resource to re-appear.

What's going on here?

[2opremio]

@JVMartin can you create a separate issue and share extra information (environment, Flux version, flux logs etc ...)? I doubt that what you are experiencing is related to the current issue.

Also, without further context it's hard to conclude whether a secret is expected to appear in the cluster through flux.

[JVMartin]

@2opremio Exactly like @jan-schumacher said, after a few minutes / hours, Flux finally notices the resource is missing, re-creates it, and all is well. With 0 intervention. It just takes a long time. This is true of ALL resources, not just HelmReleases, not just Secrets. Deployments, services, you name it. It seems like Flux should immediately notice when something has been deleted and re-create it, but it doesn't. I'll create a new issue and add more details when I get a chance.

[JVMartin]

Also: you don't have to take our word for it. Just delete any resource from your cluster, call fluxctl sync, and you'll see that it doesn't come back.

[hiddeco]

fluxctl sync only syncs a revision if it has not applied that revision yet.

Are you saying that the reconciliation loop that happens every --sync-interval does not put the resource back? Or is the --sync-interval in your case set to a very high value? As that would explain why it would take 'hours'.