Application-gateway-kubernetes-ingress: Redirect rule / Cert-manager TLS secret race condition

[DenisBiondic]

Describe the bug
A sporadic error occurs in which the redirect rule will not be create in app gw.

Issue is essentialy that the 443 redirect rule is not created because the secret could not be found (but the secret is there, created via cert-manager without any issues):

...
I0309 07:34:45.683350 1 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"traffic-health-endpoint", Name:"traffic-health-ingress-public-secure", UID:"58a0146b-61d8-11ea-8a77-f62f0564b24c", APIVersion:"extensions/v1beta1", ResourceVersion:"767058", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [traffic-health-endpoint/traffic-health-ingress-public-secure-tls]
....

E0309 07:34:45.683503 1 requestroutingrules.go:255] Will not attach default redirect to rule; SSL Redirect does not exist:

[DenisBiondic]

Similar to this: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/728 but I am unsure if it's the same issue.

[flyer299]

@DenisBiondic , What version of the AGIC are you running? We are running on kubernetes-ingress:1.0.1-rc2 and are having a very similar if not the exact same issue.

[flyer299]

I tried upgrading to 1.0.1-rc3 (not that I expected it result in anything different because the PR doesn't have anything specific for this). And it didn't fix it. But now I am running on RC3 for a while.

I am stuck on RC2 or RC3 because I am running in the Gov Cloud.

[DenisBiondic]

We are running

helmchart: "application-gateway-kubernetes-ingress/ingress-azure"
version: "1.0.0"

[andres-de-castro]

I was able to work around this by changing the name of the secret in the defined ingress.

[thomas-riccardi]

We are running helm chart 1.2.0-rc2 and got the same issue:

• cert-manager with let's encrypt
• first deployment of Ingress
• cert-manager took 6min to issue the certificate (i.e. create the k8s tls secret with it)
• it's as if AGIC stopped trying in loop to update the Application Gateway after 5min

Last logs:

I0519 18:11:38.724581       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0519 18:11:59.249915       1 mutate_app_gateway.go:185] Applied generated Application Gateway configuration
I0519 18:11:59.249936       1 mutate_app_gateway.go:200] cache: Updated with latest applied config.
I0519 18:11:59.251073       1 mutate_app_gateway.go:204] END AppGateway deployment
I0519 18:11:59.251095       1 controller.go:151] Completed last event loop run in: 20.647891202s
...
E0519 18:12:00.323329       1 event.go:316] Could not construct reference to: ... Will not report event: 'Warning' 'SecretNotFound' 'Unable to find the secret associated to secretId: [default/foo-tls]'
I0519 18:12:00.323507       1 redirects.go:44] Created redirection configuration sslr-fl-57c7702ab20d43cd6a7274ef3dc9bf89 for ([foo.default.example.com    ],443); not yet linked to a routing rule
E0519 18:12:00.323541       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/64fd4422-3372-4917-974e-6e470118daff/resourceGroups/hdd-test-app/providers/Microsoft.Network/applicationGateways/myappgw/redirectConfigurations/sslr-fl-ea122f20d31d50425c13b18fbaca28e9
I0519 18:12:00.332742       1 mutate_app_gateway.go:164] cache: Config has NOT changed! No need to connect to ARM.
I0519 18:12:00.332766       1 controller.go:151] Completed last event loop run in: 81.527897ms

Then nothing for a whole night.
(Before that we got 3 or 4 tries, without the Applied generated Application Gateway configuration log that appears at the end.)

Restarting the ingress-azure pod fixes the issue.

To conclude

• I could not find the ~5min timeout: is it configurable ?
• I would have expected an error log, not a ~success message (Applied generated Application Gateway configuration) when giving up on an Ingress definition change.
• I would also have expected an error event on the Ingress object after giving up

That being said, it seems that the new "reconcile" feature from #775 should hopefully help: could a developer confirm?

[barlass]

Facing same issue. Was anyone able to resolve?