Pulumi-kubernetes: Await logic doesn't handle networking/v1beta1:Ingress correctly

[phil-lgr]

Problem description

• Other: Deploying EKS Ingress Resources

Reproducing the issue

While deploying a Kubernetes Ingress in a CI/CI sequence, I wrote this terrible bash script to check when the load balancer is actually accessible:

external_ip=""; while [ -z $external_ip ]; do echo "waiting for endpoint..."; external_ip=$(kubectl get ingress -n erp-app-ci-$CIRCLE_BUILD_NUM -o=jsonpath='{.items[*].status.loadBalancer.ingress[0].hostname}'); [ -z "$external_ip" ] && sleep 10; done; echo "endpoint ready " && echo $external_ip; export APP_HOST=$external_ip
              until nc -vzw 2 $APP_HOST 80; do sleep 10; done

I was talking with @joeduffy today and it looks like pulumi doesn't wait for ingress resources to fully resolve before returning.

Suggestions for a fix

Idea is to have pulumi up, detects if there are new or modified ingress resources in the program and then waiting until it is fully accessible from the outsite.

The big plus is pulumi users would not have to write their own bash script to handle that.

[phil-lgr]

I think in GKE it does wait, but in EKS it doesn't. Pulumi would be a really good place to handle that.

[techmouse84]

I believe i'm running into this problem as well for EKS.
this.AppUrl = albIngress.Status.Apply(x => x.LoadBalancer.Ingress[0].Hostname);
It would complain that there's no ingress[0]
If i comment out this line, everything loads up perfectly including the ALB ingress.

Is there a work around in pulumi that i can try to do?
I kind of need this variable to perform more actions. Funny thing is that it used to work untill i added a certificate ARN in the annotations.

{ "alb.ingress.kubernetes.io/certificate-arn", "arn:aws:acm:ap-southeast-1:444146009571:certificate/XXXXXX" },

[ahilmathew]

Do we have an update on this issue. I'm also running into this issue now.
Any workarounds in place?

[lblackstone]

For issues like this, you can always work around by putting custom logic in an Output as suggested in https://github.com/pulumi/pulumi-kubernetes/issues/1056#issuecomment-636192266 and then depending on that Output.

[ahilmathew]

Just an update on my findings.
Pulumi has issues with the newer apiVersion : k8s.networking.v1beta1.Ingress. Pulumi doesn't wait for the loadBalancer hostname to be available while creating an Ingress using this apiVersion.
But if I use the old k8s.extensions.v1beta1.Ingress, it works fine. Pulumi waits for the hostname to be available before moving on with other stuff.

[lblackstone]

Ah, thanks for the update. This appears to be a bug in our await logic. Should be an easy fix.

[lblackstone]

The bug is in the decode logic here: https://github.com/pulumi/pulumi-kubernetes/blob/21410c314f384c84bc748dfedcf67c6c7b93675b/provider/pkg/await/ingress.go#L301

This function only handles the extensions/v1beta1 resource, so this logic needs to be updated to handle any version.

[techmouse84]

I'm still having problem with this.
I'm using Pulumi 2.8.1
I'm using alb ingress controller to spin the aws alb.

The behavior now is that it would fail immediately when trying to access the ingress as the ingress is not ready. There's no delay.

this.AppUrl = albIngress.Status.Apply(x => x.LoadBalancer.Ingress[0].Hostname);

In reality the ingress is actually being created successful and waiting for the alb controller to spin the ALB.
However I would need to do a pulumi up --refresh to get things to work again.

This behavior is consistent with pulumi destroy and pulumi up again.

var annotations = InfraStack.RequireOutput("MMS.CertArn").Apply(cert => { return new Dictionary<string, string>(){ { "kubernetes.io/ingress.class" , "alb" }, { "alb.ingress.kubernetes.io/tags", $"{TagConsts.CostAllocationTag}={TagConsts.BudgetTags.Mms}"}, { "alb.ingress.kubernetes.io/scheme", "internet-facing"}, { "alb.ingress.kubernetes.io/listen-ports", @"[{""HTTP"": 80}, {""HTTPS"": 443}]" }, { "alb.ingress.kubernetes.io/ssl-policy" , "ELBSecurityPolicy-TLS-1-2-2017-01"}, { "alb.ingress.kubernetes.io/certificate-arn", cert.ToString()! }, { "alb.ingress.kubernetes.io/load-balancer-attributes", "routing.http2.enabled=true" }, { "alb.ingress.kubernetes.io/target-group-attributes", "stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60,load_balancing.algorithm.type=least_outstanding_requests" }, { "alb.ingress.kubernetes.io/actions.ssl-redirect", @"{""Type"": ""redirect"", ""RedirectConfig"": { ""Protocol"": ""HTTPS"", ""Port"": ""443"", ""StatusCode"": ""HTTP_301""}}" }, { "alb.ingress.kubernetes.io/actions.landing-app", @"{""Type"": ""redirect"", ""RedirectConfig"": { ""Path"": ""/app"", ""Protocol"": ""HTTPS"", ""Port"": ""443"", ""StatusCode"": ""HTTP_301""}}" }, }; });

    `var albIngress = new K8s.Networking.V1Beta1.Ingress( "mms-ingress", new K8s.Types.Inputs.Networking.V1Beta1.IngressArgs()
    {
        Metadata = new ObjectMetaArgs(){
            Name = "mms-ingress", 
            Namespace = appNamespace.Metadata.Apply(x=>x.Name),
            Labels = new InputMap<string>() { 
                { "app", "mms-ingress" }
            },
            Annotations = annotations
        },
        Spec = new IngressSpecArgs()
        {
            Rules = new InputList<IngressRuleArgs>()
            {
                new IngressRuleArgs()
                {
                    Http = new HTTPIngressRuleValueArgs()
                    {
                        Paths = new InputList<HTTPIngressPathArgs>()
                        {
                            new HTTPIngressPathArgs()
                            {
                                Path = "/",
                                Backend = new IngressBackendArgs()
                                {
                                    ServiceName = "landing-app",
                                    ServicePort = "use-annotation"
                                }
                            },
                            new HTTPIngressPathArgs()
                            {
                                Path = "/*",
                                Backend = new IngressBackendArgs()
                                {
                                    ServiceName = "ssl-redirect",
                                    ServicePort = "use-annotation"
                                }
                            },
                            new HTTPIngressPathArgs(){
                                Path = "/app*",
                                Backend = new IngressBackendArgs()
                                {
                                    ServiceName = mvc.Service.Apply( s => s.Metadata.Apply( m => m.Name)), 
                                    ServicePort = 80
                                }
                            },
                            new HTTPIngressPathArgs(){
                                Path = "/gateway*",
                                Backend = new IngressBackendArgs()
                                {
                                    ServiceName = host.Service.Apply( s => s.Metadata.Apply( m => m.Name)),
                                    ServicePort = 80
                                }
                            }
                        }
                    }
                }
            }
        }
    }, new CustomResourceOptions
    {
        Provider = k8sProvider, DependsOn = { mvc, host }
    });`