Aws-load-balancer-controller: Unnecessary rule modifications in v1.1.3

Created on 18 Oct 2019 · 15Comments · Source: kubernetes-sigs/aws-load-balancer-controller

Since upgrading to v1.1.3, I've noticed that each rule on each ALB is upgraded exactly once an hour, even when no change is necessary. Is this intended behaviour?

Example:

I1018 01:46:50.938875       1 rules.go:82] core/core-ingress-ext-redacted-alb: modifying rule 1 on arn:aws:elasticloadbalancing:us-west-2:XXX:listener/app/XXX/XXX/XXX
I1018 01:46:50.955163       1 rules.go:98] core/core-ingress-ext-redacted-alb: rule 1 modified with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
I1018 01:46:50.955183       1 rules.go:82] core/core-ingress-ext-redacted-alb: modifying rule 2 on arn:aws:elasticloadbalancing:us-west-2:XXX:listener/app/XXX/XXX/XXX
I1018 01:46:50.976969       1 rules.go:98] core/core-ingress-ext-redacted-alb: rule 2 modified with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
I1018 01:46:51.001198       1 rules.go:82] core/core-ingress-ext-redacted-alb: modifying rule 1 on arn:aws:elasticloadbalancing:us-west-2:XXX:listener/app/XXX/XXX/XXX
I1018 01:46:51.019391       1 rules.go:98] core/core-ingress-ext-redacted-alb: rule 1 modified with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
I1018 02:46:56.487553       1 rules.go:82] core/core-ingress-ext-redacted-alb: modifying rule 1 on arn:aws:elasticloadbalancing:us-west-2:XXX:listener/app/XXX/XXX/XXX
I1018 02:46:56.508519       1 rules.go:98] core/core-ingress-ext-redacted-alb: rule 1 modified with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
I1018 02:46:56.508538       1 rules.go:82] core/core-ingress-ext-redacted-alb: modifying rule 2 on arn:aws:elasticloadbalancing:us-west-2:XXX:listener/app/XXX/XXX/XXX
I1018 02:46:56.528602       1 rules.go:98] core/core-ingress-ext-redacted-alb: rule 2 modified with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
I1018 02:46:56.556504       1 rules.go:82] core/core-ingress-ext-redacted-alb: modifying rule 1 on arn:aws:elasticloadbalancing:us-west-2:XXX:listener/app/XXX/XXX/XXX
I1018 02:46:56.603665       1 rules.go:98] core/core-ingress-ext-redacted-alb: rule 1 modified with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]

kinbug prioritimportant-soon

Source

cameronattard

👍18

Most helpful comment

I checked this today, its due to we upgraded the sdk(And the rule struct updated to support advanced routing).
So this simple check breaks (https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/internal/alb/ls/rules.go#L298).
The rules are updated with an no-op.
I'll fix this together with advanced routing support.

M00nF1sh on 6 Nov 2019

👍4 🚀2

All 15 comments

I have the same behavior with v1.1.3.

anarcher on 21 Oct 2019

👍1

I am seeing the same behavior. This is causing new ingresses to take a very long time to get created in clusters that have a large amount of ingresses.

wolfencom on 25 Oct 2019

@wolfencom My cluster has a small amount of ingress (1~3). But I am seeing the same behavior. T_T

anarcher on 28 Oct 2019

@M00nF1sh any input on this?

cameronattard on 5 Nov 2019

M00nF1sh on 6 Nov 2019

👍4 🚀2

Same here.
This can explain the controller don't add new pods to target group?

drrzmr on 27 Nov 2019

@drrzmr That shouldn't cause new pods cannot be added to target group. Would you help send me([email protected]) the logs of controller if you are see such issue?

M00nF1sh on 27 Nov 2019

@M00nF1sh sure, mail sent!

Apparently, the controller can now edit tg, adding new pod instances.
I confess, I do many random stuff, I don´t know what fix it.
I guess, removing these annotations from ingress metadata made difference:

alb.ingress.kubernetes.io/healthy-threshold-count: 5
alb.ingress.kubernetes.io/healthcheck-interval-seconds: 20

drrzmr on 28 Nov 2019

Yeah, this is biting me as well. Let me know if you'd me to send anything else also. Although, my logs look the same as the original logs at the top 😭

Edit:

Using go sdk for describe rules, it looks like aws-alb-ingress controller isn't passing in the condition completely as it should:

        Conditions: [{
          Field: "path-pattern",
          PathPatternConfig: {
            Values: ["/*"]
          },
          Values: ["/*"]
        }],

Edit:

Code to reproduce:

package main

import (
    "fmt"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/elbv2"
)

func main() {
    svc := elbv2.New(session.New(aws.NewConfig().WithRegion("us-east-1")))

    input := &elbv2.DescribeRulesInput{
        RuleArns: []*string{
            aws.String("INSERT_ARN_HERE"),
        },
    }

    result, err := svc.DescribeRules(input)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(result)
}