Aws-load-balancer-controller: Multiple separate rules with the same target group

[maxtacu]

How to create multiple separate rules with the same target group which is defined as a service.
For example something like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: <certificate_arn>
    alb.ingress.kubernetes.io/conditions.k8s-service: |
      [{"Field":"http-header","HttpHeaderConfig":{"HttpHeaderName": "AnyHeaderName", "Values":["AnyHeaderValue"]}}]
    alb.ingress.kubernetes.io/conditions.k8s-service: |
      [{"Field":"source-ip","SourceIpConfig":{"Values":["1.1.1.1/1"]}}]
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    kubernetes.io/ingress.class: alb
  labels:
    app.kubernetes.io/name:application-name
  name: some-web-ingress
spec:
  rules:
  - host: example.com
  - http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: k8s-service
          servicePort: 443
        path: /*

I want to specify a few conditions alb.ingress.kubernetes.io/conditions.k8s-service for the same service name which will create different separate rules to the same target group, but the above solution doesn't work.
A workaround which I found here is to create a second service with different name, but pointing to the same resource and then use it in alb.ingress.kubernetes.io/conditions.k8s-service2.
Workaround example I will post in comments bellow.

P.S. Just to be more clear, in my case I dont want to create something like:

alb.ingress.kubernetes.io/conditions.k8s-service: >
  [{"Field":"http-header","HttpHeaderConfig":{"HttpHeaderName": "AnyHeaderName", "Values":["AnyHeaderValue"]}},{"Field":"source-ip","SourceIpConfig":{"Values":["1.1.1.1/1"]}}]

because this creates multiple conditions in the same rule.

[maxtacu]

workaround example
Create two services pointing to the same resource:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: aplication-name
  name: k8s-service
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: aplication-name

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: aplication-name
  name: k8s-service2
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: aplication-name

Create the ingress-controller

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: <certificate_arn>
    alb.ingress.kubernetes.io/conditions.k8s-service: |
      [{"Field":"http-header","HttpHeaderConfig":{"HttpHeaderName": "AnyHeaderName", "Values":["AnyHeaderValue"]}}]
    alb.ingress.kubernetes.io/conditions.k8s-service2: |
      [{"Field":"source-ip","SourceIpConfig":{"Values":["1.1.1.1/1"]}}]
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    kubernetes.io/ingress.class: alb
  labels:
    app.kubernetes.io/name: k8s-service
  name: some-web-ingress
spec:
  rules:
  - host: example.com
  - http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: k8s-service
          servicePort: 443
        path: /*
      - backend:
          serviceName: k8s-service2
          servicePort: 443
        path: /*

This creates two different rules on ALB with two different target groups which are pointing to the same resource. These rules are working as OR - If condition is met in one rule OR another, all other requests are dropped.

[daparthi001]

@tmxak what did you install for internal pod communication ?For me target type instance is not working on cluster ip ?

[maxtacu]

@daparthi001 I'm using basic EKS cluster configurations with coredns. NodePort is needed to create an application load balancer. Share your config here

[M00nF1sh]

@tmxak 🤦‍♂️ 🤦‍♂️ 🤦‍♂️u just found a bug....it's not the intended usage of conditions annotation..
The original intended usage requires there are servicePort: use-annotation to able to use conditions. It's missing a check for servicePort here. But after all, I think it's good usage, and help user that don't want extra actions annotation. So I'll keep this behavior and update the docs instead(to "The conditions-name in the annotation must match the serviceName in the ingress rules,and servicePort must be use-annotation").

Back to your original question. You can solve it by using our actions annotation like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/conditions.fake-k8s-service1: |
      [{"Field":"http-header","HttpHeaderConfig":{"HttpHeaderName": "AnyHeaderName", "Values":["AnyHeaderValue"]}}]
    alb.ingress.kubernetes.io/conditions.fake-k8s-service2: |
      [{"Field":"source-ip","SourceIpConfig":{"Values":["1.1.1.1/1"]}}]
    alb.ingress.kubernetes.io/actions.fake-k8s-service1: |
        {"Type":"forward","ForwardConfig":{"TargetGroups":[{"ServiceName":"k8s-service","ServicePort":"443"}]}}
    alb.ingress.kubernetes.io/actions.fake-k8s-service2: |
        {"Type":"forward","ForwardConfig":{"TargetGroups":[{"ServiceName":"k8s-service","ServicePort":"443"}]}}
    alb.ingress.kubernetes.io/scheme: internet-facing
    kubernetes.io/ingress.class: alb
  labels:
    app.kubernetes.io/name: k8s-service
  name: some-web-ingress
spec:
  rules:
  - host: example.com
  - http:
      paths:
      - backend:
          serviceName: fake-k8s-service1
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: fake-k8s-service2
          servicePort: use-annotation
        path: /*

[M00nF1sh]

/kind documentation
should update the docs to "The conditions-name in the annotation must match the serviceName in the ingress rules, and servicePort must be use-annotation"

[k8s-ci-robot]

@M00nF1sh: The label(s) kind/ cannot be applied, because the repository doesn't have them

In response to this:

/kind documentation
should update the docs to "The conditions-name in the annotation must match the serviceName in the ingress rules, and servicePort must be use-annotation"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[maxtacu]

thanks, @M00nF1sh for the explanation! Glad that we found a feature :smile: and not a bug