Aws-load-balancer-controller: Traffic not routing to Service

[garfbradaz]

I have been following the documentation, and struggling to get hit my backend Service via the ALB Load balancer provisioned.

When I try and hit my Service with the correct host URL and path combo (using browser and Postman), I get a 404. Now my Service is running fine internally on the cluster and the underlying Pod readinessProbe and livenessProbe are receiving their 200 status codes as expected, so no problem there:

I'm struggling really finding out how to diagnose the issue from the Load Balancer onwards and why its not finding my action method within my service. The Target Groups Health Checks though are showing unhealthy as well, but I have configured them correctly I think:

I can also send any Ingress Controller logs from K8's if needed. Anyone spot anything with my yaml?:

alb-ingress-controller.yml (within namspace ingress-alb):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: ingress-alb
    app.kubernetes.io/part-of: ingress-alb
    app: ingress-alb
  name: ingress-alb-controller
  namespace: ingress-alb
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-alb-controller
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::X:role/aviso-alb-role
      labels:
        app.kubernetes.io/name: ingress-alb-controller
    spec:
      containers:
        - name: ingress-alb-controller
          args:
            - --ingress-class=alb
            - --cluster-name=aviso-dev-cluster
            - --aws-api-debug
            - --aws-max-retries=10
          image: docker.io/amazon/aws-alb-ingress-controller:v1.1.2
      serviceAccountName: ingress-alb-controller

default-backend.yml (within namspace ingress-alb):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-alb-backend
  namespace: ingress-alb
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-alb
        app.kubernetes.io/part-of: ingress-alb
        app: ingress-alb
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.4
        imagePullPolicy: Always
        name: default-alb-backend
        ports:
        - containerPort: 8080

service.yml (within namspace avo):

apiVersion: v1
kind: Service
metadata:
  name: unified-api-gateway-ws
  labels:
    app: unified-api-gateway-ws
    chart: unified-api-gateway-ws-0.1.0
    heritage: Tiller
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      nodePort: 
  selector:
    app: unified-api-gateway-ws

deployment.yml (within namspace avo):

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: unified-api-gateway-ws
  labels:
    app: unified-api-gateway-ws
    chart: unified-api-gateway-ws-0.1.0
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: unified-api-gateway-ws
  template:
    metadata:
      labels:
        app: unified-api-gateway-ws
    spec:
      containers:
        - name: unified-api-gateway-ws
          image: <some-image>
          imagePullPolicy: Always
          env:
            - name: ASPNETCORE_ENVIRONMENT
              value: Staging
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /ws/handshake
              port: http
            initialDelaySeconds: 30
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ws/handshake
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
            limits:
              cpu: 100m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 512Mi

ingress.yml (within namspace avo):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: unified-api-gateway-ws
  labels:
    app: unified-api-gateway-ws
    chart: unified-api-gateway-ws-0.1.0
    heritage: Tiller
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: X:certificate/X
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "10"
    alb.ingress.kubernetes.io/healthcheck-path: /ws/handshake
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "8"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets: subnet-1,subnet-2,subnet-3
    alb.ingress.kubernetes.io/success-codes: "200"
    alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "5"
    kubernetes.io/ingress.class: alb

spec:
  rules:
    - host: avo-api-dev-ws.com
      http:
        paths:
          - path: /ws/
            backend:
              serviceName: unified-api-gateway-ws
              servicePort: 80

[M00nF1sh]

Hi， sorry for the late reply.
There are multiple errors here.

1. alb Ingress controller doesn't need an default backend deployment, you can remove your default-backend.yml. The default backend is an static 404 page, and you can override it with ingress.spec.backend and actions annotation

1. In your service.yaml, you are using targetPort:8080, which should be 'targetPort:80', which is the container port of your unified-api-gateway-ws.

2. I suspect you are getting 404 because you are accessing the ALB_DNS directly. Since you specified host: avo-api-dev-ws.com in ingress.spec.rules.host, that will require you to access the ALB through avo-api-dev-ws.com instead of ALB_DNS.

   1. You should use cname/alias to point avo-api-dev-ws.com to ALB_DNS manually or use a tool like external-dns..
   2. Alternatively, you can remove the ingress.spec.rules.host from your ingress spec, and access ALB_DNS directly(which means don't validate the host header).

[garfbradaz]

@M00nF1sh :

Firstly no need to apologise, I completely understand you are busy. Thanks for replying though appreciated.

The issue was 1. and 2. in the end (I have a Rackspace DNS entry pointing to the ALB_DNS which works nicely).

Once my ports were setup, this worked (and removing the default backend), this started working and my Target Groups started reporting healthly.

Thanks for your help 👍