Ingress-nginx: Custom Errors not working as advertised

Created on 24 Oct 2019 · 9Comments · Source: kubernetes/ingress-nginx

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

NGINX Ingress controller version:

0.25.1

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:47:22Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-eks-5876d6", GitCommit:"5876d6b7429820450950ade17fe7b4bf5ccada7f", GitTreeState:"clean", BuildDate:"2019-09-24T20:54:25Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: EKS
OS (e.g. from /etc/os-release): default
Kernel (e.g. uname -a): default
Install tools: helm chart
Others:

What happened:

nginx is not forwarding X-Format X-Code etc. as advertised in https://kubernetes.github.io/ingress-nginx/examples/customization/custom-errors/ despite even the generated nginx.conf having it the correct configuration

# nginx.conf
        error_page 404 = @custom_upstream-default-backend_404;
        error_page 403 = @custom_upstream-default-backend_403;
        error_page 503 = @custom_upstream-default-backend_503;
                location @custom_upstream-default-backend_404 {
                location @custom_upstream-default-backend_403 {

I am unable to see any of the promised headers in my default backend, a dump of headers show everything as an ordinary request to my default backend. I tried hitting URLs that should've resulted in 404s

Does nginx produce an X-Code of 404 to the default backend for true 404? (i.e. hitting an URL that is not in any Ingress rules, or a service return 404)

What you expected to happen:

I expected X-Code etc. to be populated

How to reproduce it (as minimally and precisely as possible):

curl against an URL that is clearly supposed to result in 404 (something random on my cluster for sure)

Anything else we need to know:

Source

erfangc

All 9 comments

We are also experiencing the same issue and we have a very similar setup.
I have tried downgrading the versions all the way down to 0.24 and that still did not resolve the issue.

JoaAlteos on 4 Nov 2019

Having the same issue here.

IsuruDilhan on 16 Dec 2019

If someone can produce some debugging steps or bash commands we can run in the ingress-controller's Pod that would be greatly appreciated. We will post findings or even solution here for others

erfangc on 16 Dec 2019

@erfangc Agree. As per https://kubernetes.github.io/ingress-nginx/user-guide/custom-errors/ it should forward any of above mentioned headers whenever custom-http-errors field defined in the nginx configmap.

This is my configmap for nginx-controller

apiVersion: v1
data:
  allow-backend-server-header: "true"
  compute-full-forwarded-for: "true"
  custom-http-errors: 404,502
  enable-access-log-for-default-backend: "true"
  enable-vts-status: "true"
  forwarded-for-header: X-Forwarded-For
  upstream-keepalive-connections: "32"
  upstream-keepalive-requests: "100"
  upstream-keepalive-timeout: "10"
  use-forwarded-headers: "true"
  worker-processes: "1"
kind: ConfigMap
metadata:
  creationTimestamp: 2019-11-08T06:39:08Z
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.6.13
    component: controller
    heritage: Tiller
    release: nginx-ingress
  name: nginx-ingress-controller
  namespace: default

This is nginx-errors deployment yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "12"
  creationTimestamp: 2019-11-28T05:44:14Z
  generation: 12
  labels:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-errors
  namespace: default
  resourceVersion: "4503014"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx-errors
  uid: 1c41a3b5-11a2-11ea-a69e-00155da14116
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx-errors
      app.kubernetes.io/part-of: ingress-nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: nginx-errors
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      containers:
      - image: quay.io/kubernetes-ingress-controller/custom-error-pages-amd64:0.4
        imagePullPolicy: IfNotPresent
        name: nginx-error-server
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2019-12-16T06:30:12Z
    lastUpdateTime: 2019-12-16T06:30:12Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2019-11-29T06:27:14Z
    lastUpdateTime: 2019-12-16T10:57:10Z
    message: ReplicaSet "nginx-errors-d96bc4dbc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 12
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

This is nginx-ingress-controller deployment yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "7"
  creationTimestamp: 2019-11-08T06:39:08Z
  generation: 12
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.6.13
    component: controller
    heritage: Tiller
    release: nginx-ingress
  name: nginx-ingress-controller
  namespace: default
  resourceVersion: "4557030"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx-ingress-controller
  uid: 7777203a-01f2-11ea-9870-00155da14114
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress
      component: controller
      release: nginx-ingress
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
        component: controller
        release: nginx-ingress
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=default/nginx-errors
        - --publish-service=default/nginx-ingress-controller
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=default/nginx-ingress-controller
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          runAsUser: 33
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: nginx-ingress
      serviceAccountName: nginx-ingress
      terminationGracePeriodSeconds: 60
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2019-11-08T06:39:08Z
    lastUpdateTime: 2019-11-08T06:39:08Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2019-11-08T06:39:08Z
    lastUpdateTime: 2019-12-16T11:45:13Z
    message: ReplicaSet "nginx-ingress-controller-dcd9556df" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 12
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

This is how i see when i hit an invalid page.

But what expected is this.

IsuruDilhan on 17 Dec 2019

I cannot reproduce this issue.

This is what I am doing:

Install minikube (optional)
Install the ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml

Patch the ingress controller to add a custom default backend

kubectl patch deployment -n ingress-nginx nginx-ingress-controller --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--default-backend-service=default/http-svc"}]'

Configure custom http errors to intercept code 503 and send that to the default backend

kubectl patch configmap/nginx-configuration \
  -n ingress-nginx \
  --type merge \
  -p '{"data":{"custom-http-errors": "503"}}'

Install an application that will act as default backend (is just an echo app)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/http-svc.yaml

Create an ingress pointing to a non-existing service

echo "
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: www-example-com
  spec:
    rules:
    - host: www.example.com
      http:
        paths:
        - backend:
            serviceName: non-existing-service
            servicePort: 80
          path: /
" | kubectl apply -f -

make a request

POD_NAME=$(k get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o NAME)

kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Just: to check' -H 'Host: www.example.com' localhost

Output:

Hostname: http-svc-66b7b8b4c6-t4bfx

Pod Information:
    node name:  minikube
    pod name:   http-svc-66b7b8b4c6-t4bfx
    pod namespace:  default
    pod IP: 172.17.0.6

Server values:
    server_version=nginx: 1.12.2 - lua: 10010

Request Information:
    client_address=172.17.0.5
    method=GET
    real path=/
    query=
    request_version=1
    request_scheme=http
    request_uri=http://www.example.com:8080/

Request Headers:
    accept=*/*
    connection=close
    host=www.example.com
    just=to check
    user-agent=curl/7.64.0
    x-code=503
    x-format=*/*
    x-ingress-name=www-example-com
    x-namespace=default
    x-original-uri=/
    x-request-id=416f937303fc32dc36aa78f648b02b1d
    x-service-name=non-existing-service
    x-service-port=80

Request Body:
    -no body in request-

aledbf on 17 Dec 2019

👍1

@IsuruDilhan the additional headers are sent to the default backend service, not the user. It is up to your custom backend to return that. That said, is not a good idea to leak that kind of information

aledbf on 17 Dec 2019

👍1

Closing. Please reopen if you can provide reproducible steps like https://github.com/kubernetes/ingress-nginx/issues/4712#issuecomment-566548606

aledbf on 29 Dec 2019

Thanks @aledbf for your help.

IsuruDilhan on 1 Jan 2020

👍1

@aledbf should the namespace for the default-backend-service set as default? or could it be on a different namespace as well?

kubectl patch deployment -n ingress-nginx nginx-ingress-controller --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--default-backend-service=default/http-svc"}]'