Ingress-nginx: http proxy status 400 "400 Bad Request" on long URLs

Created on 22 Mar 2019 · 8Comments · Source: kubernetes/ingress-nginx

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version:
extensions/v1beta1

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:16Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-gke.1", GitCommit:"5c4fddf874319c9825581cc9ab1d0f0cf51e1dc9", GitTreeState:"clean", BuildDate:"2018-11-30T16:18:58Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration:
Google Kubernetes Engine
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

What happened:
When URL length is above a certain threshold (~4000 chars), Nginx returns 400 with no relevant information (it seems to me) other than "http proxy status 400 "400 Bad Request""

What you expected to happen:
Nginx should properly forward request to API server and return 200.

How to reproduce it (as minimally and precisely as possible):
Set up K8s Ingress with settings below, then send HTTPS request with >4000 characters in URL to it

Have tried the following solutions (increasing buffer sizes, setting secure-backends) to no avail.

Anything else we need to know:
Ingress file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    # Configuration snippets to prevent ingress-nginx from rewriting and stripping out the /forest route
    # @ this is necessary for forest admin's call to /forest to pass through to the api_server
    # @ see https://stackoverflow.com/questions/51822525/configure-kubernetes-traefik-ingress-with-different-path-rewrites-for-each-servi
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite /forest/(.*) /forest/$1 break;
      rewrite /forest /forest break;
    # Configuration for ssl certificates
    # See https://github.com/jetstack/cert-manager/blob/master/docs/reference/ingress-shim.rst
    # @ See README.md for setting ingressShim.defaultIssuerName and ingressShim.defaultIssuerKind
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # Increase client_body_buffer_size and client_max_body_size to accept larger attachments
    # nginx.ingress.kubernetes.io/client-body-buffer-size: 10m
    nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-body-size: 8k
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 8k
    nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "180"

ConfigMap file:
apiVersion: v1 kind: ConfigMap metadata: # "name" needs to match the name of the configmap that Ingress Nginx uses # To see what name this is, run `kubectl describe pod <ingress-nginx-pod-name> -o yaml` # and search for "configmap". name: ingress-nginx-ingress-controller namespace: gitlab-managed-apps labels: app: nginx-ingress chart: nginx-ingress-0.23.0 component: controller heritage: Tiller release: ingress data: # Override http-redirect-code attribute of Ingress Nginx configmap. Default values here: # https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#use-proxy-protocol http-redirect-code: "301" client-header-buffer-size: 8k client-body-buffer-size: 8k http2-max-field-size: 8k http2-max-header-size: 8k large-client-header-buffers: "8 16k" error-log-level: "debug"

Error log output from ingress-nginx pod:
2019/03/22 11:11:06 [debug] 393#393: *88 http upstream process header 2019/03/22 11:11:06 [debug] 393#393: *88 malloc: 00007FF050C30000:4096 2019/03/22 11:11:06 [debug] 393#393: *88 recv: eof:1, avail:1 2019/03/22 11:11:06 [debug] 393#393: *88 recv: fd:10 28 of 4096 2019/03/22 11:11:06 [debug] 393#393: *88 http proxy status 400 "400 Bad Request" 2019/03/22 11:11:06 [debug] 393#393: *88 http proxy header done 2019/03/22 11:11:06 [debug] 393#393: *88 headers more header filter, uri "/forest/CreditLine.csv" 2019/03/22 11:11:06 [debug] 393#393: *88 http2 header filter 2019/03/22 11:11:06 [debug] 393#393: *88 http2 output header: ":status: 400" 2019/03/22 11:11:06 [debug] 393#393: *88 http2 output header: "server: nginx/1.13.8" 2019/03/22 11:11:06 [debug] 393#393: *88 http2 output header: "date: Fri, 22 Mar 2019 11:11:06 GMT" 2019/03/22 11:11:06 [debug] 393#393: *88 http2 output header: "strict-transport-security: max-age=15724800; includeSubDomains;" 2019/03/22 11:11:06 [debug] 393#393: *88 http2:9 create HEADERS frame 00007FF053238370: len:83 2019/03/22 11:11:06 [debug] 393#393: *88 http cleanup add: 00007FF053238468 2019/03/22 11:11:06 [debug] 393#393: *88 http2 frame out: 00007FF053238370 sid:9 bl:1 len:83 2019/03/22 11:11:06 [debug] 393#393: *88 SSL buf copy: 9 2019/03/22 11:11:06 [debug] 393#393: *88 SSL buf copy: 83 2019/03/22 11:11:06 [debug] 393#393: *88 http2:9 HEADERS frame 00007FF053238370 was sent 2019/03/22 11:11:06 [debug] 393#393: *88 http2 frame sent: 00007FF053238370 sid:9 bl:1 len:83 2019/03/22 11:11:06 [debug] 393#393: *88 http proxy filter init s:400 h:0 c:0 l:-1

lifecyclrotten

Source

kaiyuanneo

Most helpful comment

I was facing the same problem. It turned out I had to set

proxy-add-original-uri-header: "false"

in the configmap, so that upstream didn't have to handle super long headers.

Reasno on 27 Sep 2019

👍2

All 8 comments

@kaiyuanneo please post the complete ingress rule, second, please update to 0.23.0.
Also please check the nginx.ingress.kubernetes.io/rewrite-target: / syntax https://kubernetes.github.io/ingress-nginx/examples/rewrite/

aledbf on 22 Mar 2019

👍1

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 20 Jun 2019

@kaiyuanneo were you able to work around the issue, just ran into the same thing.

asafpelegcodes on 18 Jul 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 17 Aug 2019

@kaiyuanneo did you get to the bottom of this problem? We have something similar happening.

jondoveston on 12 Sep 2019

I was facing the same problem. It turned out I had to set

proxy-add-original-uri-header: "false"

in the configmap, so that upstream didn't have to handle super long headers.

Reasno on 27 Sep 2019

👍2

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 27 Oct 2019

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.