Ingress-nginx: 400 error (http request to https port) on GKE

[Nick-Harvey]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

NGINX Ingress controller version: 0.9.0-beta.15

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.5-gke.0", GitCommit:"2c2a807131fa8708abc92f3513fe167126c8cce5", GitTreeState:"clean", BuildDate:"2017-12-19T20:05:45Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  GKE
• OS (e.g. from /etc/os-release):

BUILD_ID=10032.71.0
NAME="Container-Optimized OS"
KERNEL_COMMIT_ID=c4c6234ae4f384ce00819c41b48ca8f6f1fa3ba8
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=63
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Container-Optimized OS from Google"
VERSION=63
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=cos

• Kernel (e.g. uname -a):

4.4.86+ #1 SMP Thu Dec 7 20:11:11 PST 2017 x86_64 Intel(R
) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux

• Install tools:
  Helm install of the nginx-ingress - https://hub.kubeapps.com/charts/stable/nginx-ingress

• Others:
  Ingress points to a nginx reverse proxy that splits traffic to two different services (Portus application, and Docker registry)

here is the nginx conf file

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "nginx.fullname" . }}
  labels:
    app: {{ template "name" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  conf: |
    # This file is largely based on the one written by @Djelibeybi in:
    #      https://github.com/Djelibeybi/Portus-On-OracleLinux7/

    events {
      worker_connections 1024;
    }

    http {
      default_type  application/octet-stream;
      charset       UTF-8;

      # Some basic config.
      server_tokens off;
      sendfile      on;
      tcp_nopush    on;
      tcp_nodelay   on;

      # On timeouts.
      keepalive_timeout     65;
      client_header_timeout 240;
      client_body_timeout   240;
      fastcgi_read_timeout  249;
      reset_timedout_connection on;

      ## Set a variable to help us decide if we need to add the
      ## 'Docker-Distribution-Api-Version' header.
      ## The registry always sets this header.
      ## In the case of nginx performing auth, the header will be unset
      ## since nginx is auth-ing before proxying.
      map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
        '' 'registry/2.0';
      }

      upstream {{ template "portus.fullname" . }} {
        least_conn;
        server {{ template "portus.fullname" . }}:{{ .Values.portus.service.port }} max_fails=3 fail_timeout=15s;
      }

      upstream {{ template "registry.fullname" . }}:{{ .Values.registry.service.port }} {
        least_conn;
        server {{ template "registry.fullname" . }}:{{ .Values.registry.service.port }} max_fails=3 fail_timeout=15s;
      }

      server {
        server_name {{ template "nginx.fullname" . }};

        #{{- if .Values.portus.tls.enabled }}
        listen {{ .Values.nginx.service.port }} ssl http2;

        ##
        # SSL

        ssl on;

        # Certificates
        ssl_certificate /certificates/portus.crt;
        ssl_certificate_key /certificates/portus.key;

        # Enable session resumption to improve https performance
        #
        # http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;

        # Enables server-side protection from BEAST attacks
        # http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html
        ssl_prefer_server_ciphers on;

        # Disable SSLv3 (enabled by default since nginx 0.8.19)
        # since it's less secure than TLS
        # http://en.wikipedia.org/wiki/Secure_Sockets_Layer#SSL_3.0
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        # Ciphers chosen for forward secrecy and compatibility.
        #
        # http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        #{{- else }}
        #listen {{ .Values.nginx.service.port }} http2;
        #{{- end }}

        ##
        # Docker-specific stuff.

        proxy_set_header Host $http_host;   # required for Docker client sake
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Scheme $scheme;

        # disable any limits to avoid HTTP 413 for large image uploads
        client_max_body_size 0;

        # required to avoid HTTP 411: see Issue #1486
        # (https://github.com/docker/docker/issues/1486)
        chunked_transfer_encoding on;

        ##
        # Custom headers.

        # Adding HSTS[1] (HTTP Strict Transport Security) to avoid SSL stripping[2].
        #
        # [1] https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security
        # [2] https://en.wikipedia.org/wiki/SSL_stripping#SSL_stripping
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";

        # Don't allow the browser to render the page inside a frame or iframe
        # and avoid Clickjacking. More in the following link:
        #
        # https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options
        add_header X-Frame-Options DENY;

        # Disable content-type sniffing on some browsers.
        add_header X-Content-Type-Options nosniff;

        # This header enables the Cross-site scripting (XSS) filter built into
        # most recent web browsers. It's usually enabled by default anyway, so the
        # role of this header is to re-enable the filter for this particular
        # website if it was disabled by the user.
        add_header X-XSS-Protection "1; mode=block";

        # Add header for IE in compatibility mode.
        add_header X-UA-Compatible "IE=edge";

        # Redirect (most) requests to /v2/* to the Docker Registry
        location /v2/ {
          # Do not allow connections from docker 1.5 and earlier
          # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
          if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
            return 404;
          }

          ## If $docker_distribution_api_version is empty, the header will not be added.
          ## See the map directive above where this variable is defined.
          add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

          {{- if .Values.portus.tls.enabled }}
          proxy_pass https://{{ template "registry.fullname" . }}:{{ .Values.registry.service.port }};
          {{- else }}
          proxy_pass http://{{ template "registry.fullname" . }}:{{ .Values.registry.service.port }};
          {{- end }}

          proxy_set_header Host $http_host;   # required for docker client's sake
          proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_read_timeout 900;
          proxy_buffering on;
        }

        # Portus needs to handle /v2/token for authentication
        location = /v2/token {
          {{- if .Values.portus.tls.enabled }}
          proxy_pass https://{{ template "portus.fullname" . }};
          {{- else }}
          proxy_pass http://{{ template "portus.fullname" . }};
          {{- end }}

          proxy_set_header Host $http_host;   # required for docker client's sake
          proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_read_timeout 900;
          proxy_buffering on;
        }

        # Portus needs to handle /v2/webhooks/events for notifications
        location = /v2/webhooks/events {
          {{- if .Values.portus.tls.enabled }}
          proxy_pass https://{{ template "portus.fullname" . }};
          {{- else }}
          proxy_pass http://{{ template "portus.fullname" . }};
          {{- end }}

          proxy_set_header Host $http_host;   # required for docker client's sake
          proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_read_timeout 900;
          proxy_buffering on;
        }

        # Portus handles everything else for the UI
        location / {
          try_files $uri/index.html $uri.html $uri @{{ template "portus.fullname" . }};
        }

        location @{{ template "portus.fullname" . }} {
          {{- if .Values.portus.tls.enabled }}
          proxy_pass https://{{ template "portus.fullname" . }};
          {{- else }}
          proxy_pass http://{{ template "portus.fullname" . }};
          {{- end }}

          proxy_set_header Host $http_host;   # required for docker client's sake
          proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_read_timeout 900;
          proxy_buffering on;
        }
      }
    }

Section of the values.yaml file related to the ingress.

nginx:
  replicas: 1

  ## image configuration.
  ##
  image:
    repository: "library/nginx"
    tag: "alpine"
    pullPolicy: "IfNotPresent"

  ## Service configuration.
  ##
  service:
    ## Set to ClusterIP if using ingress, or NodePort if using minikube
    ##
    type: "ClusterIP"

    port: "80"
    # nodePort:

  ## in order to access the docker registry from outside of the cluster
  ## if ingress is enabled set host to the domain you are using
  ## if NodePort is being used set host to the ip address of a cluster node
  ##
  #host: foobar

  ## ingress configuration.
  ##
  ingress:
    enabled: true

    ## Anntations to be added to the web ingress
    ##
    annotations:
      kubernetes.io/ingress.class: "nginx"
      #ingress.kubernetes.io/ssl-passthrough: "true"
      kubernetes.io/tls-acme: "true"
      #kubernetes.io/ssl-passthrough: "true"


    ## TLS configuration
    ## the ingress host must be covered by the key/cert in order for TLS to work properly
    ##
    tls:
      enabled: true

      ## Secrets containing SSL key and cert must be manually created in the namespace
      ##
      secretName: portus-tls
      host:
        - reg-foobar
        - reg-foobar-registry
        - reg-foobar-nginx
  ## Resource configuration.
  ##
  resources:
    requests:
      memory: "512Mi"
      cpu: "300m"

I've tried both 80 and 443 as the cluster IP value and the results are the same.

What happened: Upon launching the application and then trying to access the application through chrome at https://foo.bar.com/, I get the following:

400 Bad Request
The plain HTTP request was sent to HTTPS port

What you expected to happen: Application to load successfully

How to reproduce it (as minimally and precisely as possible):
install the application via the following helm chart: https://github.com/kubic-project/caasp-services/tree/master/contrib/helm-charts/portus
on GKE version mentioned above

Anything else we need to know:
The logs from the ingress controller show the following, indicating that the request isn't making its way past the ingress:

PIVersion:"extensions", ResourceVersion:"101228", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/foo-bar-nginx
10.128.0.5 - [10.128.0.5] - - [08/Jan/2018:15:19:01 +0000] "GET / HTTP/2.0" 400 666 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36" 221 0.002 [default-foo-bar-nginx-80] 10.52.3.16:80 666 0.002 400
10.128.0.5 - [10.128.0.5] - - [08/Jan/2018:15:19:01 +0000] "GET /favicon.ico HTTP/2.0" 400 666 "https://foo.bar.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36" 75 0.000 [default-reg-projecticarus-nginx-80] 10.52.3.16:80 666 0.000 400

I've looked at https://github.com/kubernetes/ingress-nginx/issues/918 and the fixes there only seem to apply to AWS where this is running in GCE (more specifically GKE) so the annotations there dont seem to apply

[Nick-Harvey]

Upon further investigation, it looks like the ingress controller is making a http request:

1. ssh into the ingress controller pod
2. curl the nginx reverse proxy

* Rebuilt URL to: 10.52.0.25/
*   Trying 10.52.0.25...
* Connected to 10.52.0.25 (10.52.0.25) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.52.0.25
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 400 Bad Request
< Server: nginx
< Date: Mon, 08 Jan 2018 17:53:36 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 264
< Connection: close

[aledbf]

@Nick-Harvey if the service behind the ingress requires a TLS connection you need to add the annotation nginx.ingress.kubernetes.io/secure-backends: "true"

[tomekit]

I am using the helm installed NginX:
helm install --name ingress --namespace ingress-nginx --set rbac.create=true,controller.kind=DaemonSet,controller.service.type=ClusterIP,controller.hostNetwork=true stable/nginx-ingress

Unfortunately adding the: nginx.ingress.kubernetes.io/secure-backends: "true" annotation doesn't solve the problem for me.

Where can I find more about: secure-backends annotation?
Can't see it listed here:
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

[aledbf]

@tomekit secure-backends was removed in 0.20. Please use https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-protocol

[rajdas98]

@tomekit secure-backends was removed in 0.20. Please use https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-protocol

If we change the tartgetPort:80 then this will redirect to http. But the use case is http---> https

[liozzazhang]

@tomekit secure-backends was removed in 0.20. Please use https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-protocol

thanks, that solves my problem