Ingress-nginx: Disable URL decoding

Created on 28 Oct 2017 · 13Comments · Source: kubernetes/ingress-nginx

Is this a BUG REPORT or FEATURE REQUEST?
Maybe a bug or a question.

NGINX Ingress controller version:
gcr.io/google_containers/nginx-ingress-controller:0.8.3

Kubernetes version (use kubectl version):
1.7.2

Environment:
Kubernetes cluster on AWS. I have an ELB in front of the cluster

What happened:
I deployed scormengine inside my k8s cluster and I have an url-encoded curl request to scormengine service.

curl -u user:pwd http://xxxxxx.com/scormengine/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91701.

I can see the correct log from nginx ingress controller. Here is the logs.
GET /scormengine/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91761/launchLink?forceReview=false. You can see the encorded values in the in the url (_learning-activity%3A12721%3Buser%3A91761_).

Then nginx ingress controller proxy into scormengine service accoring to the below rule.

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: qc-scormengine-ingress
      labels:
        app: qc-scormengine
      annotations:
        kubernetes.io/ingress.class: nginx
        ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - host: "kube-qc.bluedrop360.com"
        http:
          paths:
          - path: /scormengine
            backend:
              serviceName: qc-scormengine-svc
              servicePort: 80
          - path: /scormengine/*
            backend:
              serviceName: qc-scormengine-svc
              servicePort: 80

when it comes to scormengine pod logs, what I can see is

httpMethod":"GET","url":"http://xxxxxx.com/api/v1/default/registrations/learning-activity:12721;user:91761"

So the values are decoded here. Im suspecting that nginx ingress did url decoding.

How can I disable URL decoding on nginx ingress controller ?

What you expected to happen:
I need to send the same url to scormengine without decoding it, which means I need to hit

httpMethod":"GET","url":"http://xxxxxx.com/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91761" to scormengine pod.

Anything else we need to know:
Few sections from nginx ingress controller config file
```
location ~* /scormengine/* {

    proxy_set_header Host                   $host;

    # Pass Real IP
    proxy_set_header X-Real-IP              $remote_addr;

    # Allow websocket connections
    proxy_set_header                        Upgrade           $http_upgrade;
    proxy_set_header                        Connection        $connection_upgrade;

    proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host       $host;
    proxy_set_header X-Forwarded-Port       $server_port;
    proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

    # mitigate HTTPoxy Vulnerability
    # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
    proxy_set_header Proxy                  "";

    proxy_connect_timeout                   5s;
    proxy_send_timeout                      60s;
    proxy_read_timeout                      60s;

    proxy_redirect                          off;
    proxy_buffering                         off;

    proxy_http_version                      1.1;

rewrite /scormengine/*/(.*) /$1 break;
rewrite /scormengine/* / break;
proxy_pass http://qc-qc-scormengine-svc-80;

}


location ~* /scormengine {
    proxy_set_header Host                   $host;

    # Pass Real IP
    proxy_set_header X-Real-IP              $remote_addr;

    # Allow websocket connections
    proxy_set_header                        Upgrade           $http_upgrade;
    proxy_set_header                        Connection        $connection_upgrade;

    proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host       $host;
    proxy_set_header X-Forwarded-Port       $server_port;
    proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

    # mitigate HTTPoxy Vulnerability
    # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
    proxy_set_header Proxy                  "";

    proxy_connect_timeout                   5s;
    proxy_send_timeout                      60s;
    proxy_read_timeout                      60s;

    proxy_redirect                          off;
    proxy_buffering                         off;

    proxy_http_version                      1.1;

rewrite /scormengine/(.*) /$1 break;
rewrite /scormengine / break;
proxy_pass http://qc-qc-scormengine-svc-80;

}

upstream qc-qc-scormengine-svc-80 {
    least_conn;
    server XXX.XX.X.XXX:8880 max_fails=0 fail_timeout=0;

}

```

Source

gayan415

👍2

All 13 comments

@thegayanj please update to the latest 0.9 beta version.
Also please remove the /scormengine/* path. That is not required. In nginx only when you use regex in the mappings (not supported here)

aledbf on 28 Oct 2017

Hey @aledbf - Let me try and let you know. Thank you.

gayan415 on 28 Oct 2017

Hey @aledbf.
nginx-ingress controller image changed to gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15.

Still experiencing the same issue.

New nginx ingress config

location ~* ^/scormengine\/?(?<baseuri>.*) {

    set $proxy_upstream_name "qc-qc-scormengine-svc-80";

    set $namespace      "qc";
    set $ingress_name   "qc-scormengine-ingress";
    set $service_name   "";

    port_in_redirect off;

    client_max_body_size                    "1m";

    proxy_set_header Host                   $best_http_host;
    # Pass the extracted client certificate to the backend

    # Allow websocket connections
    proxy_set_header                        Upgrade           $http_upgrade;
    proxy_set_header                        Connection        $connection_upgrade;

    proxy_set_header X-Real-IP              $the_real_ip;
    proxy_set_header X-Forwarded-For        $the_real_ip;
    proxy_set_header X-Forwarded-Host       $best_http_host;
    proxy_set_header X-Forwarded-Port       $pass_port;
    proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
    proxy_set_header X-Original-URI         $request_uri;
    proxy_set_header X-Scheme               $pass_access_scheme;

    # Pass the original X-Forwarded-For
    proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

    # mitigate HTTPoxy Vulnerability
    # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
    proxy_set_header Proxy                  "";

    # Custom headers to proxied server

    proxy_connect_timeout                   5s;
    proxy_send_timeout                      60s;
    proxy_read_timeout                      60s;

    proxy_redirect                          off;
    proxy_buffering                         off;
    proxy_buffer_size                       "4k";
    proxy_buffers                           4 "4k";
    proxy_request_buffering                 "on";

    proxy_http_version                      1.1;

    proxy_cookie_domain                     off;
    proxy_cookie_path                       off;

    # In case of errors try the next upstream server before returning an error
    proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

rewrite /scormengine/(.*) /$1 break;
rewrite /scormengine / break;
proxy_pass http://qc-qc-scormengine-svc-80;
}

upstream qc-qc-scormengine-svc-80 {
    # Load balance algorithm; empty for round robin, which is the default
    least_conn;

    keepalive 32;

    server XXX.XX.X.XXX:8880 max_fails=0 fail_timeout=0;
}

any other suggestions?

gayan415 on 28 Oct 2017

How do I know that this is an ingress issue

Get the scormengine pod internal IP by running kubectl exec -it qc-scormengine-pod -n qc cat /etc/hosts.
exec into ingress controller - kubectl exec -it ingress_ctr_pod /bin/sh
curl to scormengine with encoded URL from ingress controller - curl -u user:pwd http://XXX.XX.X.XXX:8880/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91701/launchLink?forceReview=false.
Its working as expected and I can see the url encoded curl request from scormengine pod and scormengine returned 200.

Which means ingress controller decode the URL before forwarding to scormengine pod.

gayan415 on 28 Oct 2017

@thegayanj to avoid the decoding you can:

remove the rewrite annotation (this is the reason why the uri is decoded)
create a custom template and use something like:
```
map '' $seed_uri {
default $request_uri;
}

proxy_pass http://qc-qc-scormengine-svc-80$seed_uri;
````

aledbf on 28 Oct 2017

Thank you for your suggestion @aledbf. I created a custom template and I referred this example.

map '' $seed_uri {
        default $request_uri;
}

Now I have a new map (as above) on ingres controller config file (/etc/nginx/nginx.conf).
I'm not sure how can I config my ingress to get proxy_pass http://qc-qc-scormengine-svc-80$seed_uri; Because proxy_pass created in pkg/nginx/template/template.go . I tried few ways and no luck, couldn't find a good example either.
Here is my ingress manifest.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: qc-scormengine-ingress
  labels:
    app: qc-scormengine
spec:
  rules:
  - host: "xxxxxxxx.com"
    http:
      paths:
      - path: /scormengine
        backend:
          serviceName: qc-scormengine-svc
          servicePort: 80

Can you instruct/guide me to do that please, thank you and I really appreciate that.

gayan415 on 28 Oct 2017

@aledbf , Okay.

Here is my nginx ingress controller config file now.

map '' $seed_uri {
    default          $request_uri;
}

upstream qc-qc-scormengine-svc-80 {
    # Load balance algorithm; empty for round robin, which is the default
    least_conn;

    keepalive 32;

    server xxx.xx.x.xxx:8880 max_fails=0 fail_timeout=0;
}

server {
    server_name xxxxxxx.com;
    listen 80;
    listen [::]:80;
    set $proxy_upstream_name "-";
    location /scormengine {
        set $proxy_upstream_name "qc-qc-scormengine-svc-80";

        set $namespace      "qc";
        set $ingress_name   "qc-scormengine-ingress";
        set $service_name   "qc-scormengine-svc";

        port_in_redirect off;

        client_max_body_size                    "1m";

        proxy_set_header Host                   $best_http_host;

        # Pass the extracted client certificate to the backend

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;
        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Real-IP              $the_real_ip;
        proxy_set_header X-Forwarded-For        $the_real_ip;
        proxy_set_header X-Forwarded-Host       $best_http_host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
        proxy_set_header X-Original-URI         $request_uri;
        proxy_set_header X-Scheme               $pass_access_scheme;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers to proxied server

        proxy_connect_timeout                   5s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      60s;

        proxy_redirect                          off;
        proxy_buffering                         off;
        proxy_buffer_size                       "4k";
        proxy_buffers                           4 "4k";

        proxy_http_version                      1.1;

        proxy_cookie_domain                     off;
        proxy_cookie_path                       off;

        # In case of errors try the next upstream server before returning an error
        proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
        # proxy_pass
        proxy_pass http://qc-qc-scormengine-svc-80$seed_uri;
    }
}

And my ingress rule is

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: qc-scormengine-ingress
  labels:
    app: qc-scormengine
spec:
  rules:
  - host: "xxxxxxx.com"
    http:
      paths:
      - path: /scormengine
        backend:
          serviceName: qc-scormengine-svc
          servicePort: 80

Then I ran curl - curl -u user:pwd https://xxxxxxx.com/scormengine/api/v1/default/ping.

This curl command is not hitting scormengine pod, any suggestions why?

gayan415 on 30 Oct 2017

Any updates on this?
Or a workaround of some kind?

Platzii on 13 Nov 2017

Nope 😐

gayan415 on 13 Nov 2017

@thegayanj URL decoding is disabled if you remove the rewrite annotation. There's no workaround for that (NGINX restriction)

aledbf on 13 Nov 2017

😕2

@aledbf You mentioned this could be fixed by having a custom template. Could you give some more info on that? What's the impact on other (already existing) ingresses?

I guess with every update from the ingress controller, the custom template should be updated too then?

Platzii on 14 Nov 2017