Ingress-nginx: Exact and Prefix path matching is broken in ingress-nginx-3.8.0

Created on 6 Nov 2020  路  14Comments  路  Source: kubernetes/ingress-nginx

NGINX Ingress controller version: ingress-nginx-3.8.0

Kubernetes version (use kubectl version): v1.18.8 and v1.19.1

Environment:

  • Cloud provider or hardware configuration: baremetal
  • OS: CentOS 6 and Ubuntu 18.04
  • Kernel: 5.8.11-1.el7.elrepo.x86_64 and 5.4.0-52-generic
  • Install tools: rke and kind
  • Others: n/a

What happened:
After upgrading from ingress-nginx-3.7.1 to 3.8.0, Exact and Prefix path matching behaviour changed in a incompatible way.

What you expected to happen:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /foo
        pathType: Exact
        backend:
          serviceName: foo-service
          servicePort: 5678
      - path: /foo/
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

Give above Ingress rule:

  • /foo returns 200 in 3.7.1, while in 3.8.0 it returns 301 to /foo/
  • /foo/bar returns 200 in 3.7.1, while in 3.8.0 it returns 404

I would expect this behaviour to not change in such incompatible way in v3.8.0

How to reproduce it:

Install kind

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 8080
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
EOF

Install ingress controller 3.8.0

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/ingress-nginx-3.8.0/deploy/static/provider/kind/deploy.yaml

Install a foo application with Ingress

kind: Pod
apiVersion: v1
metadata:
  name: foo-app
  labels:
    app: foo
spec:
  containers:
  - name: foo-app
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=foo"
---
kind: Service
apiVersion: v1
metadata:
  name: foo-service
spec:
  selector:
    app: foo
  ports:
  # Default port used by the image
  - port: 5678
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /foo
        pathType: Exact
        backend:
          serviceName: foo-service
          servicePort: 5678
      - path: /foo/
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

Test v3.8.0

In v3.8.0

  • curl localhost:8080/foo returns 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
  • curl localhost:8080/foo/ returns foo
  • curl localhost:8080/foo/bar returns 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Test v3.7.1

  • kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/ingress-nginx-3.8.0/deploy/static/provider/kind/deploy.yaml
  • kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/ingress-nginx-3.7.1/deploy/static/provider/kind/deploy.yaml
    In v3.7.1
  • curl localhost:8080/foo returns foo
  • curl localhost:8080/foo/ returns foo
  • curl localhost:8080/foo/bar returns foo

Anything else we need to know:

I suspect https://github.com/kubernetes/ingress-nginx/commit/cdd6437380c2085ae1321a1f66ab188376c8fd1c is the cause:

/kind bug

kinbug
Source
picture kolorful
👍1

Most helpful comment

@aledbf I tested the image and it works!

Verified that the / prefix path is chosen when the /foo prefix path does not match but the path still starts with /foo (e.g. /foob):

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: bar-service
          servicePort: 5678
      - path: /foo
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

[me@localhost ~]$ curl localhost:8080/
bar
[me@localhost ~]$ curl localhost:8080/f
bar
[me@localhost ~]$ curl localhost:8080/fo
bar
[me@localhost ~]$ curl localhost:8080/foo
foo
[me@localhost ~]$ curl localhost:8080/foob
bar
[me@localhost ~]$ curl localhost:8080/foo/
foo
[me@localhost ~]$ curl localhost:8080/foo/bar
foo

Verified that the exact path "/foo" is chosen over the prefix path "/foo" but that the prefix path still matches any path that starts with "/foo/":

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /foo
        pathType: Exact
        backend:
          serviceName: bar-service
          servicePort: 5678
      - path: /foo
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

[me@localhost ~]$ curl localhost:8080/foo
bar
[me@localhost ~]$ curl localhost:8080/foo/
foo
[me@localhost ~]$ curl localhost:8080/foo/bar
foo
[me@localhost ~]$ curl localhost:8080/foobar
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
[me@localhost ~]$ # Above 404 is expected because there is no "/" prefix path.

I also checked that a prefix path with a trailing slash is treated identically to a prefix path without a trailing slash.

Thanks very much for the quick fix!

picture ailurarctos on 9 Nov 2020
👍3

All 14 comments

I would expect this behaviour to not change in such incompatible way in v3.8.0

Agree but unfortunately, this change is required to pass the ingress conformance tests
https://github.com/kubernetes-sigs/ingress-controller-conformance/blob/master/features/path_rules.feature

aledbf picture aledbf on 6 Nov 2020

hmm, the real world scenario I'm dealing with is that I have below Ingress config in v3.7.1:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: bar-service
          servicePort: 5678
      - path: /foo
        pathType: Exact
        backend:
          serviceName: foo-service
          servicePort: 5678
      - path: /foo/
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

so that:

  • /foo hits foo
  • /foo/ hits foo
  • /foo/bar hits foo
  • /fooooo hits bar
  • / hits bar
  • /f hits bar

In v3.8.0 I tweaked it to:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: bar-service
          servicePort: 5678
      - path: /foo
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

now that:

  • /foo hits foo
  • /foo/ hits foo
  • /foo/bar hits foo
  • /fooooo returns 404
  • / hits bar
  • /f hits bar

Is the 404 behaviour really expected or it's a bug?

kolorful picture kolorful on 6 Nov 2020
  • path: /foo/

Change that to path: /foo

aledbf picture aledbf on 6 Nov 2020

Is the 404 behaviour really expected or it's a bug?

Yes. Same reason https://github.com/kubernetes-sigs/ingress-controller-conformance/blob/master/features/path_rules.feature#L161-L165

Note: you can check the output of the tests here https://aledbf.github.io/ingress-conformance-sample/features/path-rules.html

aledbf picture aledbf on 6 Nov 2020

Do you mind sharing the reason behind this behaviour, cause it feels very nature to have /foooo fallback to the secondary match / instead of returning 404.

This also seems very confusing and dangerous to user, e.g.

  • I have an existing service A that serves /
  • Some other team spins up a service B and took over /f with Prefix pathType, now suddenly all requests prefixed with /f either hit B or return 404.
kolorful picture kolorful on 6 Nov 2020

These changes are related to Ingress V1
https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20190125-ingress-api-group.md

aledbf picture aledbf on 6 Nov 2020

This is topic still open for discussion or the decision is final?

kolorful picture kolorful on 6 Nov 2020

This is topic still open for discussion or the decision is final?

There is nothing we can do about it. Ingress V1 was released in k8s 1.19.

aledbf picture aledbf on 6 Nov 2020

Is the 404 behaviour really expected or it's a bug?

I think this is an unexpected behavior in NGINX. I suggested removing that validation but it was rejected.

aledbf picture aledbf on 6 Nov 2020

ok, sigh... this 404 behaviour seems a bit wild to me cause imaging in order for a team to spin up a new service under a new path, they need to figure out all possible request paths it might overtake accidentally and add those rule individually as well.

Thanks for the explanation, I'll see what we can do

kolorful picture kolorful on 6 Nov 2020

@aledbf the test prefix /aaa does not match request /aaaccc in the output you linked does not use an Ingress with an additional / prefix path so a 404 makes sense (since no path matches).

Could you confirm that the following ingress should also return a 404 for /foobar even though there is a / prefix path?

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678
      - path: /foo
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

When I use the above Ingress with the setup steps @kolorful documented I get the following result:

[me@localhost ~]$ curl localhost:8080/
foo
[me@localhost ~]$ curl localhost:8080/f
foo
[me@localhost ~]$ curl localhost:8080/fo
foo
[me@localhost ~]$ curl localhost:8080/foo
foo
[me@localhost ~]$ curl localhost:8080/foob
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
  • / is 200 (matches / prefix path)
  • /f is 200 (matches / prefix path)
  • /fo is 200 (matches / prefix path)
  • /foo is 200 (matches / and /foo prefix paths)
  • /foob is 404 (matches / prefix path but does not match /foo prefix path)

This last point is what I'd like clarification on because NGINX is returning a 404 even though the path does match one of the ones specified.

ailurarctos picture ailurarctos on 7 Nov 2020

@kolorful @ailurarctos
Please test the fix (#6443) using the image gcr.io/k8s-staging-ingress-nginx/controller:v0.42.1-dev.0

aledbf picture aledbf on 8 Nov 2020

@aledbf I tested the image and it works!

Verified that the / prefix path is chosen when the /foo prefix path does not match but the path still starts with /foo (e.g. /foob):

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: bar-service
          servicePort: 5678
      - path: /foo
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

[me@localhost ~]$ curl localhost:8080/
bar
[me@localhost ~]$ curl localhost:8080/f
bar
[me@localhost ~]$ curl localhost:8080/fo
bar
[me@localhost ~]$ curl localhost:8080/foo
foo
[me@localhost ~]$ curl localhost:8080/foob
bar
[me@localhost ~]$ curl localhost:8080/foo/
foo
[me@localhost ~]$ curl localhost:8080/foo/bar
foo

Verified that the exact path "/foo" is chosen over the prefix path "/foo" but that the prefix path still matches any path that starts with "/foo/":

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - path: /foo
        pathType: Exact
        backend:
          serviceName: bar-service
          servicePort: 5678
      - path: /foo
        pathType: Prefix
        backend:
          serviceName: foo-service
          servicePort: 5678

[me@localhost ~]$ curl localhost:8080/foo
bar
[me@localhost ~]$ curl localhost:8080/foo/
foo
[me@localhost ~]$ curl localhost:8080/foo/bar
foo
[me@localhost ~]$ curl localhost:8080/foobar
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
[me@localhost ~]$ # Above 404 is expected because there is no "/" prefix path.

I also checked that a prefix path with a trailing slash is treated identically to a prefix path without a trailing slash.

Thanks very much for the quick fix!

ailurarctos picture ailurarctos on 9 Nov 2020
👍3

Thank you!

kolorful picture kolorful on 10 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

boazj picture boazj  路  3Comments
vdavidoff picture vdavidoff  路  3Comments
silasbw picture silasbw  路  3Comments
jwfang picture jwfang  路  3Comments
cabrinoob picture cabrinoob  路  3Comments