Ingress-nginx: Cannot get multi-path working

[erik777]

I have tried every configuration in the universe.

"work": Can access all URLs without an error in multiple browsers and with curl.

"does not work": Sends back 404 or 502 errors. If you refresh on some URLs with some configs, it can toggle between 3 different error results, two are different 404s.

I have two backend services. They work consistently with port-forward, LoadBalance in the service, and basic ingress without paths.

I get the root to map, but not any service below that in the path.

What I want is "/" to route to a node app in nginx and "/service2" and subsequent services to route to various services used by that app. I couldn't even get "/service1" and "/service2" working without the root path, which matched the examples. I tried following every example, then experimented into the unknown. No matter what I did, no path other than the root one would map successfully to a service.

I am running K8S 1.6.7 on GKE. Having ingress.kubernetes.io/rewrite-target: / doesn't help. When these errors occur, it is simply not getting to the service.

When I am inside a pod, I can curl port 80 of a service no problem. curl -v http://myservice returns the expected 200. When I get a 200, on the node app, I get a log from nginx.

Note that I did not create any controllers. I am just using gce. Nevertheless, I tried adding kubernetes.io/ingress.class: "gce" with no effect.

I spent over 8 hours trying every possibility I could think of. I deleted and re-created the ingress probably 100-200 times. I even deleted and re-created the cluster.

Going back to the original configuration, where you cannot access either service via the Ingress LB, you consistently get 404 when trying to access the service. I have noticed it is slightly different when I include a slash at the end. Without it, it returns this to curl:

```< HTTP/1.1 404 Not Found
< Content-Length: 0
< Date: Sat, 12 Aug 2017 17:42:38 GMT
< Via: 1.1 google

When I add a slash, I get that with the content `default backend - 404`.  E.g.,

    curl -v http://host/collector/

Here is the original multi-path config:

```apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-combined
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.global-static-ip-name: sc-static-ip
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: sc-admin
          servicePort: 80
      - path: /collector
        backend:
          serviceName: sc-collector
          servicePort: 80

This basic config works no matter which service I point it to:

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: basic-ingress annotations: kubernetes.io/ingress.global-static-ip-name: sc-static-ip spec: backend: serviceName: sc-admin servicePort: 80

It is only multi-path with anything other than path: / that does not work.

Thank you in advance for your help.

[erik777]

In an interesting twist, I switch the services. It invokes both services for the path root. But:

• Returns default backend - 404 if I then try to access anything other than the service root.
• includes the path instead of doing a re-write, ignoring the annotation ingress.kubernetes.io/rewrite-target: /.

I could work around the second problem by giving each service a context path if the health checks didn't require a 200 from the root.

[aledbf]

I am running K8S 1.6.7 on GKE. Having ingress.kubernetes.io/rewrite-target: / doesn't help. When these errors occur, it is simply not getting to the service.

By default you get a GCE ingress controller running. Because of this:

• kubernetes.io/ingress.class: "gce" annotation has no effect
• ingress.kubernetes.io/rewrite-target annotation is not supported by the GCE ingress controller

[erik777]

I found a workaround. Using these paths works:

  - path: /*
  - path: /test/*

Note that http://host/test still resolves to the first service, whereas http://host/test/ resolves to the second service. The services don't have to be human friendly URLs, so that is not an issue.

I clicked on every link in the Node app and couldn't locate a single 404! YAY! curl seemed happy, too.

The question I have is how come I couldn't find an example using a wildcard in the path in all the documentation on Ingress? The only reason I thought of it was because I figured out that GKE was using GCE URL Maps under the hood, and learned that it supports wild cards.

I did locate these discussions, which leave this as an open issue.

It has just been a long 10+ hour confusing ride due to documentation not being clear on this. And, to be sure, Google is #1 to earn credit here because I started out with and learned about Ingress from their GKE Setting up HTTP Load Balancing with Ingress documentation where their fanout example shows a root and a context under it pointing to two services WITHOUT asterisks! That was the example I started with, and the config I have been trying to get working. They really do need to update that page, and explain the impact of having or not having a wild card.

Hopefully, someone finds this issue in the first hour of their Ingress adventure. :)

[anderfsilva]

I am facing the same issue using Nginx Ingress on AWS. Unfortunately it took me 10+ hours to find this thread. :(

[ryan-flores]

I sorted out by adding nginx.ingress.kubernetes.io/rewrite-target annotation. I'm using LetsEncrypt and not allowing http, but https.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
namespace: default
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
ingress.kubernetes.io/service-upstream: "true"
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /

[mleneveut]

I have a similar issue with Azure AKS (v1.9.6) and a nginx (nginx-ingress-controller-0.15.0) :

I want to have different paths to route requests to different backends.

The following configuration works, but when I request "https://xxx/iv-da/swagger", the _/swagger/v0/swagger.json_ AJAX call makes a request to "https://xxx/swagger/v0/swagger.json" without the "iv-da" context.

How could it keep the context for each backend's _inside_ requests ?

{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "k8s-ingress",
    "namespace": "default",
    "annotations": {
      "certmanager.k8s.io/cluster-issuer": "letsencrypt-prod",
      "kubernetes.io/ingress.class": "nginx",
      "nginx.ingress.kubernetes.io/rewrite-target": "/"
    }
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "xxx"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "xxx",
        "http": {
          "paths": [
            {
              "path": "/iv-da",
              "backend": {
                "serviceName": "iv-da",
                "servicePort": 80
              }
            },
            {
              "path": "/",
              "backend": {
                "serviceName": "config-server",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

[crsantini]

In my case, I had two services, one running at / and another at /ui . I tried several combinations and the only one that worked with K8s 1.10 was

http://mydomain.com/*
Service 1 Ingress
kubernetes.io/ingress.class: "nginx"
path: /

http://mydonain.com/ui/app/index.html
Service 2 Ingress
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /ui
path: /ui/*

Note that for service 2, my tomcat was also initialising at /ui , so the rewrite-target did the trick otherwise, I had to add two "ui" to my url like http://mydonain.com/ui/ui/app/index.html

[lemonshow]

@erik777

I got the same issue. The base works but the multi-path not. I am using Nginx ingress controller + AWS NLB and try to bring up my two services with the Ingress. It always routes to default and I am getting 404. Anyone used with AWS EKS with luck? Use rewrite did not help either. Any other suggestion? thanks,

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
namespace: local
annotations:
ingress.kubernetes.io/rewrite-target: '/'
spec:
#backend:
#serviceName: app1-service
#servicePort: 80
rules:

• http:
  paths:
  
  
  
  • backend:
    
    serviceName: app1-service
    
    servicePort: 80
    
    path: /
  
  
  • backend:
    
    serviceName: app2-service
    
    servicePort: 80
    
    path: /app2/
  
  

[aledbf]

@lemonshow remove * from the paths

[lemonshow]

It works now!
Actually I had another issue after this where I was testing with NLB but I comment out https.
I saw 308 redirect to https. Uncomment https, things start to work.
thanks a lot,

[lemonshow]

Maybe I missed somewhere, why I can see the default backend status(I have 2 instances in 2 zones) showing one healthy, one unhealthy in the AWS NLB?

[dolphub]

@mleneveut Did you figure out your issue yet? I am also interested in this. I'm serving a web application with socket connections and am unable to get socket.io requests to persist the proper url after using the rewrite target annotation.

[mleneveut]

@dolphub I ended up doing this :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: myingress
spec:
  rules:
  - host: myhost.com
    http:
      paths:
      - backend:
          serviceName: api
          servicePort: 80
        path: /domain-api
      - backend:
          serviceName: bff
          servicePort: 80
        path: /api
      - backend:
          serviceName: front
          servicePort: 80
        path: /
  tls:
  - hosts:
    - myhost.com
    secretName: tls-secret

And for the swagger problem, it is inside the code that we need to force the context, it is not an ingress problem.

Hope it helps.

[ryanQL]

Hi, I have the Nginx Ingress running without issues in Production and HTTPS by using LetsEncrypt certificates:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/service-upstream: "true"
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.allow-http: "false"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: "1M"
spec:
  tls:
    - hosts:
      - developer.test.com
      secretName: developer-test-com-tls
    - hosts:
      - broker.test.com
      secretName: broker-test-com-tls
    - hosts:
      - dashboard.test.com
      secretName: dashboard-test-com-tls
  rules:
    - host: developer.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: developer-docs-service
              servicePort: 8080
          - path: /api
            backend:
              serviceName: developer-service
              servicePort: 8080
    - host: broker.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: broker-rabbitmq-service
              servicePort: 15672
    - host: dashboard.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: dashboard-service
              servicePort: 80

I set up a quick repo with the resources I used for the production environment. You need to replace the "test" subdomain with your own and set up the SSL certificates.

Hope it helps.

[nimendra]

I sorted out by adding nginx.ingress.kubernetes.io/rewrite-target annotation. I'm using LetsEncrypt and not allowing http, but https.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
namespace: default
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
ingress.kubernetes.io/service-upstream: "true"
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /

to make matter worse for me, I had
ingress.kubernetes.io/rewrite-target: / instead, which was hard to spot!

[gpflaum]

@nimendra Thank you for leaving that comment! We too had this, which was working until we upgraded from an old ingress-nginx version (0.9.0-beta.11 -> 0.22.0):

ingress.kubernetes.io/rewrite-target: /

After many hours, I found your comment and changed to this, which works:

nginx.ingress.kubernetes.io/rewrite-target: /

[dillilabs]

Refer to Step 6 : Serving Multiple Applications on a Load Balancer on this Google Kubernetes Engine tutorial on Setting up HTTP Load Balancing with Ingress. It exactly answers the questions raised in this thread.

To route path / to backend service web on port 8080 and path /v2 to backend service web2 on port 8080 , the spec section of the ingress yaml file will be:

spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: web
          servicePort: 8080
      - path: /v2/*
        backend:
          serviceName: web2
          servicePort: 8080

Ensure that backend services respond with a 200OK for /request. Note that web2 needs to respond with a 200OK to a root path / ( not /v2/). This is important for Google Load Balancer to perform a successful health check on the two services before making them available. If they do not pass the health checks, you would get a 502 Error.

In our case, we have a angular frontend service with an express backend. We want to angular to serve /and express to serve /api for the same host.

So, in our express request handling file, we have the following added to respond with a 200OK for path /.

const app = express();
app.use('/', function (req, res, next) {
    res.sendStatus(200);
    next();
});

Hope this helps.

[vasudev-hv]

Ingress version 0.22.0 or higher has changed the way how rewrite-target works.
You'll need to regex-match the path and add it to the rewrite-target.
nginx.ingress.kubernetes.io/rewrite-target: /$2 and

  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

Refer to changelog here
How to article here

[felipecrescencio]

In my case, I based my solution in this tutorial and I have the following:

1. FRONTEND: React Application
   
   
   
   • Path: /
   
   
   • Full URL doesn't matter for application
   
   
2. BACKEND
   
   
   
   • Path: /api
   
   
   • Full URL MATTERS for application
   
   
3. I am using GCP

The only way it worked for me was creating 2 ingress, because
If I use nginx.ingress.kubernetes.io/rewrite-target: / It breaks my backend that needs full url path

If I use nginx.ingress.kubernetes.io/rewrite-target: /$2 It breaks my frontend, showing a blank home page.

My solution:

1. FRONTEND Ingress:

nginx.ingress.kubernetes.io/rewrite-target: /
path: /

1. BACKEND Ingress:

nginx.ingress.kubernetes.io/rewrite-target: /$2
path: /api(/|$)(.*)

I attached full ingress file yaml.

Hope it helps and thank you so much, this thread helped me a lot!

[LorenzoR]

@felipecrescencio if you create 2 ingress, do you have to pay for 2 load balancers in GCP?

[aledbf]

if you create 2 ingress, do you have to pay for 2 load balancers in GCP?

@LorenzoR no with ingress-nginx

[demisx]

If you are using v0.22+, you need to use capture groups. https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-rules
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - dl-web.stg.myhost.org
      secretName: letsencrypt-prod
  rules:
    - host: dl-web.stg.myhost.org
      http:
        paths:
          - path: /
            backend:
              serviceName: dl-web
              servicePort: 80
          - path: /api(/|$)(.*)
            backend:
              serviceName: dl-api
              servicePort: 3000

[aditanix]

I as well can confirm that I spent upwards of 8 hours on the same issue. Using /* instead of / was the solution.

[DaveSlinn]

Everyone is saying that using /* or /api/* works, but it's not working for me. This is my ingress def:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hmc-ingress
  namespace: hmc
spec:
  tls:
  - hosts: 
    - hmc.gms.ca
    secretName: ssl-secret
  rules:
  - host: hmc.gms.ca
    http:
      paths:
      - path: /hmc-shell/*
        backend:
          serviceName: hmc-shell-service
          servicePort: 80

when I go to https://hmc.gms.ca/hmc-shell I end up getting the default http backend and not the service running on hmc-shell-service.
When I remove the /* from the path to - path: /hmc-shell, I manage to download the default doc from my hmc-shell-service, but any subsequent request for other files, like css and js, don't return the proper content. They each just resend the default doc content.

Argh!! What am I doing wrong?

Update: I've tried pouring over the nginx documentation, and using a regex:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hmc-ingress
  namespace: hmc
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1$2$3
spec:
  tls:
  - hosts: 
    - hmc.gms.ca
    secretName: ssl-secret
  rules:
  - host: hmc.gms.ca
    http:
      paths:
      - path: /(hmc-shell)(/|$)(.*)
        backend:
          serviceName: hmc-shell-service
          servicePort: 80

gives me this really weird nginx.conf:

server {
    server_name hmc.gms.ca ;

    listen 80;

    listen [::]:80;

    set $proxy_upstream_name "-";

    listen 443  ssl http2;

    listen [::]:443  ssl http2;

    # PEM sha: 81da239514bc48d4f2c5497126606f4289d4a2c2
    ssl_certificate                         /etc/ingress-controller/ssl/hmc-ssl-secret.pem;
    ssl_certificate_key                     /etc/ingress-controller/ssl/hmc-ssl-secret.pem;

    location ~* "^/(hmc-shell)(/|$)(.*)" {

        set $namespace      "hmc";
        set $ingress_name   "hmc-ingress";
        set $service_name   "hmc-shell-service";
        set $service_port   "80";
        set $location_path  "/(hmc-shell)(/|${literal_dollar})(.*)";    

What's with the ${literal_dollar} in the location path? Is that an nginx thing?

[JorgeGuerrero-dev]

Ingress version 0.22.0 or higher has changed the way how rewrite-target works.
You'll need to regex-match the path and add it to the rewrite-target.
nginx.ingress.kubernetes.io/rewrite-target: /$2 and

  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

Refer to changelog here
How to article here

Yesss! This worked for me Thanks you!!