Kubernetes-ingress: Controller routing only to / (root)

[ajpinedam]

Describe the bug
I am running into the same issue and just noticed it's routing correctly only on /. Whenever I use something different as the path I receive 404.

This is my Ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/client-max-body-size: 4M
    cert-manager.io/cluster-issuer: letsencrypt-prod
    cert-manager.io/acme-http01-edit-in-place: "true"
  labels:
    name: ingress
spec:
  rules:
  - host: test.mydomain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: portal-client
          servicePort: 80
      - path: /api
        backend:
          serviceName: service-api
          servicePort: 80
  tls:
  - hosts:
    - test.mydomain.com
    secretName: test-mydomain-crt

In the sample above, the service under / (portal client) is successfully rendered but the other service under /api (service-api) does not.

If I move the service-api service to / then it works.

Please advise what is wrong.

Thanks.

To Reproduce
Steps to reproduce the behavior:

1. Deploy x to '...' using some.yaml
2. View logs on '....'
3. See error

Expected behavior
Controller is able to map to all the different paths.

Your environment

• Version of the Ingress Controller -
  1.6.1
• Version of Kubernetes:
  1.13.11-gke.23
• Kubernetes platform (e.g. Mini-kube or GCP):
  GCP
• Using NGINX or NGINX Plus
  NGINX

Additional context
Add any other context about the problem here. Any log files you want to share.

kubectl describe ingress ingress

Name:             ingress
Namespace:        default
Address:          xx.xxx.xx.xxx
Default backend:  default-http-backend:80 (10.32.1.7:8080)
TLS:
  test-mydomain-crt terminates test.mydomain.com
Rules:
  Host               Path  Backends
  ----               ----  --------
  test.mydomain.com
                     /      portal-client:80 (<none>)
                     /api   service-api:80 (<none>)
Annotations:
  kubernetes.io/ingress.class:                        nginx
  nginx.org/client-max-body-size:                     4M
  cert-manager.io/acme-http01-edit-in-place:          true
  cert-manager.io/cluster-issuer:                     letsencrypt-prod

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 22m (x18 over 178m) nginx-ingress-controller Configuration for default/ingress was added or updated

kubectl get services

NAME                               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE

 service-api                        ClusterIP      10.99.11.131   <none>          80/TCP                       171m
 kubernetes                         ClusterIP      10.99.0.1      <none>          443/TCP                      2d14h
 main-nginx-ingress-nginx-ingress   LoadBalancer   10.99.15.166   xx.xxx.xx.xxx   80:30579/TCP,443:31216/TCP   8h
portal-client                      ClusterIP      10.99.13.193   <none>          80/TCP                       4h18m

kubectl get endpoints

 NAME                               ENDPOINTS                      AGE
 service-api                        10.32.0.21:8002                175m
 kubernetes                         xx.xx.xx.xx:443                2d14h
 main-nginx-ingress-nginx-ingress   10.32.0.20:80,10.32.0.20:443   8h
 portal-client                      10.32.0.22:80                  4h22m

[Dean-Coakley]

Hi @ajpinedam

The configuration you supplied looks reasonable.

As a first step I'd validate if the 404 is issued by your application by checking the logs of your service-api.
If this doesn't provide you with any insights, perhaps you could share the manifests of your services, so I can see if there's anything problematic there.

You can find out more about troubleshooting methods here.

[ajpinedam]

@Dean-Coakley thanks for your response.

I looked at the logs but could not find anything related to the service-api service.

My service-api manifest looks as follow:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-api
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      name: service-api
  template:
    metadata:
      labels:
        name: service-api
    spec:
      volumes:
      - name: secret-key
        secret:
          secretName: secret-key
      containers:
      - name: service-api
        image: gcr.io/my-project/cloud-service-api:latest
        ports:
        - name: grpc
          containerPort: 8001
        - name: gateway
          containerPort: 8002
        resources:
          requests:
            cpu: 100m
            memory: 150Mi
          limits:
            cpu: 100m
            memory: 150Mi
        volumeMounts:
        - name: secret-key
          mountPath: /var/secrets/google
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/key.json

---
apiVersion: v1
kind: Service
metadata:
  name: service-api
  namespace: default
spec:
  selector:
    name: service-api
  ports:
  - name: https
    port: 80
    targetPort: gateway

As I mentioned above. If I change my ingress definition so this service responds on the root directory ( / ) I don't see any problem.

[Dean-Coakley]

@ajpinedam Right, those manifests look fine too.

I still expect your service-api backend is returning a 404 itself, as I think I expects the request at / , not /api.

Assuming that this is the case, I would recommend adding a rewrite to try solve it: https://github.com/nginxinc/kubernetes-ingress/tree/v1.6.1/examples/rewrites

Can you give that a try?

[ajpinedam]

@Dean-Coakley that was! the rewrites made it work.

I had tried it before since for the other controller (Kubernetes/Ingress) there's something similar but honestly, I did not get it the first time.

Thanks for your help.