Kubernetes-ingress: Same host rule in multiple ingress objects

[thetechnick]

In Kubernetes it is possible to create multiple ingress objects with rules referencing the same host. How the nginx ingress controller should handle this case is not written down in the kubernetes documentation (at least I cant find it).

The official kubernetes nginx ingress controller old repo or new repo and the gce ingress controller are merging multiple ingress configurations into one nginx server object. So to specify one of the examples in multiple ingress objects is valid and leads to the expected behavior.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress1
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress2
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

The nginxinc ingress controller does not merge these rules. If there is more than one ingress object defined, it will write multiple separate server objects into the nginx config folder.

Nginx is not happy about this :P
conflicting server name "example.com" on 0.0.0.0:80, ignored

Some tools like kube-lego rely on the merging behavior to work, so simply ignoring additional ingress objects is not the best solution.

Before implementing this merge feature, it is necessary to formulate exactly how the nginx ingress controller should behave and how the order of merging is determined.

Collisions or conflicts should be dealt with gracefully, so the ingress controller is not "blocked" while a issue exists and detailed log entries should be generated, so users are able to monitor the ingress controller.

[pleshakov]

@thetechnick

Considering that we can end up having duplicated locations in multiple Ingress resources as well as annotations if we start to heavily use annotations to extend Ingress features, merging can become problematic. Agree that if we want to enable it,

it is necessary to formulate exactly how the nginx ingress controller should behave and how the order of merging is determined.

From my understanding only the contrib nginx controller currently do merging. For the gce one, a separate GCE HTTP load balancer is allocated.

I think the default behavior should be to not allow the same host names. If the user creates an Ingress with a duplicated hostname, the controller should Ignore it, reporting an Error.

However, if merging is very useful, it makes sense to add an option to enable it.

Please help me understand how kube-lego relies on the merging behavior to work.

[thetechnick]

@pleshakov
I think merging is a very useful feature, because I users would expect it to work.
How the ingress controller supports the sometimes very complex rules is up to the developer, but everything the kubernetes api allows should also be working with every ingress controller.

Kube-lego relies on the merging functionality, to add the acme challenge endpoints to the ingress points, using a separate ingress object in the namespace kube-lego runs in.

Example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme-challenge-endpoints: "true"
  name: kube-lego-nginx
spec:
  rules:
  - host: one.example.com
    http:
      paths:
      - backend:
          serviceName: kube-lego-nginx
          servicePort: 8080
        path: /.well-known/acme-challenge
  - host: two.example.com
    http:
      paths:
      - backend:
          serviceName: kube-lego-nginx
          servicePort: 8080
        path: /.well-known/acme-challenge
status:
  loadBalancer: {}

Edit:
I do not think that the merging will become too difficult, if the merging strategy is well defined.
To implement #77 I am using the oldest server (after ingress creation time) as a base and only add/override locations from the other ingress objects.
This way only the options that apply to those locations are taken over to the new merged server.

It is also possible to add config options for the merging strategy and to implement a new annotation: nginx.org/merge-weight: '200' (or similar) to give the user more power over the merging process.

[pleshakov]

@thetechnick
Thanks for the clarification with the kube-lego. I think a better approach for now is to treat it as a special case. If the kubernetes.io/tls-acme-challenge-endpoints: "true" is present, add this location to the config for the main Ingress resource, for which the kubernetes.io/tls-acme: "true" is present.

I would wait to see the results of https://github.com/kubernetes/kubernetes/issues/30151 and https://github.com/kubernetes/ingress/issues/20 so that we have a behavior that is clearly defined and consistent across the controllers.

The error reporting in case of multiple Ingresses with the same hostnames must be added and the configs must not be generated instead of what we have know.

[thetechnick]

@pleshakov
I would rather not add special cases depending on other projects, if there are other projects out there that need this feature, this will become a mess quite fast. Also if kube-lego decides to rename their annotations, it would break the special case here...

I am really curious about the future of the kubernetes ingress resource, but these new changes will not arrive in kubernetes before 1.6 or 1.7 and will very likely include changes in the API objects.
Because the official solution to the merging problems is so far away, I would welcome this ingress controller to adapt the "non official" strategy of merging the ingress objects/generated configs ordered by age, which is used by the contrib nginx ingress controller and openshift.

I know this is not ideal (I really dislike this lack of undefined behavior) ,but I see no downside of going with the established pattern until a more robust and easy to use solution is available.

After the official solution has arrived I would be glad to help with the implementation.

[kartojal]

Any work on this? Facing some issues with kube-lego and nginxinc/kubernetes-ingress. There is conflicts in my actual config, because kube-lego config creates a new configuration file, generating a name server conflict, so i can not redirect from HTTP to HTTPS.

Example:
When you go to https://example.com works good, but http://example.com does not do the redirection to the https maybe because its reading the kube-lego configuration with only the location /.well-known .

Any workaround? Should i use the kubernetes/ingress controller in my use case?

[pleshakov]

@kartojal
Please use the kubernetes/ingress controller for your use case. The necessary behavior for kube-lego to work properly is not supported by our controller.

[nikitsrj]

I have created two different namespaces for different environment. one is devops-qa and another is devops-dev. I created two ingress in different namespaces. So while creating ingress of qa env in devops-qa namespace, the rules written inside ingress of qa is working fine. Means I am able to access the webpage of qa env. The moment I will create the ingress of dev env in devops-dev namespace, I will be able to access the webpage of dev env but wont be able to access the webpage of qa. And when I delete the dev ingress then again I will be able to access the qa env website

Below is the ingree of both dev and qa env.

Dev Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress-dev
  namespace: devops-dev
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: default-token-jhtx3
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: miqpdev-svc
          servicePort: 80

QA Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress-qa
  namespace: devops-qa
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: default-token-bjg3z
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /greentea
        backend:
          serviceName: greentea-svc
          servicePort: 80
      - path: /blackcoffee
        backend:
          serviceName: blackcoffee-svc
          servicePort: 80

The token mentioned in the ingress file is of each namespace. How can i run both the ingress and will be able to get all the websites deployed in both dev and qa env ?

[pleshakov]

@nikitswaraj12345
Unfortunately, it is not possible to use multiple Ingress resources with the same host name, like cafe.example.com in your case. One of the resources will simply overwrite the others. Is it important for you to use the same host name?

If you would like to use the same host name in multiple namespaces, I can only suggest deploying a separate Ingress controller per each namespace.

[nikitsrj]

@pleshakov But with different hostname as well its overriding :( I tried both .

[pleshakov]

with different hostnames it should not override. When you make a request to the Ingress controller, are you setting the host header of the request to the correct value -- either <hostname1> or <hostname2>?

[paulwalker]

Can someone please tell me how to configure use of the kubernetes/ingress vs the nginx ingress controller? All of the links to the differing documentation seem to be broken right now.

[pleshakov]

@paulwalker the kubernetes/ingress repo has been renamed. Please check https://github.com/kubernetes/ingress-nginx

[isaachawley]

I believe this is now handled by mergable ingresses.