Ingress-nginx: [nginx] backup back-end when we have zero PODs

[yaronha]

Currently it seems like there must always be PODs up and running to serve HTTP requests, if not requests fail. Is there a way to define some back-end that will be serving the HTTP requests in case there are no end points (no PODs in ready state) ?

I assume we can use NGINX "backup" tag to mark alternate upstream server end points

[aledbf]

@yaronha that is the role of the default backend.
If you want, you can provide a custom image with a different response

[kfox1111]

@aledbf no, it isn't the same thing.

The appropriate failback behavior for a particular ingress mapping may not be to just throw the traffic at the 404 handler. For example, if you are hosting a web service, you might want parts of the particular ingress url to return static json documents or something rather then a 404 to ensure some level of service is maintained. There is only one default backend, so cramming all possible ingress rule's backends default stuff into one super webserver seems very counter to the kubernetes philosophy of using microservices/sidecars to split up work into manageable pieces. There is also the admin responsibility. The k8s admin is generally responsible for dealing with setting up the default backend and ingress controller. But the failback server content belongs squarely to the user that wrote the ingress rule.

Thinking about it a little bit, this use case could be implemented today without any patches but suboptimal. You could make two ingress classes, main and backup. point main's default at the backup controller, and then the backup at the default deployment. This would allow the user to slide in backup ingress rules as needed. Unless this pattern became the common deployment pattern though, it would not allow for very portable apps.

It would be nicer if you could just specify the backup svc right in the ingress rule and not need an intermediate ingress controller to pass traffic though.

[yaronha]

@aledbf AFAIK the default backend is used in case there is no ingress path/rule, not when there is a path w/o end points, isnt that the case?

@kfox1111 cascading ingress isnt that efficient, having backup per ingress rule is even better, i assume its not too complicated to add since "backup" is a built-in feature of Nginx upstream, wouldnt mind doing a patch myself if i know where this code reside

one use of this would be for serverless implementations or as a fail-safe mechanism that intercept and delay the transaction while the POD is initialized

[aledbf]

The appropriate failback behavior for a particular ingress mapping may not be to just throw the traffic at the 404 handler

You can do whatever you want in the default backend, like enabling custom errors in the controller and act accordingly in the backend you created (like returning a custom content type)
Please check https://github.com/aledbf/ingress/tree/master/controllers/nginx#custom-errors

But the failback server content belongs squarely to the user that wrote the ingress rule.

We cannot force that behavior, that's why a default backend is required

[aledbf]

AFAIK the default backend is used in case there is no ingress path/rule, not when there is a path w/o end points, isnt that the case?

If the ingress containes 2 paths, / and /demo and the service for demo is a service without edpoints it will use the default backend

[yaronha]

@aledbf thanks, what is the serviceName/Port i need to specify for the "/" path, same as "/demo" or my default backend service? i tried few options and getting "502 Bad Gateway"

is there a way to pass the original URL or service name to that backend (e.g. using some custom header), so i know which path or backend service failed ?

[pieterlange]

Latching on to this ticket: i've implemented a default backend with custom errors based off @aledbf's custom error controller (i believe it was in your private repo and not even in contrib back then?) and it's a really powerful feature of NGINX ingress. I don't see a lot of people using it though.

Probably worth it to write some docs & update examples for this, because it was a fair bit of trouble to figure out. I think @aledbf is already pretty busy, i'll try to write up something soon.

[aledbf]

@pieterlange here https://github.com/aledbf/contrib/tree/23352c270bcb535fb3e6caf536e955650960bd4d/images/nginx-error-server (public but in a branch)

[aledbf]

is there a way to pass the original URL or service name to that backend (e.g. using some custom header), so i know which path or backend service failed ?

We can add another header like X-Original-URL with the value of $request_uri

[kfox1111]

Let me try wording the situation maybe a little more concretely...

I'm working in a place where the team providing/maintaining the ingress-controller instance and default backend is the team providing the kubernetes cluster, and multiple other teams are provided namespaces that use ingress rules for exposing their services out to the world.

In this arrangement, the team managing the default backend for the entire cluster maybe shouldn't need to get involved when one of the teams wants to update their failure content for their specific services.

This can't be done today without a lot of extra overhead by nesting ingress controllers. While nginx natively supports this kind of thing I think.

Does that help explain the situation a little better?

There is no single team that is responsible for all the content a single backup server would need to provide, just like there is no single team that provides all the services that are being exposed through one ingress. hence the desire to use ingress's in the first place.

[yaronha]

@aledbf i have used the latest image and instructions from https://github.com/kubernetes/ingress/tree/master/examples/rbac/nginx, added "/" pointing to a valid backend and "/echo" pointing to zero POD service, still dont see how i can get it to route to something when the service have zero PODs, keep getting "503 Service Temporarily Unavailable" response. if i scale "echo" deployment i do get valid page.

i looked into the ingress nginx.conf, it has:

    # default server for services without endpoints
    server {
        listen 8181;
        set $proxy_upstream_name "-";

        location / {
            return 503;
        }
    }

can you clarify how do i need to structure the ingress rule (e.g. yaml) to point to some default server when i dont have endpoints ?

[aledbf]

@yaronha please add custom-http-errors: 404,502,503,504 (or the codes you want to customize) in the configmap of the ingress controller

[aledbf]

Does that help explain the situation a little better?

Yes. We could add a new annotation like ingress.kubernetes.io/default-backend: <ns/svc name> to allow this customization. If the annotation exists and referenced service in the annotation does not have available endpoint it will use the global default backend.

[yaronha]

We could add a new annotation like ingress.kubernetes.io/default-backend: <ns/svc name>

@aledbf thanks, that sounds better than playing with custom errors

i noticed the config with no endpoints is:

    upstream default-echoheaders-80 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 127.0.0.1:8181 max_fails=0 fail_timeout=0;
    }

so i guess we just need to replace the 127.0.0.1:8181 in the example with the default-backend:port in the no endpoint case

[aledbf]

thanks, that sounds better than playing with custom errors

Honestly, I prefer a global and consistent error handling. That said, I understand the requested use case.
Just as an example, CoreOS Tectonic uses this ingress controller with a custom default backend.

so i guess we just need to replace the 127.0.0.1:8181 in the example with the default-backend:port in the no endpoint case

No, we will create a new upstream for the custom/s default backends defined in the ingress annotations

[yaronha]

No, we will create a new upstream for the custom/s default backends defined in the ingress annotations

@aledbf this gets me confused, seems like no-endpoint case is directing traffic to 127.0.0.1:8181, not to the default backend (at least in my setup), so having a default backend per rule doesnt solve the "no-endpoint" case or require complex configuration, why not instead of writing 127.0.0.1:8181 use a "backup" address which can be passed through an annotation. having no endpoints may not always be an error, can be done on purpose to save resources (e.g. in Serverless use-case)

i noticed the line: https://github.com/kubernetes/ingress/blob/e7d2ff6fac6cea0e3be7e44484e961549b19fa74/core/pkg/ingress/controller/controller.go#L586

i assume it is possible to swap ic.cfg.Backend.DefaultEndpoint() with some other cfg parameter ?

[aledbf]

@kfox1111 please use the image quay.io/aledbf/nginx-ingress-controller:0.191
It contains the new annotation. Example: ingress.kubernetes.io/default-backend: default/http-svcn

[aledbf]

i assume it is possible to swap ic.cfg.Backend.DefaultEndpoint() with some other cfg parameter ?

@yaronha that's a bug. Please test the image ^^

[aledbf]

why not instead of writing 127.0.0.1:8181 use a "backup" address which can be passed through an annotation

Because if a pod dies (for any reason) the controller will update the configuration in few seconds. The dead pod never will be started with the same IP address.

[kfox1111]

Awesome. thanks. :)

[kfox1111]

hmm... one thought... if you can specify a namespace, you could potentially expose someone elses svc. I think the regular ingress rules only let you expose services in your current namespace preventing this security issue?

Should we just drop support for specifying the namespace in the default-backend annotation and always use the ingress rule's namespace?

[aledbf]

Should we just drop support for specifying the namespace in the default-backend annotation and always use the ingress rule's namespace?

I thought in that but the default backend flag allows you to specify the namespace. I can remove the namespace :)

[kfox1111]

the default backend flag is controlled by the k8s admin in my case, so letting them specify namespace via the flag is pretty safe I think. In the ingress rule use case, is a different matter. :)

Thanks,
Kevin

[aledbf]

In the ingress rule use case, is a different matter. :)

I will remove the namespace

[yaronha]

Because if a pod dies (for any reason) the controller will update the configuration in few seconds. The dead pod never will be started with the same IP address.

@aledbf again, in the Serverless use case i talk about (Fission, Kubeless, nuclio, ..) we want to scale down the deployment to zero when its not active, and init PODs on the first HTTP request. the current ingress cannot be used in that case forcing us to build one.

so the solution can be to have a "backup" deployment/service (can use DNS names so wont be sensitive to IP address changes), the backup service can buffer the request, scale the function deployment to 1 and pass the request to the function once its up.

another approach can be to have LUA code in the ingress to send an control request to the controller (asking to scale the function to 1), wait for it to go up, update the upstream, and pass the request(s) to it. it may be better but seems like much larger change in ingress.

so for the simple (1st) case, only thing we need is to define a "backup" end-point/DNS-name/upstream/.. to catch traffic for suspended functions, pass the original URL (or even better to pass the service name) for the backup to know which Function is it for, this backup will bring the function up, pass the request and return the response. (BTW when the conf is re-written do you lose in-process HTTP requests?)

how would you suggest to support the Serverless use-case (when service may be suspended w no endpoints) ?

Thanks, Yaron

[kfox1111]

@yaronha wont the proposed ingress.kubernetes.io/default-backend: default-svc-name annotation let you do just that? Put your serverless http process as the default backend for the ingress rule. then if it recieves a request, spawn a new pod and send the intercepted request to the new pod's svc directly. then all future traffic will go straight through the ingress?

[aledbf]

so the solution can be to have a "backup" deployment/service (can use DNS names so wont be
sensitive to IP address changes), the backup service can buffer the request, scale the function deployment to 1 and pass the request to the function once its up.

You need to rethink this. We cannot buffer request or wait for a condition to reach the backend.

how would you suggest to support the Serverless use-case (when service may be suspended w no endpoints) ?

I think this is not something Ingress can help with.

[yaronha]

@kfox1111 yes, i just tested with the new image @aledbf provided and it does seem to work (with the gcp image the traffic didnt go to the default-backend). there is still a question what will happen if the new config gets re-loaded while the responce havent returned, will Nginx preserve the request context or the reload clears all the pending context

Yaton

[yaronha]

We cannot buffer request or wait for a condition to reach the backend.

@aledbf as i wrote, the "backup" service does the buffering, not the ingress

[aledbf]

there is still a question what will happen if the new config gets re-loaded while the responce havent returned, will Nginx preserve the request context or the reload clears all the pending context

The old worker processes are keep alive until the in flight requests are returned.

[aledbf]

Closing. The default backend annotation provides the requested feature.

[kfox1111]

great. thanks for implementing this. :)

[yaronha]

@aledbf really thanks for the quick resolution

one last suggestion, it can be very helpful to pass the namespace/service-name to the default-backend via header, so the recovery or error handling logic would be able to refer to the specific failure case.

[aledbf]

one last suggestion, it can be very helpful to pass the namespace/service-name to the default-backend via header, so the recovery or error handling logic would be able to refer to the specific failure case.

Sure. I will also add the original URL in another header