Ingress-nginx: Feature Request: Load Custom Configurations from Templates

[diazjf]

The purpose of this proposal is to allow for a cloud provider to load a template containing proprietary configurations based on user needs. The templates can be further configured by the user. The feature will be enabled using 2 annotations:

1. nginx.ingress.kubernetes.io/custom-config-location will be used to set where the template is located. It will be a string containing the path to the tmpl file. It is the responsibility of the cloud provider to make sure they add this template to the container storage. For simplicity we can always start at the /etc/nginx/templates path.

2. nginx.ingress.kubernetes.io/custom-config-variables will be used to load the required variables for the template in nginx.ingress.kubernetes.io/custom-config-location

There are 2 requirements. The file specified in nginx.ingress.kubernetes.io/custom-config-location must exists and the variables it requires, if any, must be set in nginx.ingress.kubernetes.io/custom-config-variables. The variables are then applied to the template and used to generate a custom configuration, which is then applied to the given locations within the ingress resource.

Sample flow:

1. my.tmpl exists in deployment as follows:

proxy_set_header        Host            {{ host }};
proxy_set_header        X-Real-IP       {{ xrealip }};

1. User creates Ingress resource with following YAML:


apiVersion: extensions/v1beta1 
kind: Ingress

metadata: 
  annotations: 
    nginx.ingress.kubernetes.io/custom-config-location: my.tmpl 
    nginx.ingress.kubernetes.io/custom-config-variables: host=a.com, xrealip=0.0.0.0
  name: cafe-ingress
spec: 
  rules:
  - host: ok.com 
    http:
      
      paths:
      
      - backend: 
          serviceName: tea-svc 
          servicePort: 80
       path: /tea 


1. my.tmpl is loaded with provided variables

proxy_set_header        Host            a.com;
proxy_set_header        X-Real-IP      0.0.0.0;

1. nginx.conf is generated with /tea section:

Location /tea {
         proxy_set_header        Host            a.com;
          proxy_set_header        X-Real-IP      0.0.0.0;
         ….
}
…

Limitations:

1. In this current proposal, only 1 custom configuration can be applied per path

1. Custom Configuration can only be applied to location level. Maybe another annotation for the server-level and a ConfigMap change can apply it globally.

Other Questions:

1. Why do we need this?

Cloud Providers may want to provide an Ingress Controller (In this case this one) to a user by default. These Cloud Providers also want to be able to provide custom features as well as maintain a strong community integration. This feature will allow Cloud Providers to add their proprietary features, as well as enable them to keep up with the master.

1. Why not use location snippets?

We don’t want the user to have to provide a full configuration. Our configurations can be large. Hence by just providing a file and a few variables to alter, it makes the user experience a whole lot better.

1. Having to specify key/value pairs in a value doesn’t follow annotation format?

For templating there are not many alternatives, possibly CRDs can be used in this case. We can decide on implementation details.

[aledbf]

@diazjf you want to add https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#add-headers but at location level using annotations?

[diazjf]

@aledbf thanks for taking a look! add-headers was just an example. We have some large custom configurations which we would like to apply to certain locations. I think using files would be a good solution.

[aledbf]

It is the responsibility of the cloud provider to make sure they add this template to the container storage.

This is too abstract. Also, this must work in baremetal too.

Using two annotations like custom-config-location and custom-config-variables to define a text and a list of variables feels a bit weird. Related to my previous comment, we should use already existing abstractions (like a configmap) to define two objects, one with the template and another one with the key-value pair.

[aledbf]

@ElvinEfendi @antoineco WDYT about this feature request?

[diazjf]

For It is the responsibility of the cloud provider to make sure they add this template to the container storage. would be able to work on bare-metals as well. The files just needs to exist somewhere.

Good Suggestion, instead of using 2 annotations, maybe we can point to a specific configmap for this functionality. Which will have a restricted key templateFile(or something else) and the rest of the keys will be used to load the variables into the template. Make sense to me 👍

But, if using a config-map, how can I make sure only certain locations are targeted?

[antoineco]

This sounds like a nice improvement to the current snippet approach, actually it would be good in my opinion if the user could transparently pass a snippet (maybe with template elements?) _or_ a reference to a template like you suggested. (don't quote me on that, not sure about the implementation details yet, or if it really makes sense)

I like the ConfigMap approach better, using an annotation has some limits in terms of length, parsing, etc.

[diazjf]

I'm thinking of setting a ConfigMap as follows:

apiVersion: v1
data:
  <hostname>.properties: |
    template=my.tmpl
    path=/tea (optional)
  <hostname>.variables: |
    host=host
    realip=realip
    ...
kind: ConfigMap

Using the hostname we can load a template and all the required variables to a particular host and/or particular paths within a host.

@aledbf @antoineco lmk what you think.

[aledbf]

No, this should be one configmap per ingress. If you have more than one it will be applied to all the paths (like we handle all the annotation in the ingress controller).
Like

apiVersion: v1
data:
    template: |
      # content of the template
    host: host
    realip: realip

kind: ConfigMap

Edit: if you only want to apply a template to a particular path, create a new ingress only containing such path (again, this must behave like the rest of the annotations)

[diazjf]

@aledbf I have 2 questions on this:

1. How do I link a ConfigMap to just a particular Ingress Resource?
2. How would I select the file? Another specific key?

[ElvinEfendi]

I like the proposal, I think it's gonna take the flexibility of ingress-nginx to a next level!

And I think @aledbf's suggestion makes sense. We can then use annotation to attach the configmap to a particular ingress resource.

--

Do we have to declare variables though? Why can we not always expose all possible variables to the given custom template?

[aledbf]

@diazjf I am assuming you want to apply one template/vars per location? This is not the case? (you want to allow multiple templates per path?)

[diazjf]

@aledbf Just one template per path is a limitation, but I am fine with implementing that if it makes the most sense, especially in usability. Ideally a user may want multiple configurations to one path, but I don't think that is a common case.

@ElvinEfendi thanks for taking a look! An annotation to create the link make sense! Can you provide me with further details on exposing all possible variables. We have custom configurations which the user must provide some changes to. Declaring the variables in the configmap seems to be a good way.

Also for choosing a file path, I'm thinking that should be another annotation.

[aledbf]

Ideally a user may want multiple configurations to one path, but I don't think that is a common case.

We can add the support for a list in the next iteration.

[ElvinEfendi]

Can you provide me with further details on exposing all possible variables.

Someting like https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/nginx.go#L574. But since this custom template is per ingress we should expose only the variables for the given ingress. (i.e instead of exposing all ingressCfg.Servers expose only the server corresponding to the given ingress. We can then document those variables publicly as an API to be used in custom templates.

[diazjf]

@ElvinEfendi the thing is we wont know off the bat, what the keys are since there can be anything in the template which don't exist in a server, for example, our custom config can be:

proxy_set_header        Cat           {{ cat }};
proxy_set_header        Rat           {{ rat }};

so in the configmap:

apiVersion: v1
data:
    template: |
      # content of the template
    cat: meow
    rat: xxxx
kind: ConfigMap

will setup:

proxy_set_header        Cat           meow;
proxy_set_header        Rat           xxxx;

[ElvinEfendi]

@diazjf I might be missing something but what’s the point of template in this case, why don’t you put “meow” and “xxxx” in the template directly?

[diazjf]

@ElvinEfendi so that is just a simple example, but I may have a big configuration which I don't want a user to edit manually. I provide this template, but a user must configure certain items.

A better example would be:

apiVersion: v1
data:
    template: |
      # content of the template
    type: web
    my-secret:test-secret
    my-microservice:test-microservice
kind: ConfigMap

{{ if (eq (type) ("web")) }}
proxy_set_header   X-My-Secret-to-Use   {{ my-secret }}
auth_request_set ...
auth_request /_my-microservice_{{ my-microservice }};
...
...
...
{{ end }}

We have these configurations which we use to enable authentication that we would like to load. We also want users to populate specific variables within the configuration. Like the type of authentication they want to enable, the secret they want to use, namespace of the secret, etc. These can be passed as headers or used in if statements or passed in an auth-request, etc.

I'd like to allow for complete customization of a template. I'd be happy to explain in more detail on a Slack call.

[aledbf]

@diazjf from that example and explanation it seems that you want to create a template library. If that's the intention then using just one configmap makes no sense. This should could be one configmap with one or more templates and use another with a key with the name of the template to use plus the other key-values.

[diazjf]

@aledbf it would be cool if we could have a quick call sometime this week with everyone interested and we can nail out some of the details. I'll update this issue with the suggested implementation details

[diazjf]

Updated Proposal:

The purpose of this proposal is to allow for a cloud provider to load a template containing proprietary configurations based on user needs. The templates can be further configured by the user. The feature will be enabled using the following:

template-library will be a configuration map containing a one or more templates.
template-library-variables will be a configuration map containing the a key with the name of the template and the variables used for the template.

nginx.ingress.kubernetes.io/custom-config: <template-library> will be used in order to load a particular configuration into an ingress resource.
nginx.ingress.kubernetes.io/custom-config-variables: <template-library-variables> will be used in order to load the variables into the custom-configuration.

We can have many different template-libraries and template-library-variables in order to be able to have many different combinations of configurations.

Very Basic Sample flow:

my.tmpl exists in deployment as follows:

proxy_set_header        Host            {{ host }};
proxy_set_header        X-Real-IP    {{ xrealip }};

other.tmpl exists in deployment as follows:

proxy_set_header        Cat            {{ cat }};
proxy_set_header        Rat            {{ rat  }};

User create config-map containing templates:

apiVersion: v1
data:
    templates: my.tmpl, other.tmpl
kind: ConfigMap

User create config-map containing variables:

apiVersion: v1
data:
    variables: |
      host: a.com
      xrealip: 0.0.0.0
      cat: meow
      rat: xxxx
kind: ConfigMap

User creates Ingress resource with following YAML:


apiVersion: extensions/v1beta1 
kind: Ingress

metadata: 
  annotations: 
    nginx.ingress.kubernetes.io/custom-config: <template-library>
    nginx.ingress.kubernetes.io/custom-config-variables: <template-library-variables>
  name: cafe-ingress
spec: 
  rules:
  - host: ok.com 
    http:
      
      paths:
      
      - backend: 
          serviceName: tea-svc 
          servicePort: 80
       path: /tea 


my.tmpl is loaded with provided variables

proxy_set_header        Host            a.com;
proxy_set_header        X-Real-IP      0.0.0.0;

other.tmpl is loaded with provided variables

proxy_set_header        Cat            meow;
proxy_set_header        Rat            xxxx;

nginx.conf is generated with /tea section:

Location /tea {
         proxy_set_header        Host            a.com;
         proxy_set_header        X-Real-IP      0.0.0.0;
         proxy_set_header        Cat            meow;
         proxy_set_header        Rat            xxxx;
         ….
}
…

@aledbf @ElvinEfendi @antoineco what do you guys think?

[antoineco]

If a given ConfigMap is dedicated to variables, I think we should just read keys and values from it instead of that blob of text:

# no
data:
    variables: |
        host: a.com
        xrealip: 0.0.0.0
        cat: meow
        rat: xxxx

# yes
data:
    host: a.com
    xrealip: 0.0.0.0
    cat: meow
    rat: xxxx

I also think a ConfigMap is an overkill if all you want to do is pass a list of templates. In my opinion a plain annotation is sufficient for that.

You also forgot to mention template references are actual paths (absolute).

custom-config sounds vague to me. Where is it added, to a location, to a server, to the global config? I think the name is not clear enough.

All the rest looks 👌 🙆‍♂️

[aledbf]

I think we should just read keys and values from it instead of that blob of text:

That's what I did in the initial comment https://github.com/kubernetes/ingress-nginx/issues/2901#issuecomment-410845985

I also think a ConfigMap is an overkill if all you want to do is pass a list of templates. In my opinion a plain annotation is sufficient for that.

You also forgot to mention template references are actual paths (absolute).

We cannot assume that. All the things must be k8s abstractions (to be able to handle this at runtime without changing the ingress controller deployment)

If the idea is to create a "template library", then using one configmap with the key for the template is enough.

The goal here is to provide the templates and the users only be able to change the variables, right?

If that's the case then:

apiVersion: v1
data:
    template: <name of the configmap that contains the template>
    host: a.com
    xrealip: 0.0.0.0
    cat: meow
    rat: xxxx
kind: ConfigMap

Then, the annotation only contains the name of the ^^ configmap

[diazjf]

The goal here is to provide the templates and the users only be able to change the variables, right?

@aledbf thats the exact thing I had in mind. template will be a reserved key.

In template: <name of the configmap that contains the template> will that config map contains a blob containing the whole configuration? That works for me and can make things a ~little~ lot easier.

@antoineco Thanks for the input, I can change the naming during the implementation for it to make more sense. It will be applied at the location level.

[aledbf]

@diazjf

apiVersion: v1
name: cat-rat
data:
    template: |
      proxy_set_header        Cat            {{ cat }};
      proxy_set_header        Rat            {{ rat  }};
kind: ConfigMap

apiVersion: v1
name: user-configured-cat-rat
data:
    template: cat-rat
    host: a.com
    xrealip: 0.0.0.0
    cat: meow
    rat: xxxx
kind: ConfigMap

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/custom-template: user-configured-cat-rat
  name: cafe-ingress
spec:
  rules:  
  - host: ok.com
    http:
      paths:
      - backend:
          serviceName: tea-svc
          servicePort: 80
       path: /tea

[diazjf]

Awesome, this is gonna be a fun little feature!

[aledbf]

I understand the purpose of this feature but using annotation makes this a horrible hack. I would prefer to wait until we start exploring CRDs before using this approach

[diazjf]

@aledbf how long until we start using CRDs. I would like to work on this feature when possible.
Or can it be implemented with an annotation and then migrated later, not sure how feasible migration would be.

[antoineco]

The template key as you presented it @aledbf contradicts the original proposal because @diazjf mentioned the use of several templates.
In that case the key should be templates and the value should be a list of CM references, right?

Another question: what if a variable is unset?

[diazjf]

@antoineco in the example above, maybe nginx.ingress.kubernetes.io/custom-template can have multiple lists. If a variable unset, then an error should be thrown and the configuration will be ignored.

[aledbf]

The template key as you presented it @aledbf contradicts the original proposal because @diazjf mentioned the use of several templates.

https://github.com/kubernetes/ingress-nginx/issues/2901#issuecomment-410854581

Another question: what if a variable is unset?

Good question. My first reaction is that we should deal with this as a standalone template and in case of errors (unset variables) nothing will be rendered. This should be a validation during the creation of the variables configmap (not sure how feasible is)

[aledbf]

how long until we start using CRDs. I would like to work on this feature when possible.

I want to start migrating the configuration configmap first in the next one or two iterations

[diazjf]

@aledbf thanks. Let me know how I can help.

[diazjf]

I want to start migrating the configuration configmap first in the next one or two iterations

@aledbf let me know how you plan on migrating. I think I can set aside some time to help out.

[diazjf]

@aledbf any update on https://github.com/kubernetes/ingress-nginx/pull/2929
I will have some time this month to work on this feature!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.