Kustomize: Inject Host into Ingress

[lswith]

I think a reasonably common use case is to swap an ingress's host value:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: helloworld
spec:
  rules:
# This value should be editable
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: helloworld
          servicePort: 80

Can we get a feature to set this?

[Liujingfang1]

Recently we added JSON patch support, which is a good solution for this problem. Take a look at our example https://github.com/kubernetes-sigs/kustomize/blob/master/examples/jsonpatch.md
This feature is available from HEAD currently, we will release a new version soon.

[lswith]

ah right. Just create a JSON Patch and then use that to edit the build.

[iamwel]

I'm sorry @Liujingfang1, I read the example, and it does not seem like a suitable solution to what is, as @lswith mentioned, a common use case. I was thinking of incorporating Kustomize into our workflow as a low-overhead alternative to creating a helm chart, but a chart seems to be a much more elegant alternative at this point. Any opprotunity for native ingress variables in Kustomize?

[jonathanunderwood]

I agree: being able to patch the Ingress host value is super useful, and it would be preferable to be able to do it with a strategic merge. I am seeing a lot of feature requests closed with "use a JSON patch", without much consideration of the use cases.

[davinkevin]

Same for me... 👍Could we reopen this one?

[streetmapp]

Also commenting in hopes that this get looked at as something that should be supported natively. Pushing jsonpatches as the solution doesn't seem viable for all use cases. For obscure things that aren't done often sure. But configuring an ingress is quite common, so having a cleaner way to kustomize that would be extremely beneficial.

[iameli]

@davinkevin's referenced commit (https://github.com/davinkevin/Podcast-Server/commit/9ca4be54606925e3a8a4213384bc8de84ce7fbbf) illustrates the problem very nicely — how do I make three different variants with three different ingress rules applying to three different hosts? Here's how I'm currently solving the problem — can y'all see how this is inelegant?

Here's my base:

broadcaster/broadcaster.yaml

[deployment and service omitted]

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: broadcaster
spec:
  rules:
    - host: $(SERVICE_NAME).example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: broadcaster
              servicePort: 80

broadcaster/kustomization.yaml

resources:
  - broadcaster.yaml

And here's what I'm using to produce ingresses for broadcaster-bulbasaur.example.com, broadcaster-charmander.example.com, and broadcaster-squirtle.example.com.

broadcaster-pokemon/kustomization.yaml

resources:
- ./bulbasaur
- ./charmander
- ./squirtle

broadcaster-pokemon/squirtle/kustomization.yaml

resources:
- ../../broadcaster
nameSuffix: -squirtle
patchesJson6902:
  - path: hostname.yaml
    target:
      group: extensions
      kind: Ingress
      name: broadcaster
      version: v1beta1

broadcaster-pokemon/squirtle/hostname.yaml

- op: replace
  path: /spec/rules/0/host
  value: broadcaster-squirtle.example.com

broadcaster-pokemon/charmander/kustomization.yaml

resources:
- ../../broadcaster
nameSuffix: -charmander
patchesJson6902:
  - path: hostname.yaml
    target:
      group: extensions
      kind: Ingress
      name: broadcaster
      version: v1beta1

broadcaster-pokemon/charmander/hostname.yaml

- op: replace
  path: /spec/rules/0/host
  value: broadcaster-charmander.example.com

broadcaster-pokemon/bulbasaur/kustomization.yaml

resources:
- ../../broadcaster
nameSuffix: -bulbasaur
patchesJson6902:
  - path: hostname.yaml
    target:
      group: extensions
      kind: Ingress
      name: broadcaster
      version: v1beta1

broadcaster-pokemon/bulbasaur/hostname.yaml

- op: replace
  path: /spec/rules/0/host
  value: broadcaster-bulbasaur.example.com

I'd like to do something like this instead:

broadcaster/kustomization.yaml

resources:
  - broadcaster.yaml
vars:
  - name: SERVICE_NAME
    objref:
      kind: Service
      name: broadcaster
      apiVersion: v1

broadcaster-pokemon/squirtle/kustomization.yaml

resources:
- ../../broadcaster
nameSuffix: -squirtle

broadcaster-pokemon/charmander/kustomization.yaml

resources:
- ../../broadcaster
nameSuffix: -charmander

broadcaster-pokemon/bulbasaur/kustomization.yaml

resources:
- ../../broadcaster
nameSuffix: -bulbasaur

Much cleaner.

[xhanin]

Any chance to see that addressed in a future version?

When you have a bunch of subdomains in an ingress, using json patch is not acceptable. It works, but referencing the hosts by index leads to odd errors if someone changes the order of the hosts in the original ingress. So having something like we have for the images would be so nice...

In my team we ended up piping kustomize build with a sed to address that more conveniently. But it's so sad something easy is not supported out of the box for such a common use case.

[andsens]

I ran into a different use-case for the same feature yesterday:
At work we are going to have a local k8s setup on each machine. With the old VM setup we customize the hostname to have a $USER suffix. That hostname is then broadcast via mDNS on the internal network so people other than the person at the machine can test solutions from their own machine (i.e. instead of solution.local you'd have solution-$USER.local).
I am working on an mDNS ingress hostname broadcaster (currently only works with microk8s), and being able to locally apply various manifests containing ingresses without having to post-process them would be very helpful.

[coderanger]

I empathize with the Kustomize team, maybe this could be addressed in k/k with something like a spec.baseHostname field?

[esteban1983cl]

Please add support for suffixes.
For instance, I have a lot of ingress with hostname suffix .aws-test.example.com I need add an overlay for different environments or zones for get ingresses with hostname suffix .aws.example.com or .gcp.example.com

[dwmkerr]

Without features like this, being able to essentially string interpolate on fields, I'm not really sure how Kustomize really fits into the ecosystem. For me I was using it because Helm is way too complex for simple projects, Kustomize covers 99% of my needs - except that I can't configure hostnames of ingress routes.

I know there is a design goal not to make this a templating project, but without some kind of basic templating/interpolation, does this not greatly limit the potential use cases?

JSON Patch - clever, but grody for simple use cases.

[andsens]

OK, I rescind my +1 on this. Using patches my specific usecase is actually very easy to solve by templating the patch files and creating a templated kustomize overlay (using ansible):
ingress-patch.json.j2

[
    {
        "op": "replace",
        "path": "/spec/tls/0/hosts/0",
        "value": "{{ ingress.name }}-{{ ansible_env.USER }}.local"
    },
    {
        "op": "replace",
        "path": "/spec/rules/0/host",
        "value": "{{ ingress.name }}-{{ ansible_env.USER }}.local"
    }
]

kustomize.yaml.j2

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
patches:
{%- for ingress in operations.ingresses %}

- path: {{ ingress.name }}-ingress-patch.json
  target:
    group: networking.k8s.io
    version: v1beta1
    kind: Ingress
    name: {{ ingress.name }}
{%- endfor %}

I'm also beginning to think that implementing something like this in kustomize would erode some of its simplicity that I have grown really fond of.

[MichaelJCole]

Ugh, I'm on my first day of Kustomize and foiled by this fundamental challenge. I have different domains for each environment. This seems like a basic use case.

Options:
1) Copy/paste the entire file and move on.
2) patchjson using paths like path: /spec/rules/0/host which seems very fragile
3) kustomize vars but they seem to only reference other strings. Is it possible to use a constant? If not, why not?

If there is a "bug" here, is it that the elements of "rules" don't have names, so they can't be strategically merged, breaking a basic use case for reusing code with different domains?

Is there some other solution I'm missing?

# SEE: https://kubernetes.io/docs/concepts/services-networking/ingress/

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-xxx.com
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: clusterissuer-selfsigned
spec:
  tls:
  - hosts:
    - xxx.xxx.team
    - app.xxx.team
    - www.xxx.team
    - xxx.team
    secretName: tls-xxx
  defaultBackend:
    service:
      name: www
      port:
        number: 80
  rules:
  - host: xxx.xxx.team
    http:
      paths:
      - backend:
          service:
            name: echo1
            port:
              number: 80
  - host: app.xxx.team
    http:
      paths:
      - backend:
          service:
            name: echo2
            port:
              number: 80

[andsens]

@MichaelJCole the patches are just as stable as the Ingress API itself, so that shouldn't be any trouble.
I actually ended up creating a kustomize transformer instead (transformers and generators are awesome btw.).
That way an additional patch overlay is not needed.
I'm sure it can be modified to fit your usecase.

#!/usr/bin/env python3
"""IngressTransformer - Modify ingress domain names according to a template
Usage:
  IngressTransformer <config-path>
Template pattern:
  The template supports the following variables:
  {_TLD} last part of the domain name
  {_HOSTNAME} everything except the TLD
  {_FQDN} the entire domain name
  {...} any environment variable
"""
import docopt
import yaml
import os
import sys

def main():
  params = docopt.docopt(__doc__)
  config = yaml.load(open(params['<config-path>']), Loader=yaml.FullLoader)
  template = config['spec']['template']
  resources = yaml.load_all(sys.stdin, Loader=yaml.FullLoader)
  ingresses = []
  for resource in resources:
    if resource['apiVersion'] in ['networking.k8s.io/v1', 'networking.k8s.io/v1beta1'] \
      and resource['kind'] == 'Ingress':
      for entry in resource['spec']['tls']:
        for idx, domain in enumerate(entry['hosts']):
          entry['hosts'][idx] = transform_host(domain, template)
      for rule in resource['spec']['rules']:
        rule['host'] = transform_host(domain, template)
      ingresses.append(resource)
  sys.stdout.write(yaml.dump_all(ingresses))


def transform_host(domain, template):
  parts = domain.split('.')
  return template.format(**{
    **os.environ,
    '_TLD': parts[-1],
    '_HOSTNAME': '.'.join(parts[0:-1]),
    '_FQDN': '.'.join(parts),
  })

if __name__ == '__main__':
  main()

Place it in ~/.config/kustomize/plugin/APINAME/ingressdomaintransformer and put a config next to your ingress yaml:

---
apiVersion: APINAME
kind: IngressDomainTransformer
metadata:
  name: username-suffix
spec:
  template: '{_HOSTNAME}-{USER}.{_TLD}'

[Shell32-Natsu]

As @andsens mentioned, the most flexible way to do any operation in kustomize is writing your own transformer. Meanwhile, kustomize now supports KRM functions as transformer. KRM function is containerized so it will be easier to reuse. Although we are in a lack of documentation about this feature, you can get some example from the test codes.