Kustomize: Vars substitution does not work in ConfigMap

[jonathanunderwood]

Kustomize doesn't seem to substitute values of Vars into the data section of a configmap.

Reproducer below.

$ kustomize version
Version: {KustomizeVersion:2.1.0 GitCommit:af67c893d87c5fb8200f8a3edac7fdafd61ec0bd BuildDate:2019-06-19T02:13:12+01:00 GoOs:darwin GoArch:amd64}

kustomization.yaml:

resources:
  - configmap.yaml
  - ingress.yaml
  - service.yaml

vars:
  - name: CNAME
    objref:
      apiVersion: extensions/v1beta1
      kind: Ingress
      name: my-ingress
    fieldref:
      fieldpath: spec.rules[0].host

ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: CNAME.DOMAIN.COM
    http:
      paths:
      - path: /
        backend:
          serviceName: service
          servicePort: 80

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80

configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap
data:
  ZZZ: YYY
  CNAME: $(CNAME)

Note below that the $(CNAME) var has not been susbstituted in the ConfigMap resource:

$ kustomize build .
2019/06/25 10:29:56 well-defined vars that were never replaced: CNAME
apiVersion: v1
data:
  CNAME: $(CNAME)
  ZZZ: YYY
kind: ConfigMap
metadata:
  name: configmap
---
apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: CNAME.DOMAIN.COM
    http:
      paths:
      - backend:
          serviceName: service
          servicePort: 80
        path: /

Weirdly, with kubectl a different problem arises:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-20T04:49:16Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
The connection to the server localhost:6443 was refused - did you specify the right host or port?

$ kubectl kustomize .
Error: field specified in var '{CNAME extensions_v1beta1_Ingress {spec.rules[0].host}}' not found in corresponding resource


Examples:
  # Use the current working directory
  kubectl kustomize .

  # Use some shared configuration directory
  kubectl kustomize /home/configuration/production

  # Use a URL
  kubectl kustomize github.com/kubernetes-sigs/kustomize.git/examples/helloWorld?ref=v1.0.6

Usage:
  kubectl kustomize <dir> [flags] [options]

Use "kubectl options" for a list of global command-line options (applies to all commands).

field specified in var '{CNAME extensions_v1beta1_Ingress {spec.rules[0].host}}' not found in corresponding resource

[jbrette]

Hi

You are missing a config in kustomization:

in your kustomization.yaml

configurations:
- kustomizeconfig.yaml

in your kustomizeconfig.yaml

varReference:
- path: data/CNAME
  kind: ConfigMap

apiVersion: v1
data:
  CNAME: CNAME.DOMAIN.COM
  ZZZ: YYY
kind: ConfigMap
metadata:
  name: configmap
---
kind: Service
metadata:
  name: service
apiVersion: v1
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: CNAME.DOMAIN.COM
    http:
      paths:
      - backend:
          serviceName: service
          servicePort: 80
        path: /

For info there is a pull request which is attempting to do all that (kustomization.yaml var and kustomizeconfig.yaml varReference) automatically: https://github.com/kubernetes-sigs/kustomize/pull/1217.

The only requirement would be that you would have the name of your variable.
For instance

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap
data:
  ZZZ: $(Ingress.my-ingress.spec.rules[0].http.paths[0].backend.servicePort)
  CNAME: $(CNAME)

becomes after kustomize build

apiVersion: v1
data:
  CNAME: CNAME.DOMAIN.COM
  ZZZ: 80
kind: ConfigMap
metadata:
  name: configmap

[jbrette]

Added automatic testing of your issue here: https://github.com/keleustes/kustomize/tree/allinone/examples/issues/issue_1250

[jonathanunderwood]

@jbrette thank you for responding so quickly and helpfully, really appreciated.

I think this points to an omission in the kubernetes documentation. Unless I've somehow missed it, I don't think varReference is mentioned anywhere in the user docs, is it?

[jbrette]

@jonathanunderwood The only "locations" where kustomization can replace variables by default are listed here: https://github.com/kubernetes-sigs/kustomize/blob/master/pkg/transformers/config/defaultconfig/varreference.go

kustomize is using those var reference to locate (across all the yaml files of your project) where it will potentially find a variable.

Any other location is considered custom as in a CRD: Check here: https://github.com/kubernetes-sigs/kustomize/blob/master/examples/transformerconfigs/crd/README.md

There is an outstanding PR doing all that for you: When a user uses the format list bellow, he already provides the "where" to replace and with "what" to replace. This simplifies and automatizes a lot of the variables declarations. Note that you can still use the "manual" mode if you discover that the "automatic" mode has an issue.

data:
  ZZZ: $(Ingress.my-ingress.spec.rules[0].http.paths[0].backend.servicePort)

/cc @monopole @Liujingfang1 @ian-howell @pwittrock

[jonathanunderwood]

@jbrette adding the kustomizeconfig.yaml stuff does now work for kustomize itself. But with kubectl I still see this behaviour:

$ kubectl kustomize ./
Error: field specified in var '{CNAME extensions_v1beta1_Ingress {spec.rules[0].host}}' not found in corresponding resource


Examples:
  # Use the current working directory
  kubectl kustomize .

  # Use some shared configuration directory
  kubectl kustomize /home/configuration/production

  # Use a URL
  kubectl kustomize github.com/kubernetes-sigs/kustomize.git/examples/helloWorld?ref=v1.0.6

Usage:
  kubectl kustomize <dir> [flags] [options]

Use "kubectl options" for a list of global command-line options (applies to all commands).

field specified in var '{CNAME extensions_v1beta1_Ingress {spec.rules[0].host}}' not found in corresponding resource

[jbrette]

@jonathanunderwood your kubectl is probably still using on older version of kubectl. Meanwhile you should use something like

kustomize build  . | kubectl apply -f -

[jonathanunderwood]

$ kustomize version
Version: {KustomizeVersion:2.1.0 GitCommit:af67c893d87c5fb8200f8a3edac7fdafd61ec0bd BuildDate:2019-06-19T02:13:12+01:00 GoOs:darwin GoArch:amd64}

$  /usr/local/Cellar/kubernetes-cli/1.15.0/bin/kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-20T04:49:16Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
The connection to the server localhost:6443 was refused - did you specify the right host or port?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[jonathanunderwood]

/close

[Datamance]

Do varreferences work for generated configmaps? even with kustomizeconfig.yaml and the setup mentioned above, I get a "well-defined vars that were never replaced:" error.

[Liujingfang1]

@Datamance Yes, it works with generated configmaps. If it doesn't work, you can file a bug by following this instruction.

[Datamance]

@Liujingfang1 nope that's ok - I just had an extra dash in the patch value (using this functionality to add a sidecar), once I removed it everything compiled just fine. Thanks!