Kustomize: "Load from path ../<etc> failed: security, file '...' is not in or below '...'

[hach-que]

I just upgraded to the latest HEAD to get support for files in secretGenerator and it looks like there's some "security" mechanism that's been added that breaks our workload:

FATA[0011] exiting dev mode because first run failed: deploy failed: reading manifests: kustomize build: Running [kustomize build services/kubernetes/dev]: stdout , stderr: Error: rawResources failed to read Resources: Load from path ../admin.yaml failed: security; file '../admin.yaml' is not in or below '/home/.../services/kubernetes/dev'

How do I bypass this?

[hach-que]

Moving kustomization.yaml to the top-level directory of the repository isn't an option either, since we have different kustomization files for different environments, and the filename is as far as I can tell, hard-coded.

[Liujingfang1]

@hach-que The files in secretGenerators need to be relative paths in the kustomization directory or subdirectory. ../admin.yaml goes outside the kustomization directory. This is blocked in Kustomize 2.0.0.

[hach-que]

@Liujingfang1 2.0.0 blocks relative paths everywhere as far as I can tell.

The issue is that kustomization.yaml is hard-coded, so how can I have two different kustomization targets when they need to draw on the contents of the Git repository? I can't use relative parent paths anymore.

Imagine I have a directory structure like this (right now):

- dev
  - some_dev_specific_file.yaml
  - kustomization.yaml
- prod
  - some_prod_specific_file.yaml
  - kustomization.yaml
- some_files_used_by_both.yaml

With relative paths blocked, your proposal is now to put the kustomization.yaml file in the parent directory, like this:

- dev
  - some_dev_specific_file.yaml
- prod
  - some_prod_specific_file.yaml
- some_files_used_by_both.yaml
- kustomization.yaml (prod)
- kustomization.yaml (dev????? filename clash!)

Except that doesn't work because you can't have two files both named kustomization.yaml in the top level directory. There doesn't appear to be any way to override the path or name of the kustomization.yaml file either when invoking kustomize.

[mohnishkodnani]

Kubebuilder framework by default produces the following directory structure.
default

• kustomization.yaml
• patch_XYZ.yaml
  rbac:
• rbac_role.yaml
  manager:
• manager.yaml

In the default kustomization, rbac and manager folders are mentioned as resources.

Now if I have to create overlays I create
overlays/production

• kustomization.yaml

Here I put base as the default.
But then it complains that rbac/rbac_role.yaml is not under default

[Liujingfang1]

@hach-que You can put the files used by both to base and include that base in both dev and prod.

- base
   - some_files_used_by_both.yaml
   - kustomization.yaml
- dev
  - some_dev_specific_file.yaml
  - kustomization.yaml
- prod
  - some_prod_specific_file.yaml
  - kustomization.yaml

@mohnishkodnani I'm not sure I understand your problem. The base default has resources rbac and managers, but the overlay doesn't work. Is that the problem you mean?

[adeshmukh-quizlet]

@Liujingfang1 I ran into the same issue after upgrading to kustomize 2.x. I was able to fix the problem by following your tip! To summarize the change I needed to make:

I have the following folder structure:

$ tree -d
.
├── base
└── overlays
    ├── staging
    └── production

To fix the problem:

1. I added file base/kustomization.yaml with:

resources:
- my-service.yaml

1. Updated overlays/staging/kustomization.yaml and overlays/production/kustomization.yaml to replace:

resources:
- ../../base/my-service.yaml
-patches:
...

... with ...

bases:
- ../../base/
patchesStrategicMerge:
...

[jgonera]

Could you elaborate what was the motivation for blocking paths that are above given kustomization.yaml?

[bryanlarsen]

This is really annoying.

I had to go from

• base
  
  
  
  • file1.yaml
  
  
  • file2.yaml
  
  
  • file3.yaml
  
  
• dev
  
  
  
  • some_dev_specific_file.yaml
  
  
  • kustomization.yaml
  
  
• prod
  
  
  
  • some_prod_specific_file.yaml
  
  
  • kustomization.yaml
  
  

to

• base1
  
  
  
  • file1.yaml
  
  
  • kustomization.yaml
  
  
• base2
  
  
  
  • file2.yaml
  
  
  • kustomization.yaml
  
  
• base3
  
  
  
  • file3.yaml
  
  
  • kustomization.yaml
  
  
• dev
  
  
  
  • some_dev_specific_file.yaml
  
  
  • kustomization.yaml
  
  
• prod
  
  
  
  • some_prod_specific_file.yaml
  
  
  • kustomization.yaml
  
  

What I really want:

• base
  
  
  
  • file1.yaml
  
  
  • file2.yaml
  
  
  • file3.yaml
  
  
• dev-kustomization.yaml
• prod-kustomization.yaml

[jcassee]

You can put the files used by both to base and include that base in both dev and prod.

I believe it is not possible to reuse common patches in this way?

[shanna]

My use case is I have a few secrets in a shared encrypted keybase git repo used by applications not only run in kubernetes (mobile, desktop and various IOT apps) and I'd rather not have multiple copies or a makefile to shuffle files around before kustomize is run.

secrets/
    ...
platform/
  ios/
    ...
  android/
    ...
  k8s/
    base/
    next/
    production/

[monopole]

Resources should be shared by using bases - they can be located anywhere.

Patches, however, cannot be shared in v2.0.3.

One can share patches in HEAD:

https://github.com/kubernetes-sigs/kustomize/blob/master/docs/FAQ.md#security-file-foo-is-not-in-or-below-bar

The code is in, and will be in release 2.1, coming in a week or two. Need to write docs and a add some more tests for other stuff.

[develar]

shared by using bases

Ok, but secretGenerator it is not resource, it is part of kustomization.yaml. It seems that kubectl create secret generic tls --from-file=./certs/tls.cert --from-file=./certs/tls.key is the only solution now :( (security doesn't matter — it is used for local, since no doubt for production secrets created in another way).

Kustomize tries to protect developers from mistakes, it is great, but as workaround for such bugs shell script is used, that's even worse :(

As workaround, just add hard links (as symbolic links are also forbidden) :(

# https://github.com/kubernetes-sigs/kustomize/issues/766
ln -f certs/tls.cert k8s/overlays/single-node/tls.cert
ln -f certs/tls.key k8s/overlays/single-node/tls.key

[akhy]

cannot share patches anymore 😢

edit: it works with --load_restrictor flag, but somehow the flag doesn't work when using kubectl kustomize

[pavel1269]

cannot share patches anymore 😢

edit: it works with --load_restrictor flag, but somehow the flag doesn't work when using kubectl kustomize

The flag is in kustomize v3 according to this: https://github.com/kubernetes-sigs/kustomize/blob/master/docs/FAQ.md#security-file-foo-is-not-in-or-below-bar
But kubectl contains kustomize v2 according to this: https://github.com/kubernetes-sigs/kustomize#kubectl-integration