Kubebuilder: make deploy is broken for kustomize v2.0.0 release

[kfoozminus]

kustomize v2.0.0 requires that all paths other than bases have to terminate in same or subdirectory of the kustomization directory. So kustomize build config/default in make deploy fails because kustomization.yaml contains resource paths like ../rbac/rbac_role.yaml. According to kustomize v2.0.0, rbac and manager folder should be under config/default directory.

[tobbbles]

Also seeing this. A quick hack is to bring the config/default manifests into the top-level config/ directory, and updating your Makefile deploy stage to use kustomize build config | kubectl apply -f -.

Whether having the kustomize manifests in the top-level config directory is a longterm solution, I'm unsure.

[tobbbles]

I've had a play with config directory structures over the past few days and arrived at the following, feeling like it's the cleanest I've found, keeping all the kustomize builds from config/default, as from what I see manager or rbac manifests aren't used outside of this.

├── crds
│   └── your_v1beta1_sample.yaml
├── default
│   ├── kustomization.yaml
│   ├── manager
│   │   └── manager.yaml
│   ├── manager_auth_proxy_patch.yaml
│   ├── manager_image_patch.yaml
│   ├── manager_prometheus_metrics_patch.yaml
│   └── rbac
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_service.yaml
│       ├── rbac_role.yaml
│       └── rbac_role_binding.yaml
└── samples
    └── your_v1beta1_sample.yaml

[Adirio]

My suggestion:

├── apis
│   └── [GROUP]
│       └── [VERSION]
│           └── [KIND]
│               ├── crd.yaml
│               └── sample.yaml
└── manager
    ├── kustomization.yaml
    ├── patches
    │   ├── manager_auth_proxy.yaml
    │   ├── manager_image.yaml
    │   └── manager_prometheus_metrics.yaml
    └── resources
        ├── namespace.yaml
        ├── auth_proxy_role.yaml
        ├── manager_role.yaml
        ├── auth_proxy_role_binding.yaml
        ├── manager_role_binding.yaml
        ├── webhook_secret.yaml
        ├── auth_proxy_service.yaml
        ├── manager_service.yaml
        └── manager.yaml

Two main directories under config, one for API resources and the other for the manager.
The first is further subdivided in groups, versions and kinds. Two files, the crd.yaml to deine the resource in a kubernetes cluster, and the sample.yaml as an example of how to instatiate one of these resources.
The second contains a single file and two directories. The file is the kustomization.yaml file whereas the directories split resources from patches. Each resource is placed in a different file.

[figo]

at cluster-api, we are using bases instead of resources to work with kustomize 2.0,
folder structure does not need be changed, but it does need to add a kustomize.yaml in each folder.

bases:
- ../crds/
- ../rbac/
- ../manager/

please refer to https://github.com/kubernetes-sigs/cluster-api/blob/master/config/default/kustomization.yaml

[fanzhangio]

@Liujingfang1 Could you please help with it ?

[Liujingfang1]

Agree with the change suggested by @figo
We can change it to following directory structure

config/
   crds/
      kustomization.yaml
   rbac/
      kustomization.yaml
  manager/
     kustomization.yaml
  default/
     kustomization.yaml   # This one contains the bases pointing to rbac/ and manager/

[Adirio]

1. crds/ does not take part in the kustomize process.
2. What is the reason to call that folder default? (I know it is called like that right now)
3. Don't you think my scaffolding is clearer?

[figo]

@Adirio your proposal works from kustomize point of view, but it will not work for cluster-api case, i am arguing that cluster-api is not a special case.

in your design, rbac, crds, manager files generated by kube-builder stay within one kustomize folder, it can not be shared, but we have test, prod, staging environments that want to share the same rbac, crds, manager files.

thats why i think crds, rbac, manger become kustomize bases make sense to cluster-api and maybe others, those configure files/resources stay shareable.

[Adirio]

That answers 3, but 1 and 2 still hold

[alegacy]

I believe the reason for the "default" directory is so that a project can maintain multiple different kustomize overlays. "default", "production", "debug", etc. that each allow for customizations based on a different environments while still maintaining the same bases and resources. At least that is how we are using that directory level within our projects. I would like to see that continue to exist moving forward.

[Liujingfang1]

• crds/ does not take part in the kustomize process.

I took a look at the Makefile in a kubebuilder project. The CRDs are installed by

kubectl apply -f config/crds

In this case, we don't need to add kustomization.yaml to crds/.

• What is the reason to call that folder default? (I know it is called like that right now)

That folder is called default since it's provided by kubebuilder. It doesn't contain any further customization. Based on this default one, users can create different customizations for different environments, such as dev, staging.

[Liujingfang1]

For existing projects, please follow this instruction to migrate the config directory structure to the new structure, which works with kustomize 2.0.0

1. In config/rbac/ directory, add a kustomization.yaml with following content:

resources:
- rbac_role.yaml
- rbac_role_binding.yaml
  # Comment the following 3 lines if you want to disable
  # the auth proxy (https://github.com/brancz/kube-rbac-proxy)
  # which protects your /metrics endpoint.
- auth_proxy_service.yaml
- auth_proxy_role.yaml
- auth_proxy_role_binding.yaml

1. In config/manager/ directory, add a kustomization.yaml with following content:

resources:
- manager.yaml

1. In config/default/ directory, change following block in kustomization.yaml

# Each entry in this list must resolve to an existing   bases:
# resource definition in YAML.  These are the resource  - ../rbac
# files that kustomize reads, modifies and emits as a   - ../manager
# YAML string, with resources separated by document 
# markers ("---").  
resources:  
- ../rbac/rbac_role.yaml    
- ../rbac/rbac_role_binding.yaml    
- ../manager/manager.yaml   
  # Comment the following 3 lines if you want to disable    
  # the auth proxy (https://github.com/brancz/kube-rbac-proxy)  
  # which protects your /metrics endpoint.  
- ../rbac/auth_proxy_service.yaml   
- ../rbac/auth_proxy_role.yaml  
- ../rbac/auth_proxy_role_binding.yaml

to

bases:
- ../rbac
- ../manager

[droot]

Thanks @Liujingfang1 for adding the instructions for migration.

@Adirio Re:

What is the reason to call that folder default

Want to confirm what @alegacy and @Liujingfang1 said earlier that default is a sort of "default" kustomization provided by Kubebuilder, you can use this as a reference to create other kustomization staging, production etc. I realize we need to improve the docs in the kubebuilder book to convey this.

[itaysk]

@Liujingfang1 in step 3, what is ../base? I believe you meant ../rbac?

[Liujingfang1]

@itaysk Yes, thank you for catching it. I updated the steps.