Skaffold: [feature request] multiple Kustomizations

[paultiplady]

Some kustomize workflows may involve multiple kustomizations, e.g. as suggested in https://github.com/kubernetes-sigs/kustomize/issues/168, having one kustomization for a db migration, followed by another for the main deploy.

More generally, since it's supported to have multiple helm releases in the helm deployer, by analogy it seems we should support a list of kustomizations in the kustomize deployer, instead of just one.

Information

• Skaffold version: v0.25.0
• Operating system: MacOS
• Contents of skaffold.yaml:

```yaml
apiVersion: skaffold/v1beta7
kind: Config
build:
artifacts:
- image: us.gcr.io/my-image/monolith
deploy:
kustomize:
path: deploy/kustomize/overlays/minikube

profiles:

• name: builds
  deploy:
  kustomize:
  path: deploy/kustomize/overlays/builds```

[paultiplady]

I'm imagining something like

...
deploy:
  kustomize:
    - path: deploy/kustomize/overlays/minikube
    - path: deploy/kustomize/overlays/other

[corneliusweig]

@paultiplady https://github.com/kubernetes-sigs/kustomize/issues/168 talks about running kustomize twice with different kustomization.yaml. If there are two kustomizations in one deployer, how should Skaffold know which of the two kustomizations should be applied? If the answer is "both", why not have one kustomization including the others?
As far as I understand your problem, I think the right approach is to have multiple Skaffold profiles overriding the kustomization.

[tejal29]

I agree with @corneliusweig, you could create variants using overlays like this?
https://github.com/kubernetes-sigs/kustomize#2-create-variants-using-overlays

I am closing this issue for now, please re-open if you think this does not support your use case

[ItsLeeOwen]

Following the example @tejal29 linked to I organized overlays to differentiate service environments such as development, production, sre-playground, but used fully separate kustomization.yaml manifests per service such as api & ui.

api / base /...
api / overlays / development / ...

ui / base /...
ui / overlays / development / ...

My initial hope was to point skaffold at a list of kustomization paths like @paultiplady's suggestion, which appears similar in spirit to the microservices example.

I'm not sure what restructuring kustomization manifests in the manner that @corneliusweig suggests will look like, but am going to give it a go.

[kri5]

@ItsLeeOwen @paultiplady I'm facing the same issue, did you manage to work around?

If so, what does your hierarchy looks like?

@tejal29 @corneliusweig Kustomize has now some load restrictions that prevents by default to include something else in overlays other than a base, so it forces some kind of hierarchy iirc.

[davestephens]

I'd expect multiple kustomisations to work in exactly the same way you can deploy multiple kubectl manifests, or multiple helm chart. It doesn't make sense (to me) you'd support one and not the other, or expect an overarching kustomisation when this isn't the case with helm or kubectl.

My use case is that I have multiple microservices, each microservice has its own Docker image and set of manifests (each with their own kustomisations). All of these services make up the application stack that I want to deploy with Skaffold.

[corneliusweig]

@kri5 @davestephens Looking at the documentation for kubectl apply, it says

 -f, --filename=[]: that contains the configuration to apply
 -k, --kustomize='': Process a kustomization directory. This flag can't be used together with -f or -R.

So it seems quite clear to me, that one can pass multiple resources (-f), but just _one_ kustomization (-k) which needs to point to a directory with a kustomization.yaml.

Consequently, Skaffold would distort the design principle of kustomize, if we were to support multiple kustomization.yaml to define deployment resources.

Do you agree?

[davestephens]

@corneliusweig My understanding was the complete opposite, actually.

eg, if you do a kubectl apply -f <directory> - you would generally expect this to be one service - same as if you did a helm install <myapp>.

Perhaps I'm misunderstanding the use case of Skaffold, but my understanding was that you could use it to build multiple images, helm charts, manifests etc, and push them to your k8s cluster? Building and pushing multiple kustomisations feels like the missing link to me...

[pbriet]

@davestephens I totally agree with you.

I have an environment where there are multiple "projects" some that need kustomization, and other don't

So I'd like to :

• select multiple kustomize path
• use both kubectl and kustomize deployers

And I can't do any of that...

[ichtestemalwieder]

@tejal29 and @paultiplady. Could you please reopen this issue! This is an absolute basic feature as lot of people here have requested (and reacted/commented). I was so happy the first minutes about skaffold as it seemed to fulfill all developer needs but then I got so disappointed that this basic feature is missing.

Summarizing, as all the other comments already state:

• Helm supports multiple Charts
• kubectl supports multiple Ressource Config dirs

And of course kustomize should work the same way too.
Quoting paultiplady's solution/example (with slight modification). Thats exactly what would be required:

> deploy:
>   kustomize:
>     - path: devopos/mysql
>     - path: devopos/memcached
>     - path: monorepo/packages/restapi
>     - path: monorepo/packages/frontend

Thanks very much!

[corneliusweig]

Kustomize solves this by creating a new variant with several bases (see https://github.com/kubernetes-sigs/kustomize/tree/master/examples/multibases). So @ichtestemalwieder's example just needs a kustomization.yaml with the following content:

resources:
- devopos/mysql
- devopos/memcached
- monorepo/packages/restapi
- monorepo/packages/frontend

Maybe someone can explain, why this solution does not work for you?

[ichtestemalwieder]

@corneliusweig: Thanks very much for your answer!

As far as I understeand multibases, it is the equivalent of "helm umbrella charts". Yes it seems to be a solution, but not an ideal one. As it requires to setup an additional and "external" (to skaffold) "meta/aggregator" configuration file (namely the multibases kustomization.yaml file).

Conclusion and potential Improvements:
1.) The multibases "workaround" should be better documented in skaffold (and even in the kustomize docs). This is not obvious at all, when reading the skaffolld docs. (But at least it is now documented here).
2.) Nevertheless, skaffold has an inconsistency and something that reduces Developer Experience/Effectiveness: Multiple Helm charts (and kubernetes files...) can be imported directly (without using umbrealla charts). However with kustomize we need an additional "aggregator/umbrella" kustomization.yaml.

[corneliusweig]

This issue was discussed during the Skaffold bi-weekly meeting on Jan 22. The core team agrees that Skaffold should treat helm and kustomize in the same way, see #3585 for details.

Btw, everybody is welcome to bring up issues like these during that meeting. That's what it is there for :)