Flux: Flux sync fails to run if there are any duplicate yamls in sync repo

[latchmihay]

Describe the bug
Flux sync fails to run if there are any duplicate yamls in sync repo

To Reproduce
create a duplicate file in a repo which flux syncs from, in my example i have place the same namespace file twice

Expected behavior
Ignore if they are duplicate and just apply over and over, or at-least have exceptions for namespace duplicates.
Perhaps duplicate checking could be a flag that operators can opt out of during flux daemon startup.

Logs

ts=2019-10-09T14:31:58.422718237Z caller=images.go:23 component=sync-loop error="getting unlocked automated resources: duplicate definition of '<cluster>:namespace/logging' (in logging/logging-namespace.yaml and logging-namespace.yaml)"

ts=2019-10-09T14:32:08.222398153Z caller=loop.go:101 component=sync-loop err="loading resources from repo: duplicate definition of '<cluster>:namespace/logging' (in logging/logging-namespace.yaml and logging-namespace.yaml)"

Additional context

• Flux version: 1.15.0
• Helm Operator version: null
• Kubernetes version: 1.15.3
• Git provider: github
• Container registry provider: null

[hiddeco]

Flux is a tool that automatically ensures that the state of a cluster matches the config in git.
— README.md

As namespaces can still contain different labels or other type of metadata, the outcome by either applying over-and-over, or having exceptions for namespace duplicates will both result in an unpredictable cluster state outcome.

Ignoring one of the two isn't an option either, as Flux isn't able to tell what file contains the actual truth.

Flux blocking this is thus simply to ensure a reproducible cluster state, which is one of the main goals.

[pierretasci]

As a counter point, I am trying to use Flux with Kustomize. It fails with the error message:

duplicate definition of '<cluster>:kustomization/' (in cluster/overlays/minikube/foo/bar/kustomization.yaml and cluster/overlays/minikube/laurem/ipsum/kustomization.yaml)

Each of those kustomize files setups a different controller in my cluster yet Flux believes they are applying the same change.

[latchmihay]

@pierretasci are the contents of the kustomization.yaml the same?

I assume they are.

[pierretasci]

They are very different.

The first one:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../../base/internal/foo

# Ensure all components are deployed into system namespace
namespace: system

patchesStrategicMerge:
  - patch-webhook.yaml

secretGenerator:
  - name: injector-tls
    type: "kubernetes.io/tls"
    files:
      - tls.crt=certs/system.svc.pem
      - tls.key=certs/system.svc-key.pem

The second one:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../../base/internal/bar

# Ensure all components are deployed into system namespace
namespace: system

Obviously both import further kustomize files but it seems to be complaining about the duplication of a Kustomize kind more than anything?

But at no point is there any duplication in what is being applied to the cluster.

[hiddeco]

@pierretasci your --git-path should be configured in such a way that it excludes Kustomization overlay directories (or any kustomization.yaml files in general).

It would however be friendlier from us to ignore the Kustomization resources when we encounter them and log a warning instead of aggressively slamming on the brake of the sync, which has been taken note of as a future improvement.

[ntx-ben]

I'm in the same boat as @pierretasci.

Using manifests factorization with kustomize. I have a single repository containing all my workloads for different overlays. Flux is instructed to read from a specific overlay using --git-path=overlays/staging. I would assume Flux would be able to recognize that there are multiple workloads to manage under the following setup:

/bases
---/app1
    kustomization.yaml (and other manifests)
---/app2
    kustomization.yaml (and other manifests)
/overlays
---/staging
------/app1
       .flux.yaml
       flux-patch.yaml
       kustomization.yaml
------/app2
       .flux.yaml
       flux-patch.yaml
       kustomization.yaml

The contents of each .flux.yaml is as such:

version: 1
patchUpdated:
  generators:
    - command: kustomize build ./app1 # $CWD is set to /overlays/staging
  patchFile: flux-patch.yaml

Getting the same error:

loading resources from repo: duplicate definition of '<cluster>:kustomization/' (in overlays/staging/app1/kustomization.yaml and overlays/staging/app2/kustomization.yaml

Is there any way to make this work using one instance of flux monitoring that /overlays/staging folder. I don't want to have an instance of flux per micro-service...

[tomjohnburton]

Exactly the same problem as @ntx-ben and @pierretasci

Is there no way for you to specify which environment flux should build based on where the flux pod is deployed?
+1

-fluxcd
|__base
|__overlays
-frontends
|__app1
  |__base
  |__overlays
   |__.flux.yaml
   |__stage
    |__kustomization.yaml
   |__sandbox
    |__kustomization.yaml
   |__production
|__app2
  |__base
  |__overlays

Same duplication error

[2opremio]

Getting the same error:

loading resources from repo: duplicate definition of '<cluster>:kustomization/' (in overlays/staging/app1/kustomization.yaml and overlays/staging/app2/kustomization.yaml

@ntx-ben Either you are running with --manifest-generation=true, or you are not supplying
--git-path correctly. Otherwise Flux wouldn't attempt to read overlays/staging/app1/kustomization.yaml nor overlays/staging/app2/kustomization.yaml

[2opremio]

This is derailing from the original proposal from the OP.

1. The OP was indicating the desire for Flux to tolerate duplicate Kubernetes resources (i.e same kind, name and namespace if applicable). I don't think we can do this. Flux simply can't know which of the two duplicates to apply. Only if they are completely identical (all keys and values are exactly the same) Flux could discard one of them. But, even then, I think that having a complete duplicate could indicate problem in the repository structure, so I am reluctant to do this since duplicates should be easily avoidable even in legitimate cases. Finally, also note that tools like Kustomize don't accept duplicates either.

2. On the other hand there is people finding problems because of their kustomization.yaml files are being read by Flux and interpreted as kubernetes resources (they aren't). This should be avoided by using correct values of --manifest-generation , --git-path and a directory structure which prevents Flux from scanning kustomization.yaml files. As @hiddeco has already pointed out, we could print a warning instead of aborting the sync. I am reluctant to do that since, in cases like @ntx-ben's , it can make the situation worse, like Flux applying partial merge patches(which can be valid kubernetes resources) inside the instead of the applying the patched resource through Kustomize.

[tomjohnburton]

@2opremio please see above for updated comment.

The error I am getting is the same as @ntx-ben. I can confirm --manifest-generation=true .
The way to fix it was to set my --gitpath to /frontends/app1/overlays/staging. But then obviously it just looks in that one directory.

Maybe there is a recommended way of structuring the repo?

[2opremio]

Is there no way for you to specify which environment flux should build based on where the flux pod is deployed?

Yes, the --git-path argument passed to Flux. See https://docs.fluxcd.io/en/latest/references/fluxyaml-config-files.html

[tomjohnburton]

so with my git path like this:
--git-path=kustomize/frontends/app1/overlays/staging,kustomize/frontends/app2/overlays/staging

It successfully deploys app1 but does nothing with app2

[2opremio]

It should work, with the right configuration and file layout.

Is there a .flux.yaml file in kustomize/frontends/app2/overlays/staging or any of its parents? Are you taking into account that the CWD will be set to kustomize/frontends/app2/overlays/staging ?

Please read https://docs.fluxcd.io/en/latest/references/fluxyaml-config-files.html in detail

[tomjohnburton]

I have a .flux.yaml in app1/overlays/staging and app2/overlays/staging

version: 1
commandUpdated:
  generators:
    - command:  kustomize build . 

Apologies if I am being super ignorant

[2opremio]

@tomjohnburton it's really hard to help you without a complete example, and we are abusing this issue since the OP was referring to something else.

Would you mind creating another issue or using the Slack channel? Please provide a self-contained example.

BTW, you will soon be able to debug manifest generation locally (as soon as https://github.com/fluxcd/flux/pull/2659 is merged)

[wmgroot]

Is it possible to simply ignore both duplicate resources, and continue to apply other resources normally?

My use case is that I have a multi-tenant cluster repository, and am serving multiple teams with the same flux process. If any team accidentally commits a duplicate resources, they block applies for all other teams. Having a feature (optional, non-default would be fine), to simply ignore both duplicate resources (with an error message) would be fantastic. I believe this functionality would be more intuitive and in line with the way that invalid yaml errors thrown by the kube api are handled.

[darktempla]

I am new to kustomize but pretty much I am just using it rather than vanilla YAML files to orchestrate order. I want the namespace created before other objects for example.

My current directory structure is as below and I set the --git-path=manifests/kube-ops-view,manifests/argocd. The idea here is I run k apply -k manifests/kube-ops-view to test against a kube instance running locally. Once happy I push to GIT and then run an update on fluxcd to look at the new path. This gives me control over what gets deployed and when.

When running the command above it works fine so I am not sure why Flux has issues running the command and getting a duplicate error. I would assume and likely incorrectly that the paths I set in config should be considered the working directory and anything outside the path should be ignored?

So my question is, is this a bug?

If not than could someone please recommend a better structure for Kustomize that allows me to pretty much do the same local test -> push approach with the ability to get Flux to ignore some roll-outs (dirs) if I choose to do so.

Error Message

ts=2020-10-25T03:02:45.6116684Z caller=sync.go:61 component=daemon info="trying to sync git changes to the cluster" old=974296c5f8472f7576e0d12783474c73caa8a9ad new=06cc458ec3412b8231def4edda89acd52f1e8583
ts=2020-10-25T03:02:45.671316Z caller=loop.go:108 component=sync-loop err="loading resources from repo: duplicate definition of '<cluster>:kustomization/' (in manifests/kube-ops-view/kustomization.yaml and manifests/argocd/kustomization.yaml)"

Directory Structure

├── README.md
├── install-flux.sh
└── manifests
    ├── argocd
    │   ├── install.yaml
    │   ├── kustomization.yaml
    │   ├── namespace.yaml
    │   └── update-to-latest.sh
    └── kube-ops-view
        ├── deployment.yaml
        ├── kustomization.yaml
        ├── namespace.yaml
        ├── rbac.yaml
        ├── redis-deployment.yaml
        ├── redis-service.yaml
        └── service.yaml

.flux.yaml

Get the same error with or without this file existing

version: 1
commandUpdated:
  generators:
    - command:  kustomize build . 

[Frederik-Baetens]

@darktempla idk if it's relevant to your use case, but as far as I know it is not possible to apply yaml files normally up until a certain point in the directory structure where a flux.yaml file is encountered, and only then start with manifest generation.

If you need to apply some yamls normally and do manifest generation for others, you either need to:

• Contain the files that need to applied normally (together with their own special .flux.yaml) in a subdirectory of the directory with the manifest generating .flux.yaml as explained here: https://docs.fluxcd.io/en/latest/references/fluxyaml-config-files/#the-scanforfiles-directive
• Or you need to just have two entirely separate directories and specify 2 git paths for those two directories.

[alexanderturner]

I'm having a similar issue.

I have a Kustomize repository with a base directory then a directory for each environment which patchStrategicMerges elements in the base directory. I also have a core directory which deploys things such as the ingress controller which are shared and untouched between app deployments.

Running kustomize build . in the root works fine, my Kustomize patch files changes namespaces for each environments. I'd like flux to run kustomize build . in the root dir, like I've specified in .flux.yaml and deploy the app following any locks or taints specified in flux-patch.yamls found in their respective sub directories.

I think the user should be able to specify whether Kustomize is used where flux trawls the directory to find yamls or give the user to option to specific their build command.

[Frederik-Baetens]

@alexanderturner you definitely can choose whether or not flux will crawl or generate a manifest from your command. (See docs about manifest generation and flux.yaml)