Helm-operator: Helm operator : Could not read from remote repository.

[Docteur-RS]

Describe the bug

Problem 1
For some reason the Flux helm operator can't connect to our gitlab anymore...

ts=2019-10-15T15:45:26.832760944Z caller=chartsync.go:533 component=chartsync info="chart repo not ready yet" resource=logging:helmrelease/fluentd-elasticsearch status=new err="git clone --mirror: fatal: Could not read from remote repository., full output:\n Cloning into bare repository '/tmp/flux-gitclone975152289'...\nGitLab: The project you were looking for could not be found.\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"

However Flux still works and deploys standalone yamls perfectly. Its only the helm operator that has connectivity issues. Even after being restarted multiple times by myself the error persists.

The above warning repeats itself for each helm release present on the cluster. Even for the ones that were NOT deployed by this helm operator.
For instance the fluentd-elasticsearch chart in the above message is not part of the git repository. I don't get why Flux would try to clone something that is NOT in the repository...

Problem 2

I also have this issue on charts that are in the repository:

ts=2019-10-15T15:45:17.433825329Z caller=chartsync.go:533 component=chartsync info="chart repo not ready yet" resource=opa-integ:helmrelease/gitlab-runner-integ status=new err="git clone --mirror: fatal: Could not read from remote repository., full output:\n Cloning into bare repository '/tmp/flux-gitclone597278666'...\nGitLab: The project you were looking for could not be found.\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"

Always the same error on each deployed charts ...

FYI, I have a second Flux which is working perfectly in another namespace and synchronizing a second repository connected to the same Gitlab instance.

To Reproduce

Steps to reproduce the behaviour:

• Install 2 Flux via Helm in your cluster
• Connect them to the same hosted Gitlab on 2 separate repositories
• Install and remove charts
• Wait for something to go wrong (quick step)

Expected behavior

Helm operator should still work and sync ressources.

Additional context

Add any other context about the problem here, e.g

• Flux version: 1.13.2
• Helm Operator version: 0.10.0
• Kubernetes version: 1.14.3
• Git provider: Gitlab
• Container registry provider: Nexus

It must be an error on my end.
Any ideas are welcome ;-)
Thx

[hiddeco]

What are the configurations of your Flux and Helm operator instances?

[Docteur-RS]

After having destroyed and restarted everything I still got the issue and I only have one flux running. So its not about having multiple flux running in the same cluster.

Moreover when I kubectl exec inside the helm operator pod, I can clone the repository without any problem.

[Docteur-RS]

@hiddeco I made a gist with the configuration of the flux and helm operator pods : https://gist.github.com/Docteur-RS/36e05dd93853d9d588fa70fc9bb19626

And this is the values.yaml of Flux : https://gist.github.com/Docteur-RS/d8e6d16f89c9fa60cce0d5f02d297f1f

Even though "ssh>know_hosts" is empty, the hosts keys are correctly mounted in the pod. I have checked the content of .ssh/know_hosts and have manualy cloned the repo from inside the container. Everything runs fine...

[Docteur-RS]

I found the issue. As expected it was my fault...

Because I have so many HelmRelease files I messed up the spec>chart>git value's.
I set the wrong repository at the end of the URL.

...
spec:
  releaseName: someName
  chart:
    git: ssh://[email protected]:10022/<THIS_PART_WAS_WRONG>
    path: charts/flux
    ref: master
  values:
...

[hiddeco]

@Docteur-RS I already had my suspicions that it was due to some mistake in one of the HelmRelease resources. Glad you were able to get to the bottom of it by yourself, as I wouldn't have been able to go through all the files for you.

[Docteur-RS]

@hiddeco yep ! Thx for your time.