Helmfile: Go-getter URLs from #1374 are not working

[Nuru]

I am trying to use go-getter URLs as described in #1374 in Helmfile v0.125.2 but they are not working for me:

releases:
  - name: "forecastle"
    chart: git::https://github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54
    forceGoGetter: true

Helmfile does not seem to intercept this URL, it just passes it along to helm, which of course cannot handle it.

I tried http instead of https, with and without .git, and replacing @ with // as described in the go-getter docs.

Error output

$ helmfile -b /usr/bin/helm3 template
Fetching git::http://github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54
Removing remove .helmfile/cache/http_github_com_stakater_Forecastle_git.ref=v1.0.54/deployments/kubernetes/chart/forecastle: directory not empty: %!v(MISSING)
in ./helmfile.yaml: [command "/usr/bin/helm3" exited with non-zero status:

PATH:
  /usr/bin/helm3

ARGS:
  0: /usr/bin/helm3 (14 bytes)
  1: fetch (5 bytes)
  2: git::http://github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54 (98 bytes)
  3: --version (9 bytes)
  4: v1.0.54 (7 bytes)
  5: --untar (7 bytes)
  6: --untardir (10 bytes)
  7: /tmp/934991620/portal/v1.0.54/git::http:/github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54 (127 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Error: non-absolute URLs should be in form of repo_name/path_to_chart, got: git::http://github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54

COMBINED OUTPUT:
  Error: non-absolute URLs should be in form of repo_name/path_to_chart, got: git::http://github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54]

[mumoshu]

@Nuru Thanks as always for your support 🙏 This was definitely a bug, which is being fixed via #1405 soon.

// BTW, I'm planning to enhance our integration test suite so that regressions like this won't happen in the future.

[lukasmrtvy]

@mumoshu Can You take a look at something like this, please?

releases:
- name: gitlab
  namespace: gitlab
  chart: git::https://gitlab.com/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master&sparse=0
  forceGoGetter: true

in ./helmfile.yaml: [fetching "git::https://gitlab.com/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master&sparse=0": get: error downloading 'https://gitlab.com/gitlab-org/charts/gitlab.git?ref=master&sparse=0': /usr/bin/git exited with 128: Cloning into '.helmfile/cache/https_gitlab_com_gitlab-org_charts_gitlab_git.ref=master_sparse=0'...
fatal: unable to access 'https://gitlab.com/gitlab-org/charts/gitlab.git?sparse=0/': Maximum (20) redirects followed

Thanks ! <3

EDIT:
Weird thing is that I am getting different error without sparse arg
``` releases:

• name: gitlab
  namespace: gitlab
  chart: git::https://gitlab.com/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master
  forceGoGetter: true

Removing remove .helmfile/cache/https_gitlab_com_gitlab-org_charts_gitlab_git.ref=master/charts/gitlab: no such file or directory: %!v(MISSING)
Templating release=gitlab, chart=git::https://gitlab.com/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master
in ./helmfile.yaml: command "/usr/local/bin/helm" exited with non-zero status:

PATH:
/usr/local/bin/helm

ARGS:
0: helm (4 bytes)
1: template (8 bytes)
2: gitlab (6 bytes)
3: git::https://gitlab.com/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master (77 bytes)
4: --namespace (11 bytes)
5: gitlab (6 bytes)

ERROR:
exit status 1

EXIT STATUS
1

STDERR:
Error: failed to download "git::https://gitlab.com/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master" (hint: running helm repo update may help)

COMBINED OUTPUT:
Error: failed to download "git::https://gitlab.com/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master" (hint: running helm repo update may help)
```

[mumoshu]

@lukasmrtvy Could you try chart: git::ssh://[email protected]/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master&sparse=0 instead? I believe go-getter doesn't support https-access for gitlab

[lukasmrtvy]

SSH:
chart: git::ssh://[email protected]/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master&sparse=0

in ./helmfile.yaml: [fetching "git::ssh://[email protected]/gitlab-org/charts/gitlab.git@charts/gitlab?ref=master&sparse=0": get: error downloading 'ssh://[email protected]/gitlab-org/charts/gitlab.git?ref=master&sparse=0': /usr/bin/git exited with 128: Cloning into '.helmfile/cache/ssh_gitlab_com_gitlab-org_charts_gitlab_git.ref=master_sparse=0'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
]

Imported to Github:
chart: git::https://github.com/lukasmrtvy/gitlab-chart.git@charts/gitlab?ref=master

COMBINED OUTPUT:
  Error: failed to download "git::https://github.com/lukasmrtvy/gitlab-chart.git@charts/gitlab?ref=master" (hint: running `helm repo update` may help)

With sparse=0 arg:
chart: git::https://github.com/lukasmrtvy/gitlab-chart.git@charts/gitlab?ref=master%sparse=0

COMBINED OUTPUT:
  Error: failed to download "git::https://github.com/lukasmrtvy/gitlab-chart.git@charts/gitlab?ref=master%sparse=0" (hint: running `helm repo update` may help

[mumoshu]

@lukasmrtvy Which version of Helmfile are you using? The fix for this issue #1405 isn't released yet as a helmfile release. So you'd need to build your own binary to test it.

[mumoshu]

Anyway, I've just released v0.125.4 😃 Would u mind testing it out, if you weren't using your own binary?

[lukasmrtvy]

sorry, my mistake ..

releases:
  - name: "forecastle"
    chart: git::https://github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54
    forceGoGetter: true

is working OK in v0.125.4 !

I did not find any info about sparse arg in go-getter repository, maybe its not supported..

I had to switch chart from gitlab to forecastle, gitlab has somehow specific chart btw..

repositories:
- name: fook
  url: git+https://gitlab.com/gitlab-org/charts/gitlab@.?ref=master&sparse=0

releases:
- name: gitlab
  namespace: gitlab
  chart: fook/gitlab
  values:
    - certmanager-issuer:
        email: [email protected]

is working OK !

This syntax:

releases:
- name: gitlab
  namespace: gitlab
  chart: git::https://gitlab.com/gitlab-org/charts/gitlab@
  forceGoGetter: true
  values:
    - certmanager-issuer:
        email: [email protected]

outputs:

COMBINED OUTPUT:
  Error: found in Chart.yaml, but missing in charts/ directory: cert-manager, prometheus, postgresql, gitlab-runner, grafana, redis

no sure how its working for deps...

Thanks

[lukasmrtvy]

@mumoshu

Btw ( Related: https://github.com/roboll/helmfile/pull/1374 ), this is working now:

releases:
- name: cert-manager-crds
  namespace: cert-manager
  chart: git::http://github.com/jetstack/cert-manager.git@deploy/crds?ref=v0.15.2

[mumoshu]

@lukasmrtvy Thanks for your detailed reports! Glad to hear it (almost) worked for you.

releases:
- name: gitlab
  namespace: gitlab
  chart: git::https://gitlab.com/gitlab-org/charts/gitlab@
  forceGoGetter: true
  values:
    - certmanager-issuer:
        email: [email protected]

Re this one, I'm wondering if the gitlab chart is incomplete that we need to run helm dep build after git-cloning it? Could you check if the gitlab chart has Chart.lock?

Recently, Helmfile has been updated in #1400, to NOT run helm dep build on the fetched chart to avoid blocking on a "broken" chart that has an outdated Chart.lock. Apparently, we need to run helm dep build on a fetched chart without Chart.lock, while it shouldn't be run on a chart with an outdated Chart.lock?

[mumoshu]

So I submitted #1408 to run helm dep build on a chart fetched with go-getter. With that chart: git::https://gitlab.com/gitlab-org/charts/gitlab@ worked for me.