kustomize 3.5.5 breaks installing ArgoCD from URL

[chancez]

With kustomize 3.5.4, the following command succeeds and produces the YAML as expected:

kustomize build 'github.com/argoproj/argo-cd//manifests/ha/cluster-install?ref=v1.5.2

With kustomize 3.5.5 I now get the following error when running the command:

$ kustomize build 'github.com/argoproj/argo-cd//manifests/ha/cluster-install?ref=v1.5.2'
Error: accumulating resources: accumulateFile "accumulating resources from '../namespace-install': evalsymlink failure on '/private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/namespace-install' : lstat /private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/namespace-install: no such file or directory", loader.New "Error loading ../namespace-install with git: url lacks host: ../namespace-install, dir: evalsymlink failure on '/private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/namespace-install' : lstat /private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/namespace-install: no such file or directory, get: invalid source string: ../namespace-install"

[chancez]

May be related to https://github.com/kubernetes-sigs/kustomize/issues/2494

[monopole]

likely a dupe of #2444. Fix is in #2537, which is in v3.6.1
please give it a try. reopen if it doesn't work.

@Shell32-Natsu FYI

[chancez]

This doesn't seem resolved:

~/p/kubernetes-gitops(⎈ |dev-2447-us-west-2:jupyterhub) czibolski ❯❯❯ ~/Downloads/kustomize version                                                                          ⏎staging_prod_shared_logging_overlay ✭
{Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:darwin GoArch:amd64}
~/p/kubernetes-gitops(⎈ |dev-2447-us-west-2:jupyterhub) czibolski ❯❯❯ ~/Downloads/kustomize build 'github.com/argoproj/argo-cd//manifests/ha/cluster-install?ref=v1.5.2'      staging_prod_shared_logging_overlay ✭
Error: accumulating resources: accumulateFile "accumulating resources from '../namespace-install': evalsymlink failure on '/private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/kustomize-731707729/namespace-install' : lstat /private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/kustomize-731707729/namespace-install: no such file or directory", loader.New "Error loading ../namespace-install with git: url lacks host: ../namespace-install, dir: evalsymlink failure on '/private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/kustomize-731707729/namespace-install' : lstat /private/var/folders/0_/gjp5mdvn64lgly0qsrlzq1vc0000gp/T/kustomize-731707729/namespace-install: no such file or directory, get: invalid source string: ../namespace-install"

[chancez]

And I cannot seem to re-open this issue either.

[Shell32-Natsu]

I think the issue is different from #2444. Kustomize has changed to use go-getter by default. From the documentation of go-getter:

If you want to download only a specific subdirectory from a downloaded directory, you can specify a subdirectory after a double-slash //. go-getter will first download the URL specified before the double-slash (as if you didn't specify a double-slash), but will then copy the path after the double slash into the target directory.

So only manifests/ha/cluster-install will be copied and ../namespace-install is not there. You can try to replace // with /.

[seans3]

Re-opening so that we can triage it.

/reopen

[k8s-ci-robot]

@seans3: Reopened this issue.

In response to this:

Re-opening so that we can triage it.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chancez]

@Shell32-Natsu That does seem to make a difference. I'll try to make this change to upstream argo-cd install docs so that it's hopefully avoided for others. Though I imagine this might be an issue for other remote bases/overlays that people could be using.

[monopole]

Thanks @chancez and @seans3, and thanks @Shell32-Natsu for taking this on.

[jameshochadel]

I am encountering this issue on 3.6.1, but I am not using // anywhere in my URLs. However, the issue only occurs when the protocol is not specified.

E.g., where my-repo/dev/kustomization.yaml references a resource ../base, this fails:

kustomize build my-github-org/my-repo/dev

but this succeeds:

kustomize build https://my-github-org/my-repo/dev

This was not previously (before 3.6.1) the case.

[Shell32-Natsu]

@jameshochadel Hi James, thanks for the feedback. It looks like this is a go-getter issue. I have created issue here.

For your case, the problem is go-getter has bug with sub directory. Adding https works since go-getter fails with https scheme as well (because it's ambiguous I guess) and a normal git clone happens.

And the reason for why @chancez can fix the issue by replace // with / is go-getter doesn't work when you are using / and ref at the same time. So actually there is a normal git clone as well.

I am surprised that go-getter has these problems. I think we may need to clearly document the url pattern if we want to continue using go-getter.

[monopole]

@jameshochadel

please provide

• provide the output of the tree command,
• include the contents of the kustomization files (at least the resources: and/or bases: field)
• full command and value of working directory when running kustomize build
• error message

[jameshochadel]

@monopole

Here is a repo that reproduces the issue: https://github.com/jameshochadel/kustomize-issue-2538

Instructions are included in the readme (just running kustomize build against the repo with and without the protocol specified).

I think that covers everything. tree for completeness:

.
├── README.md
├── base
│   └── kustomization.yaml
├── dev
│   └── kustomization.yaml
├── no-https.sh
└── with-https.sh

[Shell32-Natsu]

@jameshochadel My speculation was correct. In no-https.sh , go-getter finishes with no error and only download the path (currently, / and // work nearly same in go-getter, which is different from their doc). If you add https, the go-getter fails and kustomize will clone the entire repo.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[docwhat]

/remove-lifecycle stale

[thatsmydoing]

Is it even possible using valid go-getter syntax to work with relative imports from parent directories? In the example, using github.com/jameshochadel/kustomize-issue-2538/dev results in only getting the dev folder which would cause referencing ../base to fail.

For now, the workaround seems to be to use a url that go-getter rejects so that it falls back to kustomize's old git clone strategy, but how is this supposed to actually work?

Should kustomize properly resolve ../base relative to the original url? In this case, to github.com/jameschochadel/kustomize-issue-2538/base? This seems like how I would expect it to work.

Otherwise, it seems like we need to specify 2 separate things, the go-getter url to fetch, and then the subdirectory within to use for the build.

[Shell32-Natsu]

@thatsmydoing Copy directory dev from the repo github.com/jameshochadel/kustomize-issue-2538. That is how go-getter works now. As they said in my issue these problems may be addressed in go-getter v2 but v2 is not released yet.

[thatsmydoing]

I think we're talking about 2 different things. The problem I'm trying to highlight is that when you have a resource like github.com/jameshochadel/kustomize-issue-2538//dev, what go-getter does is give you just the dev folder hence resources like ../base will fail. The issue linked seems to just be fixing issues where it's not parsing urls correctly, but in the end doesn't change the behavior of only getting the subdirectory.

What I instead usually want is to copy the whole repo github.com/jameshochadel/kustomize-issue-2538 and then build the kustomization at folder dev. This is how the cloner does it and that's why putting in an invalid go-getter url works.

I want to clarify because it all seems by chance that any of this works at all. Using go-getter syntax necessarily means that we are only getting the contents of a subdirectory (which does not work for anything that remotely uses the recommended kustomize project structure).

The way I see it, there are a couple ways forward here:

• separately specify what we want copied with go-getter and then the subdirectory we want to use inside what was copied
• make the loader smart by taking into account the parent loader when resolving relative imports

As it is now, saying that kustomize uses go-getter syntax just causes confusion because actually using legitimate go-getter syntax breaks anything that uses relative imports. This also means that when go-getter starts accepting more urls that it currently doesn't now, we'll run into this entire issue again of kustomize breaking relative imports.

[Shell32-Natsu]

Since go-getter has some annoying issues like this, I am happy to remove it from kustomize temporarily. What I am not sure about is the performance reduction that caused by calling external git exec. And also user will need to have a git executable in they PATH.

Perhaps using a go git library like go-git is better.

[thatsmydoing]

I'm not proposing getting rid of go-getter at all, removing it would probably break someone else's workflow which I think we want to avoid doing any more of. Note that go-getter itself uses external git so I'm not sure there's that much benefit to using go-git.

I think it would be better to aim for a proper fix that is consistent. I believe that requires clarifying the semantics of what should happen when you have remote resources and what happens with parent imports.

[thatsmydoing]

I did a bit more digging into this and I might be more persuaded to just drop go-getter at this point. Here is how kustomize currently handles fetching certain urls with go-getter. All of these urls work with the cloner. I don't know what the extent of usage of the following urls are, but I've certainly used some variation of these that doesn't work with go-getter.

URL | go-getter result
-- | --
github.com/jameshochadel/kustomize-issue-2538 | fetch success
github.com/jameshochadel/kustomize-issue-2538/ | fetch success
github.com/jameshochadel/kustomize-issue-2538/dev | fetch success
github.com/jameshochadel/kustomize-issue-2538//dev | fetch success
https://github.com/jameshochadel/kustomize-issue-2538 | wrong fetch
https://github.com/jameshochadel/kustomize-issue-2538/ | fallback to cloner
https://github.com/jameshochadel/kustomize-issue-2538/dev | fallback to cloner
https://github.com/jameshochadel/kustomize-issue-2538//dev | wrong fetch
[email protected]:jameshochadel/kustomize-issue-2538 | fetch success
[email protected]:jameshochadel/kustomize-issue-2538/ | fallback to cloner
[email protected]:jameshochadel/kustomize-issue-2538/dev | fallback to cloner
[email protected]:jameshochadel/kustomize-issue-2538//dev | fetch success
[email protected]/jameshochadel/kustomize-issue-2538 | fallback to cloner
[email protected]/jameshochadel/kustomize-issue-2538/ | fallback to cloner
[email protected]/jameshochadel/kustomize-issue-2538/dev | fallback to cloner
[email protected]/jameshochadel/kustomize-issue-2538//dev | fallback to cloner
ssh://[email protected]:jameshochadel/kustomize-issue-2538 | fallback to cloner
ssh://[email protected]:jameshochadel/kustomize-issue-2538/ | fallback to cloner
ssh://[email protected]:jameshochadel/kustomize-issue-2538/dev | fallback to cloner
ssh://[email protected]:jameshochadel/kustomize-issue-2538//dev | fallback to cloner
ssh://[email protected]/jameshochadel/kustomize-issue-2538 | fallback to cloner
ssh://[email protected]/jameshochadel/kustomize-issue-2538/ | fallback to cloner
ssh://[email protected]/jameshochadel/kustomize-issue-2538/dev | fallback to cloner
ssh://[email protected]/jameshochadel/kustomize-issue-2538//dev | fallback to cloner
git::ssh://[email protected]/jameshochadel/kustomize-issue-2538 | fetch success
git::ssh://[email protected]/jameshochadel/kustomize-issue-2538/ | fallback to cloner
git::ssh://[email protected]/jameshochadel/kustomize-issue-2538/dev | fallback to cloner
git::ssh://[email protected]/jameshochadel/kustomize-issue-2538//dev | fetch success

This means that even with #3244, the following urls will still be broken

https://github.com/jameshochadel/kustomize-issue-2538
https://github.com/jameshochadel/kustomize-issue-2538//dev

go-getter does not treat these as git repositories but as just standard http downloads.

There can obviously be more variations of this, and this doesn't cover the special handling kustomize has for azure and AWS repositories. It has been suggested that the special handling can be made into a go-getter detector but this doesn't perfectly work either because go-getter treats urls with schemes (https://, ssh://) as already normalized so they don't get transformed anymore. Additionally, even if kustomize worked around this by transforming the url itself before passing it to go-getter then there doesn't seem to be much point using go-getter at all.

Where go-getter works but the cloner doesn't is for archive downloads, but even then the usage of the feature is not obvious.

URL | go-getter result
-- | --
https://github.com/jameshochadel/kustomize-issue-2538/archive/master.zip | wrong fetch
https://github.com/jameshochadel/kustomize-issue-2538/archive/master.zip//kustomize-issue-2538-master | fetch success

I see that @yujunz added go-getter into kustomize, maybe they could shed more light on how they use this?

[yujunz]

A brief history of go-getter support

1. a full version of go-getter library was integrated in kustomize at the very beginning
2. go-getter was replaced by git exec later on because of massive transitive dependency
3. a script based plugin was created to integrate go-getter as binary
4. a lite version of go-getter is integrated into kustomize

In fact go-getter can support git clone. But we tried to keep backward compatibility to old cloner URLs when implementing step 4. I think that could be one reason of the chaos we are facing. Dropping go-getter will probably break the backward compatibility to current version.

Let me have a look of current implementation to see what is the best way out.

[yujunz]

I see that @yujunz added go-getter into kustomize, maybe they could shed more light on how they use this?

It was tested against the examples in https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md#url-format . But I roughly remembered the example tests were once excluded from CI. Not sure if it is covered now or not.

See also some history conversion in the two PRs for the problems go-getter tries to resolve and the tradeoffs

• https://github.com/kubernetes-sigs/kustomize/pull/2169
• https://github.com/kubernetes-sigs/kustomize/pull/2278

[thatsmydoing]

Dropping go-getter will probably break the backward compatibility to current version.

Aside from archive downloads and urls that use the depth query parameter, I don't think any url will be broken if go-getter is removed.

Either way though, I do think kustomize needs to be clear what the supported urls are. If the official position is that only go-getter urls are supported, then we would have just dropped the old cloner entirely but that didn't pan out. Some people have suggested removing go-getter but even then, there should be a spec as to what urls are supported since it accepts way too many variations and any kind of refactoring would be difficult without breaking someone's workflow. The repospec code is very convoluted and it's difficult to tell what kinds of urls it accepts and how it transforms them.

As a user, I just want to know what url formats are supported and that I can reasonably expect to never break and that kustomize will immediately fix if broken.

[yujunz]

Either way though, I do think kustomize needs to be clear what the supported urls are.

Yeah. It should be documented and covered by integration test.

We may need an exhaustive list in https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md#url-format which is covered by https://github.com/kubernetes-sigs/kustomize/blob/master/hack/testExamplesAgainstKustomize.sh

[yujunz]

The test seems triggered by https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/kustomize/kustomize-presubmit-master.yaml before submit any PR to master.

[yujunz]

I just realized the tests in remote build were skipped because of an extra blank line...

Proposed a fix in https://github.com/kubernetes-sigs/kustomize/pull/3253