Dep: Issue in dep check with files dropped by .gitignore

[tariq1890]

What version of dep are you using (dep version)?

dep 0.5.0

What dep command did you run?

dep check

Issue: We are trying to integrate dep check into our ci/cd workflows. But we face a problem when of our vendor directories have files that are included in the .gitignore. When we run dep ensure in our local machine, the computed hash will always take that ignored file into account. In our ci/cd, however, the gitignored file will not be present and during the dep check, the computed hash will always end up being different from the hash in the gopkg.lock.

I feel that this issue poses a challenge for those of us who'd want to use dep check in ci/cd pipelines. One solution could be to just prune the files that are included in .gitignore.

[tariq1890]

@sdboyer @carolynvs Is it possible to have some discussion about this? dep check is very useful for us and this issue precludes us from using it in our CI processes.

I am happy to implement the required changes through a PR if we come to some consensus on the required approach here.

[ondrej-fabry]

For us it happens for this dependency: https://github.com/go-errors/errors

It contains cover.out file which is by default ignored by .gitignore files, especially for Go: https://github.com/github/gitignore/blob/master/Go.gitignore#L12

The dep check, which is run by Travis, always fails because of this.

[AlekSi]

dep ensure
git add --force vendor/
git commit

That should do the trick.

[sdboyer]

hi! i'm sorry, but there's no reasonable way to make dep check accommodate .gitignore files. It accounts for your prune settings - that's as good as can be done, and is usually good enough when it comes to trimming down the size of vendor. (Unfortunately, using .gitignore in this way was a bit of a hack from the beginning).

If there are specific projects that you're trying to exclude from verification checks because there are .gitignored parts, you can use the new noverify Gopkg.toml field. This will make your dep ensures less efficient, as it will revert to the old behavior of always writing out that project on every run. But you'll be able to use dep check without erroneous failures in CI.

For cases like @ondrej-fabry's, where a default .gitignore is ignoring a file, the one-time operation @AlekSi describes is the best solution - thanks for that!

[tariq1890]

@sdboyer @AlekSi , Thanks for your inputs.

To clarify, my case is similar to that of @ondrej-fabry 's. In my case we are gitignoring an .editorconfig file and one of our vendored repositories has that file added to their repo. This case will not be uncommon I am sure.

While the solution provided works, it seems very hacky. To other devs who may fork/clone our repositories, this may not be obvious and can possibly cause confusion esp. If they wanted to add a new dependency which had gitingnore conflicts. They would have to force add the vendor as well. Developer/User experience wise, this is counter-intuitive. I am also reluctant to normalize the practice of git force adds. It's certainly not elegant.

[sdboyer]

I get it. I do - again, there's the noverify workaround, if needed. I'd also be happy to accept a PR that makes a note on the command docs suggesting that .gitignore may be the problem.

i could also imagine even just a shell script that checks for any files being ignored in such a way, as a helper.

but the whole design of dep check is based on the idea that any input that could affect the state in vendor is structured and well-defined - e. g. prune opts change, or the package list, or the version. it is that property that lets us trivially determine when toml/code/lock/vendor are out of sync.

gitignore, OTOH, is arbitrary, which means we can't tell from one dep run to the next if its contents have changed, and therefore vendor should be regenerated even if hashes match.

its convoluted to think about, I know. think about it this way - if pruneopts wasn't in the lock, how would we know when it changed in the toml, thereby necessitating regenerating vendor?

[ondrej-fabry]

@tariq1890 this is how I dealt with it: https://github.com/ligato/vpp-agent/blob/pantheon-dev/.gitignore#L36

I've added this line to .gitignore:

!/vendor/**/*

This will also prevent other developers, working on this repo, from doing similar mistake.

[bogdanpetrea]

@ondrej-fabry: Unfortunately that only works for stuff ignored in the same .gitignore file (and probably levels above). If the dependencies have their own .gitignore files it doesn't override them, at least for me. I'm using git version 2.18.0.

[ondrej-fabry]

@bogdanpetrea Not sure I understand your problem correctly. You mean one of the deps in vendor directory has its own .gitignore file in which they defined some files to be ignored and still commit the file anyway? Because otherwise I'm not sure how the different hash could be calculated.

[tariq1890]

@sdboyer Appreciate your detailed response. I think I am fine with Ondrej's solution. I might just go forward with that. Like you had mentioned earlier, I am happy to raise a PR to highlight these dep-check related caveats in the dep docs.

@ondrej-fabry Thanks! This workaround certainly seems pragmatic.

[AlekSi]

Did you consider to ask maintainers to remove files that should not be committed, or were committed and then gitignore patterns were erroneously updated to include them?

[bogdanpetrea]

@ondrej-fabry Yes, I think you are right.
@AlekSi It could be a solution, although not a definitive one. Might just do git add --force /vendor instead. Thanks!