dep ensure local packages

[avelino]

dep support create vendor based on local packages?

like:

govendor add +e

[prasannavl]

Yes. I'm interested in this as well. Gives you the ability to checkout code anywhere and just build it as one contained unit, rather than having to check it out to a GOPATH to build it relying on the assumption that it has all the external dependencies not already vendored - which seems quite fragile from a project management point of view.

[themartorana]

Came here to ask for this. My projects look something like:

$GOPATH/src/davelogging
$GOPATH/src/daveerrors
$GOPATH/src/daveproject

daveproject imports davelogging and daveerrors - but even when converting from govendor they are not dropped in to the vendor directory, which obviously breaks CI if I commit the repository. I could put everything in its appropriate github repo like $GOPATH/src/github.com/Dave/davelogging, but considering goimports supports local projects, I'd like dep to as well. (Also, we're lucky enough to use GH which gives us a workaround, but as mentioned in #286 not everyone is.)

Thanks!

[sdboyer]

ugh, i'm sorry, i don't know why i never got around to directly answering this in the first place. thanks for your (multi-month) patience.

so, this is a tricky thing for dep to do safely and maintain its safety guarantees. projects like vg may facilitate some hacks sorta in the interim, but the biggest issue here is that trusting arbitrary local state really alters dep's model, and in doing so introduces a lot of new failure modes for us to deal with, and for users to potentially run into.

if nothing else, consider this: when we operate only on upstream repositories, we know that the revisions we operate on actually do immutably represent a code tree. if we're looking at local state, though, then what the e.g. git repository says the current revision is doesn't necessarily match the code tree, because it could be dirty (modified files), or just have untracked files that would still be picked up.

this effectively means that we don't have a revision we could accurately put into Gopkg.lock, as it wouldn't actually correspond to the code that's on disk - anybody trying to dep ensure that lock on a different machine wouldn't actually get the same vendor directory contents. and that's still assuming the revision on local disk has been pushed to a canonical location.

we can make workarounds for these things - other language package managers certainly have, to enable this kind of behavior. and that is something i want to do - but sadly, i've been saying that for ~15 months now, as there's a whole bunch of design issues that would need to be considered for this; other, more targeted issues have continued to be more pressing. tbh, it's likely to continue like that for a while. however, if there's an intrepid Gopher out there who wants to win fame in and love of the entire Go community by taking on the job of grokking the problem and designing out a solution, then i am 💯💯 willing to guide that 🎉 amazing human 🎉 through the process 😄

[avelino]

@sdboyer I agree with the possible security flaw in adding local files.
Security liability must be the person who decided to use local files, so we take the responsibility of the dep.

[leaxoy]

This may be helpful feature. In lots of case, we use internal package which store in local, but dep doesn't support this.

[roadrunner]

other DM tools figure out this problem with hacky ways; and since we're all somewhat developers, we're hacking tools if there is no other way around.
i'm pretty sure most of us working on something like project A depends on project B and both depends on same open source projects.
every iteration like; modify dependency source code -> submit changes -> push origin -> dep ensure literally kills the productivity.

safety guaranty is an awesome feature of dep, everyone of us loves that, and no one thinks any other alternative for the production environments.

however, in the dev/local environment, we all develop or try to implement new features; most of the iterations end up dirty state with broken things.
since we're spending most of our times in the dirt; obviously, we like to use tools play nice with the dirt as well.

[JelteF]

To solve this problem at my company I created virtualgo: https://github.com/GetStream/vg
It's a small wrapper around dep that moves vendor directory to a project specific GOPATH after running ensure. It also adds a vg localInstall <package> [path] command that makes it super easy to try a modified local checkout of a dependency.

[hasufell]

if nothing else, consider this: when we operate only on upstream repositories, we know that the revisions we operate on actually do immutably represent a code tree. if we're looking at local state, though, then what the e.g. git repository says the current revision is doesn't necessarily match the code tree, because it could be dirty (modified files), or just have untracked files that would still be picked up.

I don't understand what you mean with this. If you clone from a local git repository you end up with the same guarantees as cloning from a remote git repository. It doesn't matter whether there are unstaged/dirty files. That's not how git works.

[sdboyer]

right, and in this case, that's the problem: what is retrieved into vendor is not the same as what is on in the working tree on disk. however, what's on disk is what many people would be expecting out of this use pattern, because they're looking for a comfortable flow for rapid hacking.

so, either they get the "wrong" code, or we don't move a tree into vendor at all, and they get the "right" code, but the only got commit hash we have to put into Gopkg.lock does not describe reality.

On January 30, 2018 7:29:37 AM EST, Julian Ospald [e-mail] wrote:

if nothing else, consider this: when we operate only on upstream
repositories, we know that the revisions we operate on actually do
immutably represent a code tree. if we're looking at local state,
though, then what the e.g. git repository says the current revision is
doesn't necessarily match the code tree, because it could be dirty
(modified files), or just have untracked files that would still be
picked up.

I don't understand what you mean with this. If you clone from a local
git repository you end up with the same guarantees as cloning from a
remote git repository. It doesn't matter whether there are
unstaged/dirty files. That's not how git works.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/golang/dep/issues/935#issuecomment-361579029

[hasufell]

what is retrieved into vendor is not the same as what is on in the working tree on disk. however, what's on disk is what many people would be expecting out of this use pattern, because they're looking for a comfortable flow for rapid hacking.

How is that a problem? That is totally expected when working with git.

[sdboyer]

asked and answered, please reread

On January 30, 2018 8:23:56 AM EST, Julian Ospald [e-mail] wrote:

what is retrieved into vendor is not the same as what is on in the
working tree on disk. however, what's on disk is what many people would
be expecting out of this use pattern, because they're looking for a
comfortable flow for rapid hacking.

How is that a problem? That is totally expected when working with
git.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/golang/dep/issues/935#issuecomment-361591858

[hasufell]

I did read it, but it appears to me those are just assumptions. Basically you are excluding a certain kind of git uri scheme, because you think some people might not know what it means.

[niondir]

Any news on this?

[Adrien-Luxey]

I'm also interested.

@roadrunner put it well:

every iteration like; modify dependency source code -> submit changes -> push origin -> dep ensure literally kills the productivity.

I keep pushing untested/buggy code to my repo; this is certainly not idiomatic git usage. Ideally I wouldn't even want to commit the code I am testing before I tested it by executing my program.

[maja42]

@roadrunner / @Adrien-Luxey
I have the same issues in my projects. Is there are workaround for now?

[Adrien-Luxey]

Nope, that's why I'm complaining :grin:

[JelteF]

Like I already commented before, I created (and still use) vg to workaround this issue: https://github.com/GetStream/vg

[krzysztofantczak]

@JelteF does it work well with multiple GO versions?

[JelteF]

@krzysztofantczak, not built in. But it should work when you set the GOROOT manually.

[krzysztofantczak]

@JelteF yea, i already tested. It works pretty well with GVM ;-)

[furstenheim]

In case it is of any help. Node package managers (npm and yarn) work as follows: in the folder of the project that want to be depended on, ỳarn link will register the project. Then in the project that depends on the first one yarn link $name_of_first_project will replace the dependency folder by a symlink, so every change is automatically detected. https://yarnpkg.com/lang/en/docs/cli/link/

[aspotashev]

I use a workaround mentioned here: https://github.com/golang/dep/issues/1775

More specifically, I added this into Gopkg.toml to make "dep ensure" skip download of gocv and let golang compiler use local version from $GOPATH/src/gocv.io/x/gocv:

ignored = ["gocv.io/x/gocv"]

[th317erd]

So what if "ignored" in Gopkg.toml could just fall back to an environment variable if not set (or merged with an environment variable if set)? That way in my projects I could just set an environment variable for the packages that I want it to ignore. This way the modified "Gopkg.toml" (adding "ignored") won't accidentally make it into source control, and as a developer I could easily make scripts locally to set the environment variables automatically.

[themartorana]

@kevinburke is this solved?

[kevinburke]

I'm not sure, I just went through and closed all old issues yesterday, see #2166. It's entirely possible this is still an issue; it's also possible it's been fixed!