glide get did not fetch recursive dependencies

[mh-cbon]

Hi,

I hope i did not misread the readme in

When glide get is used it will introspect the listed package to resolve its dependencies including using Godep, GPM, Gom, and GB config files.

$ go version
go version go1.6.2 linux/amd64
$ glide -v
glide version 0.10.2-86-g5865b8e
$ mkdir test
$ cd test
$ glide create
[INFO]  Generating a YAML configuration file and guessing the dependencies
[INFO]  Attempting to import from other package managers (use --skip-import to skip)
 $ glide get github.com/mh-cbon/verbose
[INFO]  Preparing to install 1 package.
[INFO]  Importing github.com/mh-cbon/verbose
[INFO]  Downloading dependencies. Please wait...
[INFO]  Fetching updates for github.com/mh-cbon/verbose.
[INFO]  Resolving imports
[INFO]  Downloading dependencies. Please wait...
$ ll vendor/github.com/mh-cbon/
total 4
drwxrwxr-x 6 mh-cbon mh-cbon 4096 10 juin  14:12 verbose
$ ll vendor/
total 4
drwxrwxr-x 3 mh-cbon mh-cbon 4096 10 juin  14:12 github.com
$ 

Notes: glide update works fine and pull dependencies recursively.

[mh-cbon]

I have similar problems with latest on a different repo, this time glide update did not help

$ glide create
[INFO]  Generating a YAML configuration file and guessing the dependencies
[INFO]  Attempting to import from other package managers (use --skip-import to skip)
[INFO]  Scanning code to look for dependencies
[INFO]  Writing configuration file (glide.yaml)
[INFO]  Would you like Glide to help you find ways to improve your glide.yaml configuration?
[INFO]  If you want to revisit this step you can use the config-wizard command at any time.
[INFO]  Yes (Y) or No (N)?
y
[INFO]  Looking for dependencies to make suggestions on
[INFO]  --> Scanning for dependencies not using verion ranges
[INFO]  --> Scanning for dependencies using commit ids
[INFO]  Gathering information on each dependency
[INFO]  --> This may take a moment. Especially on a codebase with many dependencies
[INFO]  --> Gathering release information for dependencies
[INFO]  --> Looking for dependency imports where versions are commit ids
[INFO]  No proposed changes found. Have a nice day.
$ ll
total 16
drwxrwxr-x 2 mh-cbon mh-cbon 4096  9 juin  13:38 cmd
-rw-rw-r-- 1 mh-cbon mh-cbon   44 10 juin  15:22 glide.yaml
-rw-rw-r-- 1 mh-cbon mh-cbon   13  9 juin  13:41 gump.go
-rw-rw-r-- 1 mh-cbon mh-cbon   86  9 juin  13:39 README.md
$ glide get github.com/mh-cbon/go-repo-utils
[INFO]  Preparing to install 1 package.
[INFO]  Attempting to get package github.com/mh-cbon/go-repo-utils
[INFO]  --> Gathering release information for github.com/mh-cbon/go-repo-utils
[INFO]  --> Adding github.com/mh-cbon/go-repo-utils to your configuration %!s(MISSING)
[INFO]  Downloading dependencies. Please wait...
[INFO]  Fetching updates for github.com/mh-cbon/go-repo-utils.
[INFO]  Resolving imports
[INFO]  Downloading dependencies. Please wait...
$ glide update
[INFO]  Downloading dependencies. Please wait...
[INFO]  Fetching updates for github.com/mh-cbon/go-repo-utils.
[INFO]  Resolving imports
[INFO]  Downloading dependencies. Please wait...
[INFO]  Setting references for remaining imports
[INFO]  Versions did not change. Skipping glide.lock update.
[INFO]  Project relies on 1 dependencies
$ glide -v
glide version 0.10.2-109-gc25976f
$ ll vendor/github.com/
total 4
drwxrwxr-x 3 mh-cbon mh-cbon 4096 10 juin  15:23 mh-cbon
$ ll vendor/github.com/mh-cbon/
total 4
drwxrwxr-x 9 mh-cbon mh-cbon 4096 10 juin  15:23 go-repo-utils
$ cat vendor/github.com/mh-cbon/go-repo-utils/glide.yaml 
package: github.com/mh-cbon/go-list-tags
import:
- package: github.com/docopt/docopt.go
- package: github.com/Masterminds/semver
- package: github.com/mh-cbon/verbose

[mattfarina]

Glide scans the code to know where in the import tree to fetch. It starts with references in your application and follows the referenced import tree. Do you have references in your code?

If you want to follow all possible trees, not just the referenced imports, you can use the --all-dependencies flag.

Is this what's going on?

[mh-cbon]

Glide scans the code to know where in the import tree to fetch. It starts with references in your application and follows the referenced import tree. Do you have references in your code?

Do you mean import statements ? Like this

Indeed glide get github.com/mh-cbon/verbose --all-dependencies find the recursive dependencies and import them.

$ glide -v
glide version 0.10.2-109-gc25976f

BTW, notice how glide tries to update '.'

$ pwd
test
$ glide get github.com/mh-cbon/verbose --all-dependencies
[INFO]  Preparing to install 1 package.
[INFO]  Attempting to get package github.com/mh-cbon/verbose
[INFO]  --> Gathering release information for github.com/mh-cbon/verbose
[INFO]  --> Adding github.com/mh-cbon/verbose to your configuration %!s(MISSING)
[INFO]  Downloading dependencies. Please wait...
[INFO]  Fetching updates for github.com/mh-cbon/verbose.
[INFO]  Resolving imports
[INFO]  Fetching test into /home/mh-cbon/gow/src/github.com/mh-cbon/test/vendor
[ERROR] Failed to fetch test: Cannot detect VCS
[INFO]  Fetching github.com/fatih/color into /home/mh-cbon/gow/src/github.com/mh-cbon/test/vendor
[INFO]  Setting version for github.com/fatih/color to 533cd7fd8a85905f67a1753afb4deddc85ea174f.
[INFO]  Found GB manifest file in vendor/github.com/fatih/color
[INFO]  --> Parsing GB metadata...
[INFO]  Fetching github.com/mattn/go-colorable into /home/mh-cbon/gow/src/github.com/mh-cbon/test/vendor
[INFO]  Setting version for github.com/mattn/go-colorable to 9cbef7c35391cca05f15f8181dc0b18bc9736dbb.
[INFO]  Fetching github.com/mattn/go-isatty into /home/mh-cbon/gow/src/github.com/mh-cbon/test/vendor
[INFO]  Setting version for github.com/mattn/go-isatty to 56b76bdf51f7708750eac80fa38b952bb9f32639.
[INFO]  Fetching golang.org/x/sys/unix into /home/mh-cbon/gow/src/github.com/mh-cbon/test/vendor
[INFO]  Downloading dependencies. Please wait...
[INFO]  Fetching updates for test.
[WARN]  Unable to checkout test
[ERROR] Update failed for test: Cannot detect VCS
[ERROR] Could not update packages: Cannot detect VCS

thanks !

[ulrichSchreiner]

hi,
i think this is not an ideal solution. when i need a new library in my code i first install it and then use it. when invoking the current version of glide with glide get github.com/spf13/cobra it installs the cobra library but not the dependencies.

Now i use the cobra library in my code and get compilation errors, because the dependencies of cobra are not installed.

I used to work with tools like goimports so i normally do not write the imports by myself. i simply use the library in the code and the tool does the import. but this only works if the library is installed. and here is the chicken-egg problem: glide wants my code to contain a import statement and other tools want a correctly installed library.

imho it is a bug of glide to not install a library like go get. In your documentation you write glide get is similiar to go get. But it is not! It does install the library code, but it does only install the dependencies if i have an import statement.

It would be better to not install anything or install everything (including dependencies). but not only some parts where a subsequent compilation will fail because of missing dependencies.

[sdboyer]

@ulrichSchreiner yep, i can't think of any real use case for NOT fetching transitive deps here.

(note that this issue is also fixed by #384, as it stands right now)

[huygn]

I need to get all dependencies for a new packages because my CI is failing, but --all-dependencies flag caused errors:

$ glide --version
glide version 0.12.1

$ glide get github.com/solher/arangolite --all-dependencies                                         15:46:21
[INFO]  Preparing to install 1 package.
[INFO]  Attempting to get package github.com/solher/arangolite
[INFO]  --> Gathering release information for github.com/solher/arangolite
[INFO]  The package github.com/solher/arangolite appears to have Semantic Version releases (http://semver.org).
[INFO]  The latest release is v1.3.0. You are currently not using a release. Would you like
[INFO]  to use this release? Yes (Y) or No (N)
n
[INFO]  --> Adding github.com/solher/arangolite to your configuration
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for github.com/solher/arangolite.
[INFO]  --> Fetching updates for google.golang.org/grpc.
[INFO]  --> Fetching updates for gopkg.in/yaml.v2.
[INFO]  --> Fetching updates for golang.org/x/net.
[INFO]  --> Fetching updates for github.com/golang/protobuf.
[INFO]  --> Fetching updates for github.com/go-kit/kit.
[INFO]  Resolving imports
[ERROR] Could not find /Users/huygiang/src/gitlab.com/my/pj/vendor/github.com/go-kit/kit/endpoint: open /Users/huygiang/.glide/cache/src/https--Users-huygiang-src-gitlab.com-my-pj-vendor-github.com-go-kit-kit-endpoint: no such file or directory
[ERROR] Dependency /Users/huygiang/src/gitlab.com/my/pj/vendor/github.com/go-kit/kit/endpoint failed to resolve: open /Users/huygiang/.glide/cache/src/https--Users-huygiang-src-gitlab.com-my-pj-vendor-github.com-go-kit-kit-endpoint: no such file or directory.
[ERROR] Failed to retrieve a list of dependencies: open /Users/huygiang/.glide/cache/src/https--Users-huygiang-src-gitlab.com-my-pj-vendor-github.com-go-kit-kit-endpoint: no such file or directory

/Users/huygiang/src/gitlab.com/my/pj/vendor/github.com/go-kit/kit/endpoint is present and I have no idea why glide said such things :/

[sdboyer]

@gnhuy91 huh...that's some weirdness in those errors. have you set up any mirrors, e.g. via glide mirror?

[huygn]

@sdboyer not at all, also my/pj is my private repo.

[karalabe]

I'm getting the same errors:

[ERROR] Could not find /home/karalabe/.glide/cache/src/https-golang.org-x-tools/imports: open /home/karalabe/.glide/cache/src/https--home-karalabe-.glide-cache-src-https-golang.org-x-tools-imports: no such file or directory
[ERROR] Dependency golang.org/x/tools/imports (/home/karalabe/.glide/cache/src/https-golang.org-x-tools/imports) failed to resolve: open /home/karalabe/.glide/cache/src/https--home-karalabe-.glide-cache-src-https-golang.org-x-tools-imports: no such file or directory.
[ERROR] Failed to retrieve a list of dependencies: open /home/karalabe/.glide/cache/src/https--home-karalabe-.glide-cache-src-https-golang.org-x-tools-imports: no such file or directory

[leeola]

Same,

[ERROR] Could not find /Users/leeolayvar/.glide/cache/src/https-golang.org-x-sys/unix: open /Users/leeolayvar/.glide/cache/src/https--Users-leeolayvar-.glide-cache-src-https-golang.org-x-sys-unix: no such file or directory
[ERROR] Dependency golang.org/x/sys/unix (/Users/leeolayvar/.glide/cache/src/https-golang.org-x-sys/unix) failed to resolve: open /Users/leeolayvar/.glide/cache/src/https--Users-leeolayvar-.glide-cache-src-https-golang.org-x-sys-unix: no such file or directory.
[ERROR] Failed to retrieve a list of dependencies: open /Users/leeolayvar/.glide/cache/src/https--Users-leeolayvar-.glide-cache-src-https-golang.org-x-sys-unix: no such file or directory

When trying to run:

glide get --all-dependencies github.com/inconshreveable/log15

[nsilberman]

Same error for me !

[nsilberman]

hi @mattfarina , it 's a very strange bug
This is what I've done (on mac os 10.11)

• glide --version => glide version 0.12.3
• glide install => OK but missing dependencies used by vendor libs (???)
• glide up --all-dependencies (should be great to do that on install step) => error:

$ glide up --all-dependencies
...
[INFO]  Resolving imports
[INFO]  --> Fetching updates for golang.org/x/sys.
...
[ERROR] Could not find /Users/nicolas/.glide/cache/src/https-golang.org-x-sys/unix: open /Users/nicolas/.glide/cache/src/https--Users-nicolas-.glide-cache-src-https-golang.org-x-sys-unix: no such file or directory
[ERROR] Dependency golang.org/x/sys/unix (/Users/nicolas/.glide/cache/src/https-golang.org-x-sys/unix) failed to resolve: open /Users/nicolas/.glide/cache/src/https--Users-nicolas-.glide-cache-src-https-golang.org-x-sys-unix: no such file or directory.
[ERROR] Failed to retrieve a list of dependencies: open /Users/nicolas/.glide/cache/src/https--Users-nicolas-.glide-cache-src-https-golang.org-x-sys-unix: no such file or directory

[sdboyer]

yeah, glide is clearly chasing its own tail, here - treating the repo in the source cache as an upstream, rather than being the actual cache repo.

i know we've been slow to move on this - sorry. i can't promise an immediate resolution, either :( but the output everyone's reporting is pretty strongly indicative of the class of problem. hopefully one of us can find time to get to this soon.

in the meantime, if someone could create some steps to reproduce, using publicly-available code, that would be TREMENDOUSLY helpful.

[neocortical]

Does anyone have a good workaround for this issue?

EDIT: I was able to work around this issue in my case by hacking a one-line change into Glide and rebuilding. Here is the diff:

diff --git a/dependency/resolver.go b/dependency/resolver.go
index bf59c3e..5f339f8 100644
--- a/dependency/resolver.go
+++ b/dependency/resolver.go
@@ -827,7 +827,7 @@ func (r *Resolver) imports(pkg string, testDeps, addTest bool) ([]string, error)
                if testDeps {
                        _, imps, err = IterativeScan(r.Handler.PkgPath(pkg))
                } else {
-                       imps, _, err = IterativeScan(r.Handler.PkgPath(pkg))
+                       imps, _, err = IterativeScan(pkg)
                }

                if err != nil {

WARNING: I know next to nothing about Glide internals and am not totally sure what I did there. YMMV.

[nsilberman]

@neocortical your fix works for me to ! maybe you can send a PR and we'll discuss about it with glide team ?

[neocortical]

@nsilberman Let me dig in a little into _why_ my fix worked. Then I'll submit a PR if I can. I suspect that my fix hacked out a feature that is needed in other use cases and/or there are some deeper reasons why the method call I removed exists.

[sdboyer]

So, it is quite interesting that @neocortical's modification makes a difference. But I don't think it's the right solution. The IterativeScan() path deals with packages that have slightly non-vanilla structures - there are different package <name>s declared in the same dir (and not <name>_test; pulling out that call would cause glide to begin failing in those cases.

But it still tells us a lot about where the actual problem is - so that's quite helpful :) My guess is that somewhere higher up in the call chain, an import path SHOULD be used, but an absolute path is being used instead.

[nsilberman]

My guess is that somewhere higher up in the call chain, an import path SHOULD be used, but an absolute path is being used instead.

yep, seems to be that. We know "just" have to find where

[neocortical]

A couple more notes.

1. I just tried my glide update command using v0.11.1 and it worked. Then I upgraded to v0.12.3 and it failed. (I'm on a different laptop from when I rebuilt from source).

2. I traced the issues to r.Handler.PkgPath(pkg) by debugging and noting that the value of pkg pointed to a valid location in the glide cache until PkgPath() was applied. The specific Error I get is:

[ERROR] Could not find /Users/nate/.glide/cache/src/https-github.com-lib-pq/oid: open /Users/nate/.glide/cache/src/https--Users-nate-.glide-cache-src-https-github.com-lib-pq-oid: no such file or directory

However, lib pq/oid does exist here:

/Users/nate/.glide/cache/src/https-github.com-lib-pq/oid

So the issue is the package path mutation that occurs in r.Handler.PkgPath(pkg).

[lucaswxp]

Any workarounds to this yet? +1

[hraban]

Im having the same problem, and it essentially makes glide unusable for large projects. @neocortical's fix breaks it on the first private repository dependency, unfortunately.

[brikis98]

Honest question for the glide maintainers: is this project abandoned now? This (blocking) bug has been open for ~9 months and there have been no new releases of glide for ~6 months. I'm grateful for all the work that's been put into this project, and I use it in a ton of my projects, but it would be great to know if there are any plans to keep Glide alive, or if I should be migrating away from it.

[hraban]

@brikis98 we literally just migrated to govendor (https://github.com/kardianos/govendor) since my last comment. It handles recursive dependencies well and is under active development. I'd say give it a shot, if nothing else.

[mh-cbon]

Honest question for the glide maintainers: is this project abandoned now?

I m not maintainer, but i don't think so.

This (blocking) bug has been open for ~9

As OP, i can say it happened once or two to me, but since them, imho, its fixed.

So unless someone can provide a useful trace to reproduce the problem (including the full state of each vendored repo, which i missed to add originally), and understand that issue,
i prefer to close the thread as it bikesheding without much useful feedback.

[brikis98]

The same sort of issue was reported in #640 also about 9 months ago. So it's definitely not some isolated problem. I can't post an easy repro case because it involves Go code in a private repo.

[franciscocpg]

@brikis98
I'm not the maintainer neither but I know the community is putting some effort on https://github.com/golang/dep.
I wouldn't say that this project is abandoned now, but it certainly has low activity.

[priscila225]

For me, it works after did the command glide install before glide update, don't know why but ok.