Glide: Nested dependencies vendor directories not created

[brycefisher]

Demonstrates Soft Flattening Bug

Copied from https://github.com/brycefisher/nested .

Steps to reproduce:

1. go get -d github.com/brycefisher/nested
2. cd $GOPATH/src/github.com/brycefisher/nested
3. glide up
   

   Expected Result

In this case, glide up _should_ notice that the github/brycefisher/nested and github.com/brycefisher/glide-nested-dependency require difference versions of the package github.com/aws/aws-sdk-go. The vendor ought to look something like this:

$GOPATH/src/github.com/brycefisher/nested/

 * glide.lock
 * glide.yaml
 * main.go
 * vendor/
   * github.com/
     * aws/
       * aws-sdk-go/
         * ...
     * brycefisher/
       * glide-nested-dependency
         * get_newer_s3_client.go
         * glide.lock
         * glide.yaml
         * vendor/
           * aws/
             * aws-sdk-go/
               * ...

Actual Result

Output using glide 0.9.3 on OSX El Capitan, go version 1.6.0:

nesting $  glide up
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/aws/aws-sdk-go.
[INFO] Fetching updates for github.com/brycefisher/glide-nested-dependency.
[INFO] Setting version for github.com/aws/aws-sdk-go to v0.9.3rc4.
[INFO] Setting version for github.com/brycefisher/glide-nested-dependency to master.
[INFO] Resolving imports
[INFO] Fetching github.com/vaughan0/go-ini into /Users/bfisher-fleig/work/src/github.com/brycefisher/nesting/vendor
[ERROR] Error scanning github.com/aws/aws-sdk-go/aws/session: open /Users/bfisher-fleig/work/src/github.com/brycefisher/nesting/vendor/github.com/aws/aws-sdk-go/aws/session: no such file or directory
[INFO] Downloading dependencies. Please wait...
[INFO] Setting references for remaining imports
[INFO] Project relies on 3 dependencies.
An Error has occurred
nesting $ tree -L 5
nesting master 🍻  tree -L 5
.
├── README.md
├── glide.lock
├── glide.yaml
├── main.go
└── vendor
    └── github.com
        ├── aws
        │   └── aws-sdk-go
        │       ├── Gemfile
        │       ├── LICENSE.txt
        │       ├── Makefile
        │       ├── NOTICE.txt
        │       ├── README.md
        │       ├── apis
        │       ├── aws
        │       ├── awsmigrate
        │       ├── doc-src
        │       ├── internal
        │       ├── sdk.go
        │       └── service
        ├── brycefisher
        │   └── glide-nested-dependency
        │       ├── get_newer_s3_client.go
        │       └── glide.yaml
        └── vaughan0
            └── go-ini
                ├── LICENSE
                ├── README.md
                ├── ini.go
                ├── ini_linux_test.go
                ├── ini_test.go
                └── test.ini

14 directories, 18 files

[brycefisher]

Yes, I use 🍻 instead of $ in my shell.

[dmitris]

@brycefisher - I believe flattening the vendor/ directory is a core glide feature and design decision, see
https://glide.readthedocs.org/en/latest/vendor/ and
http://engineeredweb.com/blog/2015/glide-flatten-dependencies/

The following may be relevant (even if partly outdated):
https://github.com/Masterminds/glide/issues/108
https://github.com/Masterminds/glide/issues/73

My appeal to the glide owners would be to please keep the vendor/ directory flattened and not get in the mess of including different versions of the same package in the executable - which is known to likely lead to problems with init() functions and overall brittleness and complexity (see the "Problems" in the first link). I believe it is the right thing to let the user (building package main) to figure out which single version of the dependency will work for them and if needed to specify it explicitly in glide.yaml.

/cc @mastermind @technosophos

[mattfarina]

@brycefisher Nested vendor directories is complicated. There are a couple things to know.

1. Instances created from a version of a package in one vendor directory are _NOT_ compatible with the same package, even the same version, in another vendor directory. I wrote up an example.
2. Each instance of a package in a vendor directory will end up in the final binary. That makes it easy to get to binary bloat.

That means the safest method is to use one vendor directory at the top of an application. This is where we are today.

I'd like to be able to be smarter with the vendor directory handling. For example, if a package is using a different version from others and isn't exposing any instances put it in a different vendor directory. We've had some recent conversations about it and if you'd like to join us we're on Gitter.

We've learned, through a bit of experience, to craft a solution that brings about the fewest support issues and problems. We experimented with different solutions early on. A top level vendor directory does that. Moving to a more nuanced solution is something I'm open to but we need to provide a great experience.

[mattfarina]

@brycefisher Also, I've not looked into why you didn't get to the error about a conflict. I think the previous error stopped it from getting to that step. There is a point, if code didn't call a package not there in the checked out version, that a message comes up notifying you about the conflict.

[brycefisher]

@mattfarina your insights are much appreciated. I totally understand points 1. and 2. you raise above -- somewhat counterintuitive at first, but I completely understand the desire to flatten dependencies as much as possible. I don't think I'm asking you to rewrite glide completely from scratch.

My understanding from reading the glide docs though is that if I specify a direct dependency on FOO v1 and BAR v1, and BAR v1 vendors FOO v2, glide will _not_ try to flatten BAR's vendored copy of FOO v2. This was my understanding from reading https://github.com/Masterminds/glide/issues/303 and from https://glide.readthedocs.org/en/latest/vendor/ .

Do I misunderstand the intended behavior of glide in the case I describe here?

[brycefisher]

@mattfarina any response on the expected behavior of glide?

[technosophos]

The intention is to flatten all of the dependencies, with the only exceptions being cases where the import paths specifically reference the vendored copy. We aren't currently planning on changing this.

[brycefisher]

@technosophos -- thanks for clarifying my question about the intended functionality.