Dep: Project double-up in Gopkg.lock when private package is both a transitive and direct dependency

[lwc]

What version of Go and dep are you using?

$ go version
go version go1.8.3 linux/amd64

$ git describe --tags
v0.3.0-19-g52ed251

Background

I'm attempting to migrate a project with private dependencies from glide to dep.

In glide, the private dep is specified like this:

- package: github.com/private/package
  repo: https://username:[email protected]/private/package

After running dep init, dep parses the glide.yml successfully and the Gopkg.toml includes:

[[constraint]]
  name = "github.com/private/package"
  source = "https://username:[email protected]/private/package"

👌 Everything looks fine, until...

What dep command did you run?

$ dep ensure -v
Gopkg.lock was already in sync with imports and Gopkg.toml, recreating vendor/ directory
Username for 'https://github.com':

Upon terminating:

grouped write of manifest, lock and vendor: error while writing out vendor tree: error while exporting github.com/private/package: no valid source could be created:
    failed to set up "https://github.com/private/package", error remote repository at https://github.com/private/package does not exist, or is inaccessible
    failed to set up "ssh://[email protected]/private/package", error context canceled
    failed to set up "git://github.com/private/package", error context canceled
    failed to set up "http://github.com/private/package", error context canceled

It appears that dep has forgotten the auth details. Looking at Gopkg.lock:

[[projects]]
  name = "github.com/private/package"
  packages = ["transitivedep"]
  revision = "a122ad8ed6e057df58a1addb47febe70d3494f91"

[[projects]]
  name = "github.com/private/package"
  packages = ["directone","directtwo"]
  revision = "a122ad8ed6e057df58a1addb47febe70d3494f91"
  source = "https://username:[email protected]/private/package"

The repo had two project entries, and after digging through the deps I realised that another dep (let's call it github.com/other/package) depended on a different package out of the private dep:

                   +-----------------+
                   |                 |
                   | /private/package|
                   |                 |
depends on         +-------^-^-------+
transitivedep              | |
         +-----------------+ |
         |                   |
+--------+--------+          |
|                 |          | depends on
| /other/package  |          | directone and directtwo
|                 |          |
+--------^--------+          |
         |                   |
         +-----------------+ |
                           | |
                   +-------+-+-------+
                   |                 |
                   |   Application   |
                   |                 |
                   +-----------------+

Looks like the dependency resolver can't figure out that the transitive dep lives with the two direct dependencies.

Testing

I tested my hypothesis by adding transitivedep to the project as a direct dependency and rerunning dep ensure. Sure enough:

[[projects]]
  name = "github.com/private/package"
  packages = ["directone","directtwo","transitivedep"]
  revision = "a122ad8ed6e057df58a1addb47febe70d3494f91"
  source = "https://username:[email protected]/private/package"

The solver got it right.

Dep of deps

Worth noting that github.com/other/package actually has Gopkg files outlining the dependency on github.com/private/package/transitivedep, complete with auth.

Not sure what the plans are for dep to see those and parse them while solving?

[sdboyer]

ooh, thanks for the highly detailed report 😄

it seems like there's two, unrelated things here - dependency duplication, and auth issues. they might turn out to be connected, but that doesn't immediately strike me as likely.

auth first, as its simpler - our handling of the source strings has a few known issues in it that we haven't gotten around to fixing quite yet. in the meantime, the preferred mechanism (as it avoids putting passwords in the clear into a committed file) is described in the FAQ.

now, duplication.

[[projects]]
  name = "github.com/private/package"
  packages = ["transitivedep"]
  revision = "a122ad8ed6e057df58a1addb47febe70d3494f91"

[[projects]]
  name = "github.com/private/package"
  packages = ["directone","directtwo"]
  revision = "a122ad8ed6e057df58a1addb47febe70d3494f91"
  source = "https://username:[email protected]/private/package"

this is quite surprising! the solver is designed to allow for the possibility of duplicates for reasons that aren't likely to become relevant for at least another six months to a year - but such duplication should not be achievable with any inputs dep allows today.

there are two factors that seem relevant here:

1. two dependers import entirely disjoint sets of packages from a single dependent project
2. one depender specifies an alternate source rule, and the other does not

we have tests for both of these cases individually: unifying disjoint package sets and reconciling variant source URLs (note that this latter case is kinda dicey in general - #428). so, my guess was that this has to be a _combination_ of these two properties varying at once that's causing the duplication. i drew up a quick test case, and sure enough, you've found a solver bug 😄!

fortunately, i'm pretty sure i know right where the fix for this one is, and i don't think it's bad.

[sdboyer]

found the bug. you should be all set with this now. good catch! this is the first actual solver correctness bug i think anyone else has reported 😄

[lwc]

Yay! Glad to help 😁

Thanks for the quick turn around!