Dep: manifest.json is empty after dep init

[dcelasun]

I was asked to open this issue in this thread.

After running dep init with an existing project (located at $GOPATH/src/bitbucket.org/company_name/project_name), manifest.json has an empty JSON object ({}), but lock.json has all the correct dependencies.

$ go version
go version go1.7.4 linux/amd64
$ go env
GOARCH="amd64"
GOBIN="/home/dcelasun/go/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/dcelasun/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build159479626=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

I can provide more info if needed.

[disq]

Same thing here, project located at $GOPATH/src/github.com/company/project
Retried after rm -rf vendor, same results. lock.json looks cool and correct but manifest.json has just {} in it.

$ go version
go version go1.7 darwin/amd64
$ go end
go: unknown subcommand "end"
Run 'go help' for usage.
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kemal/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/81/jzjhd4m15sq1cj31pt52f0gh0000gn/T/go-build586013334=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
$

[pksunkara]

Just confirming that I also hit this and wrote about it in the google group. I can verify that dep init command here is in migration mode according to the spec written down previously.

[raytung]

Are your dependencies under $GOPATH?
Looks like dep is only adding dependencies that exist in $GOPATH/src
See init.go#L302, context.go#L143, and context.go#L128

[pksunkara]

Nope, I have an empty project with a fresh install of go. So, dependencies are nowhere to be found. Cloned https://github.com/pksunkara/whitespaces and ran dep init

[raytung]

@pksunkara Yep, that's it I think! It wouldn't save into the manifest.json file if the source doesn't already exist under $GOPATH/src.

Steps to reproduce:

• Remove all dependencies from $GOPATH/src
• Run dep init

EDIT: Removed workaround - not relevant to this convo

[pksunkara]

Well, dep init did pull the source into vendor repo, so, shouldn't it take that code after that step? Or shouldn't it take from github directly? I am not looking for a workaround. I am trying to determine how the tool should behave in this case.

I guess, first of all, we probably need to separate dep init and dep migrate out.

[jbrodriguez]

It happened to me too.

So, I did:

• dep init
  manifest.json: only dependencies that exist in $GOPATH/src
  lock.json: ok

• dep ensure -update
  creates/populates vendor

• mv vendor vendor.old
  (actually had to move it one folder up the hierarchy, otherwise the next dep ensure would get confused)
  This simulates leaving vendor out of source control, then cloning the repo back

• dep ensure -update
  recreates vendor correctly

• diff -r vendor vendor.old
  no differences

so, what exactly is the purpose of manifest ?

[pksunkara]

This analogy with npm helps me remember.

package.json => manifest.json
npm-shrinkwrap.json => lock.json
node_modules => vendor

[jbrodriguez]

Right, but package.json does hold all dependencies.

I know dep is pre-alpha, just wondering if manifest.json will also hold all dependencies as it matures.

[pksunkara]

Yeah, according to the design spec I read previously, it is supposed to.

[sdboyer]

(sorry, I've not had time to get to this one yet)

I know dep is pre-alpha, just wondering if manifest.json will also hold all dependencies as it matures.

Standard constraint declarations in manifest.json (under the dependencies key) are only applied to direct dependencies of the current project. It's important to understand, though, that if you declare a constraint in manifest.json, but don't actually import it, that constraint will be ignored. This is still Go - the import graph is queen. (If you want to force something to show up, even if it's not imported, then you want a require. There's no command for that yet, but you can write them into the manifest directly and they work fine 😄 )

Overrides are applied globally, superceding constraints declared by all dependencies (including any dependencies in your own manifest.json, if you did a weird thing and have both). lock.json holds a transitively complete list of all deps.

If you're accustomed to npm, especially older versions (pre-flattening), this all works a bit differently because dep doesn't allow duplication of dependencies.

We don't really have docs optimized for end users yet (pre-alpha, sorry!), but there's more info about the mechanics in the gps docs.

[pksunkara]

Ah, my above statement was ambiguous. I meant that according to the spec manifest.json was supposed to hold the top level dependencies and any other overrides. Sorry about the wording "all dependencies"

[jbrodriguez]

Standard constraint declarations in manifest.json (under the dependencies key) are only applied to direct dependencies of the current project

Sure, that's the most important thing. As it stands right now, this is not fulfilled, as it expects these direct dependencies to exist in $GOPATH/src.

In any case, the project is awesome and I feel it's going in the right direction, so I'll keep trying to poke holes in it (in a very friendly manner :) 👍

[sdboyer]

In any case, the project is awesome and I feel it's going in the right direction, so I'll keep trying to poke holes in it (in a very friendly manner :)

Thanks! please do :) 🎉

As it stands right now, this is not fulfilled, as it expects these direct dependencies to exist in $GOPATH/src.

mm, that's not the case. The only part of dep that touches GOPATH, at all, is dep init - and it only does so to scan what you have on your GOPATH in order to set initial constraints, which it then records in manifest.json. However, if any of those direct (or transitive) dependencies are missing, it'll reach out and try to make a default decision on what version to use by picking the "newest", as defined by the ordering rules. So, if you run dep init on a project in an otherwise empty GOPATH, it'll reach out for everything.

Of course, this issue is about cases in which that fails, and for some reason the resulting manifest is empty...and that's a bug, and I'm not sure why it's doing it. But if you're seeing behavior that makes it seem like dep is in any way requiring anything to be on the GOPATH, please please, do explain.

[sdboyer]

@raytung

Yep, that's it I think! It wouldn't save into the manifest.json file if the source doesn't already exist under $GOPATH/src.

Oh wait...maybe this is it. If this is all it is, then it's a super-easy fix. I'll look tonight :)

[sdboyer]

Ah, so, it seems like we actually did this intentionally for the first round with init - if it's not on disk, we don't put a constraint in the manifest. So, I want to check in with @adg before we go changing that.

[peterbourgon]

For the record, I find it highly surprising (read: bad) that first-level deps are only in manifest.json conditionally. I expect all first-level deps to be there.

[sdboyer]

Agreed, and that's definitely the plan for the revisited command set - #277

[kevingo]

I also found this issue with empty manifest.json {}. And as for @raytung described, if a package does not in the $GOPATH/src, it will not save into the manifest.json file.

I just want to make sure that it is a bug and will be fix before release, right ? @sdboyer

By the way, it's a nice tool and we will happy for supporting dep will be put into the go toolchain in the near future 👍

[sdboyer]

Yes, we do intend to change this behavior with the respeccing of init.

[sdboyer]

This should, finally, be fixed 😄