Dep: Error handling when running `dep init` from project subdirectory

[catatsuy]

I tried to adopt dep instead of https://github.com/Masterminds/glide in https://github.com/isucon/isucon6-final/tree/master/bench.

glide.yml did not include the own repository (github.com/isucon/isucon6-final), but manifest.json contains the own repository.

glide.yml

package: github.com/isucon/isucon6-final/bench
import:
- package: github.com/PuerkitoBio/goquery
  version: v1.0.1
- package: github.com/Songmu/timeout
- package: github.com/mitchellh/go-homedir

manifest.json

{
    "dependencies": {
        "github.com/PuerkitoBio/goquery": {
            "branch": "master"
        },
        "github.com/Songmu/timeout": {
            "branch": "master"
        },
        "github.com/andybalholm/cascadia": {
            "branch": "master"
        },
        "github.com/isucon/isucon6-final": {
            "branch": "master"
        },
        "github.com/mitchellh/go-homedir": {
            "branch": "master"
        },
        "golang.org/x/crypto": {
            "branch": "master"
        },
        "golang.org/x/net": {
            "branch": "master"
        },
        "golang.org/x/oauth2": {
            "branch": "master"
        }
    }
}

It is generated by dep init on the directory of https://github.com/isucon/isucon6-final/tree/master/bench. I want not to include the own repository like glide.yml in manifest.json.

I use go version go1.7.3 darwin/amd64
I use macOS Sierra 10.12.2

[sdboyer]

Hm, that's odd. When you ran dep init, was it from the directory $GOPATH/src/github.com/isucon/isucon6-final?

[catatsuy]

@sdboyer No. I ran dep init from the directory $GOPATH/src/github.com/isucon/isucon6-final/bench.

[mattn]

I tried this on the subdirectory bench on github.com/isucon/isucon6-final.

manifest.json

{
    "dependencies": {
        "github.com/PuerkitoBio/goquery": {
            "branch": "master"
        },
        "github.com/Songmu/timeout": {
            "revision": "f7382821bee4009254ba6003475762aa5e740807"
        },
        "github.com/isucon/isucon6-final": {
            "branch": "master"
        },
        "github.com/mitchellh/go-homedir": {
            "branch": "master"
        },
        "golang.org/x/net": {
            "branch": "master"
        }
    }
}

lock.json

{
    "memo": "",
    "projects": [
        {
            "name": "github.com/PuerkitoBio/goquery",
            "branch": "master",
            "revision": "4126223a86db8598275b29db04d511cfdab67cf1",
            "packages": [
                "."
            ]
        },
        {
            "name": "github.com/Songmu/timeout",
            "revision": "f7382821bee4009254ba6003475762aa5e740807",
            "packages": [
                "."
            ]
        },
        {
            "name": "github.com/andybalholm/cascadia",
            "branch": "master",
            "revision": "3ad29d1ad1c4f2023e355603324348cf1f4b2d48",
            "packages": [
                "."
            ]
        },
        {
            "name": "github.com/isucon/isucon6-final",
            "branch": "master",
            "revision": "46fc7c511444dcb13765b9a9d6b72e0f49e3ae77",
            "packages": [
                "portal/job"
            ]
        },
        {
            "name": "github.com/mitchellh/go-homedir",
            "branch": "master",
            "revision": "b8bc1bf767474819792c23f32d8286a45736f1c6",
            "packages": [
                "."
            ]
        },
        {
            "name": "golang.org/x/net",
            "branch": "master",
            "revision": "f2499483f923065a842d38eb4c7f1927e6fc6e6d",
            "packages": [
                "http2/hpack",
                "lex/httplex",
                "html"
            ]
        }
    ]
}

[sdboyer]

@catatsuy it's expected that you run dep init from the project root. From dep init -help:

Initialize the project at filepath root by parsing its dependencies and writing manifest and lock files. If root isn't specified, use the current directory.

Subsequent commands can be run from anywhere beneath the top-level directory, but the initial run has to be in the directory you want to initialize, or pointing to it. This is also how git works - git init either takes the target dir to init, or uses cwd, and then you can run git from anywhere beneath that repo.

We might be able to improve error handling around running from a subdirectory, though.

[wslulciuc]

@sdboyer @carolynvs : I'll focus on this issue.

[carolynvs]

@wslulciuc Are you still working on this one?

[wslulciuc]

@carolynvs : Yeah, still plan to patch this, just haven't had the time I'd hoped to focus on this issue. Finally have some free time now, so I'll be visiting this issue later tonight!

[wslulciuc]

@carolynvs : After further reviewing this issue, I think more details around what specifically we are attempting to resolve is needed. That is, any attempts to determine whether dep init is executed in a subdirectory assumes user intent (more specifically, that the current directory is not the intended root directory). So my question is can we reasonably determine the project root directory? For reinitialization, this is, possibly, less of an issue. That is, we can search for the presence of a Gopkg.toml and Gopkg.lock _n_ levels from the current directory, warning the user that this might be a subdirectory and whether they would wish to proceed with initialization.

Thoughts?

[ibrasho]

I still can't think of any reliable method to detect if we are in the project directory or not.

Gopkg.toml and Gopkg.lock only exists after dep init (unless the user creates them manually). So relying on these files is not enough in this case (findProjectRoot does that, but we only use it after dep init).

I think it might be worth it to show a warning when running dep init within a subdirectory of an already initialized project.

We can look for signs like a .git directory in the cwd or expect to be 3 levels under GOPATH (GOPATH/github.com/user/package) and show a warning instead of an error in these case. They are not a definite indicator but this could help a bit.

[wslulciuc]

@ibrasho : Thanks. That's helpful. What you outlined were possible approaches I was also considering. I'll work on adding them to dep init.

[carolynvs]

I am hesitant to have init implement heuristics to detect the root of a repo, e.g. checking for .git and other tricks. We already have similar logic which I am hoping could be used to detect this scenario, e.g. dep.Context.SplitAbsoluteProjectRoot and gps.SourceManager.DeduceProjectRoot.

To be clear, I don't know if that will 100% work. 😀 Just pointing out another direction that, if we get it to work, would be preferable.

[wslulciuc]

@carolynvs : I'll explore the possibility of reusing the logic defined in SplitAbsoluteProjectRoot() and DeduceProjectRoot() as a solution.

[lanzafame]

...expect to be 3 levels under GOPATH (GOPATH/github.com/user/package)...

@ibrasho I may be missing a lot of context, sorry in advance if my assumptions are way off. Won't doing a check against the depth of the project in the GOPATH result in warnings when project use vanity urls, i.e. rsc.io/goversion sits at $GOPATH/src/rsc.io/goversion? Again, sorry if this has already been discussed and decided on.

[sdboyer]

@lanzafame nope, you're right - we can't rely on a particular depth within GOPATH. there's no guarantees about that, at all.

[ibrasho]

@lanzafame I was just listing possible methods. But, as I said in the original comment, none of my suggestions are definite signs sadly.

When keeping in mind that we are trying to print a helpful warning, checking the first segment after GOPATH/src (for domains like github.com, bitbucket.com, gopkg.io or others with structured project names) might not be a bad idea but I'm not sure if it's worth it to implement this check in dep. :grin: