Go: proposal: Go 2: cmd/go: allow relative imports

[cch123]

I think this is a useful feature for library writers, sometimes dependency can not be avoided for a lib.
eg. a proxy lib may use a sql parser written by third party.

The library package layout may look like follows:

myproxy: host on github or sth
  main.go
  └── import "github.com/xxx/sqlparser"
  └── import "github.com/yyy/sometcppoolpackage"

If I want to make the dependency invisible to lib endusers, currently I have to move the dependencies to internal package, and manually change the import path to, maybe:

import "github.com/myname/dbproxy/internal/github.com/xxx/sqlparser"

It's quite ugly and inconvenient.

If internal can be imported easily, then import path will be,

import "internal/github.com/xxx/sqlparser"

quite clear

I noticed that some libs also have inner package, and they currently layout their project like the following:

mqtool is hosted on github.com/xxx/mqtool

mqtool/
  main.go
   └─ import "github.com/xxx/mqtool/cli"
  otherfile.go
  └─ import "github.com/xxx/mqtool/cli"
  └─ import "github.com/xxx/mqtool/logutil"
  ... many many file
  otherfileend.go
  └─ import "github.com/xxx/mqtool/bytes2"

If someone wants to fork this mqtool lib to fix bugs or maintain it as his own, then he will meet a lot problems (to modify endless import paths).

Someone may say that he can git clone this project to the correct path, but go get can not support such case, which is really weird (because go get will put this project to the forked path, not the original path).

If internal package can be found in relative path, I think things can go much better :-)

Then the above project may become like this:

mqtool is hosted on github.com/xxx/mqtool

mqtool/
  main.go
   └─ import "internal/cli"
  otherfile.go
  └─ import "internal/cli"
  └─ import "internal/logutil"
  ... many many file
  otherfileend.go
  └─ import "internal/bytes2"

[rsc]

This is not really about internal at all, since you could have the same situation with non-internal subdirectories you want to import. For better or worse we've decided that having fully-qualified paths is useful for being able to understand imports when you see them. It's not clear we want to break that now.

[leonklingele]

Can this be reopened as a proposal for Go 2 or should I create another proposal?

I'm especially asking for something similar to this:

mqtool is hosted on github.com/xxx/mqtool

mqtool/
  main.go
   └─ import "internal/cli"
  otherfile.go
  └─ import "internal/cli"
  └─ import "internal/logutil"
  ... many many file
  otherfileend.go
  └─ import "internal/bytes2"

It doesn't make sense to hard-code the platform / domain I'm going to host my code on inside the code itself.

[ianlancetaylor]

I'll reopen for Go 2.

[leonklingele]

While I'm not a fan of internal/.., something like this ("project-relative imports") should be considered for Go 2. internal could be a package name, can it? So this could potentially break existing code.
Instead, I'd prefer import "/x/y" (leading slash indicates project-relative import, / is the root of the project)

Hard-coding the platform / domain one intends to host code under should not be part of the code itself, moving code between different hosting providers (or even renaming the project) should be easier and should not require a.) changes to the source code itself (s/old-domain/new-domain/) b.) the use of a self-hosted go-import-providing server: example which gets the source from diverse hosting platforms but uses a single, static custom domain name.

This proposal basically is a requirement for proposal: incrementally modify the Go toolchain to work without GOPATH

[dlsniper]

I would discourage relative imports to begin with. They can lead to unexpected results. The current model of paths being GOPATH dependent works really well and can be adapted towards a package oriented workflow rather easily.

This proposal basically is a requirement for proposal: incrementally modify the Go toolchain to work without GOPATH

I would disagree about that.

[leonklingele]

I would discourage relative imports to begin with. They can lead to unexpected results

For example?

I would disagree about that.

Why? How else would the Go compiler toolchain know if an import should be retrieved from $GOPATH / vendor/ (i.e. external packages) or from the current project?

[johnDoe2018]

I'd share another example where this could be useful. Let's assume there is a project with the following structure:

---

github.com/janeRoe/myapp

• 
  
  • main.go - imports models, handlers, and routes:
  
  

import (
    "github.com/janeRoe/myapp/handlers"
    "github.com/janeRoe/myapp/models"
    "github.com/janeRoe/myapp/routes"
)

• 
  
  • handlers/
  
  
• 
  
  • models/
  
  
• 
  
  • routes/
  
  

---

I want to fork the project and make a few changes to, say, routes. I click GH "Fork" button and get a copy of the project that now lives under github.com/johnAnonDoe/myapp. But the main.go's imports still point to the old "github.com/janeRoe/myapp/...". That means the following requirements cannot be satisfied at the same time:

1. Changes to the routes/ subpackage must affect my fork (which, in case of absolute import paths, requires update of the main.go).
2. My fork should be go getable.
3. It should be easy to merge the changes back to the initial repo.

[ianlancetaylor]

Because currently Go packages are distinguished by import paths, support for relative import paths would mean that very early on in the build process the relative imports would have to be converted to full paths.

As far as I can see the only time it is useful to have a relative import path is when you are going to move a set of packages around. That seems to happen relatively rarely, and it would be straightforward to write a tool to move a set of packages while rewriting the import paths. I'm not aware of any such tool today, but if we had one would there still be a need for relative import paths?

[cch123]

@ianlancetaylor , if I fork other people's project on github, and want to contribute back to this project.It seems there is still no way to keep the forked project go gettable and the import paths inside this forked project to be same with the original's.

If I want to use it as my own, I have to rewrite the import path, and if I want to contribute, I have to rewrite the import path again. So I think it's not quite friendly to the current github workflow.(but if you can provide a rewriting tool, this annoying procedure can be more convenient than now.

[ianlancetaylor]

@cch123 Seems like you may be assuming that the other person's project is already using relative imports, which it most likely is not.

[cch123]

@ianlancetaylor ，for example，I forked etcd project，this file https://github.com/coreos/etcd/blob/master/discovery/discovery.go contains imports of the same project's paths:

    "github.com/coreos/etcd/client"
    "github.com/coreos/etcd/pkg/transport"
    "github.com/coreos/etcd/pkg/types"

After forked to my own repo, the import path should be:

    "github.com/cch123/etcd/client"
    "github.com/cch123/etcd/pkg/transport"
    "github.com/cch123/etcd/pkg/types"

If I don't change the import paths and then execute

go get github.com/cch123/etcd

Then the project will appear at $GOPATH/src/github.com/cch123/etcd

But the import paths inside this project are not correct. I'm not assuming other people is using relative imports, but I mean if we could support some way(eg.relative import? or some way of internal usage) to unbind the import path with the host address, then I will not be forced to change the import paths back and forth.

The original issue is that I want to make use of internal feature to bind dependencies together with my library. Because dependencies on site like github cannot avoid risks to be deleted by original author(like the sad story of leftpad in npm community). So I think feature of this could make my library more safe in such situation. But if I make these dependencies as my library internals , it will also meet the import path problem. I have to change all the import paths of these dependencies if they use the import path like the above case.

[dlsniper]

@cch123 I recommend you read http://blog.sgmansfield.com/2016/06/working-with-forks-in-go/ on how to work with forks if you want to contribute changes to a project. If you want to use your fork instead of the original place, dep supports this (and I think vgo may support this at some point, if not already).

Relative import paths is something I would love to see gone in Go2 as they are a constant source of pain.

[nkovacs]

I agree with @cch123 forking packages with subpackages in them is a huge pain.
Changing origins is not a solution (you can't go get them). Dep's support for forks is a band-aid.

You can actually abuse the vendor system to solve this problem. Just put your subpackages into vendor, and you've got project-relative imports, you can fork the package easily without having to rewrite everything. Obviously that's an ugly hack, but I'd like to see something like that as a solution to this problem, maybe not full relative imports.

[dlsniper]

I agree with @cch123 forking packages with subpackages in them is a huge pain.
Changing origins is not a solution (you can't go get them). Dep's support for forks is a band-aid.

@nkovacs this is not a band-aid, nor would relative import paths solve anything in this case. dep allows something that's considered basic functionality for package managers. There is nothing new or different for Go compared to other languages.

Just put your subpackages into vendor, and you've got project-relative imports, [...] Obviously that's an ugly hack

This is not why we have the vendor directory. We have it for reproducible builds. It allows guaranteed that, if committed, everyone gets the same identical source code regardless of environment.

but I'd like to see something like that as a solution to this problem, maybe not full relative imports.

I don't understand how relative imports solve any problem that you have.

[nkovacs]

If I fork a php package, I can publish it on packagist without first having to change a bunch of imports inside it.
If I fork a go package with subpackages, I have to change all the imports to its subpackages in it if I want people to go get it, or govendor fetch it, or dep ensure it, or whatever.

Both composer and npm allow you to specify a temporary fork instead of the canonical package. But in both ecosystems you can just publish your forked package without having to rewrite a bunch of imports inside it, because they're all relative. You can't do that in go.

This is not why we have the vendor directory. We have it for reproducible builds. It allows guaranteed that, if committed, everyone gets the same identical source code regardless of environment.

Of course not. I said you can _abuse_ it to fix the problem.

I don't understand how relative imports solve any problem that you have.

I wouldn't have to do this: https://github.com/nkovacs/ifacemaker/commit/c327e8df5f99e0ba97fda4acde3ca972d36963d1
Or this: https://github.com/nkovacs/counterfeiter/commit/d0d3dfca20c0f2f399dc15c1750db58a9789a629

[dominikh]

If I fork a go package with subpackages, I have to change all the imports to its subpackages in it if I want people to go get it, or govendor fetch it, or dep ensure it, or whatever.

Which, compared to the effort of forking, modifying and maintaining a project, takes absolutely zero effort.

[dlsniper]

But in both ecosystems you can just publish your forked package without having to rewrite a bunch of imports inside it, because they're all relative. You can't do that in go.

To give you the counter-point for the PHP example. You would have to override the autoload process to tell it that your project provides now the reference for the namespace you want to use, rather than the original package, see https://github.com/Seldaek/monolog/blob/master/composer.json#L46-L48

Now Go being different than PHP is is allowed to behave in a different way. If you ask me, instructing dep to fetch the source from a different place is a lot more cleaner and obvious than digging in the package.json and figuring out where the autoloader will perform the operation for loading certain package.

And even in PHP or in Go world, allowing relative import paths suffer greatly when introducing symlinks into play (which at least for me in the PHP world have been a source of needlessly spent hours of debugging why something doesn't work in a particular setup of a developer / environment or in bug reports that I had to handle).

As far as I can tell, the whole argument is: please allow relative import paths so we don't have to run a search & replace operation for forks we want to use as main source but we don't want to use dep and we don't want to have a commit for that operation and we'll never backport changes to upstream (due to the import path change). Would this be an accurate depiction of the case?

[cch123]

@dlsniper , if the official vgo can support publish and get a forked library without manually replace the imports in src code, I think it's ok.
But before vgo, you cannot control your user's package management tool, maybe some people choose godep or some other tool which doesn't support such compile time import search. They'll be in trouble.

[dlsniper]

@dlsniper , if the official vgo can support publish and get a forked library without manually replace the imports, I think it's ok.

@cch123 that's what dep can do today (and I think glide as well).

But before vgo, you cannot control your user's package management tool, maybe some people choose godep or some other tool which doesn't support such compile time import search. They'll be in trouble.

dep, which is the recommended solution for package management until vgo matures, supports this. Much like gofmt or other tools, people can either use it or not, but if the majority of the community uses that, then the problem is on the users that choose to ignore it, not on the rest.

I hope this helps understanding where my point of view comes from and why I think it's important to address the underlying issue that causes the need for this change as well, not just agreeing to do the change.

[nkovacs]

Ok, let's take npm as an example.

You have a library called awesome-package, which contains a subdirectory utils. In your code, you can just use require("awesome-package") or require("awesome-package/utils").
If I fork the package and call it super-awesome-package, you have to change your code to require("super-awesome-package") or require("super-awesome-package/utils"). No confusion there. But I don't have to change the code in super-awesome-package, because it uses require("./utils").

The same thing could work with go. You would import github.com/user/awesome-package or github.com/user/awesome-package/utils, and if you want to use the fork,
github.com/anotheruser/super-awesome-package or github.com/anotheruser/super-awesome-package/utils.

If you ask me, instructing dep to fetch the source from a different place is a lot more cleaner and obvious than digging in the package.json and figuring out where the autoloader will perform the operation for loading certain package.

I don't think it's any better (or worse) to have to dig into dep's lockfile (or wherever it stores this info) to find out that it's not actually importing github.com/user/awesome-package, but github.com/anotheruser/super-awesome-package.
I would much prefer it if my code actually imported github.com/anotheruser/super-awesome-package.

And even in PHP or in Go world, allowing relative import paths suffer greatly when introducing symlinks into play

What do symlinks have to do with relative imports? You can use symlinks with absolute imports too.

As far as I can tell, the whole argument is: please allow relative import paths so we don't have to run a search & replace operation for forks we want to use as main source but we don't want to use dep and we don't want to have a commit for that operation and we'll never backport changes to upstream (due to the import path change). Would this be an accurate depiction of the case?

No, I think that's a bit of a loaded question. Maybe I don't want to use dep because it's slow or not mature yet or because it can't be used for installing tools. Also, instructing it to use a fork is an extra step and makes it not obvious that you're importing the fork (see above).
Maybe I _do_ want to backport changes upstream, but the upstream project is inactive, or they don't agree with the direction I've taken and I have to fork.
Sure, I can run a search & replace, but it feels like I'm working around a deficiency in the language.

[myitcv]

@ianlancetaylor - I just happened upon this issue to add my weight to the argument for removing relative imports in Go 2.

As this issue stands, I think the title is somewhat confusing because there doesn't appear to be the context referenced anywhere that the current plan is to remove them (unless I'm missing something).

Can I suggest we tweak the title to _proposal: Go 2: cmd/go: continue to allow relative imports_? Because:

• it's currently possible (yet very much discouraged, with good reason) to use relative imports
• the plan as things stand is to remove them for Go 2

[ianlancetaylor]

Relative imports barely work today. They don't work inside of GOPATH. They only really work when compiling individual files. I think this proposal is about allowing relative imports inside of GOPATH.

I'm not myself aware of any particular plan regarding relative imports in Go 2. Perhaps there has been some discussion of them in the vgo context, I don't know.

[myitcv]

Relative imports barely work today. They don't work inside of GOPATH. They only really work when compiling individual files. I think this proposal is about allowing relative imports inside of GOPATH.

👍 - that (a single file via go run) happens to be _exactly_ the scenario I "tested" to see to what extent things worked today. Indeed they don't work within GOPATH - thanks, I hadn't realised that distinction.

Title is fine as is therefore. Apologies for the noise.

[metakeule]

Since the functionality is already available via the vendor directory, it would be sufficient to allow the import of vendor/internal/...'. As of Go2 thevendor` directory could then be renamed to something more matching (maybe just 'r' for 'relative').

[nyetwurk]

It is inexcusable that the golang devs didn't think about the impact of forcing absolute imports on the fork/branch/pull-request github workflow.

[dominikh]

It is inexcusable that the golang devs didn't think about the impact of forcing absolute imports on the fork/branch/pull-request github workflow.

Not nearly as inexcusable as people not understanding the concept of git remotes: http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html

[davecheney]

I’m locking this issue to limit conversation until the proposal committee respond.

[bcmills]

The proposal committee doesn't usually respond to Go 2 issues. I'm going to unlock the issue, but folks, please be mindful of our gopher values and keep it constructive!

[sebastianhaberey]

I, too, ran into the issues pointed out by @johnDoe2018 and @cch123 recently. I tried the proposed solution involving _dep ensure_, but I could not get it to work for internal packages. I tend to agree with @cch123 that this dep functionality is a bit of a band-aid. From the documentation:

source rules are generally brittle and should only be used when there is no other recourse.

I am convinced that there are good reasons to completely remove relative paths from Go and so I'll leave that discussion to people more familiar with Go. On the other hand I think that forking repositories, creating pull requests and validating them on CI systems are part of daily developer life. The current need to adress packages inside my own project by their full repository path seems to conflict with that workflow.

[tv42]

The underlying problem seems to be solved by go modules and their support for "github-forking etcd", with a git clone fetching whatever clone you want, while avoiding all the downsides of relative imports. And writing code using forked libraries is supported with replace directive in go.mod.

[surajbarkale]

How about treating import paths that start with / to be relative to the root of module? So the following code blocks are equivallant?

    "github.com/coreos/etcd/client"
    "github.com/coreos/etcd/pkg/transport"
    "github.com/coreos/etcd/pkg/types"

With module relative import

    "/client"
    "/pkg/transport"
    "/pkg/types"

Since the module name is present from go.mod, this will not require any significant changes to the import machinery.