Go: proposal: add golang.org/x/internal repository

[bcmills]

Some of the code in the golang.org/x repositories is shared between multiple repositories: for example, the program to generate the lookup tables in golang.org/x/net/idna needs access to a number of utility packages that are shared with (and currently internal to) golang.org/x/text, including Unicode tables containing large constant strings and arrays.

I suspect that there may be similar opportunities for factoring out internal packages from x/build and x/tools.

Ideally, we should be able to share that data through some common module dependency, rather than by copying code from one module to another. And we have a mechanism for that: the internal path component works in module mode, too, so we could add a golang.org/x/internal component and move any shared dependencies there.

I propose that we add a new repository, golang.org/x/internal, to contain these sorts of project-internal cross-repo dependencies.

Note that the packages within x/internal may need to import from a large number of the other golang.org/x modules, so we would need to be careful to minimize its dependencies. This may be a reasonable use-case for a multi-module repository, since packages within x/internal may be only very loosely coupled to each other.

CC @bradfitz @mpvl @dmitshur @jayconrod @matloob @hyangah @ianthehat @rsc

[matloob]

Speaking for myself, I think the behavior that golang.org/x/internal would create packages internal to the "golang.org/x/" "namespace" is surprising. I'm worried that our users and contributors will get confused by this too.

[bcmills]

@matloob, why would that be surprising? It's the same way the internal/ path component works everywhere...

[matloob]

@bcmills My personal intuitive model is that internal/ is used within a repo or module, so it's surprising to me according to that model. I wonder if there are others who think with that model.

[bradfitz]

I'm fine with this as long as it's multi-module.

If this is going to be a single module 'util' dumping ground then I'm firmly opposed.

[mpvl]

At least in the x/text case it seems to solve only half the problem. Most of the cross-repo generation is from x/text to core, which this would not solve. This could be mitigated with "vendoring", but not for everything (e.g. core unicode). Also, most of these packages, like unicode/norm, would ideally live as a stripped down version in core that x/text can depend on (this can reduce table size significantly.) The same perhaps for net/idna. This would diminish the benefit of x/internal greatly from x/text's perspective.

[andybons]

Per discussion with @golang/proposal-review

What are the initial set of packages in this repo and how would they help? Would they each have their own module?

[rsc]

I think a "golang.org/x/internal" repo is a type error. I understand why others would think not and have done exactly this, but if you have to put version numbers on it and manage references to it from other repos, it's not particularly internal anymore. I also worry about the tendency to think "oh we'll just put this in internal for now instead of thinking about an API we are willing to commit to", which would lead to the internal repo being a bit of a dumping ground. It would also end up with bits of lots of different subject domains and cut useful git history out of the more specific repos. All of this cautions against an internal repo. I'd rather address case-by-case specific examples as they come up.

[bcmills]

The specific example I had in mind was x/text, but Marcel notes that it would not actually help for that. On that basis, I'll withdraw this proposal until/unless we find a more compelling use-case.

[bcmills]

I also worry about the tendency to think "oh we'll just put this in internal for now instead of thinking about an API we are willing to commit to", which would lead to the internal repo being a bit of a dumping ground.

Even worse, an internal repo doesn't actually let us avoid committing to an API: the internal repo would have to have a stable API to maintain forward-compatibility, and at that point I guess we may as well make the packages public rather than internal.