Go: circular dependency in golang.org/x/net and golang.org/x/crypto repos

[mem]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?

Go 1.6.2

1. What operating system and processor architecture are you using (go env)?

linux amd64

1. What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

$ mkdir test
$ cd test
$ export GOPATH=$PWD
$ git clone https://go.googlesource.com/net src/golang.org/x/net
$ go install golang.org/x/net/...
src/golang.org/x/net/html/charset/charset.go:20:2: cannot find package "golang.org/x/text/encoding" in any of:
        /usr/lib/go-1.6/src/golang.org/x/text/encoding (from $GOROOT)
        /home/marcelo/devel/go/test/src/golang.org/x/text/encoding (from $GOPATH)
src/golang.org/x/net/html/charset/charset.go:21:2: cannot find package "golang.org/x/text/encoding/charmap" in any of:
        /usr/lib/go-1.6/src/golang.org/x/text/encoding/charmap (from $GOROOT)
        /home/marcelo/devel/go/test/src/golang.org/x/text/encoding/charmap (from $GOPATH)
src/golang.org/x/net/html/charset/charset.go:22:2: cannot find package "golang.org/x/text/encoding/htmlindex" in any of:
        /usr/lib/go-1.6/src/golang.org/x/text/encoding/htmlindex (from $GOROOT)
        /home/marcelo/devel/go/test/src/golang.org/x/text/encoding/htmlindex (from $GOPATH)
src/golang.org/x/net/html/charset/charset.go:23:2: cannot find package "golang.org/x/text/transform" in any of:
        /usr/lib/go-1.6/src/golang.org/x/text/transform (from $GOROOT)
        /home/marcelo/devel/go/test/src/golang.org/x/text/transform (from $GOPATH)
src/golang.org/x/net/http2/h2i/h2i.go:38:2: cannot find package "golang.org/x/crypto/ssh/terminal" in any of:
        /usr/lib/go-1.6/src/golang.org/x/crypto/ssh/terminal (from $GOROOT)
        /home/marcelo/devel/go/test/src/golang.org/x/crypto/ssh/terminal (from $GOPATH)

the missing golang.org/x/text/ packages are easy to solve. When trying to solve the crypto ones, though, you get this:

$ rm -rf src
$ git clone https://go.googlesource.com/crypto src/golang.org/x/crypto
$ go install golang.org/x/crypto/...
src/golang.org/x/crypto/acme/internal/acme/acme.go:25:2: cannot find package "golang.org/x/net/context" in any of:
        /usr/lib/go-1.6/src/golang.org/x/net/context (from $GOROOT)
        /home/marcelo/devel/go/test/src/golang.org/x/net/context (from $GOPATH)

Why am I trying to do it like this? Because what I'd like to do is provide a method where single repositories are fetched, built and installed independently of other repositories, much like a regular Linux distribution would do.

This is a bootstrapping issue that doesn't show up with the regular workflow because go get will fetch the dependencies, and then build the binaries:

$ rm -rf src
$ go get golang.org/x/net/...

This fetches the three repos (net, crypto, text) and builds the packages.

This particular issue is going to be solved when 1.7 is released and golang.org/x/crypto/acme/internal/acme removes the dependency on golang.org/x/net/context in favor of net/context.

The way I work around this issue in my use case is that I build a single "project" out of the three repositories, fetch the three repositories and build everything in one go (no pun intended). The other workaround is to build packages in a specific order (x/net/context first, then golang.org/x/crypto/... then golang.org/x/text/... and then the rest of golang.org/x/net/...), but this doesn't map well to a Linux distribution package building process (you end up with "two" packages for the golang.org/x/net repo, one for golang.org/x/net/context and another one for the rest).

1. What did you expect to see?

That golang.org/x/net can be built before or after golang.org/x/crypto, and not have a circular dependency between the repositories.

1. What did you see instead?

There's a circular dependency between the repositories golang.org/x/net and golang.org/x/crypto. I understand that there's no circular dependency between packages.

Just to be clear, I understand there's no issue with the regular workflow. This is an issue only for workflows that cannot depend on "go get" or having a something akin to a monorepo.

My specific use case arises when trying to provide Go packages in the context of a Yocto-derived Linux distribution (https://www.yoctoproject.org/).

[bradfitz]

I don't think there's anything we're going to do here.

Properties on repo-level dependencies are not something we want to promise. If you have your own tools, you also have the responsibility of fixing your own tools to do the things you need.

We don't intend for git repos to be a unit of software distribution. Go packages are the unit that should be distributed.

[mem]

Ok, that's fine.

Just in case I didn't make myself clear: in general yes, the packages in the distribution correspond to Go packages (plus/minus details). The source packages, though, correspond to whatever is a single entity (think for example a tarball). In this particular case what you clone is a repository containing a whole bunch of Go packages that get mapped to a bunch of distribution packages, all coming out of a single source. The issue here is bootstrapping.

Thanks for the feedback!

[mwhudson]

I'd like to talk about this a little more if we could. This sort of thing really does make life more awkward for distro maintainers. It can be worked around (in this case, probably by moving to 1.7 so it's really not that hard), but it can be awkward.

Currently in Debian, the repo == distribution package is the norm, and works out fine for most things we distribute, although it already results in lots of relatively speaking tiny packages. Splitting x/crypto into 32 separate package or x/net into 24 would be considered at best surprising in this world. You might argue that Debian is broken here, and maybe even be correct, but it still is where we are and won't be quick to fix.

(Possibly this isn't the ideal forum for this discussion...)

[sanmai]

@mwhudson
Isn't there vendor to solve this issue? Every package could (and probably should) include their own dependencies with their source making their building a breeze in terms of dependency management. And builds are static anyway.

[nbraud]

@sanmai Debian does not vendor things, and (more or less) systematically removes vendored dependencies, for several reasons:

• this makes it much, much easier to update a library that had a security upgrade, compared to hunting down all its vendored copies;
• this makes handling copyright attributions and license metadata a pain, since the work is multiplied by the number of times a given library appears in dependencies;
• this is inefficient.

Of course, this only impacts library packages (and source packages): since the produced binaries are (currently) statically-linked, they do not depend on Go library packages.

[mwhudson]

On 13 July 2016 at 09:02, Nicolas Braud-Santoni [email protected]
wrote:

@sanmai https://github.com/sanmai Debian does not vendor things
https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles,
and (more or less) systematically removes vendored dependencies, for
several reasons:

• this makes it much, much easier to update a library that had a
  security upgrade, compared to hunting down all its vendored copies;
• this is inefficient.

Of course, this only impacts library packages (and source packages): since
the produced binaries are (currently) statically-linked, they do not depend
on Go library packages.

Yes indeed, and the "(currently)" is important too -- I'm working on
building Go packages as shared libraries in Ubuntu too, and circular
dependencies are a problem then too!

[bradfitz]

@mwhudson, you're going to build the entire "net" git repo as a single shared library?

And "tools", the dumping ground? librandomcrap.so?

[mem]

I'd think in the context of Linux distributions there could be a single golang.org-x-net package providing multiple shared libraries (libx-net-context.so, etc). In that case you still have circular dependencies because golang.org-x-net depends on golang.org-x-crypto which depends on golang.org-x-net. This can be fixed by having a binary package called golang.org-x-net-context, etc. But this still doesn't fix the bootstrapping issue.

[mwhudson]

On 13 July 2016 at 12:16, Brad Fitzpatrick [email protected] wrote:

@mwhudson https://github.com/mwhudson, you're going to build the entire
"net" git repo as a single shared library?

That's my current plan, yes. The x/foo repositories are perhaps not well
suited to this approach but it works OK for most things I've had to touch
so far.

And "tools", the dumping ground? librandomcrap.so?

Yeah, it would be especially ridiculous to do that for tools. We don't have
any packages that depend on x/tools though.

Even with that said, though, build-dependency loops are painful to work
with even if there ends up being no dependency loops between the resulting
shared objects.

Cheers,
mwh

[bradfitz]

Nothing is being proposed in this bug.

@mwhudson, you reopened this bug, so it's on you to come up with a concrete proposal here. If we just want to lament the situation, we can move it to a mailing list.

[bradfitz]

Timeout. It's been over 6 months since this bug was last updated, and still nothing has been proposed.