bazel leaks sockets into the source file system

[Q-Lee]

Description of the problem / feature request:

bazel occasionally leaves sockets behind in the middle of source code. I've run into this problem in the kubernetes/kubernetes repo since at least bazel v0.6.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This is intermittent, and I don't know the underlying reason. I'll see an error like this:
ERROR: /home/qlee/kubernetes/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/BUILD:87:1: //vendor/k8s.io/apiserver/pkg/registry/generic/registry:package-srcs: invalid label '127.0.0.1:2104981928' in element 0 of attribute 'srcs' in 'filegroup' rule: invalid target name '127.0.0.1:2104981928': target names may not contain ':'

When I investigate the file system, a few sockets are just sitting there.

$ ls staging/src/k8s.io/apiserver/pkg/registry/generic/registry/
127.0.0.1:2104981928 BUILD decorated_watcher_test.go localhost:1058126989664801089 storage_factory.go store_test.go
127.0.0.1:2105081928 decorated_watcher.go doc.go localhost:10581269896648010890 store.go

After a bazel clean, the issue goes away.

What operating system are you running Bazel on?

Ubuntu 14.04

What's the output of bazel info release?

release 0.10.0

Have you found anything relevant by searching the web?

Not even ... "remotely"....

[philwo]

Wait, what? Bazel creates unix sockets with names resembling an IP:port pair (except the port number is way too large) in random places in the output_base? I assume when you wrote "When I investigate the file system, a few sockets are just sitting there." you were looking inside the output base and not your workspace?

Now that's a pretty mysterious bug. I don't think Bazel itself is directly the one at fault here, as we can't create Unix sockets from Java and our C++ code doesn't create any - we use TCP/IP to localhost for communication.

I'm not even sure where to start debugging this. Does this only ever happen when you build k8s? Or also with other projects?

@lberki Any idea?

[lberki]

I'm somewhat skeptical, for two reasons:

• You state that this goes away after bazel clean -- that command does not touch the source tree, so I don't see how that could fix this issue. I could imagine there being odd symlinks in the source tree that somehow point to the output directory, but I don't see evidence of that in the k8s source tree.
• Bazel does not create sockets, ever, or for that matter, any file in the source tree except the bazel-* symlinks. Your ls output doesn't show that these are indeed sockets (ls -l would), either.

My guess is that these files are put there either by some tool Bazel runs (outside of the sandbox, probably) or by some mechanism independent of Bazel.

[BenTheElder]

I'm pretty sure this is some tests that spin up etcd in process, I don't think this is a bazel bug.

[BenTheElder]

EG: https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-bazel-test#8f40be4f108b60484c33

[hlopko]

Closing since we don't plan to investigate more and it seems like this is not a bazel bug.

[jaypipes]

Still investigating, but I'm seeing this exact issue occur when running bazel build -- //... -//vendor/... on a fresh checkout of k/k@b479c7c7ee.

jaypipes@thelio:~/src/github.com/kubernetes/kubernetes$ bazel build --jobs=6 -- //... -//vendor/...
ERROR: /home/jaypipes/src/github.com/kubernetes/kubernetes/pkg/master/BUILD:191:1: //pkg/master:package-srcs: invalid label '127.0.0.1:2100114261' in element 0 of attribute 'srcs' in 'filegroup' rule: invalid target name '127.0.0.1:2100114261': target names may not contain ':'
ERROR: package contains errors: pkg/master
ERROR: error loading package 'pkg/master': Package 'pkg/master' contains errors

After running, there are two socket files left in the source tree:

jaypipes@thelio:~/src/github.com/kubernetes/kubernetes$ git log --oneline | head -n1
bf79c7c7ee Merge pull request #77719 from feiskyer/fix-service-tag
jaypipes@thelio:~/src/github.com/kubernetes/kubernetes$ find -name "127.0.0.1:*"
./pkg/master/127.0.0.1:2100114261
./pkg/master/127.0.0.1:2100214261

[jaypipes]

Also note that bazel clean does not remove the leaking files

[BenTheElder]

I don't think bazel produced these. if you remove them and then build again do they re-appear?

bazel should not be modifying the source tree, and bazel clean will clean up the output in the cache, not anything in the source tree as far as I know

[jaypipes]

Yeah, I suspect something in either make verify or make update is leaving these files in there. I've been following the instructions verbatim from the dev environment setup docs and reproduced this on both a 17.10 and 19.04 ubuntu host. After removing the two socket files, bazel build is running A-OK. I'll do some digging into make update and make verify to see if I can determine where those files are coming in.