Test-infra: ERROR: infinite symlink expansion detected

[hpandeycodeit]

What happened:
Steps to reproduce:

Run ./hack/update-gofmt.sh

Above scripts create following symlinks:

bazel-bin -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out/darwin-fastbuild/bin
bazel-testlogs -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out/darwin-fastbuild/testlogs
bazel-genfiles -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out/darwin-fastbuild/bin
bazel-out -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out
bazel-test-infra -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra

and then run ./hack/verify-gofmt.sh you will see the error infinite symlink expansion detected.
If I run bazel build //prow/cmd/plank after the above command, it will fail as well with the same error:

./hack/verify-gofmt.sh 
+ bazel test --test_output=streamed @io_k8s_test_infra//hack:verify-gofmt
Starting local Bazel server and connecting to it...
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
DEBUG: /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:256:5: rbe_default is using checked-in configs 'struct(config_repos = [], create_cc_configs = True, create_java_configs = True, env = {"ABI_LIBC_VERSION": "glibc_2.19", "ABI_VERSION": "clang", "BAZEL_COMPILER": "clang", "BAZEL_HOST_SYSTEM": "i686-unknown-linux-gnu", "BAZEL_TARGET_CPU": "k8", "BAZEL_TARGET_LIBC": "glibc_2.19", "BAZEL_TARGET_SYSTEM": "x86_64-unknown-linux-gnu", "CC": "clang", "CC_TOOLCHAIN_NAME": "linux_gnu_x86"}, java_home = "/usr/lib/jvm/java-8-openjdk-amd64", name = "9.0.0")'
ERROR: infinite symlink expansion detected
[start of symlink chain]
/Users/hp/test-infra
/Users/hp/test-infra/bazel-test-infra/external/io_k8s_test_infra
/private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/external/io_k8s_test_infra
/private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/external/io_k8s_test_infra
[end of symlink chain]
Internal error thrown during build. Printing stack trace: java.lang.IllegalStateException: @io_k8s_test_infra//hack:verify-gofmt BuildConfigurationValue.Key[9ab889dfb22a196fe63d08c1789c9a0c] false -> ErrorInfo{exception=com.google.devtools.build.lib.skyframe.FileSymlinkInfiniteExpansionException: Infinite symlink expansion, rootCauses={FILE:[/Users/hp/test-infra]/[bazel-test-infra/external/io_k8s_test_infra]}, cycles=[], isCatastrophic=false, rootCauseOfException=FILE:[/Users/hp/test-infra]/[bazel-test-infra/external/io_k8s_test_infra], isDirectlyTransient=false, isTransitivelyTransient=false}
    at com.google.common.base.Preconditions.checkState(Preconditions.java:823)
    at com.google.devtools.build.lib.skyframe.SkyframeBuildView.assertSaneAnalysisError(SkyframeBuildView.java:675)
    at com.google.devtools.build.lib.skyframe.SkyframeBuildView.processErrors(SkyframeBuildView.java:541)
    at com.google.devtools.build.lib.skyframe.SkyframeBuildView.configureTargets(SkyframeBuildView.java:453)
    at com.google.devtools.build.lib.analysis.BuildView.update(BuildView.java:399)
    at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.runAnalysisPhase(AnalysisPhaseRunner.java:198)
    at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:110)
    at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:141)
    at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:268)
    at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:131)
    at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:111)
    at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:524)
    at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:192)
    at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:573)
    at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$2(GrpcServerImpl.java:624)
    at java.base/java.lang.Thread.run(Unknown Source)

INFO: Elapsed time: 5.555s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (8 packages loaded, 16 targets configured)

What you expected to happen:

It should not throw this error.

How to reproduce it (as minimally and precisely as possible):

1. Run ./hack/update-gofmt.sh
2. Run ./hack/verify-gofmt.sh It will fail on this step.

[stevekuznetsov]

Are you getting this on a clean clone of master?

/cc @fejta @BenTheElder @Katharine

[hpandeycodeit]

yes

[BenTheElder]

Bazel Version?

[hpandeycodeit]

bazel 0.28.1

[BenTheElder]

I cannot reproduce this on a clean clone on bazel 0.28.1. I've run both ./hack/verify-gofmt.sh and bazel build //prow/cmd/plank alternately a few times.

[hpandeycodeit]

I just figured out the exact steps that are causing this. Edited the Issue as well but here is the case.

1. Run ./hack/update-gofmt.sh
   Above scripts create following symlinks:

bazel-bin -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out/darwin-fastbuild/bin
bazel-testlogs -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out/darwin-fastbuild/testlogs
bazel-genfiles -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out/darwin-fastbuild/bin
bazel-out -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra/bazel-out
bazel-test-infra -> /private/var/tmp/_bazel_hp/65b75a3f48b341da305d48973ede8655/execroot/io_k8s_test_infra

and then run ./hack/verify-gofmt.sh you will see the error infinite symlink expansion detected.

[BenTheElder]

Thanks, this replicates on my machine.

[BenTheElder]

rm bazel-test-infra after update-gofmt.sh removes the infinite symlink, but i'm not sure why this is populated in the first place.

running very-gofmt repeatedly works fine after this. running update-gofmt again adds this symlink and breaks everything.

[BenTheElder]

https://github.com/kubernetes/test-infra/pull/13910 should fix it.

[BenTheElder]

Can you try now that we've merged #13910? You may need to rm bazel-test-infra first.

[BenTheElder]

please re-open if you see this again

[hpandeycodeit]

Thanks! It worked after the above changes.