Test-infra: rpmbuild is not supported in bazel images

Created on 9 Nov 2017 · 10Comments · Source: kubernetes/test-infra

It appears make bazel-build works fine on my linux workstation against k/k master, but inside planter it does not now. It looks like something changed with packaging for the images.

ERROR: /Users/bentheelder/go/src/k8s.io/kubernetes/cluster/images/hyperkube/BUILD:3:1: no such package '@debian-hyperkube-base-amd64//image': Pull command failed: Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/__main__.py", line 93, in <module>
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/__main__.py", line 80, in main
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py", line 324, in __enter__
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 193, in __init__
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 272, in _Refresh
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/docker_creds_.py", line 153, in Get
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
 (/usr/bin/python /Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par --directory /Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/debian-hyperkube-base-amd64/image --name gcr.io/google-containers/debian-hyperkube-base-amd64@sha256:10546d592e58d5fdb2e25d79f291b8ac62c8d3a3d83337ad7309cca766dbebce) and referenced by '//cluster/images/hyperkube:hyperkube-internal'.
ERROR: Analysis of target '//cluster/images/hyperkube:hyperkube' failed; build aborted: no such package '@debian-hyperkube-base-amd64//image': Pull command failed: Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/__main__.py", line 93, in <module>
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/__main__.py", line 80, in main
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py", line 324, in __enter__
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 193, in __init__
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 272, in _Refresh
  File "/Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par/containerregistry/client/docker_creds_.py", line 153, in Get
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
 (/usr/bin/python /Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/puller/file/puller.par --directory /Users/bentheelder/.cache/bazel/_bazel_bentheelder/c99a7ed2e9dbc7676356495197988838/external/debian-hyperkube-base-amd64/image --name gcr.io/google-containers/debian-hyperkube-base-amd64@sha256:10546d592e58d5fdb2e25d79f291b8ac62c8d3a3d83337ad7309cca766dbebce).
____Elapsed time: 32.367s
make: *** [bazel-build] Error 1
Makefile:544: recipe for target 'bazel-build' failed

arebazel areplanter kinbug

Source

BenTheElder

All 10 comments

/area planter
/kind bug
/cc @ixdy
FYI @leifmadsen
I think this is a relatively recent regression, I've been periodically running a full build with planter against test-infra and kubernetes with the now default 0.7.0-1 image.

BenTheElder on 9 Nov 2017

@BenTheElder ah ok great, thanks so much for trying and filing this. I wasn't sure if I was doing something wrong, or not running some pre-install stuff. That looks similar to the error I'm getting. If I clone kubernetes and test-infra, and do the make bazel-build I get this:

____[6,748 / 6,952] Creating source manifest for //cmd/kubemark:kubemark
ERROR: /home/lmadsen/src/go/src/k8s.io/kubernetes/build/rpms/BUILD:40:1: MakeRpm build/rpms/kubernetes-cni-x86_64.rpm failed (Exit 1): make_rpm failed: error executing command 
  (cd /home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/execroot/__main__ && \
  exec env - \
  bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm '--name=kubernetes-cni' '--version=@bazel-out/local-fastbuild/genfiles/build/cni_version' '--arch=x86_64' '--spec_file=build/rpms/kubernetes-cni.spec' '--out_file=bazel-out/local-fastbuild/bin/build/rpms/kubernetes-cni-x86_64.rpm' CHANGELOG.md external/kubernetes_cni/file/cni-plugins-amd64-v0.6.0.tgz)

Use --sandbox_debug to see verbose messages from the sandbox.
Building RPM for kubernetes-cni at bazel-out/local-fastbuild/bin/build/rpms/kubernetes-cni-x86_64.rpm
Traceback (most recent call last):
  File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py", line 228, in <module>
    main(FLAGS(sys.argv))
  File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py", line 223, in main
    return builder.Build(FLAGS.spec_file, FLAGS.out_file)
  File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py", line 214, in Build
    status = self.CallRpmBuild(dirname)
  File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py", line 182, in CallRpmBuild
    p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
____Elapsed time: 125.915s, Critical Path: 5.13s
Makefile:544: recipe for target 'bazel-build' failed
make: *** [bazel-build] Error 1

...and when I run the test in test-infra...

 lmadsen  rover  ~  …  test-infra  master  $  ./planter/planter.sh bazel test //
Unable to find image 'gcr.io/k8s-testimages/planter:0.7.0-1' locally
Trying to pull repository gcr.io/k8s-testimages/planter ... 
sha256:1f4b619036072a7342df0ec9eb5fea3d9e3a564fe7e86c3f87266b33567c3a84: Pulling from gcr.io/k8s-testimages/planter
c20dc0c5afeb: Pull complete 
7d45c8fbd867: Pull complete 
b35ea3e7d4eb: Pull complete 
4ab96b2ab340: Pull complete 
101f36dc9c48: Pull complete 
b90ae78a37c0: Pull complete 
Digest: sha256:1f4b619036072a7342df0ec9eb5fea3d9e3a564fe7e86c3f87266b33567c3a84
Status: Downloaded newer image for gcr.io/k8s-testimages/planter:0.7.0-1
.
ERROR: the empty string is not a valid target.
____Elapsed time: 0.195s
ERROR: Couldn't start the build. Unable to run tests.
 lmadsen  rover  ~  …  test-infra  master  ERROR  $  pwd
/home/lmadsen/src/go/src/k8s.io/test-infra

leifmadsen on 9 Nov 2017

👍1

For the second error test-infra currently needs bazel test -- //... -//vendor/... (there is an open issue to get back to bazel test //...).
// is not a valid target pattern.

Generally you build/test with //... but both k/k and test-infra have bad go
test code in vendor/ so they exclude vendor with: //... -//vendor/...
which must come after -- to disambiguate the flags from the negative
target.

https://docs.bazel.build/versions/master/user-manual.html#target-patterns

The failures with kubernetes make bazel-build are definitely a regression
though, which I am looking into today.

On Nov 9, 2017 10:54, "Leif Madsen" notifications@github.com wrote:

@BenTheElder https://github.com/bentheelder ah ok great, thanks so much
for trying and filing this. I wasn't sure if I was doing something wrong,
or not running some pre-install stuff. That looks similar to the error I'm
getting. If I clone kubernetes and test-infra, and do the make bazel-build
I get this:

____[6,748 / 6,952] Creating source manifest for //cmd/kubemark:kubemark
ERROR: /home/lmadsen/src/go/src/k8s.io/kubernetes/build/rpms/BUILD:40:1:
MakeRpm build/rpms/kubernetes-cni-x86_64.rpm failed (Exit 1): make_rpm
failed: error executing command
(cd /home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/execroot/__main__
&& \
exec env - \
bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm
'--name=kubernetes-cni'
'--version=@bazel-out/local-fastbuild/genfiles/build/cni_version'
'--arch=x86_64' '--spec_file=build/rpms/kubernetes-cni.spec'
'--out_file=bazel-out/local-fastbuild/bin/build/rpms/kubernetes-cni-x86_64.rpm'
CHANGELOG.md external/kubernetes_cni/file/cni-plugins-amd64-v0.6.0.tgz)

Use --sandbox_debug to see verbose messages from the sandbox.
Building RPM for kubernetes-cni at
bazel-out/local-fastbuild/bin/build/rpms/kubernetes-cni-x86_64.rpm
Traceback (most recent call last):
File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py",
line 228, in
main(FLAGS(sys.argv))
File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py",
line 223, in main
return builder.Build(FLAGS.spec_file, FLAGS.out_file)
File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py",
line 214, in Build
status = self.CallRpmBuild(dirname)
File "/home/lmadsen/.cache/bazel/_bazel_lmadsen/ec8246b7f01bf839e9b64a0dc5cd798b/bazel-sandbox/8130683712985612099/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/make_rpm.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/make_rpm.py",
line 182, in CallRpmBuild
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
____Elapsed time: 125.915s, Critical Path: 5.13s
Makefile:544: recipe for target 'bazel-build' failed
make: * [bazel-build] Error 1

...and when I run the test in test-infra...

lmadsen  rover  ~  …  test-infra  master  $ 
./planter/planter.sh bazel test //
Unable to find image 'gcr.io/k8s-testimages/planter:0.7.0-1' locally
Trying to pull repository gcr.io/k8s-testimages/planter ...
sha256:1f4b619036072a7342df0ec9eb5fea3d9e3a564fe7e86c3f87266b33567c3a84:
Pulling from gcr.io/k8s-testimages/planter
c20dc0c5afeb http://gcr.io/k8s-testimages/planterc20dc0c5afeb: Pull complete
7d45c8fbd867: Pull complete
b35ea3e7d4eb: Pull complete
4ab96b2ab340: Pull complete
101f36dc9c48: Pull complete
b90ae78a37c0: Pull complete
Digest: sha256:1f4b619036072a7342df0ec9eb5fea3d9e3a564fe7e86c3f87266b33567c3a84
Status: Downloaded newer image for gcr.io/k8s-testimages/planter:0.7.0-1
.
ERROR: the empty string is not a valid target.
____Elapsed time: 0.195s
ERROR: Couldn't start the build. Unable to run tests.
lmadsen  rover  ~  …  test-infra  master  ERROR  $  pwd
/home/lmadsen/src/go/src/k8s.io/test-infra

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/test-infra/issues/5429#issuecomment-343254384,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA4Bq-snBf63xA3hWAaWNLCNUrt6IE_Kks5s00pPgaJpZM4QYaU2
.

BenTheElder on 9 Nov 2017

It seems the MakeRpm shells out to rpmbuild:

https://github.com/bazelbuild/bazel/blob/9135b7bfc4245e16bd6599e22dce0ba077e0d10a/tools/build_defs/pkg/make_rpm.py#L170-L184

  def CallRpmBuild(self, dirname):
    """Call rpmbuild with the correct arguments."""

    args = [
        'rpmbuild',
        '--define',
        '_topdir %s' % dirname,
        '--define',
        '_tmppath %s/TMP' % dirname,
        '--bb',
        self.spec_file,
    ]
    p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    output = p.communicate()[0]

pull-kubernetes-bazel-build doesn't appear to build building build/rpms/..., though it is on bazel 0.6.1 (TAG=0.6.1-1 planter.sh ... equivalent) and our CI scripts actually do a query for build targets which seems to avoid these being included...

BenTheElder on 9 Nov 2017

Next steps imho are:

MakeRpm might need to to be made more hermetic (?)
make bazel-build ought to match the presubmit (why on earth is this not failing in CI?)
follow up with Planter to make it match what happens in CI more closely pending the two points above

BenTheElder on 9 Nov 2017

@leifmadsen in the meantime:
$GOPATH/src/k8s.io/test-infra/planter/planter.sh bazel build -- //... -//vendor/... -//build/rpms/...
should work for building kubernetes with bazel/planter until we get this sorted out.

BenTheElder on 9 Nov 2017

@BenTheElder instead of calling rpmbuild directly, would it actually make more sense to leverage mock so that we could build the RPMs in a chroot environment? I always use mock when building RPMs so that you don't need to have the host machine worry about all the dependencies. That would probably solve your desire to make it hermetic (I had to look up that word; good one :))

leifmadsen on 10 Nov 2017

Possibly? We're not actually calling it directly per se (one of the stock rules "pkg_rpm" is doing this internally), and bazel does actually sandbox the call quite a bit. Unfortunately though unlike say, the go compiler toolchain, it is depending on rpmbuild already being available on your system instead of managing it's own copy.

For now I think we're going to tweak CI and planter to support rpmbuild but I'd really prefer we didn't have default targets that depend on external tooling as much as possible

BenTheElder on 10 Nov 2017

https://github.com/kubernetes/test-infra/pull/5456 has a fix for planter, changes to CI will follow after doing a little bit more local testing.

BenTheElder on 10 Nov 2017

RPM tooling is in the planter image, I'll get these building in CI soonish

BenTheElder on 16 Nov 2017

👍1

Was this page helpful?

0 / 5 - 0 ratings