Mediapipe: Is `rules_closure` tensorflow url correct in current WORKSPACE?

Created on 5 Oct 2020 · 8Comments · Source: google/mediapipe

Current WORKSPACE contains

# Needed by TensorFlow
http_archive(
    name = "io_bazel_rules_closure",
    sha256 = "e0a111000aeed2051f29fcc7a3f83be3ad8c6c93c186e64beb1ad313f0c7f9f9",
    strip_prefix = "rules_closure-cf1e44edb908e9616030cc83d085989b8e6cd6df",
    urls = [   "http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz",
        "https://github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz",  # 2019-04-04
    ],
)

but first url returns 404 code

curl -IL "http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz"
HTTP/1.1 404 Not Found
X-GUploader-UploadID: ABg5-UwzejVMmnLr5TOer3jaXu4Hu2dFSEMRoUmeeQE9XBuxlIuHnf9i96iagGG7hlc28c2nadEEy8bLPQuiZe2toVc
Content-Type: application/xml; charset=UTF-8
Content-Length: 127
Date: Sun, 04 Oct 2020 22:13:40 GMT
Expires: Sun, 04 Oct 2020 22:13:40 GMT
Cache-Control: private, max-age=0
Server: UploadServer

I tried do it from Google Cloud VM and from some other cloud providers.
Thanks!

And on the page http://mirror.tensorflow.org/ I can not find any url start with github.com/bazelbuild/rules_closure/archive/ with fixed date 2019-04-04, but there are another dates

Source

iamsimakov

All 8 comments

TensorFlow might removed "http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz". But that should not block you from using mediapipe since "https://github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz" is still valid.

```
wget https://github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz
--2020-10-05 09:53:20-- https://github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/bazelbuild/rules_closure/tar.gz/cf1e44edb908e9616030cc83d085989b8e6cd6df [following]
--2020-10-05 09:53:21-- https://codeload.github.com/bazelbuild/rules_closure/tar.gz/cf1e44edb908e9616030cc83d085989b8e6cd6df
Resolving codeload.github.com (codeload.github.com)... 140.82.113.10
Connecting to codeload.github.com (codeload.github.com)|140.82.113.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz’

cf1e44edb908e9616030cc83d085989 [ <=> ] 451.95K 1.16MB/s in 0.4s

2020-10-05 09:53:22 (1.16 MB/s) - ‘cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz’ saved [462798]
```

jiuqiant on 5 Oct 2020

👀1

when I delete first url

Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
INFO: SHA256 (https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip) = f358144776d3dfc5a928ef32a0e4fbe93c99b55772d70cca77a6478e34d96aa7
DEBUG: Rule 'rules_foreign_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "f358144776d3dfc5a928ef32a0e4fbe93c99b55772d70cca77a6478e34d96aa7"
DEBUG: Call stack for the definition of repository 'rules_foreign_cc' which is a http_archive (rule definition at /root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
 - <builtin>
 - /mediapipe/WORKSPACE:39:1
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: mediapipe/examples/desktop/autoflip
INFO: SHA256 (https://github.com/bazelbuild/rules_cc/archive/master.zip) = e1b6d845f9473ececdfb8dbd8c4c18e0339f4e6a45c4aa52e6a265e6f5040933
DEBUG: Rule 'rules_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "e1b6d845f9473ececdfb8dbd8c4c18e0339f4e6a45c4aa52e6a265e6f5040933"
DEBUG: Call stack for the definition of repository 'rules_cc' which is a http_archive (rule definition at /root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
 - <builtin>
 - /mediapipe/WORKSPACE:33:1
ERROR: /root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/external/rules_cc/cc/private/rules_impl/cc_flags_supplier.bzl:28:21: rule() got unexpected keyword argument 'incompatible_use_toolchain_transition'
ERROR: Skipping 'mediapipe/examples/desktop/autoflip:run_autoflip': error loading package 'mediapipe/examples/desktop/autoflip': in /mediapipe/mediapipe/framework/port/build_config.bzl: in /root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/external/com_google_protobuf/protobuf.bzl: in /root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/external/rules_cc/cc/defs.bzl: Extension file 'cc/private/rules_impl/cc_flags_supplier.bzl' has errors
WARNING: Target pattern parsing failed.
ERROR: error loading package 'mediapipe/examples/desktop/autoflip': in /mediapipe/mediapipe/framework/port/build_config.bzl: in /root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/external/com_google_protobuf/protobuf.bzl: in /root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/external/rules_cc/cc/defs.bzl: Extension file 'cc/private/rules_impl/cc_flags_supplier.bzl' has errors
INFO: Elapsed time: 62.932s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded)
FAILED: Build did NOT complete successfully (1 packages loaded)

should I change sha256 / strip_prefix or what else in WORKSPACE?

iamsimakov on 5 Oct 2020

I delete the same line and still can successfully build autoflip on my machine with Bazel 3.5.1. What's your bazel version?
Moreover, the invalid url should have no effect since Bazel automatically goes to the other url to download io_bazel_rules_closure if it finds that the first url returns 404. Your problem seems to be caused by something else.

$ bazel version
Build label: 3.5.1
...
$ bazel clean --expunge && bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/autoflip:run_autoflip
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Starting local Bazel server and connecting to it...
INFO: SHA256 (https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip) = f358144776d3dfc5a928ef32a0e4fbe93c99b55772d70cca77a6478e34d96aa7
DEBUG: Rule 'rules_foreign_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "f358144776d3dfc5a928ef32a0e4fbe93c99b55772d70cca77a6478e34d96aa7"
DEBUG: Repository rules_foreign_cc instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /usr/local/google/home/jqtang/.cache/bazel/_bazel_jqtang/90903d0f81681cb2a489037dd534e716/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: SHA256 (https://github.com/bazelbuild/rules_cc/archive/master.zip) = e1b6d845f9473ececdfb8dbd8c4c18e0339f4e6a45c4aa52e6a265e6f5040933
DEBUG: Rule 'rules_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "e1b6d845f9473ececdfb8dbd8c4c18e0339f4e6a45c4aa52e6a265e6f5040933"
DEBUG: Repository rules_cc instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /usr/local/google/home/jqtang/.cache/bazel/_bazel_jqtang/90903d0f81681cb2a489037dd534e716/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: SHA256 (https://chromium.googlesource.com/libyuv/libyuv/+archive/refs/heads/master.tar.gz) = 5383d87306bf5bc1814522f8ec1d962f27b8f5dca3966947c18db85e9cda927b
DEBUG: Rule 'libyuv' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "5383d87306bf5bc1814522f8ec1d962f27b8f5dca3966947c18db85e9cda927b"
DEBUG: Repository libyuv instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /usr/local/google/home/jqtang/.cache/bazel/_bazel_jqtang/90903d0f81681cb2a489037dd534e716/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: Analyzed target //mediapipe/examples/desktop/autoflip:run_autoflip (123 packages loaded, 5825 targets configured).
INFO: Found 1 target...
......
INFO: From Linking mediapipe/examples/desktop/autoflip/run_autoflip:
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/lite/nnapi/_objs/nnapi_implementation/nnapi_implementation.o:nnapi_implementation.cc:function (anonymous namespace)::ASharedMemory_create(char const*, unsigned long): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
bazel-out/k8-opt/bin/mediapipe/calculators/video/_objs/opencv_video_decoder_calculator/opencv_video_decoder_calculator.o:opencv_video_decoder_calculator.cc:function mediapipe::OpenCvVideoDecoderCalculator::Open(mediapipe::CalculatorContext*): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
bazel-out/k8-opt/bin/mediapipe/calculators/video/_objs/opencv_video_encoder_calculator/opencv_video_encoder_calculator.o:opencv_video_encoder_calculator.cc:function mediapipe::OpenCvVideoEncoderCalculator::Close(mediapipe::CalculatorContext*): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
Target //mediapipe/examples/desktop/autoflip:run_autoflip up-to-date:
  bazel-bin/mediapipe/examples/desktop/autoflip/run_autoflip
INFO: Elapsed time: 57.846s, Critical Path: 40.79s
INFO: 2394 processes: 2394 linux-sandbox.
INFO: Build completed successfully, 2526 total actions

jiuqiant on 5 Oct 2020

I built it using Docker, just docker build . at last commit master branch

iamsimakov on 5 Oct 2020

Try the following change to update the bazel version in the Dockerfile

diff --git a/Dockerfile b/Dockerfile
index d622191..5572557 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -54,7 +54,7 @@ RUN pip3 install tf_slim
 RUN ln -s /usr/bin/python3 /usr/bin/python

 # Install bazel
-ARG BAZEL_VERSION=3.0.0
+ARG BAZEL_VERSION=3.4.0
 RUN mkdir /bazel && \
     wget --no-check-certificate -O /bazel/installer.sh "https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/b\
 azel-${BAZEL_VERSION}-installer-linux-x86_64.sh" && \

It should fix the issue you mentioned. I can successfully build autoflip in a docker container:

root@f908926a984a:/mediapipe# bazel version
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Build label: 3.4.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Jul 13 15:29:42 2020 (1594654182)
Build timestamp: 1594654182
Build timestamp as int: 1594654182
root@f908926a984a:/mediapipe# bazel clean --expunge && bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/autoflip:run_autoflip
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Starting local Bazel server and connecting to it...
......
INFO: Elapsed time: 76.411s, Critical Path: 43.45s
INFO: 2394 processes: 2394 processwrapper-sandbox.
INFO: Build completed successfully, 2526 total actions

jiuqiant on 6 Oct 2020

👀1

Yes, it was build successfully!
Thank you!
You can close this issue.
But if you answer pair my questions I will be more very thankful to you.
1) My build was ended in 659.494s and I have 8 CPU and 16 GB memory, but your build build took much less time. If I want to build only autoflip how could I optimize build time without changing hardware?
2) Result image size is over 3 GB disk space. I would like to have only autoflip in my image. I think I could optimize docker layers, but I don't understand which layers are is not necessary in my case, maybe. In general, are any special best practices or docs to decrease image with mediapipe if I need only one instrument(autoflip)? I know about collapsing layers in one and dropping cache of apt or pip, but is there something else?

Ok, finally I use bazel 3.4.1 to avoid errors on CI https://github.com/bazelbuild/bazel/issues/11756#issuecomment-658134754
I caught described errors on gitlab.

iamsimakov on 7 Oct 2020

Yes, it was build successfully!
Thank you!
You can close this issue.
But if you answer pair my questions I will be more very thankful to you.

My build was ended in 659.494s and I have 8 CPU and 16 GB memory, but your build build took much less time. If I want to build only autoflip how could I optimize build time without changing hardware?

Bazel downloads and builds everything from source when it's the first time you build MediaPipe targets. But, Bazel caches everything it built. In other words, if you modify the autoflip source code and rerun the build command again, it should be super fast. Since you use Docker, you probably can add the autoflip build command to the Dockerfile so that the Docker container will have all the caches for you to use when it gets started.

Result image size is over 3 GB disk space. I would like to have only autoflip in my image. I think I could optimize docker layers, but I don't understand which layers are is not necessary in my case, maybe. In general, are any special best practices or docs to decrease image with mediapipe if I need only one instrument(autoflip)? I know about collapsing layers in one and dropping cache of apt or pip, but is there something else?

The autoflip binary (bazel-bin/mediapipe/examples/desktop/autoflip/run_autoflip) is the only thing you need. You can move the binary to the mediapipe root directory in the docker container, and then run bazel clean --expunge to remove all the build caches, which I believe will significantly reduce the disk size.