Mediapipe: Fail to pull in MediaPipe's external dependencies due to unstable internet connection

[chenloveheimei]

Hello,My name is Long.
Description:
version:newest(2aaf469)
bazel:0.27.0
ubuntu:16.04
ndk:r17c
opencv:3.4.1(from source code include origin and contrib)

this is my Error:
long@long:/media/long/data/android/mediapipe$ bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu
ERROR: /media/long/data/android/mediapipe/third_party/BUILD:24:1: no such package '@android_opencv//': java.io.IOException: Error downloading [https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/downl
oad] to /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/android_opencv/download.zip: connect timed out and referenced by '//third_party:opencv'
ERROR: Analysis of target '//mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu:facedetectiongpu' failed; build aborted: Analysis failed
INFO: Elapsed time: 16.197s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (156 packages loaded, 6354 targets configured)
currently loading: @org_tensorflow//tensorflow/lite/experimental/ruy
Fetching @com_google_common_flogger; fetching 7s
Fetching @com_google_common_flogger_system_backend; fetching 7s

Can I use version 3.4.1 of opencv?
Is it possible not to upgrade?
thx.

[jiuqiant]

Hi Long, the error you pasted above is more like a network issue. More specifically, your machine is not able to download the pre-compiled OpenCV android SDK from https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download.
Unfortunately, we use the pre-compiled OpenCV Android SDK directly, rather than building the libraries from source.

Could you verify that you can access https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download? If so, can you retry the build command?

[chenloveheimei]

Hi Long, the error you pasted above is more like a network issue. More specifically, your machine is not able to download the pre-compiled OpenCV android SDK from https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download.
Unfortunately, we use the pre-compiled OpenCV Android SDK directly, rather than building the libraries from source.

Could you verify that you can access https://sourceforge.net/projects/opencvlibrary/files/4.0.1/opencv-4.0.1-android-sdk.zip/download? If so, can you retry the build command?

i can acces your website.
now i try it again. and generate a new Error:
INFO: Call stack for the definition of repository 'FP16' which is a third_party_http_archive (rule definition at /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/third_party/repo.bzl:206:28):

• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/third_party/FP16/workspace.bzl:6:5
• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/workspace.bzl:37:5
• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/workspace.bzl:73:5
• /media/long/data/android/mediapipe/WORKSPACE:96:1
  ERROR: An error occurred during the fetch of repository 'FP16':
  java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz, https://github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068
  997.tar.gz] to /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/FP16/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]
  INFO: Call stack for the definition of repository 'android_opencv' which is a http_archive (rule definition at /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/bazel_tools/tools/build_defs/repo/http.bzl:237:
  16):
• /media/long/data/android/mediapipe/WORKSPACE:112:1
  INFO: Call stack for the definition of repository 'eigen_archive' which is a tf_http_archive (rule definition at /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/third_party/repo.bzl:126:19):
• /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/workspace.bzl:148:5
• /media/long/data/android/mediapipe/WORKSPACE:96:1
  ERROR: /home/long/.cache/bazel/_bazel_long/2855d1951dc2684b853376a37e34b96b/external/org_tensorflow/tensorflow/lite/delegates/gpu/common/BUILD:149:1: no such package '@FP16//': java.io.IOException: Error downloading [https://mirror.baze
  l.build/github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz, https://github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz] to /home/long/.cache/bazel/_bazel_long/2855d1951dc
  2684b853376a37e34b96b/external/FP16/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out] and referenced by '@org_tensorflow//tensorflow/lite/delegates/gpu/common:types'
  ERROR: Analysis of target '//mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu:facedetectiongpu' failed; build aborted: no such package '@FP16//': java.io.IOException: Error downloading [https://mirror.bazel
  .build/github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz, https://github.com/Maratyszcza/FP16/archive/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz] to /home/long/.cache/bazel/_bazel_long/2855d1951dc2
  684b853376a37e34b96b/external/FP16/febbb1c163726b5db24bed55cc9dc42529068997.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]
  INFO: Elapsed time: 10.975s
  INFO: 0 processes.
  FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
  Fetching @com_google_common_flogger; fetching 10s
  Fetching @com_google_common_flogger_system_backend; fetching 10s

I am in China. I can't open the VPN recent.

[jiuqiant]

Hi Long, sorry about the inconvenience.
https://mirror.bazel.build/ is a google owned website, and it is likely to be blocked by the Great Firewall. But, https://github.com/ should not be blacklisted. If you don't have VPN, the best solution might be retry the Bazel build command later. Bazel caches the downloaded libraries, so you don't need to worry about losing them by retrying the same build command.
Hope this helps.

[chenloveheimei]

Hi Long, sorry about the inconvenience.
https://mirror.bazel.build/ is a google owned website, and it is likely to be blocked by the Great Firewall. But, https://github.com/ should not be blacklisted. If you don't have VPN, the best solution might be retry the Bazel build command later. Bazel caches the downloaded libraries, so you don't need to worry about losing them by retrying the same build command.
Hope this helps.

My windows PC can opens VPN, but the Ubuntu can't.so I think I'll download the libraries which project needs from Windows and put them into the Ubuntu to setup?

[jiuqiant]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

[chenloveheimei]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

thanks.I hope have a good news and can you @me when you update the new version?

[camillol]

@jiuqiant it seems like this would be a common issue for projects using Bazel. There are similar issues on the TensorFlow and Bazel trackers; we should check in with them and see if there's an existing workaround or plan.

[jiuqiant]

@chenloveheimei and I tested the current version of MediaPipe. We verified that MediaPipe demo apps can be successfully built in China now. However, due to the unstable/slow internet connection, we prefer to ask Bazel to cache the external repositories locally and search for the local archives before accessing the network.

For the users with unstable/slow internet connection, you may need to try the following build command several times to download all the needed external repositories:

$ bazel build -c opt --config=android_arm64 \ 
  --distdir=<your cache directory> \
  --repository_cache=<your cache directory> \
  --keep_going \
  mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu

[chenloveheimei]

@chenloveheimei and I tested the current version of MediaPipe. We verified that MediaPipe demo apps can be successfully built in China now. However, due to the unstable/slow internet connection, we prefer to ask Bazel to cache the external repositories locally and search for the local archives before accessing the network.

For the users with unstable/slow internet connection, you may need to try the following build command several times to download all the needed external repositories:

$ bazel build -c opt --config=android_arm64 \ 
  --distdir=<your cache directory> \
  --repository_cache=<your cache directory> \
  --keep_going \
  mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu

I am now able to build object and face detection demos from Shenzhen, China. The problem is solved.
Thank you , you are good guy.

[hippyk]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

Any others work but the 'opencv-3.4.3-android-sdk.zip'.

[Papageno2]

Yes, it's possible to cache the external dependencies and use the cached version later. We are testing the following workflow:

# On the workstation that has the full access to the Internet, fetch all the dependencies to a cache directory by doing:
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ bazel fetch //...  --keep_going --repository_cache=<your cache directory> 

# Then, copy the cache directory to the machine that has restricted Internet access and do:
$  bazel build -c opt --config=android_arm64  mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu --distdir=<your cache directory> 

If the workflow works, we may host an archive that contains all the MediaPipe external dependencies for the users in China.

Stay tuned.

hi, thanks for the above hints.

I want to install MediaPipe on CentOS7, but the internet connection error occurs (Beijing now), so I try to fetch the cache dependencies on my PC(macOS), but the above workflow didn't work on my PC, anywhere I can download all the MediaPipe externel dependencies directly ?

the following is part of the warnings and errors on PC fetching(* is my username):

WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/unicode-org/icu/archive/release-64-2.zip failed: class java.io.IOException connect timed out
WARNING: Download from https://github.com/unicode-org/icu/archive/release-64-2.zip failed: class javax.net.ssl.SSLProtocolException Read timed out
ERROR: An error occurred during the fetch of repository 'icu':
   java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/unicode-org/icu/archive/release-64-2.zip, https://github.com/unicode-org/icu/archive/release-64-2.zip] to /private/var/tmp/_bazel_***/392f0b43f8832c7859e2f502bc85241d/external/icu/release-64-2.zip: Read timed out
ERROR: /private/var/tmp/_bazel_***/392f0b43f8832c7859e2f502bc85241d/external/org_tensorflow/third_party/icu/data/BUILD.bazel:39:1: no such package '@icu//': java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/unicode-org/icu/archive/release-64-2.zip, https://github.com/unicode-org/icu/archive/release-64-2.zip] to /private/var/tmp/_bazel_***/392f0b43f8832c7859e2f502bc85241d/external/icu/release-64-2.zip: Read timed out and referenced by '@org_tensorflow//third_party/icu/data:conversion_data'
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/open-source-parsers/jsoncpp/archive/1.8.4.tar.gz failed: class java.io.IOException connect timed out
WARNING: Download from https://github.com/open-source-parsers/jsoncpp/archive/1.8.4.tar.gz failed: class java.io.IOException connect timed out
ERROR: An error occurred during the fetch of repository 'jsoncpp_git':
   java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/open-source-parsers/jsoncpp/archive/1.8.4