Rules_docker: Passing flag --network=host to docker_flags of docker_toolchain_configure

[whs-dot-hk]

In jenkins of kubernetes in order to run rules like install_pkgs the flag --network=host is need to pass to docker build but there is no way to supply that flag

podTemplate(yaml: """
apiVersion: v1
kind: Pod
spec:
  serviceAccountName: ecr
  containers:
  - name: bazel
    image: gcr.io/cloud-marketplace-containers/google/bazel@sha256:bea7cec14f05aea5a8a8ead0ddbf9a8ef66b4ad2a0f997c8d1d0217804044b8a
    command:
    - cat
    tty: true
    volumeMounts:
    - name: dockersock
      mountPath: /var/run/docker.sock
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock
"""
) {
    node(POD_LABEL) {
      checkout scm

      container("bazel") {
        sh "bazel run //:go_image"
      }
    }
}

download_pkgs(
    name = "cacerts_pkg",
    image_tar = "@debian_image//image",
    packages = [
        "ca-certificates",
    ],
)

install_pkgs(
    name = "cacerts_pkg_image",
    image_tar = "@debian_image//image",
    installables_tar = ":cacerts_pkg.tar",
    installation_cleanup_commands = "rm -rf /var/lib/apt/lists/*",
    output_image_name = "cacerts_pkg_image",
)

container_image(
    name = "cacerts_wrapper",
    base = ":cacerts_pkg_image.tar",
)

go_image(
    name = "go_image",
    base = ":cacerts_wrapper",
    embed = [":go_default_library"],
    visibility = ["//visibility:public"],
)

[austince]

Also running into this issue when building in GitLab CI, where the docker daemon is running as a service. Are you seeing an error message similar to this?

+ /usr/local/bin/docker load -i bazel-out/k8-fastbuild/bin/external/node_lts_slim/image/image.tar
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Trying to register a new toolchain, but the toolchain_type is private here: https://github.com/bazelbuild/rules_docker/blob/9e9f57a171d5aa576e9d5eb937e060a5858f748a/toolchains/docker/BUILD

I'm not very familiar with the toolchain APIs, but this seems like a necessary feature for adding platform-specific docker toolchains.

My approach so far:

tools/platforms/BUILD

package(default_visibility = ["//visibility:public"])

constraint_setting(
    name = "environment",
)

constraint_value(
    name = "ci",
    constraint_setting = ":environment",
)

platform(
    name = "linux_ci",
    constraint_values = [
        ":ci",
    ],
    host_platform = True,
    os_constraints = [
        "@bazel_tools//platforms:linux",
    ],
)

/tools/docker/BUILD

package(default_visibility = ["//visibility:public"])

# Register a docker toolchain for use in CI where the docker daemon is not on the host
load("@io_bazel_rules_docker//toolchains/docker:toolchain.bzl", "docker_toolchain")

docker_toolchain(
    name = "ci_toolchain_impl",
    docker_flags = ["--network=host"],
    tool_path = "/usr/local/bin/docker",
    xz_path = "/usr/bin/xz",
)

toolchain(
    name = "ci_toolchain",
    exec_compatible_with = [
        "//tools/platforms:linux_ci",
        "@platforms//cpu:x86_64",
    ],
    toolchain = ":ci_toolchain_impl",
    toolchain_type = "@io_bazel_rules_docker//toolchains/docker:toolchain_type",
)

Is there another approach?

[whs-dot-hk]

Hi, @austince I didn't came across the error because I supplied the docker socket through hostPath.

Can you config the docker though docker_toolchain_configure?

[austince]

Hey @whs-dot-hk -- yes, I can with docker_toolchain_configure, but only for the entire workspace. Is there a way to only configure it differently in some environments? I'd like to pass the --host_platform //tools/platforms:linux_ci flag to bazel when running in CI.

[smukherj1]

Hi @austince

I think it makes sense to set the docker toolchain_type as public so that users can define and register their own toolchain. Feel free to send a PR

@whs-dot-hk It seems your original question was resolved by using the hostPath? Although I'm not entirely sure where you specified this. Do you mind explaining for my curiosity?

[whs-dot-hk]

@smukherj1 I use jenkins setuped on kubernetes as build system. I specified hostPath in a build Pod.

To use the docker of the bazel image gcr.io/cloud-marketplace-containers/google/bazel, I pass in the dockersock through hostPath.

I could build something like

---
container_pull(
    name = "awscli_image",
    digest = "sha256:895b976cde0e3ab62746f8ed6f9ca239a3eccf5032c2b28d01a30add0e8930ab",  # 2.0.30
    registry = "index.docker.io",
    repository = "amazon/aws-cli",
)

---
load("@io_bazel_rules_docker//container:container.bzl", "container_image", "container_layer")
load("@io_bazel_rules_docker//docker/util:run.bzl", "container_run_and_commit")

container_layer(
    name = "kubectl-layer",
    files = [
        "@kubectl_archive//file",
    ],
)

container_image(
    name = "kubectl_installer",
    base = "@awscli_image//image",
    cmd = "",
    entrypoint = "",
    layers = [":kubectl-layer"],
)

container_run_and_commit(
    name = "kubectl_install",
    commands = [
        "chmod a+x /kubectl",
        "cp /kubectl /usr/local/bin/kubectl",
        "rm /kubectl",
    ],
    image = ":kubectl_installer.tar",
)

container_image(
    name = "my_image",
    base = ":kubectl_install_commit.tar",
)

without problem, because it do not require any network.

But for rules like download_pkgs,
because it rely on apt-get, so docker run --network=host is needed.

When I add the flag to docker_toolchain_configure there will be error, because e.g. docker load doesn't support --network=host.

[whs-dot-hk]

@smukherj1 I fixed it temporary with

https://github.com/henrywong-seekers/rules_docker/commit/89f051dd1befd46c2b69b8129112c0aca864e1ab#diff-d16d07acca7047ffdd9084adffe2e82f

[austince]

@smukherj1, would you accept @whs-dot-hk's current patch in a PR?

EDIT:
I've merged @whs-dot-hk's patch here https://github.com/austince/rules_docker/commit/edbea2e8b691d5703b29c1d9d2ca540661920d7d, but am still getting the same error with the docker load used in download_pkgs

The only way I've found to get docker load to connect to the remote socket is using the DOCKER_HOST env var. I'm not sure of the best way to integrate this, but I'll play around with it.

[whs-dot-hk]

Hi, @austince

I created a dockerfile like so to test if it works

# 3.4.1
FROM gcr.io/cloud-marketplace-containers/google/bazel@sha256:f670e9aec235aa23a5f068566352c5850a67eb93de8d7a2350240c68fcec3b25

COPY . .

This is one example https://github.com/henrywong-seekers/bazel_gpgme_test/blob/master/Dockerfile

and run the follow commands to test it

$ docker build --network=host -t test .
$ docker run -v/var/run/docker.sock:/var/run/docker.sock --network=host test run //:my_image

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

[carolguo-dd]

Hi, I have the same issue running on Gitlab Ci docker gitlab-runner. I was able to get the error working by setting DOCKER_HOST with --action_env flag bazel run --action_env=DOCKER_HOST=xxx. However I ran into another error docker: Error response from daemon: privileged mode is incompatible with user namespaces. You must run the container in the host namespace when running privileged mode. I think it's bc of --privileged flag https://github.com/austince/rules_docker/blob/master/docker/package_managers/run_download.sh.tpl#L31
@whs-dot-hk @austince have you encountered same problems?

[manika1511]

@whs-dot-hk @austince I am using the download_pkgs rule like this:

container_image(
    name="base_image_with_network_flag",
    base = "@pytorch_base_image//image",
    docker_run_flags= "--network=host",
    env = {"DEBIAN_FRONTEND": "noninteractive", "TZ": "America/Denver"},
    creation_time = "{BUILD_TIMESTAMP}",
)

download_pkgs(
    name = "downloaded_apt_packages",
    image_tar = ":base_image_with_network_flag.tar",
    packages = [
        "xxx",
        "yyy"
    ],
)

But, when I run build, the downloaded_apt_packages.sh file created doesn't have the DOCKER_FLAGS variable set. Can you suggest how I can make sure that the DOCKER_FLAGS gets set.