rules_docker can't run on arm based targets

[AustinSchuh]

The presence of the x86_64 binary golang puller (and friends) results in very esoteric errors when running on aarch64.

[nlopezgi]

Thanks for creating this issue. We will probably need to release aarch64 binaries for this to work. @smukherj1 fyi, something else unexpected for the migration

[smukherj1]

For container_pull, the puller_linux or puller_darwin attributes can be overridden as necessary to point to a custom puller Go binary. So a way to get this to work would be, build the //container/go/cmd/puller target on a arm system by doing
bazel build --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64 //container/go/cmd/puller and upload the newly produced binary somewhere and add a http_file invocation to your WORKSPACE right after the call to

load(
    "@io_bazel_rules_docker//repositories:repositories.bzl",
    container_repositories = "repositories",
)
container_repositories()

to download the arm binary instead. So something like:

load(
    "@io_bazel_rules_docker//repositories:repositories.bzl",
    container_repositories = "repositories",
)
container_repositories()
http_file(
            name = "go_puller_linux",
            executable = True,
            sha256 = <insert the sha256 digest of the uploaded binary here>,
            urls = [<add URL to the location you uploaded the arm64 linux binary here>],
        )
load("@io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")

container_deps()
...

Let us know if this works. container_load currently doesn't support this which will need to be done. Also, maybe we can allow overriding the label for the puller/loader binary used via an environment variable. This will prevent users from having to specify the puller/loader via the attribute for every rule invocation.

[AustinSchuh]

I should have been a bit more aggressive on my proposal.

I wrote a repository rule to replace go_puller_linux.

http_file_architecture(
    name = "my_deb",
    urls = {
        "x86_64":["http://example.com/package_x86_64.deb"],
        "aarch64":["http://example.com/package_aarch64.deb"],
    },
    sha256 = {
        "x86_64": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
        "aarch64": "b720c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    },
)

I can share the details if they would be helpful, but it is a small delta off of http_file.

I've got this working, but in the spirit of keeping upstream deltas small for my sanity, I opened this ticket.

[nlopezgi]

Hey @AustinSchuh ,
we just finished most of the migration to go and are ready to get back to resolving this issue. Would you want to collaborate with us to create a PR to use your http_file_architecture properly? Let me know so we can figure out how to get started, probably a PR with just your repo rule so we can look at it and see what else would be needed for us to publish all binaries for the different archs.
Thanks!

[AustinSchuh]

Sure, though I'm overwhelmed right now... Does this make sense to put in the core rules with http_file ? Or keep it in rules_docker? I essentially stole the implementation from core and modified it to use a map.

[AL-Cui]

@AustinSchuh
i add the following code
http_file_architecture( name = "my_deb", urls = { "x86_64":["http://example.com/package_x86_64.deb"], "aarch64":["http://example.com/package_aarch64.deb"], }, sha256 = { "x86_64": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "aarch64": "b720c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", }, )

then,error.name 'http_file_architecture' is not defined

[AustinSchuh]

Put this rule somewhere and load it.

_HTTP_FILE_BUILD = """ 
package(default_visibility = ["//visibility:public"])

filegroup(
    name = "file",
    srcs = ["{}"],
)
"""

# Mostly shamelessly stollen from http.bzl from bazel.  The only difference
# being that we look up which list of urls and which sha256 based on the
# architecture.
def _http_file_architecture(ctx):
    repo_root = ctx.path(".")
    forbidden_files = [ 
        repo_root,
        ctx.path("WORKSPACE"),
        ctx.path("BUILD"),
        ctx.path("BUILD.bazel"),
        ctx.path("file/BUILD"),
        ctx.path("file/BUILD.bazel"),
    ]   

    downloaded_file_path = ctx.attr.downloaded_file_path
    download_path = ctx.path("file/" + downloaded_file_path)
    if download_path in forbidden_files or not str(download_path).startswith(str(repo_root)):
        fail("'%s' cannot be used as downloaded_file_path in http_file_architecture" % ctx.attr.downloaded_file_path)

    architecture = ctx.execute(["uname", "-m"]).stdout.strip()
    urls = ctx.attr.urls[architecture]
    sha256 = ctx.attr.sha256[architecture]

    download_info = ctx.download(
        urls,
        "file/" + downloaded_file_path,
        sha256,
        ctx.attr.executable,
    )
    ctx.file("WORKSPACE", "workspace(name = \"{name}\")".format(name = ctx.name))
    ctx.file("file/BUILD", _HTTP_FILE_BUILD.format(downloaded_file_path))

    return update_attrs(ctx.attr, _http_file_architecture_attrs.keys(), {"sha256": download_info.sha256})

_http_file_architecture_attrs = {
    "executable": attr.bool(
        doc = "If the downloaded file should be made executable.",
    ),
    "downloaded_file_path": attr.string(
        default = "downloaded",
        doc = "Path assigned to the file downloaded",
    ),
    "sha256": attr.string_dict(
        mandatory = True,
        doc = """The expected SHA-256 of the file downloaded.

This is a mapping from architecture to sha256.

This must match the SHA-256 of the file downloaded. _It is a security risk
to omit the SHA-256 as remote files can change._ At best omitting this
field will make your build non-hermetic. It is optional to make development
easier but should be set before shipping.""",
    ),
    "urls": attr.string_list_dict(
        mandatory = True,
        doc = """A list of URLs to a file that will be made available to Bazel.

This is a mapping from architecture to urls.

Each entry in the map must be a file, http or https URL. Redirections are
followed.  Authentication is not supported.""",
    ),
}

http_file_architecture = repository_rule(
    attrs = _http_file_architecture_attrs,
    doc =
        """Downloads a file from a URL and makes it available to be used as a file
group per architecture.

Examples:
  Suppose you need to have a debian package for your custom rules. This package
  is available from http://example.com/package_x86_64.deb or
  http://example.com/package_aarch64.deb. Then you can add to your
  WORKSPACE file:

  python
  load("///util.bzl", "http_file_architecture")

  http_file_architecture(
      name = "my_deb",
      urls = {
          "x86_64":["http://example.com/package_x86_64.deb"],
          "aarch64":["http://example.com/package_aarch64.deb"],
      },
      sha256 = {
          "x86_64": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
          "aarch64": "b720c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      },
  )

  Targets would specify `@my_deb//file` as a dependency to depend on this file.
""",
    implementation = _http_file_architecture,
)

[AL-Cui]

Put this rule somewhere and load it.

_HTTP_FILE_BUILD = """ 
package(default_visibility = ["//visibility:public"])

filegroup(
    name = "file",
    srcs = ["{}"],
)
"""

# Mostly shamelessly stollen from http.bzl from bazel.  The only difference
# being that we look up which list of urls and which sha256 based on the
# architecture.
def _http_file_architecture(ctx):
    repo_root = ctx.path(".")
    forbidden_files = [ 
        repo_root,
        ctx.path("WORKSPACE"),
        ctx.path("BUILD"),
        ctx.path("BUILD.bazel"),
        ctx.path("file/BUILD"),
        ctx.path("file/BUILD.bazel"),
    ]   

    downloaded_file_path = ctx.attr.downloaded_file_path
    download_path = ctx.path("file/" + downloaded_file_path)
    if download_path in forbidden_files or not str(download_path).startswith(str(repo_root)):
        fail("'%s' cannot be used as downloaded_file_path in http_file_architecture" % ctx.attr.downloaded_file_path)

    architecture = ctx.execute(["uname", "-m"]).stdout.strip()
    urls = ctx.attr.urls[architecture]
    sha256 = ctx.attr.sha256[architecture]

    download_info = ctx.download(
        urls,
        "file/" + downloaded_file_path,
        sha256,
        ctx.attr.executable,
    )
    ctx.file("WORKSPACE", "workspace(name = \"{name}\")".format(name = ctx.name))
    ctx.file("file/BUILD", _HTTP_FILE_BUILD.format(downloaded_file_path))

    return update_attrs(ctx.attr, _http_file_architecture_attrs.keys(), {"sha256": download_info.sha256})

_http_file_architecture_attrs = {
    "executable": attr.bool(
        doc = "If the downloaded file should be made executable.",
    ),
    "downloaded_file_path": attr.string(
        default = "downloaded",
        doc = "Path assigned to the file downloaded",
    ),
    "sha256": attr.string_dict(
        mandatory = True,
        doc = """The expected SHA-256 of the file downloaded.

This is a mapping from architecture to sha256.

This must match the SHA-256 of the file downloaded. _It is a security risk
to omit the SHA-256 as remote files can change._ At best omitting this
field will make your build non-hermetic. It is optional to make development
easier but should be set before shipping.""",
    ),
    "urls": attr.string_list_dict(
        mandatory = True,
        doc = """A list of URLs to a file that will be made available to Bazel.

This is a mapping from architecture to urls.

Each entry in the map must be a file, http or https URL. Redirections are
followed.  Authentication is not supported.""",
    ),
}

http_file_architecture = repository_rule(
    attrs = _http_file_architecture_attrs,
    doc =
        """Downloads a file from a URL and makes it available to be used as a file
group per architecture.

Examples:
  Suppose you need to have a debian package for your custom rules. This package
  is available from http://example.com/package_x86_64.deb or
  http://example.com/package_aarch64.deb. Then you can add to your
  WORKSPACE file:

  python
  load("///util.bzl", "http_file_architecture")

  http_file_architecture(
      name = "my_deb",
      urls = {
          "x86_64":["http://example.com/package_x86_64.deb"],
          "aarch64":["http://example.com/package_aarch64.deb"],
      },
      sha256 = {
          "x86_64": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
          "aarch64": "b720c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      },
  )

  Targets would specify `@my_deb//file` as a dependency to depend on this file.
""",
    implementation = _http_file_architecture,
)

hi, boy. now, i want the Arm version of the tar.gz package.Do you know how to build?or,where to find it?thank you vary much

[bretmckee]

I encountered this problem building on 32 bit arm, and eventually found my way to this issue.

I'm trying to understand the motivation for the root cause of this issue --- that rules_docker is trying to download a pre-compiled binary. How did we end up here given that this used to work?

Since this problem showed up recently, I'm guessing it is the result of some migration away from an old tool to a new version written in go? Any pointers to the discussion/design/etc. that lead to this point would be appreciated.

I was surprised to learn that plan is for the package to downloading precompiled binaries, since this assumes that the binary I need already exists, which can not be guaranteed to be true.

This reported issue is caused by an assumption about processor type, and the proposed fixes seem to be centered on that specific issue (adding better binary selection), but once that is resolved the same problem will occur for processor variants, as well as for OS, then for the specific version of the OS, and possibly others as well. The number of binaries required will be very large -- at least (processor,revision) * (OS, version), where both terms have a non-trivial number of legal values.

It seems like safer approaches are to either download the source and build the tools on the target, or or use an interpreted language that works everywhere (python comes to mind since bazel already requires it).

Building on the target will work, but will cause other issues, such as suddenly requiring the go build environment to be downloaded in order to build a container that does to contain any go.

[smukherj1]

See #580 for why rules_docker was migrated from the python containerregistry library that didn't have this problem to go-containerregistry. tl; dr the python containerregistry library no longer accepts feature requests and users of rules_docker that needed new features were out of luck. Thus we made the decision to move to go-containerregistry. Unfortunately, a side effect of using go-containerregistry which is based on Go and produces native binaries is the problem highlighted in this issue.

Now going back to python containerregistry isn't an option due to it no longer being maintained. As for the path forward here are the options:

1. Use rules_docker v0.10.0 which is the last release where all rules use the python binaries.
2. Before calling the repositories macro in @io_bazel_rules_docker//repositories:repositories.bzl here, replace the Go binary repositories to point to a location that has the binaries built for the architecture you wish to run rules_docker on. The binary source code is available here. This option assumes the user of rules_docker has pre-built the binaries for their desired platform and uploaded it somewhere.
3. Figure out if it's possible for our http_archive rule to detect the target architecture and use this to download the correct binary. I'm not sure how to do this. If detecting the target architecture becomes possible, we would be open to publishing pre-built binaries for more architectures beyond linux-amd64 and darwin-amd64.

[strikosn]

This is a completely irresponsible decision making process. Raspberry Pi is a very popular ARM platform for which those docker rules become useless and break the build once the repository gets upgraded to a recent version. Things that used to work suddenly have ugly bugs that require ugly workarounds. In my opinion you should have tried harder to maintain compatibility before deciding to dump everything for Go.

[nlopezgi]

@strikosn , we are very sorry for the issue. Unfortunately, decisions like this one come with trade offs, and while we understand that this tradeoff seems inconsiderable to you, the arm use case is one that we rarely see (if at all) in most of our users / critical use cases. We understand that this is problematic for you, but we do not have atm the bandwidth to provide support for all possible use cases and have to rely on community contributions to help provide support for use cases that we don't. We would be happy to work with you to fix this using, likely, the proposal made higher up in this thread.
Please keep in mind that most of the work on these rules currently should come from community contributions, so you cannot ever expect us to solely be responsible for providing full coverage of all use cases with these rules.

[vielmetti]

@nlopezgi -

From the backlinks here, it looks like there's been a considerable amount of activity over the past 10 months, and also that @tcnghia has managed to put together some patches that address this issue.

The external project that this is blocking is kind, at https://github.com/kubernetes-sigs/kind/issues/166 - which should fit your criteria for a substantial use case with active users who would validate that the work would test correctly.

Happy to help in any way that I can - @worksonarm currently provides arm64 infra for the Bazel project (cc @ariellea @philwo who are contributing to that effort).

[tcnghia]

I think this is now fixed after https://github.com/bazelbuild/rules_docker/pull/1671

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"