Buildx: error in python3.9 pip install via qemu for arm64
Hi everybody,
looks like I face with a known issue when trying to build a container for ARM64.
prerequisites:
1) Dockerfile.arm
FROM python
RUN /usr/sbin/adduser --disabled-password --system flask
WORKDIR /app
COPY requirements.txt /app
RUN /usr/local/bin/pip install -r requirements.txt --no-cache-dir
COPY . /app
EXPOSE 5000
USER flask
CMD ["uwsgi","app.ini"]
requirements.txt
alembic==1.4.3
cachelib==0.1.1
click==7.1.2
Flask==1.1.2
Flask-Migrate==2.5.3
Flask-SQLAlchemy==2.4.4
itsdangerous==1.1.0
Jinja2==2.11.2
Mako==1.1.3
MarkupSafe==1.1.1
psycopg2-binary==2.8.6
python-dateutil==2.8.1
python-dotenv==0.14.0
python-editor==1.0.4
six==1.15.0
SQLAlchemy==1.3.18
SQLAlchemy-Utils==0.36.8
uWSGI==2.0.19.1
Werkzeug==1.0.1
versions
~/.docker/cli-plugins/docker-buildx version
github.com/docker/buildx v0.5.1 11057da37336192bfc57d81e02359ba7ba848e4a
Docker version 20.10.1, build 831ebeae96
What I run:
docker buildx build --platform=linux/arm64 -f=Dockerfile.arm -t ep4sh/myimage:arm --push .
What I expect:
Docker build successfully
What I actually get:
app (master) ✗ docker buildx build --platform=linux/arm64 -f=Dockerfile.arm -t ep4sh/myimage:arm --push .
[+] Building 20.6s (9/10)
=> [internal] load build definition from Dockerfile.arm 0.0s
=> => transferring dockerfile: 36B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 34B 0.0s
=> [internal] load metadata for docker.io/library/python:latest 2.1s
=> [1/6] FROM docker.io/library/python@sha256:39c16d1a064c0239939d4ed52923b736c25b389e6ea439d5652b8fc9564ede76 0.0s
=> => resolve docker.io/library/python@sha256:39c16d1a064c0239939d4ed52923b736c25b389e6ea439d5652b8fc9564ede76 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 981B 0.0s
=> CACHED [2/6] RUN /usr/sbin/adduser --disabled-password --system flask 0.0s
=> CACHED [3/6] WORKDIR /app 0.0s
=> CACHED [4/6] COPY requirements.txt /app 0.0s
=> ERROR [5/6] RUN /usr/local/bin/pip install -r requirements.txt --no-cache-dir 18.3s
------
> [5/6] RUN /usr/local/bin/pip install -r requirements.txt --no-cache-dir:
#9 18.00 ERROR: Exception:
#9 18.00 Traceback (most recent call last):
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 224, in _main
#9 18.00 status = self.run(options, args)
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
#9 18.00 return func(self, options, args)
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 271, in run
#9 18.00 session = self.get_default_session(options)
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 78, in get_default_session
#9 18.00 self._session = self.enter_context(self._build_session(options))
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 88, in _build_session
#9 18.00 session = PipSession(
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/session.py", line 248, in __init__
#9 18.00 self.headers["User-Agent"] = user_agent()
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/session.py", line 131, in user_agent
#9 18.00 zip(["name", "version", "id"], distro.linux_distribution()),
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 125, in linux_distribution
#9 18.00 return _distro.linux_distribution(full_distribution_name)
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 681, in linux_distribution
#9 18.00 self.version(),
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 741, in version
#9 18.00 self.lsb_release_attr('release'),
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 903, in lsb_release_attr
#9 18.00 return self._lsb_release_info.get(attribute, '')
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 556, in __get__
#9 18.00 ret = obj.__dict__[self._fname] = self._f(obj)
#9 18.00 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 1014, in _lsb_release_info
#9 18.00 stdout = subprocess.check_output(cmd, stderr=devnull)
#9 18.00 File "/usr/local/lib/python3.9/subprocess.py", line 420, in check_output
#9 18.00 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#9 18.00 File "/usr/local/lib/python3.9/subprocess.py", line 524, in run
#9 18.00 raise CalledProcessError(retcode, process.args,
#9 18.00 subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
#9 18.02 Traceback (most recent call last):
#9 18.02 File "/usr/local/bin/pip", line 8, in <module>
#9 18.02 sys.exit(main())
#9 18.02 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py", line 75, in main
#9 18.02 return command.main(cmd_args)
#9 18.02 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 117, in main
#9 18.02 return self._main(args)
#9 18.02 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 261, in _main
#9 18.03 self.handle_pip_version_check(options)
#9 18.03 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 147, in handle_pip_version_check
#9 18.03 session = self._build_session(
#9 18.03 File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 88, in _build_session
#9 18.03 session = PipSession(
#9 18.03 File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/session.py", line 248, in __init__
#9 18.03 self.headers["User-Agent"] = user_agent()
#9 18.03 File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/session.py", line 131, in user_agent
#9 18.03 zip(["name", "version", "id"], distro.linux_distribution()),
#9 18.03 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 125, in linux_distribution
#9 18.03 return _distro.linux_distribution(full_distribution_name)
#9 18.03 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 681, in linux_distribution
#9 18.03 self.version(),
#9 18.03 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 741, in version
#9 18.04 self.lsb_release_attr('release'),
#9 18.04 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 903, in lsb_release_attr
#9 18.04 return self._lsb_release_info.get(attribute, '')
#9 18.04 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 556, in __get__
#9 18.04 ret = obj.__dict__[self._fname] = self._f(obj)
#9 18.04 File "/usr/local/lib/python3.9/site-packages/pip/_vendor/distro.py", line 1014, in _lsb_release_info
#9 18.04 stdout = subprocess.check_output(cmd, stderr=devnull)
#9 18.04 File "/usr/local/lib/python3.9/subprocess.py", line 420, in check_output
#9 18.05 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#9 18.05 File "/usr/local/lib/python3.9/subprocess.py", line 524, in run
#9 18.05 raise CalledProcessError(retcode, process.args,
#9 18.05 subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
------
Dockerfile.arm:5
--------------------
3 | WORKDIR /app
4 | COPY requirements.txt /app
5 | >>> RUN /usr/local/bin/pip install -r requirements.txt --no-cache-dir
6 |
7 | COPY . /app
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c /usr/local/bin/pip install -r requirements.txt --no-cache-dir]: exit code: 1
Comment: As you can see buildx fails on RUN /usr/local/bin/pip install -r requirements.txt --no-cache-dir.
I have checked the same inside a running container on an ARM machine - the step was played well.
Could somebody help with that issue? Thanks in advance
qemu
ep4sh
👍3
All 5 comments
Hey guys,
do you have any thoughts on this?
ep4sh
on 28 Dec 2020
Looks like a qemu issue.
tonistiigi
on 29 Dec 2020
I am also seeing failed builds on arm64 buildx which succeed on the image natively. In my case it's the python:3.9 image and pip install is not recognizing the arch and attempting to build all packages rather than installing the wheels.
EDIT: The specific pinned version of the package in requirements.txt didn't have an arm64 wheel, updated and it's working.
brontide
on 3 Jan 2021
I get a very similar error, but only in CodeBuild, and unrelated to python. Locally the same Dockerfile works.
In CodeBuild I have:
Client: Docker Engine - Community
Version: 19.03.11
Server: Docker Engine - Community
Version: 19.03.11
github.com/docker/buildx v0.5.1 11057da37336192bfc57d81e02359ba7ba848e4a814
What does "a qemu issue" mean in this context, and how does one fix this?
autarchprinceps
on 5 Jan 2021
If error happens on running a specific process only under emulation it is likely the emulation layer (qemu) that has the issue on executing this specific code. Just make sure that your nodes have emulators installed properly. https://github.com/tonistiigi/binfmt#installing-emulators is the best way to guarantee it. If you installed binfmt/qemu-user packages via your distro package manager but still see an error starting with dev/.buildkit_qemu_emulator that means it is not properly installed to work in containerized environments and you should use tonistiigi/binfmt image to uninstall the previous emulators first and then reinstall them.
tonistiigi
on 5 Jan 2021
👍3
Was this page helpful?
0 / 5 - 0 ratings
Related issues
carlosedp
·
12Comments
richarvey
·
13Comments
securisec
·
9Comments
dubo-dubon-duponey
·
10Comments
iav
·
7Comments
Most helpful comment
If error happens on running a specific process only under emulation it is likely the emulation layer (qemu) that has the issue on executing this specific code. Just make sure that your nodes have emulators installed properly. https://github.com/tonistiigi/binfmt#installing-emulators is the best way to guarantee it. If you installed binfmt/qemu-user packages via your distro package manager but still see an error starting with
dev/.buildkit_qemu_emulatorthat means it is not properly installed to work in containerized environments and you should usetonistiigi/binfmtimage to uninstall the previous emulators first and then reinstall them.