Scikit-image: Unable to build on Jetson Nano (arm64): UnicodeDecodeError: 'ascii' codec ...

Created on 13 May 2020 · 39Comments · Source: scikit-image/scikit-image

Trying to build on Jetson Nano, encountering a build error with 0.17.1

Installation using pip3 install scikit-image==0.17.1

Collecting scikit-image==0.17.1
  Downloading scikit-image-0.17.1.tar.gz (29.8 MB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-lj9nm9y7/scikit-image/setup.py'"'"'; __file__='"'"'/tmp/pip-install-lj9nm9y7/scikit-image/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-0kn8yhc5
         cwd: /tmp/pip-install-lj9nm9y7/scikit-image/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "", line 1, in 
      File "/tmp/pip-install-lj9nm9y7/scikit-image/setup.py", line 30, in 
        LONG_DESCRIPTION = f.read()
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4029: ordinal not in range(128)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Way to reproduce

Generated a small Dockerfile for this purpose to demonstrate the error (because of the base container being nano specific, might need one to run it):

FROM nvcr.io/nvidia/l4t-base:r32.3.1
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update -y \
  && apt-get install -y --no-install-recommends apt-utils \
  && apt-get install -y \
    build-essential python3-dev libpython3-dev python-pil python3-tk python-imaging-tk                                                                                                                                      
RUN wget -q -O /tmp/get-pip.py --no-check-certificate https://bootstrap.pypa.io/get-pip.py \
  && python3 /tmp/get-pip.py \
  && pip3 install -U pip
RUN pip3 install -U testresources setuptools
RUN apt install -y 
RUN pip3 install -U numpy
RUN pip3 install scikit-image==0.17.1

Version information

Using the temporary container built right before the build log (next section)

docker run --rm -it de732dc209a4
root@a8b1e7122edb:/# python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> import sys; print(sys.version)
3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0]
>>> import platform; print(platform.platform())
Linux-4.9.140-tegra-aarch64-with-Ubuntu-18.04-bionic
>>> import skimage; print("scikit-image version: {}".format(skimage.__version__))
Traceback (most recent call last):
  File "", line 1, in 
ModuleNotFoundError: No module named 'skimage'
>>> import numpy; print("numpy version: {}".format(numpy.__version__))

Build log

docker build --tag ski171:builderror -f scikit-image-0.17.1-Dockerfile .
Sending build context to Docker daemon   2.56kB
Step 1/8 : FROM nvcr.io/nvidia/l4t-base:r32.3.1
 ---> aaaa63e7b12d
Step 2/8 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> b35780f1b3af
Step 3/8 : RUN apt-get update -y   && apt-get install -y --no-install-recommends apt-utils   && apt-get install -y     python3-dev libpython3-dev python-pil python3-tk python-imaging-tk
 ---> Using cache
 ---> a96cbdbb8f3e
Step 4/8 : RUN wget -q -O /tmp/get-pip.py --no-check-certificate https://bootstrap.pypa.io/get-pip.py   && python3 /tmp/get-pip.py   && pip3 install -U pip
 ---> Using cache
 ---> b6852764a3b4
Step 5/8 : RUN pip3 install -U testresources setuptools
 ---> Using cache
 ---> d895bea13fb3
Step 6/8 : RUN apt install -y build-essential
 ---> Using cache
 ---> 4c25ca5a497d
Step 7/8 : RUN pip3 install -U numpy
 ---> Using cache
 ---> de732dc209a4
Step 8/8 : RUN pip3 install scikit-image==0.17.1
 ---> Running in 5b135c7e3b0d
Collecting scikit-image==0.17.1
  Downloading scikit-image-0.17.1.tar.gz (29.8 MB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-lj9nm9y7/scikit-image/setup.py'"'"'; __file__='"'"'/tmp/pip-install-lj9nm9y7/scikit-image/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-0kn8yhc5
         cwd: /tmp/pip-install-lj9nm9y7/scikit-image/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "", line 1, in 
      File "/tmp/pip-install-lj9nm9y7/scikit-image/setup.py", line 30, in 
        LONG_DESCRIPTION = f.read()
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4029: ordinal not in range(128)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
The command '/bin/sh -c pip3 install scikit-image==0.17.1' returned a non-zero code: 1

I will try to see if another arm64 base image can be used to reproduce the error.

Source

mmartial

All 39 comments

can you try with pip3 install scikit-image==0.17.1 -vv (more verbose)

hmaarrfk on 13 May 2020

i think your default encodings in your image are not set to UTF-8, python probably assumes that

hmaarrfk on 13 May 2020

Trying with -vv added but happy to report that a very similar arm64v8/ubuntu:18.04 based container also fails with the same error

FROM arm64v8/ubuntu:18.04

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update -y \
  && apt-get install -y --no-install-recommends apt-utils \
  && apt-get install -y \
    python3-dev libpython3-dev python-pil python3-tk python-imaging-tk \
    build-essential wget

RUN wget -q -O /tmp/get-pip.py --no-check-certificate https://bootstrap.pypa.io/get-pip.py \
  && python3 /tmp/get-pip.py \
  && pip3 install -U pip
RUN pip3 install -U testresources setuptools

RUN pip3 install -U numpy
RUN pip3 install scikit-image==0.17.1

mmartial on 13 May 2020

Going to skip many non-match below, but, here is the output:

Step 7/7 : RUN pip3 install scikit-image==0.17.1 -vv
 ---> Running in 7b995315307e
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-vbqed1sg
Created temporary directory: /tmp/pip-req-tracker-jn879wcc
Initialized build tracking at /tmp/pip-req-tracker-jn879wcc
Created build tracker: /tmp/pip-req-tracker-jn879wcc
Entered build tracker: /tmp/pip-req-tracker-jn879wcc
Created temporary directory: /tmp/pip-install-j_4vctbr
1 location(s) to search for versions of scikit-image:
* https://pypi.org/simple/scikit-image/
Fetching project page and analyzing links: https://pypi.org/simple/scikit-image/
Getting page https://pypi.org/simple/scikit-image/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/scikit-image/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/scikit-image/ HTTP/1.1" 200 23960
Updating cache with response from "https://pypi.org/simple/scikit-image/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/2e/2d/f8a2f9e135d716a66328f48711445795a97bc470f9ddf2cf0b762f9fbb89/scikit-image-0.7.2.tar.gz#sha256=dad28ba772ccd1ddf70a72c3ecfbf27eb1a7ddf7b2e46e2ce56
7fc0c6b603931 (from https://pypi.org/simple/scikit-image/), version: 0.7.2
[...]
  Skipping link: none of the wheel's tags match: cp38-cp38-win_amd64: https://files.pythonhosted.org/packages/86/85/a4764a459982c42e843544ff7b09fa7c155dfc9457cfbc7c783cf54ff58c/scikit_image-0.17.1-cp38-cp
38-win_amd64.whl#sha256=f4cd4464792d744c6f4c1cf7f0aea9945ea7664eaf5124f783703a1df80560f5 (from https://pypi.org/simple/scikit-image/) (requires-python:>=3.6)
Given no hashes to check 1 links for project 'scikit-image': discarding no candidates
Using version 0.17.1 (newest of versions: 0.17.1)                                                                                                                                                           Collecting scikit-image==0.17.1
  Created temporary directory: /tmp/pip-unpack-_xjmf45y
  Looking up "https://files.pythonhosted.org/packages/3d/b3/b9fdd4dead798cf9c654f2ffee24caa8e398ee27921914539d1e5525b754/scikit-image-0.17.1.tar.gz" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/3d/b3/b9fdd4dead798cf9c654f2ffee24caa8e398ee27921914539d1e5525b754/scikit-image-0.17.1.tar.gz HTTP/1.1" 200 29770107
  Downloading scikit-image-0.17.1.tar.gz (29.8 MB)
  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/3d/b3/b9fdd4dead798cf9c654f2ffee24caa8e398ee27921914539d1e5525b754/scikit-image-0.17.1.tar.gz"
  Caching due to etag
  Added scikit-image==0.17.1 from https://files.pythonhosted.org/packages/3d/b3/b9fdd4dead798cf9c654f2ffee24caa8e398ee27921914539d1e5525b754/scikit-image-0.17.1.tar.gz#sha256=1e2e2cf2572549bdb20b88a0f0ac275eea9f04f78b2b6973afdc3f329a73c75c to build tracker '/tmp/pip-req-tracker-jn879wcc'
    Running setup.py (path:/tmp/pip-install-j_4vctbr/scikit-image/setup.py) egg_info for package scikit-image
    Created temporary directory: /tmp/pip-pip-egg-info-8jsqky8c
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "", line 1, in 
      File "/tmp/pip-install-j_4vctbr/scikit-image/setup.py", line 30, in 
        LONG_DESCRIPTION = f.read()
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4029: ordinal not in range(128)
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 333, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 431, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 85, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_install.py", line 550, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_install.py", line 530, in _generate_metadata
    details=self.name or "from {}".format(self.link)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/build/metadata_legacy.py", line 73, in generate_metadata
    command_desc='python setup.py egg_info',
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/utils/subprocess.py", line 241, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Removed scikit-image==0.17.1 from https://files.pythonhosted.org/packages/3d/b3/b9fdd4dead798cf9c654f2ffee24caa8e398ee27921914539d1e5525b754/scikit-image-0.17.1.tar.gz#sha256=1e2e2cf2572549bdb20b88a0f0ac275eea9f04f78b2b6973afdc3f329a73c75c from build tracker '/tmp/pip-req-tracker-jn879wcc'
Removed build tracker: '/tmp/pip-req-tracker-jn879wcc'
The command '/bin/sh -c pip3 install scikit-image==0.17.1 -vv' returned a non-zero code: 1

mmartial on 13 May 2020

Notice how:
https://github.com/conda-forge/docker-images/blob/master/linux-anvil-aarch64/Dockerfile#L8

Includes a UTF-8 tag.

Can you confirm you do something similar for Ubuntu (i remember it being a little different)
I think it was locale-gen
https://github.com/Archiconda/python-feedstock/blob/aarch64/shippable.yml#L36

hmaarrfk on 13 May 2020

I also tried with pip3 install scikit-image==0.16.2 -vv and it failed in the container because of numpy.distutils.system_info.NotFoundError: No lapack/blas resources found (which is expected because I did not install it to test this error).

I noticed the glitch on 0.17.1 when trying to build a new version of https://github.com/datamachines/cuda_tensorflow_opencv/tree/master/JetsonNano
Because scikit-image was updated, it tried to install the latest version and failed.

When I forced scikit-image==0.16.2 in the Dockerfile it built safely.

I will try the UTF-8 tag details.

mmartial on 13 May 2020

Confirming adding the ENV as requested but same error

[...]
Step 7/9 : ENV LANG en_US.UTF-8
 ---> Running in 25e5a30f0e4d
Removing intermediate container 25e5a30f0e4d                                                                                                                                                                 ---> 342835fbf6c5
Step 8/9 : ENV LANGUAGE=en_US.UTF-8
 ---> Running in 7ad522bd8277
Removing intermediate container 7ad522bd8277
 ---> 49c27592b408
Step 9/9 : RUN pip3 install scikit-image==0.17.1 -vv
 ---> Running in 67518442cbe5
[...]
 Caching due to etag
  Added scikit-image==0.17.1 from https://files.pythonhosted.org/packages/3d/b3/b9fdd4dead798cf9c654f2ffee24caa8e398ee27921914539d1e5525b754/scikit-image-0.17.1.tar.gz#sha256=1e2e2cf2572549bdb20b88a0f0ac275eea9f04f78b2b6973afdc3f329a73c75c to build tracker '/tmp/pip-req-tracker-hfpl_ex_'
    Running setup.py (path:/tmp/pip-install-4_a0dfd1/scikit-image/setup.py) egg_info for package scikit-image
    Created temporary directory: /tmp/pip-pip-egg-info-vymkyvyv
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "", line 1, in 
      File "/tmp/pip-install-4_a0dfd1/scikit-image/setup.py", line 30, in 
        LONG_DESCRIPTION = f.read()
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4029: ordinal not in range(128)
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 333, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 431, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 85, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_install.py", line 550, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_install.py", line 530, in _generate_metadata
    details=self.name or "from {}".format(self.link)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/build/metadata_legacy.py", line 73, in generate_metadata
    command_desc='python setup.py egg_info',
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/utils/subprocess.py", line 241, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Removed scikit-image==0.17.1 from https://files.pythonhosted.org/packages/3d/b3/b9fdd4dead798cf9c654f2ffee24caa8e398ee27921914539d1e5525b754/scikit-image-0.17.1.tar.gz#sha256=1e2e2cf2572549bdb20b88a0f0ac275eea9f04f78b2b6973afdc3f329a73c75c from build tracker '/tmp/pip-req-tracker-hfpl_ex_'
Removed build tracker: '/tmp/pip-req-tracker-hfpl_ex_'
The command '/bin/sh -c pip3 install scikit-image==0.17.1 -vv' returned a non-zero code: 1

mmartial on 13 May 2020

Please, run python -c 'import locale; print(locale.getpreferredencoding())' to check your encoding. It must be utf8, the default with python3.

sciunto on 13 May 2020

The errrorcomes from the European characters in the documentation .

hmaarrfk on 13 May 2020

Is there a way for us to specify the encoding in the read function?

hmaarrfk on 13 May 2020

@hmaarrfk what are the European characters? Also interested in knowing how you did the debugging here.

emmanuelle on 13 May 2020

It fails on the "é" of Stéfan. The char number is displayed in the reported error above. WIth vim (or any other good editor), you can move to a specific character position ;)

sciunto on 13 May 2020

👍1

But python3 is supposed to be natively utf8, and support this, So that's why I asked to check what's the encoding.

sciunto on 13 May 2020

I got bit by this ALOT when developing Archiconda (the precursor to conda-foge's arm64 builds).

Basically, docker images use minimal configuration, which leaves the encoding as some weird latin encoding on machines. Most of python failed to build on this since developers (like myself), hate seeing

# -*- coding: utf-8 -*-

https://www.python.org/dev/peps/pep-0263/

It was most telling that the traceback is in reading the long description, near the end of it

    Traceback (most recent call last):
      File "", line 1, in 
      File "/tmp/pip-install-j_4vctbr/scikit-image/setup.py", line 30, in 
        LONG_DESCRIPTION = f.read()
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]

hmaarrfk on 13 May 2020

👍1

I don't think my PR will solve all the issues, but it will get past the setup phase.

hmaarrfk on 13 May 2020

python -c 'import locale; print(locale.getpreferredencoding())' within the container returns ANSI_X3.4-1968 when not using the ENV.
It seems to be the same when using the ENV.

Based on this, I found the following https://stackoverflow.com/a/38553499
and am adding

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y locales

RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend=noninteractive locales && \
    update-locale LANG=en_US.UTF-8

ENV LANG en_US.UTF-8

to the Dockerfile and testing again.

Will report back with updates.

mmartial on 13 May 2020

@mmartial we would love to have some arm pip wheels. I think some people started to ship wheels for arm.

PRs for infrastructure would be greatly appreciated.

Alternatievely, you could make a PR that tests for an environment running a different locale. I feel like you would jump into a rabbit hole at that stage.

hmaarrfk on 13 May 2020

The lack of binary wheels for many arm64 is what have me build those indeed :)

Here is the updated Dockerfile that is building right now (and so far is still building past what was the UnicodeDecodeError)

FROM arm64v8/ubuntu:18.04

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update -y \
  && apt-get install -y --no-install-recommends apt-utils \
  && apt-get install -y \
    python3-dev libpython3-dev python-pil python3-tk python-imaging-tk \
    build-essential wget locales liblapack-dev

RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend=noninteractive locales && \
    update-locale LANG=en_US.UTF-8
ENV LANG en_US.UTF-8


RUN wget -q -O /tmp/get-pip.py --no-check-certificate https://bootstrap.pypa.io/get-pip.py \
  && python3 /tmp/get-pip.py \
  && pip3 install -U pip
RUN pip3 install -U testresources setuptools

RUN pip3 install -U numpy
RUN pip3 install scikit-image==0.17.1 -vv

mmartial on 13 May 2020

@mmartial, I've been there I can tell you that!

the UTF-8 default is probably some legacy thing that linux can't move away from.

On conda-forge, the scope creep made us stuck on compiling qt5.

so I can't really recommend that.

I can say that you would have trouble with much more than just python without setting UTF-8

hmaarrfk on 13 May 2020

Looking at this https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives related to wheel generation (might be possible from the container as well)

The container is still building in the meanwhile :)

mmartial on 13 May 2020

@mmartial in addition to that:

Look at numpy's testing on "shippable" strategry:
Make a PR to scikit-image
Add shippable to your own fork of scikit-image.
Demo things working on shippable.
Get things building on scikit-image-wheels

hmaarrfk on 13 May 2020

the UTF-8 default is probably some legacy thing that linux can't move away from.
[...]
I can say that you would have trouble with much more than just python without setting UTF-8

Agreed; I will for sure add the extra locale lines in the main container to ensure this is functional

mmartial on 13 May 2020

I just sent them the following email for scikit-image:

Hi XXX, XXXX,

Working on an other project, scikit-image, that would love to build and test out on shippable.

We've had users ask about arm64 support for a while and helping them out would be great!

Let me know if you need any more info.

My github handle is
hmaarrfk
interested the project is
scikit-image

Best,

Mark

I sent that to the support personnel that contacted me in the past, but you may send your request to: [email protected]

hmaarrfk on 13 May 2020

I am a little confused about how the Dockerfile would constitute a PR that said: I am not modifying the scikit-image code, simply finding a way to get it working for building within a container

mmartial on 13 May 2020

so the Dockerfile will need to be extended some, it failed (not too surprising since it has a ton of dependencies NOT installed):

Successfully built scikit-image PyWavelets kiwisolver
Failed to build scipy matplotlib pillow imagecodecs

But the extra locales setup in https://github.com/scikit-image/scikit-image/issues/4705#issuecomment-627985005 fixed the original UnicodeDecodeError error listed.

mmartial on 13 May 2020

The PR would be more to show us that you can run the tests on arm, and that our code is functioning.
We don't test on arm whatsover, and then problems are simply detected by end users like yourself.

As for your immediate needs:

I would suggest you start off by using miniforge:
https://github.com/conda-forge/miniforge/
and install as much of the dependencies as you can from conda-forge

For example,we have
https://github.com/conda-forge/scipy-feedstock
We don't have quite a few more, so you still have building to do.

Honestly, the sad truth is that:

Jetsons, and raspberry Pis are not the same as your laptop

and it will take time to compile for them.

You will have to take care to build up all the dependencies yourself with care.

Or the longer term fix, is to help projects like conda-forge! that aim to have a common platform for linux,mac, windows + aarch64 more recently!

For conda-forge we need a few special packages typically offered by the operating system, so things might take time without your help!

hmaarrfk on 13 May 2020

The other option is to use system provided packages through apt.

I believe Ubuntu provides some.

While I don't recommend users use sudo and pip together, for a contained docker image on a platform we don't really support, it might be the best bet.

hmaarrfk on 13 May 2020

Thanks @hmaarrfk will take a look at it as soon as I am able to

mmartial on 13 May 2020

It may be worth taking a look at what the folks at https://www.piwheels.org/ do too.

stefanv on 14 May 2020

👍1

Yeah piwheels is 32 bit. So for 64 but Jetson, it doesn't always help. You may be able to use a 32 bit docker image though.

hmaarrfk on 14 May 2020

Got it down to only missing dependencies for imagecodecs so far, so should have a minimalistic Dockerfile to build it soon.

mmartial on 14 May 2020

you can probably use imagecodecs-lite

hmaarrfk on 14 May 2020

While working on it, I discovered that same imagecodecs issue that I see listed above.
I can try to build from 20.04

https://github.com/Czaki/imagecodecs/issues/6

mmartial on 14 May 2020

@cgohlke stop using imagecodecs-lite but You can try to install tifffile in version 2019.12.31

Czaki on 15 May 2020

I have a tentative Dockerfile (needs to be cleaned up), noting that because of 18.04 I have to limit to imagecodecs==2019.12.31

FROM arm64v8/ubuntu:18.04

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update -y \
  && apt-get install -y --no-install-recommends apt-utils \
  && apt-get install -y \
    python3-dev libpython3-dev python-pil python3-tk python-imaging-tk \
    build-essential wget locales libfreetype6-dev \
    libopenblas-dev liblapack-dev libatlas-base-dev gfortran \
    libjpeg-dev libpng-dev libtiff5-dev libopenjp2-7-dev libopenjp2-tools


RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend=noninteractive locales && \
    update-locale LANG=en_US.UTF-8
ENV LANG en_US.UTF-8


RUN wget -q -O /tmp/get-pip.py --no-check-certificate https://bootstrap.pypa.io/get-pip.py \
  && python3 /tmp/get-pip.py \
  && pip3 install -U pip
RUN pip3 install -U testresources setuptools

RUN pip3 install -U Cython
RUN pip3 install -U pillow

RUN pip3 install -U numpy
RUN pip3 install -U scipy
RUN pip3 install -U matplotlib
RUN pip3 install -U PyWavelets
RUN pip3 install -U kiwisolver

RUN apt-get install -y libaec-dev
#RUN apt-get install -y libhdf5-serial-dev libhdf5-dev hdf5-tools
RUN apt-get install -y libblosc-dev
#RUN apt-get install -y zlib1g-dev libzmq3-dev
RUN apt-get install -y libbrotli-dev
RUN apt-get install -y libbz2-dev
RUN apt-get install -y libgif-dev
RUN apt-get install -y imagemagick
RUN apt-get install -y liblcms2-dev
RUN apt-get install -y libjxr-dev
RUN apt-get install -y liblz4-dev
RUN apt-get install -y libsnappy-dev
RUN apt-get install -y libopenjp2-7-dev libopenjp2-tools
RUN apt-get install -y libfreetype6-dev
RUN apt-get install -y libzstd-dev
RUN apt-get install -y libwebp-dev

RUN pip3 install imagecodecs==2019.12.3
RUN pip3 install scikit-image==0.17.1 -vv

Caveat being that scikit-image uninstalls imagecodecs==2019.12.3 :)

Restarting with --no-deps

mmartial on 15 May 2020

at this point, 0.17.1 builds :)

Step 30/30 : RUN pip3 install scikit-image==0.17.1 --no-deps
 ---> Running in c5af64e1a982
Collecting scikit-image==0.17.1
  Downloading scikit-image-0.17.1.tar.gz (29.8 MB)
Building wheels for collected packages: scikit-image
  Building wheel for scikit-image (setup.py): started
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): still running...
  Building wheel for scikit-image (setup.py): finished with status 'done'
  Created wheel for scikit-image: filename=scikit_image-0.17.1-cp36-cp36m-linux_aarch64.whl size=24601776 sha256=283ca49e0be2317bc60da614f92044dd5946f85587c93b4514d42c192848a365
  Stored in directory: /root/.cache/pip/wheels/19/59/f2/832c8af2b3d1173911b3549ee4f6f585b6def3b67bfec8c36f
Successfully built scikit-image
Installing collected packages: scikit-image
Successfully installed scikit-image-0.17.1
Removing intermediate container c5af64e1a982
 ---> c1104255a5be
Successfully built c1104255a5be
Successfully tagged sciki:build

Next is likely to test building libtiff from source so we can also use the latest version of imagecodecs on an 18.04 (and consolidate the Dockerfile in the process)

mmartial on 15 May 2020

ok, i'm going to close since the "installation" issue is solved.

I'm pretty sure we have UTF-8 sprinkled everywhere, but lets let somebody else open that issue with a targetted error message!

hmaarrfk on 15 May 2020

pleas see this comment
https://github.com/cgohlke/tifffile/issues/7#issuecomment-629094668

especially:

Imagecodecs can be build without any external C library dependencies (build with --lite option).

Czaki on 15 May 2020

👍1

Updating the Dockerfile with the version that compiles both a full imagecodecs and scikit-image, posted at https://gist.github.com/mmartial/0514b714d1928c518cff2af1ea011686

mmartial on 15 May 2020

Was this page helpful?

0 / 5 - 0 ratings