Black: Installing black 19.10b0 without gcc installed causes an error

[jonadaly]

Installing black version 19.10b0 in an environment without gcc causes an error.

To Reproduce:

1. Build a docker image (docker build .) using this Dockerfile:

FROM python:3.7-slim
RUN pip install black

1. See error:

Building wheels for collected packages: pathspec, regex
  Building wheel for pathspec (setup.py): started
  Building wheel for pathspec (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/62/b8/e1/e2719465b5947c40cd85d613d6cb33449b86a1ca5a6c574269
  Building wheel for regex (setup.py): started
  Building wheel for regex (setup.py): finished with status 'error'
  ERROR: Complete output from command /usr/local/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-t05gdu90/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-2uuo3rjn --python-tag cp37:
  ERROR: BASE_DIR is /tmp/pip-install-t05gdu90/regex
  /usr/local/lib/python3.7/site-packages/setuptools/dist.py:472: UserWarning: Normalizing '2019.08.19' to '2019.8.19'
    normalized_version,
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/regex
  copying regex_3/regex/__init__.py -> build/lib.linux-x86_64-3.7/regex
  copying regex_3/regex/regex.py -> build/lib.linux-x86_64-3.7/regex
  copying regex_3/regex/_regex_core.py -> build/lib.linux-x86_64-3.7/regex
  creating build/lib.linux-x86_64-3.7/regex/test
  copying regex_3/regex/test/__init__.py -> build/lib.linux-x86_64-3.7/regex/test
  copying regex_3/regex/test/test_regex.py -> build/lib.linux-x86_64-3.7/regex/test
  running build_ext
  building 'regex._regex' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/regex_3
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.7m -c regex_3/_regex.c -o build/temp.linux-x86_64-3.7/regex_3/_regex.o
  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for regex
  Running setup.py clean for regex
Successfully built pathspec
Failed to build regex

Expected behavior:

We should be able to install black without needing to have gcc installed separately.

Environment (please complete the following information):

• Version: 19.10b0
• OS and Python version: Docker image (python:3.7-slim) running on MacOS/Linux hosts

[miguelsimon]

It seems black's regex dependency fails to build without gcc, here's reproducing in ubuntu in virtualenvs (this is where it broke in our systems):

FROM ubuntu:18.04
RUN apt-get update && apt-get install -y python3-venv
RUN python3 -m venv env && env/bin/pip install black

output:

Failed to build pathspec regex
Installing collected packages: attrs, appdirs, typed-ast, toml, pathspec, click, regex, black
  Running setup.py install for pathspec: started
    Running setup.py install for pathspec: finished with status 'done'
  Running setup.py install for regex: started
    Running setup.py install for regex: finished with status 'error'
    Complete output from command /env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o_hpuz1z/regex/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-tzk8s1zz-record/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.6/regex:
    BASE_DIR is /tmp/pip-build-o_hpuz1z/regex
    /env/lib/python3.6/site-packages/setuptools/dist.py:397: UserWarning: Normalizing '2019.08.19' to '2019.8.19'
      normalized_version,
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/regex
    copying regex_3/regex/__init__.py -> build/lib.linux-x86_64-3.6/regex
    copying regex_3/regex/regex.py -> build/lib.linux-x86_64-3.6/regex
    copying regex_3/regex/_regex_core.py -> build/lib.linux-x86_64-3.6/regex
    creating build/lib.linux-x86_64-3.6/regex/test
    copying regex_3/regex/test/__init__.py -> build/lib.linux-x86_64-3.6/regex/test
    copying regex_3/regex/test/test_regex.py -> build/lib.linux-x86_64-3.6/regex/test
    running build_ext
    building 'regex._regex' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/regex_3
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/env/include -I/usr/include/python3.6m -c regex_3/_regex.c -o build/temp.linux-x86_64-3.6/regex_3/_regex.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o_hpuz1z/regex/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-tzk8s1zz-record/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.6/regex" failed with error code 1 in /tmp/pip-build-o_hpuz1z/regex/
The command '/bin/sh -c python3 -m venv env && env/bin/pip install black' returned a non-zero code: 1

Including python3-dev headers and build-essential makes the build succeed:

FROM ubuntu:18.04
RUN apt-get update && apt-get install -y python3-venv build-essential python3-dev
RUN python3 -m venv env && env/bin/pip install black

output:

    Running setup.py install for regex: finished with status 'done'
Successfully installed appdirs-1.4.3 attrs-19.3.0 black-19.10b0 click-7.0 pathspec-0.6.0 regex-2019.8.19 toml-0.10.0 typed-ast-1.4.0

[mwysokin]

I wouldn't say it's a bug, but it's definitely an inconvenience which might have an actual impact on the global docker related traffic.

[zwing99]

This makes using black pretty hard on modern CI/CD which usually uses docker. Can we please find a better tool other than regex or MR something into that library, so that it builds binary linux wheels

[kvikshaug]

The regex author doesn't seem to be willing to add pre-built linux wheels. We also find that requiring a source build is inconvenient, both in terms of added complexity and build times.

[zsol]

Instead of looking for an alternative to regex itself, I would offer to make it better by contributing a CI pipeline to it that builds wheels on all platforms.

[jezdez]

Since I've been involved with some wheel building platforms I just wanted to chime in and say that it's a non-zero amount of effort to maintain such systems and it's totally okay for the author of regex to decline doing that.

And IMO it's not super fair to reply to the legitimate concern about this new requirement of the C compiler that we should simply help build this non-trivial system either.

I haven't seen efforts to look for alternatives (in the public issues/PRs at least) that would be vastly less time-intensive. @zsol Do you know of any work in that direction?

[zsol]

I don't think anyone has spent time looking at regex alternatives while keeping binary wheel availability in mind. I certainly haven't.

I'd like to clarify - I don't think it's unacceptable to replace Black's dependency on regex with something that has all of: correct unicode handling, similar performance characteristics, binary wheels published. If someone wants to do that, I'm all ears and happy to review PRs. I personally think it's worth investing in regex this way because otherwise it's a high quality library.

[impredicative]

I ran into this issue in an image that uses continuumio/miniconda3:latest which in turn uses debian:latest. I addressed it by adding the following line:

RUN apt-get update && apt-get -y install gcc

I didn't need to install all of build-essential. In any case, this resolution is now going to stress the debian repositories.

[cs01]

This was recently solved for pipx's CI in https://github.com/pipxproject/pipx/pull/269. pipx uses travis, and the fix was to add the path /usr/bin to the PATH environment variable. Just wanted to mention that here in case it helps anyone.

[impredicative]

pipx uses travis, and the fix was to add the path /usr/bin to the PATH environment variable.

Builds are often containerized, and I doubt that many containers have /usr/bin/gcc by default.

---

I don't know what regex offers anyway that black can't do with re.

[JelleZijlstra]

@impredicative regex has better Unicode support. Using regex fixed a bug where Black couldn't deal with certain Unicode identifiers.

[jdufresne]

If an example of the problematic Unicode pattern and input string is known, perhaps it could be shared here so someone could report it to https://bugs.python.org/ in the hopes that is could be improved in the re module.

[hugovk]

https://github.com/psf/black/pull/1047 is the PR which added regex, and has tests for the tricky Unicode bits.

https://github.com/psf/black/issues/455 is the issue it fixed, with several examples.

[vemel]

I added a PR https://github.com/psf/black/pull/1172 that makes typed_ast optional and adds PyPy support.

It would be great if you could help me to test my changes. assert_equivalent is potentially broken for Python <3.8 with no typed_ast installed. I could not find an input that triggers it, so please help me.

[evolutics]

For people looking for a compact Alpine Docker image with Black 19.10b0, installing Black with GCC can be done in a separate build stage to reduce the final image size. Example (see original Dockerfile):

FROM alpine:3.10.3 AS black
RUN apk add --no-cache \
    gcc~=8.3.0 \
    musl-dev~=1.1.22 \
    python3-dev~=3.7.5 \
  && pip3 install --target /opt/black black==19.10b0

FROM alpine:3.10.3
RUN apk add --no-cache python3~=3.7.5
COPY --from=black /opt/black /opt/black
ENV PATH="${PATH}:/opt/black/bin" \
  PYTHONPATH="${PYTHONPATH}:/opt/black"

This one results in an image of about 64.8MB in size. I hope this helps.

[impredicative]

For people looking for a compact Alpine Docker image with Black 19.10b0

Why is there a reason for black to be in a final release image at all? I use black, among other analyzers, in an intermediate throwaway test image, not in the release image. Dockerfile

[vemel]

@impredicative I have a library that generates Python source code from jinja templates and then formats it with black, so it is a direct dependency.

[cas--]

There should be an alternative to a hard dependency of regex for a year old low priority bug that causes widespread disruption to users. Also relying on third-party library in pull requests and packaging availability must be questioned.

The author of regex is not going to provide wheels:

https://bitbucket.org/mrabarnett/mrab-regex/issues/343/wheel-for-linux

@Seth Morton: I, personally, am not interested in building wheels for anything other than Windows (and I’m not sure how long I’ll be building ones for Python 2, what with it about to reach end-of-life), but I would accept a contribution that would enable others to do build them.

Why not revert the fix and dependency to reconsider a better solution?

[zwing99]

I love the suggestion of give up on weird edge cases of Unicode, it is not even 1% of the folks using black if I had to guess that face this odd edge case. That being said, I think that you could also take the library of regex internally into black (Much like urllib3 is embedded in requests) and build black as wheels for all deploy targets.

But whatever the decision something needs to happen or folks will start to abandon ship.

[vemel]

Everywhere in code regex can be safely replaced by re, so regex can be imported with a fallback to re and installed as an extra.

[jezdez]

FWIW, I've just asked @ambv to revert the change again given the low priority nature of the bug and the impact it has on many people.

[zsol]

There's a pull request merged that will soon produce linux wheels for regex.

[ghost]

@zsol , I couldn't find the pull request, but looks like there are wheels for mrab-regex :tada: