Black: pre-commit fails when setting up the black environment.

[bellisk]

Describe the bug

pre-commit fails when setting up the black environment.

To Reproduce

When using pre-commit with a version of black tagged later than 18.9b0 (including stable or master, the initial setup of the black environment fails with the following output.

[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('/home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/bin/python3.7', '/home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/bin/pip', 'install', '.')
Return code: 1
Expected return code: 0
Output:
    Processing /home/rae/.cache/pre-commit/reporrf_4ype
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'error'

Errors:
        ERROR: Command errored out with exit status 1:
         command: /home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/bin/python3.7 /home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp7yzty2sc
             cwd: /tmp/pip-req-build-1ksmcb94
        Complete output (36 lines):
        Traceback (most recent call last):
          File "/home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
            main()
          File "/home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
            return hook(metadata_directory, config_settings)
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 156, in prepare_metadata_for_build_wheel
            self.run_setup()
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
            exec(compile(code, __file__, 'exec'), locals())
          File "setup.py", line 65, in <module>
            "blackd=blackd:patched_main [d]",
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
            return distutils.core.setup(**attrs)
          File "/home/rae/.pyenv/versions/3.7.3/lib/python3.7/distutils/core.py", line 108, in setup
            _setup_distribution = dist = klass(attrs)
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 448, in __init__
            k: v for k, v in attrs.items()
          File "/home/rae/.pyenv/versions/3.7.3/lib/python3.7/distutils/dist.py", line 292, in __init__
            self.finalize_options()
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 740, in finalize_options
            ep.load()(self)
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 747, in _finalize_setup_keywords
            ep.load()(self, ep.name, value)
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools_scm/integration.py", line 17, in version_keyword
            dist.metadata.version = get_version(**value)
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 150, in get_version
            parsed_version = _do_parse(config)
          File "/tmp/pip-build-env-2tuvz0xl/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py", line 113, in _do_parse
            "use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
        LookupError: setuptools-scm was unable to detect version for '/tmp/pip-req-build-1ksmcb94'.

        Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

       For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
        ----------------------------------------
    ERROR: Command errored out with exit status 1: /home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/bin/python3.7 /home/rae/.cache/pre-commit/reporrf_4ype/py_env-python3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp7yzty2sc Check the logs for full command output.

Check the log at /home/rae/.cache/pre-commit/pre-commit.log

A longer log, with export SETUPTOOLS_SCM_DEBUG=1, is here: https://pastebin.com/YfYsxPPC.

I noticed that the following line in black's setup.py seemed to be involved: "blackd=blackd:patched_main [d]",. When I specified a tagged version of black from just before that line was last edited (18.9b0), the environment setup ran through correctly.

Expected behavior

pre-commit should set up the black environment and then run black on the code to be committed.

Environment (please complete the following information):

• Version: Anything later than 18.9b0, including stable or master
• OS and Python version: Linux Mint 19.1 / Python 3.7.3

Does this bug also happen on master?

Yes.

[asottile]

I tried but couldn't reproduce this, what version of git are you using and could you share your .pre-commit-config.yaml?

[bellisk]

My git version is 2.17.1 and this is my .pre-commit-config.yaml:

repos:
-   repo: https://github.com/psf/black
    rev: stable  # also fails with master or 19.10b0
    hooks:
    - id: black
      language_version: python3.7
-   repo: https://github.com/pre-commit/mirrors-isort
    rev: v4.3.21
    hooks:
    - id: isort
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.4.0
    hooks:
    - id: flake8

I just noticed that if I run pre-commit, the environment initialises OK. It only has the problem when running git commit -am foo.

[asottile]

hmm that's an interesting data point -- but even with that I still can't reproduce :S

I wonder if you could maybe make a reproduction with docker? I have a hunch at what the fix is (probably need to apply something like this to setuptools-scm)

I wonder if this is also maybe the version of virtualenv you have installed adjacent to pre-commit -- that might be the difference (since that controls what version of pip is used)

[Siecje]

I have been trying to create a reproducible environment but I have not been able to yet.

One thing is that it happens when I commit inside of a submodule.

[asottile]

that's very helpful! it also crashes for me inside of a submodule

here's a more minimal reproduction:

#!/usr/bin/env bash
set -euxo pipefail

rm -rf y z home

export PRE_COMMIT_HOME="$PWD/home"
git init y
git init z
git -C z commit --allow-empty -m 'commit!'
git -C y submodule add "$PWD/z"
cat > "$PWD/y/z/.pre-commit-config.yaml" <<EOF
repos:
-   repo: https://github.com/psf/black
    rev: 19.10b0
    hooks:
    -   id: black
EOF
cd y/z
pre-commit install
git commit -m "test"

[asottile]

and here is a minimal reproduction not involving pre-commit:

#!/usr/bin/env bash
set -euxo pipefail

rm -rf y z home black venv

git clone https://github.com/psf/black

export PRE_COMMIT_HOME="$PWD/home"
git init y
git init z
git -C z commit --allow-empty -m 'commit!'
git -C y submodule add "$PWD/z"
cat > "$PWD/y/.git/modules/z/hooks/pre-commit" <<EOF
#!/usr/bin/env bash
virtualenv "$PWD/venv"
"$PWD/venv/bin/pip" install "$PWD/black"
EOF
chmod +x "$PWD/y/.git/modules/z/hooks/pre-commit"
cd y/z
git commit -m "test"

[asottile]

I've created https://github.com/pypa/setuptools_scm/issues/381 to track this

[asottile]

A workaround is to call pre-commit install-hooks manually once outside of git commit (pre-commit will reuse that cache)

[Siecje]

Running pre-commit install-hooks after pre-commit install works. Is this always going to be safe?

pre-commit install
pre-commit install-hooks

[asottile]

yes, you can even combine the two as pre-commit install --install-hooks

we're fixing the root cause in setuptools-scm though so that workaround shouldn't be necessary indefinitely

[RajUnikie]

Manually executing pre-commit install --install-hooks didn't work for me. However downgrading black in .pre-commit-config.yaml worked.
Env:
On macOS Catalina,
Tried with both python3.7.7 and python3.7.4, pip 19.0.3
Black version change to .pre-commit-config.yaml that worked:

-    rev: 19.10b0
+    rev: 19.3b0

[asottile]

@RajUnikie can you show your output? is it identical to the original post? (otherwise it might be a different issue entirely)

[asottile]

this should be closable now, I've fixed this in setuptools-scm

[JelleZijlstra]

Thanks @asottile!