Black: Binary releases

[cxong]

I'm interested in adopting black for our dev team, but we are currently on python 2.7 and 3.5, so we cannot easily use black as we'll need to set up python 3.6+. This is not very convenient for a large dev team.

It would be more convenient if this project also distributed binaries for each release. Build for Windows, macOS and Linux. This can be done using CI.

[ambv]

I'm not opposed to this but it depends on how much additional manual work I'd have to do during releases. Accepting PRs.

[orn688]

I have a PR ready to go into homebrew/homebrew-core so Black can be installed on MacOS via brew. Keeping this updated when doing releases would involve running the following command locally on a Mac or in a MacOS Travis pipeline after releasing to PyPI.

brew bump-formula-pr black --url={new_version_pythonhosted_url}

(see homebrew's contributing info)
@ambv is this ok with you? Thanks for an amazing tool!

[ambv]

Yes, I will be happy to upgrade brew as part of new releases. This is great!

[orn688]

Great, done! (#29382)

If/when that PR gets merged I can submit a Black PR documenting that it can be installed with brew.

[skorokithakis]

You can also generate a self-contained binary that runs great on Linux (even without Python) with PyInstaller with one command:

pyinstaller --clean -F --add-data blib2to3/:blib2to3 black.py

04:13:24 $ docker run --rm -i -t -v (pwd):/app ubuntu /bin/bash
root@f6d758f00235:/# cd /app/
root@f6d758f00235:/app# python --version
bash: python: command not found
root@f6d758f00235:/app# ./dist/black black.py 
reformatted black.py
All done! \u2728 \U0001f370 \u2728
1 file reformatted.

[cxong]

I've managed to make a Windows exe. Here's how I did it from scratch:

Install python 3.6+ (https://www.python.org/downloads/)
Open cmd
pip3 install pipenv
pipenv install --dev --python 3.6
git clone https://github.com/ambv/black.git
cd black
pre-commit install
python3 setup.py test # just checking
pip3 install pyinstaller
pipenv run pyinstaller --clean -F --add-data blib2to3/;blib2to3 black.py

When I run tests, I got 70 run, 3 failed, 1 skipped. Not sure if this is expected or needs to be looked into?

Since pyinstaller can't cross-compile, next step would be to find a CI that supports Windows, like AppVeyor.

Attached built exe for anyone interested.
black.zip

[zsol]

We do run appveyor and it's passing on master. What are the failed tests you see?

[sloria]

Might be relevant: I use the following script to send a homebrew release update PR for doitlive. It takes care of passing the --sha256 argument.

#!/bin/sh
# Sends a PR to homebrew-core for a new version of doitlive
# Usage: ./release_homebrew.sh https://files.pythonhosted.org/packages/8c/41/b08e2883c256d52f63f00f622cf8a33d3bf36bb5714af337e67476f8b3fe/doitlive-2.8.0.tar.gz

# Validate argument
[ "$#" -eq 1 ] || { echo 'ERROR: Must pass a URL'; exit 1; }
echo $1 | grep -q '^https://files\.pythonhosted\.org' || { echo 'ERROR: URL must start with https://files.pythonhosted.org'; exit 1; }
URL=$1

# Create a temporary directory
WORK_DIR=`mktemp -d`
# check if tmp dir was created
if [[ ! "$WORK_DIR" || ! -d "$WORK_DIR" ]]; then
  echo "Could not create temp dir"
  exit 1
fi
# deletes the temp directory
function cleanup {
  echo "Cleaning up..."
  rm -rf "$WORK_DIR"
  echo "Done."
}
# remove temporary directory after exiting
trap cleanup EXIT

OUTPUT_PATH="$WORK_DIR/doitlive.tar.gz"
curl $URL -o $OUTPUT_PATH
SHA256=`shasum -a 256 $OUTPUT_PATH | head -1 | grep -o '^\S\+'`
echo "URL: $URL"
echo "SHA: $SHA256"

brew update
echo '*** Sending PR to homebrew-core... ***'
brew bump-formula-pr --strict --sha256=$SHA256 --url=$URL doitlive

https://github.com/sloria/doitlive/blob/dev/release_homebrew.sh

Replace "doitlive" with "black" and the script should just work.

[cxong]

The tests failing on Windows are

======================================================================
FAIL: test_check_diff_use_together (tests.test_black.BlackTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Cong.Xu\AppData\Local\Programs\Python\Python36-32\Lib\contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "c:\users\cong.xu\documents\black\tests\test_black.py", line 962, in test_check_diff_use_together
    self.assertEqual(result.exit_code, 1, result.output)
AssertionError: -1 != 1 :

======================================================================
FAIL: test_multi_file_force_py36 (tests.test_black.BlackTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Cong.Xu\AppData\Local\Programs\Python\Python36-32\Lib\contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "c:\users\cong.xu\documents\black\tests\test_black.py", line 1078, in test_multi_file_force_py36
    self.assertEqual(result.exit_code, 0)
AssertionError: -1 != 0

======================================================================
FAIL: test_multi_file_force_pyi (tests.test_black.BlackTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Cong.Xu\AppData\Local\Programs\Python\Python36-32\Lib\contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "c:\users\cong.xu\documents\black\tests\test_black.py", line 1024, in test_multi_file_force_pyi
    self.assertEqual(result.exit_code, 0)
AssertionError: -1 != 0

----------------------------------------------------------------------
Ran 70 tests in 11.417s

FAILED (failures=3, skipped=1)
Test failed: <unittest.runner.TextTestResult run=70 errors=0 failures=3>
error: Test failed: <unittest.runner.TextTestResult run=70 errors=0 failures=3>

But these tests were run using python setup.py test; if I specify a test runner e.g. python -m pytest, or nose/unittest, the tests pass. I see that AppVeyor is running the test script directly. Must be a test runner issue.

[cxong]

I'm trying to set up binary releases for linux using Travis CI, but I've noticed that the build is failing for 3.6, even on the latest master. Has something changed with one of the dependencies perhaps? Here's the build:
https://travis-ci.org/cxong/black/jobs/404762422

and the error messages:

The command "coverage run tests/test_black.py" exited with 0.
5.22s$ if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then mypy black.py tests/test_black.py; fi
tests/test_black.py:164:18: error: Argument "input" to "invoke" of "CliRunner" has incompatible type "str"; expected "Optional[IO[Any]]"
tests/test_black.py:182:18: error: Argument "input" to "invoke" of "CliRunner" has incompatible type "str"; expected "Optional[IO[Any]]"
tests/test_black.py:880:22: error: Argument "input" to "invoke" of "CliRunner" has incompatible type "str"; expected "Optional[IO[Any]]"
tests/test_black.py:1038:18: error: Argument "input" to "invoke" of "CliRunner" has incompatible type "str"; expected "Optional[IO[Any]]"
tests/test_black.py:1092:18: error: Argument "input" to "invoke" of "CliRunner" has incompatible type "str"; expected "Optional[IO[Any]]"
tests/test_black.py:1171: note: unused 'type: ignore' comment
tests/test_black.py:1173: note: unused 'type: ignore' comment
The command "if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then mypy black.py tests/test_black.py; fi" exited with 1.

[zsol]

Indeed it looks like the tests don't respect Pipfile.lock, but instead install the latest version of everything.

[orn688]

Black is now available as a Homebrew formula (#29382)!

As mentioned above, brew bump-formula-pr will need to be run for each new release to keep it updated. Let me know if I can do anything to help automate that.

[ambv]

@cxong I merged your two pull requests. Thank you! Now, operationally, is there something I need to do manually to mark a particular commit a release, or is a tag on the master branch enough to trigger a release build?

[cxong]

Binaries are built for releases. Both travis and appveyor require github auth tokens which are specific to the repo; the one I used for my fork won't work for ambv/black. I provided instructions in the pull requests; for travis follow this: https://docs.travis-ci.com/user/deployment/releases/ and for appveyor follow this: https://www.appveyor.com/docs/deployment/github/

For travis you can see it attempted to do the release but the auth token was the one for my fork: https://travis-ci.org/ambv/black/jobs/433723373#L594

[briceparent]

For information: I use Black like this, when my repo doesn't have the right version of python:
$ docker run -v pwd:/data bricep/simple_black

It's a tiny docker container I made, which only embeds the minimal requirements. So you need to have docker installed, which is not the solution for everyone. But if you already have docker, you can use Black with this single command line, executed from the folder you want to reformat.

There are probably many ways to improve this container, like to pass the other command line arguments, but for many cases it works as is.

[markuskramerIgitt]

Regarding deployment of the Windows exe, is the auth_token still missing (https://github.com/psf/black/pull/422#discussion_r205653006)?

[ichard26]

Linux and Windows are (once again) covered by #1743, so Mac OS is the only one left. I personally didn't add Mac OS support as I don't own a Mac machine so there was no way to test my config.