Pillow: Release Pillow 8.0.0 on October 15, 2020

Created on 7 Jul 2020 · 32Comments · Source: python-pillow/Pillow

A major bump because the next release will drop EOL Python 3.5 (https://github.com/python-pillow/Pillow/pull/4746). Will also be the first version to support Python 3.9.

Needs release notes:

Release Checklist

Main Release

Released quarterly on January 2nd, April 1st, July 1st and October 15th.

[x] Open a release ticket e.g. https://github.com/python-pillow/Pillow/issues/3154
[x] Develop and prepare release in master branch.
[x] Check GitHub Actions,
Travis CI and
AppVeyor to confirm
passing tests in master branch.
[x] Check that all of the wheel builds Pillow Wheel Builder pass the tests in Travis CI.
[x] In compliance with PEP 440, update version identifier in src/PIL/_version.py
[x] Update CHANGES.rst.
[x] Run pre-release check via make release-test in a freshly cloned repo.
[x] Create branch and tag for release e.g.:
bash git branch 5.2.x git tag 5.2.0 git push --all git push --tags
[x] Create source distributions e.g.:
bash make sdist
[x] Create binary distributions
[x] Upload all binaries and source distributions e.g. twine upload dist/Pillow-5.2.0*
[x] Create a new release on GitHub
[x] In compliance with PEP 440, increment and append .dev0 to version identifier in src/PIL/_version.py

Binary Distributions

Windows

[x] Contact @cgohlke for Windows binaries via release ticket e.g. https://github.com/python-pillow/Pillow/issues/1174.
[x] Download and extract tarball from @cgohlke and twine upload *.

Mac and Linux

[x] Use the Pillow Wheel Builder:
bash git clone https://github.com/python-pillow/pillow-wheels cd pillow-wheels ./update-pillow-tag.sh [[release tag]]
[x] Download wheels from the Pillow Wheel Builder release.

Publicize Release

[x] Announce release availability via Twitter e.g. https://twitter.com/PythonPillow/status/1013789184354603010

Documentation

[x] Make sure the default version for Read the Docs is up-to-date with the release changes

Docker Images

[x] Update Pillow in the Docker Images repository
bash git clone https://github.com/python-pillow/docker-images cd docker-images ./update-pillow-tag.sh [[release tag]]

Release

Source

hugovk

👍3

Most helpful comment

https://twitter.com/PythonPillow/status/1319361076463546370

Pinned!

aclark4life on 22 Oct 2020

🚀2

All 32 comments

Review of deprecations:

| Deprecation | |
| - | - |
| Image.show command parameter | Deprecated since version 7.2.0 (June 2020).
Not long enough to remove. |
| Image._showxv | Deprecated since version 7.2.0 (June 2020).
Not long enough to remove. |
| ImageFile.raise_ioerror | Deprecated since version 7.2.0 (June 2020).
Not long enough to remove. |
| PILLOW_VERSION constant | Deprecated since version 5.2.0 (July 2018).
It was initially removed in Pillow 7.0.0, but brought back in 7.1.0 to give projects more time to upgrade.
Deprecation warnings since 7.1.0 (April 2020).
Not long enough to remove. |
| ImageCms.CmsProfile attributes | Deprecated since version 3.2.0 (April 2016).
Deprecation warnings since 6.0.0 (April 2019).
Can be removed: PR #4768. |

hugovk on 9 Jul 2020

👍2

Will begin the release process a bit later this evening so we're ready to release later tonight or tomorrow morning.

hugovk on 14 Oct 2020

@cgohlke Please could we have Windows binaries for 8.0.0?

hugovk on 14 Oct 2020

Here you go.

cgohlke on 14 Oct 2020

👍1

Thanks!

Release is out!

Misc. notes:

The Travis wheel builder took ages, there are meant to be 5 parallel jobs but there were only 2 or 4 running. Other accounts I checked on https://travis-ci.org/ only had 4 parallel.

It took 1 hr 42 min 52 sec (tag). The last master, 3 days ago, took 1 hr 44 min 59 sec, and that ran twice as much (tag + latest).

I downloaded wheels from https://github.com/python-pillow/pillow-wheels/releases/tag/8.0.0 with:

# brew install fetch
mkdir /tmp/assets
fetch --repo https://github.com/python-pillow/pillow-wheels --release-asset="\.whl" --progress --tag 8.0.0 /tmp/assets

hugovk on 14 Oct 2020

🚀1

Travis has been running at almost full capacity the last few days. There was a large uptick in the queue length just before the release started: https://www.traviscistatus.com#system-metrics

nulano on 14 Oct 2020

Yeah, sure that didn't help. But it's dropped quite a lot, and again only 2 jobs are running.

hugovk on 14 Oct 2020

But it's dropped quite a lot, and again only 2 jobs are running.

True, I don't think I've seen 5 concurrent jobs on my repo at all this week.

nulano on 14 Oct 2020

😕1

Announce release availability via Twitter e.g. https://twitter.com/PythonPillow/status/1013789184354603010

Can you pin the new tweet (and maybe update the release checklist)? The 7.2.0 release is still pinned on Twitter.

nulano on 14 Oct 2020

I can't pin via TweetDeck, @aclark4life please could you do it? ^

hugovk on 14 Oct 2020

😕1 👍1

Done! Thanks all ❤️

aclark4life on 14 Oct 2020

🚀2

Hi, I see the following in our CI that uses Ubuntu Xenial:

Collecting pillow>=4.1.1 (from torchvision==0.5.0)

  Downloading https://files.pythonhosted.org/packages/74/15/93be74c3124ad183ea3f25251a66d3c7f8641065537973c276e81f703a0b/Pillow-8.0.0.tar.gz (44.6MB)

    Complete output from command python setup.py egg_info:

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

      File "/tmp/pip-build-q4jwbs9a/pillow/setup.py", line 42

        f"Pillow {PILLOW_VERSION} does not support Python "

                                                          ^

    SyntaxError: invalid syntax

I guess Xenial is no longer supported?

pevogam on 15 Oct 2020

Xenial is supported but Python 3.5 and lower are not:

https://pillow.readthedocs.io/en/latest/installation.html

What Python do you have? Are you using pip older than version 9? If you update pip to 9+, it should fetch the correct version of Pillow for you.

python3 -m pip install --upgrade pip
python3 -m pip install --upgrade Pillow

hugovk on 15 Oct 2020

Yes, the python version is 3.5. The pip package that is fetched is indeed older:

Selecting previously unselected package python-pip-whl.

Preparing to unpack .../python-pip-whl_8.1.1-2ubuntu0.6_all.deb ...

However, it is the package of choice on Xenial since we don't specify the pip version when installing:

apt-get -y install gcc libx11-dev libxtst-dev python3-dev libpng-dev python3-pip

I guess this means that pillow now requires/assumes non-default setup on Xenial if one has to manually upgrade the pip?

pevogam on 15 Oct 2020

I recommend upgrading pip to make sure you have the latest security updates (here's one), and to use the latest features of pip, notably the python_requires metadata that would prevent this problem.

Anyway, you can still use old pip by pinning, for example: python -m pip install "pillow < 8"

hugovk on 15 Oct 2020

I am not insisting on using the old pip and our only goal in the CI is to use as minimal and standard dependency setup as possible. My point above was that default Ubuntu Xenial and every user on it will get the aforementioned error if they run the most standard setup

apt-get -y install python3-pip
pip3 install pillow

but this is a minor problem. I guess these users are a small circle by now and hopefully won't open too many duplicate issues.

pevogam on 15 Oct 2020

Looks like the Windows wheels do not have CBDT support enabled: https://github.com/python-pillow/Pillow/pull/4998#issuecomment-712753694

SKIPPED [2] C:\Git\Pillow\Tests\test_imagefont.py:951: freetype compiled without libpng or unsupported

nulano on 20 Oct 2020

Cross-posting @nulano's https://github.com/python-pillow/Pillow/pull/4998#issuecomment-712753694:

From (emphasis mine): sourceforge.net/projects/freetype/files/freetype2/2.10.4

CHANGES BETWEEN 2.10.3 and 2.10.4
I. IMPORTANT BUG FIXES

A heap buffer overflow has been found in the handling of embedded
PNG bitmaps, introduced in FreeType version 2.6.
cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-15999
If you use option FT_CONFIG_OPTION_USE_PNG you should upgrade
immediately.

Does this affect Pillow wheels? FT_CONFIG_OPTION_USE_PNG was added in #4955.

Edit: Windows wheels for 8.0.0 don't use this option, with PyPI wheel I get:
SKIPPED [2] C:\Git\Pillow\Tests\test_imagefont.py:951: freetype compiled without libpng or unsupported
Edit2: Tests on pillow-wheels run without verbose mode, but I don't see the skip message, suggesting that it is enabled there: travis-ci.org/github/python-pillow/pillow-wheels/jobs/736355165
Is this a reason to release 8.0.1?

Does this affect Pillow wheels and do we need an 8.0.1 release?

hugovk on 21 Oct 2020

Here is the FreeType discussion about the CVE: https://savannah.nongnu.org/bugs/?59308
The report mentions that the CVE is being actively exploited in Chrome, with more details to be published on 2020-10-26.

The report has an attached test font file; I think the following snippet should trigger the exploit. Running it in a build with address sanitization enabled should report an issue, but I can't test that on Windows. Just enabling heap verification with gflags.exe has no effect in a build from 8.0.0 tag with libpng.

>>> from PIL import ImageFile
# font.ttf from the FreeType report
>>> ImageFile.truetype("font.ttf", 150).getmask("ABC")
# heap corruption happens here
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\git\pillow\src\PIL\ImageFont.py", line 572, in getmask
    ink=ink,
  File "c:\git\pillow\src\PIL\ImageFont.py", line 652, in getmask2
    text, mode, direction, features, language, anchor
OSError: broken file
# I think this exception is raised after the heap corruption

It is not necessary to pass draw.text(..., embedded_color=True) to trigger the exploit, FreeType will load the PNG even with embedded_color=False and just convert it to grayscale. FreeType being compiled with libpng should be enough to trigger it.

As I mentioned in the comment above, the Travis log suggests that Linux and macOS wheels are exploitable (the test_imagefont.py:test_cbdt test is not skipped); Windows was built without libpng. I'm not sure, but this might affect earlier versions as well.

It sounds to me like an 8.0.1 release is needed, ideally before the Chrome exploit details are made public on 2020-10-26.

nulano on 22 Oct 2020

I think we should ship a new version, and if we do one binary release, we have to do all of them.

wiredfool on 22 Oct 2020

Okay, I'll do 8.0.1.

We've already updated FreeType to 2.10.4 in the Mac/Linux wheel builder.
https://github.com/python-pillow/pillow-wheels/pull/168

It's not need for Windows wheels, but let's include https://github.com/python-pillow/Pillow/pull/4998 for consistency.

Will also include https://github.com/python-pillow/Pillow/pull/4992 to help with the release process.

Anything else we need (other than release notes etc.)?

hugovk on 22 Oct 2020

Is there the potential that any older releases are affected, or is this something that got enabled with the color font support?

We should have a list of potentially vulnerable releases, and specifically mention the last python 2 release.

wiredfool on 22 Oct 2020

Is there the potential that any older releases are affected, or is this something that got enabled with the color font support?

Before #4955 bitmap fonts were disabled with FT_LOAD_NO_BITMAP, but it is not clear to me whether this prevents the exploit. The FreeType documentation states:

FT_LOAD_NO_BITMAP: Ignore bitmap strikes when loading. Bitmap-only fonts ignore this flag.

Other than that, it is sufficient for FreeType to be compiled with libpng enabled. I do not see any changes specific to colour font support in the git blame of config.sh which makes me think this was enabled automatically by the FreeType build based on libpng being present.

nulano on 22 Oct 2020

It's not need for Windows wheels, but let's include #4998 for consistency.

It would be nice to actually add libpng / CBDT font support to the Windows wheels, but it is obviously not necessary. This requires @cgohlke adding the library to the build and maybe release notes.

nulano on 22 Oct 2020

Feedback welcome on release notes: https://github.com/python-pillow/Pillow/pull/5000

hugovk on 22 Oct 2020

Release Checklist

Point Release

Released as needed for security, installation or critical bug fixes.

[x] Make necessary changes in master branch.
[x] Update CHANGES.rst.
[x] Check out release branch e.g.:
bash git checkout -t remotes/origin/5.2.x
[x] Cherry pick individual commits from master branch to release branch e.g. 5.2.x.
[x] Check GitHub Actions,
Travis CI and
AppVeyor to confirm
passing tests in release branch e.g. 5.2.x.
[x] In compliance with PEP 440, update version identifier in src/PIL/_version.py
[x] Run pre-release check via make release-test.
[x] Create tag for release e.g.:
bash git tag 5.2.1 git push git push --tags
[x] Create source distributions e.g.:
bash make sdist
[x] Create binary distributions
[x] Upload all binaries and source distributions e.g. twine upload dist/Pillow-5.2.1*
[x] Create a new release on GitHub

Binary Distributions

Windows

[x] Contact @cgohlke for Windows binaries via release ticket e.g. https://github.com/python-pillow/Pillow/issues/1174.
[x] Download and extract tarball from @cgohlke and twine upload *.

Mac and Linux

[x] Use the Pillow Wheel Builder:
bash git clone https://github.com/python-pillow/pillow-wheels cd pillow-wheels ./update-pillow-tag.sh [[release tag]]
[x] Download wheels from the Pillow Wheel Builder release.

Publicize Release

[x] Announce release availability via Twitter e.g. https://twitter.com/PythonPillow/status/1013789184354603010

Documentation

[x] Make sure the default version for Read the Docs is up-to-date with the release changes

Docker Images

[x] Update Pillow in the Docker Images repository
bash git clone https://github.com/python-pillow/docker-images cd docker-images ./update-pillow-tag.sh [[release tag]]

hugovk on 22 Oct 2020

Cherry picked these:

0045cca027a3cf2e9cf1069b62edd571deef6a90 (#4992 release process)
919fdcc49c6f9f803d6c3086c7ec7cf9a75c15a3 (#4998 FreeType update)
558b2e6cf6143acd6323f0887c112043f34b5fb4 & ac348ea3b1807f805c165cd29a54734000f14c3a (#5000 release notes)

GHA Window CI failed with "Overwrite D:\a\Pillow\Pillow\Tests\images\string_dimension.tiff (Yes/No/All)?", will be because of https://github.com/python-pillow/Pillow/pull/4993 / https://github.com/python-pillow/pillow-depends/pull/34, so will include that too: f886bc973bb972a796756d9c0a728795d203dd97.

Edit: passing now 👍

hugovk on 22 Oct 2020

GHA Window CI failed with "Overwrite D:\a\Pillow\Pillow\Tests\images\string_dimension.tiff (Yes/No/All)?", will be because of #4993 / python-pillow/pillow-depends#34, so will include that too: f886bc9.

See #5001 to avoid this in the future.

nulano on 22 Oct 2020

👍1

The Mac and Linux wheel builder is running, currently with one "parallel" job, so it's going to take a long time.

https://travis-ci.org/github/python-pillow/pillow-wheels/builds/738067859

@cgohlke Please could you make Windows binaries for Pillow 8.0.1, making sure to first update to FreeType 2.10.4 to include the security fix? Thank you!

hugovk on 22 Oct 2020

Here you go.

cgohlke on 22 Oct 2020

👍1

Thank you!

Travis CI only had 0-3 parallel jobs and took 3.5 hours instead of the usual 5 parallel jobs and half an hour.

Released!

Thanks all!

hugovk on 22 Oct 2020

👍2