Pillow: Release Pillow 4.0.0 on Jan 1, 2017

[wiredfool]

• [x] Release notes for dropping 2.6/3.2 #2192
• [x] Release notes for dropping olefile in favor of a vendored version #2199
• [x] Release notes for handles_eof #2223
• [x] Release notes for stretch #2276
• [x] Have bikeshed discussion about version number.

[hugovk]

That bikeshed won't paint itself. So:

We don't really have a formal policy, other than a tradition to use a major.minor.micro versioning. We update the micro version for https://github.com/python-pillow/Pillow/blob/master/RELEASING.md#point-release for security, installation or critical bug fixes, but we're not covering the micro number here.

We're not following semantic versioning, as we release quarterly with whatever has been merged, which inevitably contains both functionality added in a backwards-compatible manner (semver minor bump) and backwards-compatible bug fixes (semver micro bump). We generally try to avoid incompatible API changes (semver major bump), but they sometimes come along too.

I'm not too bothered about 3.5.0 or 4.0.0, but here's some pros and cons.

Reasons for 3.5.0:

• Last release was 3.4.0.
• In https://github.com/python-pillow/Pillow/issues/1087 we decided 3.0 would follow 2.9 instead of 2.10.

Reasons for 4.0.0:

• Makes it clearer something big has changed: Pillow's no longer backwards compatible with Python 2.6 and 3.2.
• It follows suit with Pillow 2.0.0 in dropping Python versions 2.4 and 2.5, and also be much easier to remember.
• In https://github.com/python-pillow/Pillow/issues/1087 we decided 3.0 would follow 2.9 instead of 2.10, i.e. bumping major version when minor reached 9, but we didn't talk about or rule out other big reasons for bumping major versions.
• From https://github.com/python-pillow/Pillow/issues/1087: "Is the release after 2.9 big or important enough to be given a 3.0 version number? Well, 10 releases of Pillow is a great achievement in itself, especially with quarterly releases."
  
  
  
  • This suggests other "big" things merit a major version bump.
  
  
• From https://github.com/python-pillow/Pillow/issues/1087: "Having said that, if anything "big" comes along before 2.9, it might make sense to bump up to 3.0. For example Python 3 support warranted the bump to Pillow 2.0."
  
  
  
  • Dropping support for two Python versions (2.6/3.2) may be big enough.
  
  

More welcome for either version.

[aclark4life]

I am pro "make the version number noticeably different so folks notice the change". There are likely some projects in which I'd prefer sequential "don't draw attention" versioning, but Pillow is probably not one of them.

[aclark4life]

Is this the "release 3.5 (or 4) on Jan 1 2017" issue? If so, can we standardize the title? E.g. "Release Pillow VERSION on DATE"?

[aclark4life]

🏏 s… Do we have any consensus on 3.5 vs. 4? Note that while I've stated my opinions, I'm not doing all the work so I can easily be overridden. But please let me know because I want to update the milestones, etc.

[wiredfool]

I was leaning to 4.0. I think that's the general consensus, and there are no objections that I've heard.

[gpraceman]

Will this add support for Python 3.6?

[aclark4life]

@gpraceman I believe so, yes. When in doubt, check the master branch.

[wiredfool]

I'm ready to increment the version and tag 4.0. @cgohlke Do you need the source uploaded to Pypi to build? I'm thinking that we might avoid some questions if we tag and build, then upload binaries at the same time as source.

[wiredfool]

@cgohlke Works for me. Thanks for building for us.

[wiredfool]

Tagged, OSX and Linux wheels built, but without the PR for libpng. (which is odd, unless it's a dependency of one of the dependencies, because it's not required for the library)

[keikoro]

Having never uploaded anything to PyPI myself before, I'm not really familiar with the jargon (also, it looks like some messages were deleted from this thread?) and now I'm a little confused about the version change vs. the new version of Pillow not (yet) being available on PyPI. Maybe one of you could help?

[wiredfool]

@cgohlke I assume you fixed up the typo'd version number manually?

I'm rebuilding the other wheels now, and will upload when they're done.

[hugovk]

@kerstin The new Pillow 4.0.0 release is in progress now, and will be up on PyPI later today.

[keikoro]

@hugovk Thank you!

[wiredfool]

Everything but the source .zip was uploaded to pypi, that one is failing with: HTTPError: 400 Client Error: Duplicate file upload detected. I don't see it in the list, and I'm not seeing an MD5 list anywhere.

At any rate, we're released.

[hugovk]

Had you already uploaded a source zip before fixing the typo?

I believe you cannot upload a filename that's already been uploaded, even if it was deleted: https://github.com/pypa/packaging-problems/issues/74

[aclark4life]

@hugovk Sounds familiar, maybe we need a 4.0.1 ?

[wiredfool]

Nope, it failed on the first twine upload.

[cgohlke]

Everything but the source .zip was uploaded to pypi, that one is failing with: HTTPError: 400 Client > Error: Duplicate file upload detected. I don't see it in the list, and I'm not seeing an MD5 list anywhere.

That might be on purpose. There is already one source distribution, Pillow-4.0.0.tar.gz.
See https://www.python.org/dev/peps/pep-0527/#limiting-number-of-sdists-per-release

[aclark4life]

Maybe @dstufft knows if that's been implemented yet. If so, then I guess we should use only zip from now on?

[dstufft]

It is implemented, you have to pick either a single .tar.gz or a single .zip. You are only allowed one. In general .tar.gz should probably be preferred unless there is a specific reason to use .zip, but it's up to each individual project.

[wiredfool]

We've been uploading both because different packagers have requested source dists in their native compression -- .tar.gz for unix and .zip for windows.

[aclark4life]

I've spent zero time thinking about this but at a glance it seems like an odd restriction that's going to cause our project trouble for dubiously "good" reasons… not to mention the ensuing bikeshed discussion: how about bzip2 instead of gzip? And so on…

[dstufft]

The PEP already dealt with the bikeshed for the most part, you can't pick bzip2 or xz or anything but .tar.gz and .zip. Everything else requires a (different) optional C library but those two both only rely on zlib. In addition those two were the only ones that had any real amount of usage. The only thing left to do on a per-project basis is pick .zip or .tar.gz, and the recommendation is to pick .tar.gz unless you have a compelling reason to pick something different (in the latest versions of Python will always default to .tar.gz.

The benefit of having a single sdist is it completely eliminates a class of problems that relate to what happens if you have two sdists with different contents.

[aclark4life]

@dstufft tar.gz it is then for us, I guess, unless anyone objects. Thanks

[hugovk]

@dstufft It's not very clear that "HTTPError: 400 Client Error: Duplicate file upload detected" in this case means "Duplicate source file upload detected, ignoring .zip or .tar.gz extensions."

Perhaps a clearer message and/or adding "See PEP 527" would help others in the future. Thanks!

[dstufft]

Yea, Warehouse has a better error message, I just hadn't done so on legacy PyPI yet.

[radarhere]

Is this right to be closed then?

[joshma]

Just my 2c, but given the non-semver versioning it'd be really useful for external consumers of the library if CHANGES.rst highlighted backwards incompatible changes. I don't mind reading through, but a lot of the changes don't mean much to me and it's hard to tell if upgrading to 4.0 will break things terribly.

(On a related note, if semver were respected, I would just stay within minor patches and upgrade major versions less frequently, so it'd be less of a problem.)

[wiredfool]

Significant changes are noted in the release notes: http://pillow.readthedocs.io/en/4.0.x/releasenotes/4.0.0.html

[aclark4life]

@joshma Where do you see "non-semver" (aside from ad hoc bump to 4)? If I understand correctly, we mostly adhere to "semver".

[joshma]

@wiredfool thanks for pointing me to those, didn't realize there were two sources

@aclark4life Yeah, I think I picked the wrong term there, you're right. A few pieces around versioning:

• I mostly thought 4.0 didn't have any backwards-incompatible things and was an arbitrary bump, but the changelog as linked makes it much clearer. The initial comments on this thread gave me the impression things were being arbitrarily bumped.
• 3.1.2 was published to address CVE-2016-3076, but 3.0, 2.7, and 2.8 weren't bumped. Can you clarify which older versions are expected to get security fixes? (I understand you might not have resources to fix all of them, just wondering which ones are being supported.)
• Similarly, are other bugfixes backported to older lines? I guess where "semver" came in was that minor fixes are being bundled in with larger backwards-incompatible pieces, but it's true that it's not a violation of semver as I understand it.

[wiredfool]

Generally:

• Point releases happen for two reasons -- either due to a regression on a scheduled release, or a security issue.
• CVE level security issues are generally handled with a point release, but sometimes the scheduling is such that they get rolled into a scheduled release. If that's the case, we will do a source only point release for the previous version, as much to provide a clean patch for distributors as anything else. We do coordinate with distros on CVE releases.
• We don't backport fixes.
• We try to not break existing code. Sometimes we do, accidentally, and sometimes it's pretty bad. (Tiff Metadata comes to mind. There was never a point regression fix on that because it took me 3 months to get the fix done.)