Marshmallow: Drop Python 2 support

[sloria]

marshmallow 3 will be the last major version to support Python 2. Per discussion below, Python 2 support will be dropped in marshmallow 3.

[rooterkyberian]

Why not drop Python 3 in marshmallow 3?

• Python 2 EoL is in 8 months, marshmallow 2 was being supported for 4 years and counting
• marshmallow 3 already contains multiple breaking changes
• supporting old Python version costs development time
  
  
  
  • duh
  
  
  • there are still open bugs for marshmallow 2 concerning Python 2-3 compatibility that there is low interest in fixing (#797, #493)
  
  
• ~python suffers from relatively high function call overhead, and that is how compat module works - not arguing it would be a game changer, but should provide measurable benefits~
  
  
  
  • ~performance is arguably one of biggest drawbacks of marshmallow and this seems like an easy win~
  
  
• switch to Python 3-only should be easy enough - AFAIK it just a matter of removal of compat module
• opens doors for more non-breaking improvements as part of 3.x line (which are NOT required to be a part of 3.0.0 release); for example - type annotations

---

Initial tests after Python 2 compat removal #1204 show performance improvement that is well within margin of error using marshmallow benchmark. I.e. no actual improvement, so my statement about "easy win" has proven to be false.

[deckar01]

Although the most recent python developer survey indicates that developers use python 3 more than python 2 80% of the time, python 2 is still used a lot. Assuming the developers on average use their less-used python version in 25% of their active projects, the stats would come out to python 3 65%, python 2 35%. I suspect the percentage of actual development activity would be even more skewed towards python 2 considering how many python 2 projects are still being actively maintained.

https://www.jetbrains.com/research/python-developers-survey-2018/#python-3-adoption

[rooterkyberian]

If someone wants to use outdated software they can keep on doing that (Python interpreter, older marshmallow) - I'm just arguing that we shouldn't hold the project back for minority that is unlikely to use the newer version of a package that brings breaking changes on its own.

Which reminds me:
after Python 3 upgrade a python_requires should be added to setup.py (https://packaging.python.org/guides/distributing-packages-using-setuptools/?highlight=python_requires#python-requires)
this will ensure that Python 2.7 users will be unaffected even if they don't pin marshmallow version.

[hugovk]

Here's the pip installs for marshmallow from PyPI for March 2019:

| category | percent | downloads |
|----------|--------:|----------:|
| 3.6 | 38.63% | 362,726 |
| 2.7 | 34.13% | 320,445 |
| 3.7 | 17.88% | 167,918 |
| 3.5 | 7.68% | 72,111 |
| null | 0.86% | 8,090 |
| 3.4 | 0.78% | 7,286 |
| 2.6 | 0.02% | 230 |
| 3.8 | 0.01% | 87 |
| 3.3 | 0.01% | 75 |
| Total | | 938,968 |

Source: pypistats python_minor marshmallow --last-month # pip install pypistats

These projects pledge to drop Python 2 support in or before 2020: https://python3statement.org

Which reminds me:
after Python 3 upgrade a python_requires should be added to setup.py (packaging.python.org/guides/distributing-packages-using-setuptools/?highlight=python_requires#python-requires)
this will ensure that Python 2.7 users will be unaffected even if they don't pin marshmallow version.

Good idea, and python_requires is also useful now, please see PR https://github.com/marshmallow-code/marshmallow/pull/1194.

[sloria]

I'm not opposed to dropping Python 2. 2020 is getting near...

Besides, I am planning on supporting marshmallow 2 (bug fixes only) at least until 2020.

@lafrech @deckar01 Any objections?

[lafrech]

I tend to agree with @rooterkyberian. People stuck on PY2 would just be stuck on MA2 as well, which is not the end of the world, especially if it is maintained for a while.

If they just didn't bother porting their own code to PY3 yet, well... it's about time. It shouldn't be that difficult. And perhaps easier than porting to MA3. If they're stuck there due to third-party dependencies, then perhaps they should be worried about those dependencies.

It would suck for people who've been developing with MA3 on PY2, but I doubt there's much of those. People develop on a beta MA version when starting a project, and then they shouldn't be using PY2. Unless someone with a PY2/MA2 project moved to MA3 to benefit from a feature or fix hoping it would be released as stable soon and with few changes. We've been pushing the use of MA3 in its beta state for a while, as it is close to stable. What would be interesting to know is the number/proportion of users of marshmallow 3 on Python 2 but I don't think PyPI gives access to this.

Overall, I'm in favour of dropping PY2 in MA3.

We'll probably be living with MA3 for a while, and it would be nice to benefit from PY3.x improvements and get the burden of MA2 out of the way.

Ultimately, a most compelling statistic that speaks for itself:

| thumb direction in reaction to OP | percent | number |
|----------|--------:|----------:|
| up | 100.00% | 4 |
| down | 0.00% | 0 |

[deckar01]

In the last 30 days there have been 59,862 downloads of marshmallow 3.x from python 2 and 154,161 from python 3. That means ~28% of new marshmallow 3 installs are coming from python 2. I would not advocate dropping python 2 support this far into the release candidate process.

Marshmallow 3: Python 2 Usage
Marshmallow 3: Python 3 Usage

The queries I ran should be public, but it requires logging in with a Google account. If you have any trouble accessing them let me know.

[lafrech]

Wow, I find this quite surprising.

Could it be a side-effect of CI tests on some libs running all configurations, including PY2/MA3 (e.g. marshmallow-sqlalchemy)? I'm not sure how caching affects those stats. I suppose even when using a cache there must be a request on PyPI to see if a new version was released.

[deckar01]

One way to control for CI downloads would be to ignore Linux downloads and assume downloads from Windows and MacOS are a representative sample of real developer downloads. That would put python 2 at 708 downloads, and python 3 at 3,719 (in the last 30 days). So, ~12% of Windows downloads and ~19% of MacOS downloads are coming from python 2 (~16% combined).

Marshmallow 3: Python 2 Usage by OS
Marshmallow 3: Python 3 Usage by OS

[sloria]

One possible path would be to release marshmallow 4 alpha that drops Python 2, shortly after the 3.0 final release.

[lafrech]

@deckar01 has a point, that would be an important change for a RC phase:

I would not advocate dropping python 2 support this far into the release candidate process.

I would have thought no one used PY2/MA3 but the stats prove me wrong.

One possible path would be to release marshmallow 4 alpha that drops Python 2, shortly after the 3.0 final release.

A few months after MA3 will be released, PY2 will be officially unmaintained. We can then remove PY2 support from MA3 in a minor release, like we did for PY3.3 (https://github.com/marshmallow-code/marshmallow/pull/987).

https://pythonclock.org/

[sloria]

A few months after MA3 will be released, PY2 will be officially unmaintained. We can then remove PY2 support from MA3 in a minor release, like we did for PY3.3 (#987).

Dropping a major Python version in a minor marshmallow release might be too drastic. I'd be more comfortable dropping in the RC stage than the stable line. Do you disagree?

[sloria]

Following up on this. I see two viable options:

🎉 Drop Python 2 for the marshmallow 3 final release.
❤️ Support Python 2 in the marshmallow 3 line. Release a 4.0a without Python 2 support.

Cast your votes.

cc @lafrech @deckar01

[deckar01]

I do like the idea of major releases more often, but I think dropping python 2 in v3 will be fairly painless. If this ends up being a problem for anyone I would recommend forking v3 and sharing any bug fixes you port in a community release for the other holdouts.

[lafrech]

I think dropping python 2 in v3 will be fairly painless

So do I. As I wrote above, I don't see who would be using MA3 on a PY2 project. PY2 means old project, while people should only pick MA3 beta for a new project.

Since we intend to support MA3 for a while, it would be a pity to release it with the burden of a Python version planned to be dropped a few months later. MA3 is not ready yet and the more time passes, the more this is true. (https://pythonclock.org/)

[sloria]

It's decided, then. marshmallow 3 will be Python 3-only. 🎉

After https://github.com/marshmallow-code/marshmallow/pull/1204 is merged, I'll update the marshmallow-code libs so that they don't test marshmallow 3 in Python 2 environments. Then I'll cut another RC release.

[hugovk]

Consider signing https://python3statement.org/#sections60-sign too!

[rooterkyberian]

I would suggest removing python-2 topic from github repository description as well

[ThiefMaster]

Sorry, but WTF?! :angry:

People with Python 2 projects already switched to 3.0rcX (or started with it) since this was recommended in various places. I don't remember where but some place, probably in the docs, actually mentioned that it's better to start using MM3 and fix the small things that might break during the RCs instead of starting with MM2 and then having to change many things.

Now this decision means we'll be stuck for months with a release candidate, and since there's no way to do a patch release on top of a RC, there's absolutely no hope for bug fixes no matter how badly something might be broken in rc5 (maybe we are lucky and there are no such bugs, so far everything seems fine!) except by putting a fork on PyPI (which I REALLY want to avoid because of how much of a mess it would be).

And no, switching a large project to Python 3 doesn't happen within a few weeks, and not everyone can drop their work on everything else just to move to Python 3 (as much as I would love to do this).

Many large Linux distros (CentOS 7 and Debian 9 for example) don't even have Python 3.6 as their easily available Python 3 version yet, so if we were to switch to Python 3 right now our users would have to bother with SCL, backports, etc. or we'd have to support a Python version that doesn't even have nice things like f-strings...

---

So why not keep Python 3 in MM 3.0, and release the stable MM 3.0 (2.7 + 3.x) and MM 4.0 (3.x only) at the same time? MM 3.0 could then receive bugfixes (3.0.x), while MM 4 would get both bugfixes and new features.

[deckar01]

If this ends up being a problem for anyone I would recommend forking v3 and sharing any bug fixes you port in a community release for the other holdouts.

https://github.com/marshmallow-code/marshmallow/issues/1120#issuecomment-488417854

[ThiefMaster]

The problem with forking is that this is a mess - people WILL put multiple forks on PyPI for sure.
It already happens for Python 3 stdlib backports to Python 2, so usually you have to check their repos to see which one is the most up to date one...

[deckar01]

I would be willing to host a stable fork. I'm not sure if a link here would be enough for discovery or if we would need to mention it in the readme.

[ThiefMaster]

But then again, what's the benefit of having multiple packages in PyPI instead of just doing 3.0 and 4.0?

[rooterkyberian]

@ThiefMaster please read thru the whole discussion. Like mentioned already: developer time is not free, even if it seems it that way, for us, end user of this opensource project. I'm also not happy of marshmallow 3 taking so long - already half a year passed from first rc release, but I don't think only right we have here is offer our work to make it happen faster or just thank the other contributors for their hard work.

You have mentioned also a "con" of problematic Python 3.6 install on some distros, which I would like to refute:

• Python 3.6 is not required, 3.5 is
• installation of Python 3.6 on CentOS 7:

sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm
sudo yum update
sudo yum install -y python36u python36u-libs python36u-devel python36u-pip

• installation of Python 3.5 on the Debian 9 is just apt-get install python3, 3.6 is just "as complex" as under CentOS 7

Now the difference with these solutions and the one you are proposing (keeping Python 2 support) is that for above end users which make decision have to pay the price for their decision instead of voluntary developers.

[lafrech]

@ThiefMaster I'm sorry to hear that. As you can see, there's been a lot of discussion before taking this decision.

The rationale was that PY2 projects are doomed in the short term, as PY2 will be terminated a few months after MA3 is out.

some place, probably in the docs, actually mentioned that it's better to start using MM3 and fix the small things that might break during the RCs instead of starting with MM2 and then having to change many things.

This is correct. I mentioned that above. MA3 has been advertised as recommended version for starting projects for a while. But so is PY3.

I'm still curious what project, recent enough to use MA3, would pick PY2. Not making a judgement, everyone has his constraints. Just curious what those constraints would be.

Many large Linux distros (CentOS 7 and Debian 9 for example) don't even have Python 3.6

I'm running Debian Stretch (currently stable, oldstable in a few months when Buster "is ready") with 3.5, which is supported by MA3.

So why not keep Python 3 in MM 3.0, and release the stable MM 3.0 (2.7 + 3.x) and MM 4.0 (3.x only) at the same time?

The issue with releasing many major releases is not only the need to maintain several versions, but also the added difficulty for all third party libs to support all major MA versions. (See https://github.com/marshmallow-code/marshmallow/issues/928#issuecomment-417087761 and below.)

[ThiefMaster]

I'm still curious what project, recent enough to use MA3, would pick PY2. Not making a judgement, everyone has his constraints. Just curious what those constraints would be.

https://github.com/indico/indico - it exists for more than 15 years now, but we only started using marshmallow somewhat recently. Without going too much off-topic, many of our users used a step-by-step guide to install it, so switching to Python 3 will be hard for them (some have little Linux administration knowledge.. and yes I know that this is bad for other reasons like security - but that's their problem in the end).

So we intend to do it once (require 2.7 -> require 3.6+). If we were to do it now, we'd probably need to support 3.5 as well, and it would be just as awful as in the past when 2.6 was still supported even though every developer wanted to use the new features from 2.7 but couldn't (like f-strings or type annotations).

[deckar01]

I took a quick look at indico. It should be pretty easy to switch to marshmallow 2. marshmallow-flask and flask-sqlalchemy support both versions. The only thing I saw that was marshmallow 3 only was Pluck and the same behavior is available in marshmallow 2 with a different syntax.

[ThiefMaster]

Wouldn't be great, not only because we only updated 2 months ago but also because the MM3 API is much more pleasant. I'll rather stay with rc5.

[deckar01]

It's still a rc, but marshmallow-code/marshmallow#1154 recommends using v3 for new stuff

Users starting a new project are encouraged to use v3

indico - it exists for more than 15 years now

[sloria]

@ThiefMaster I'm sorry this decision left you stuck on an RC release. But dropping Python 2 for the 3.x line was the best course. Given that we don't yet have any planned release timeline for the 4.x line, keeping Py2 in marshamallow 3 would mean we'd be stuck supporting Python 2 way past its EOL date of 2020-01-01.

The best I can offer is to keep a community fork of marshmallow 3 with Python 2 support under the marshmallow-code org. I wouldn't actively maintain it myself, but I'd be happy to merge PRs now and then. I would be liberal about giving commit privs (perhaps using the PR Hack as we've done with some of the other repos). Would you like to be a part of this?