Hack: Switch to Python 3

[paride]

Python 3 is now 10 years old, certainly mature enough to be the first choice for new projects. In the case of Hack switching seems rather easy:

• call python3 instead of python in the scripts;
• use pip3 instead of pip;
• drop the from future ... imports in the python scripts.

This could really be sufficient.

[anthrotype]

yes!
fontmake and all its dependencies are py2.py3 compatible. If you see any differences in the fonts generated with py2 and py3 than that's a bug and I'll gladly fix it.

[chrissimpkins]

Are you testing against both Py2 and Py3 in fontTools and fontmake (and by extension all of your dependencies...) Cosimo?

[anthrotype]

of course we are

[chrissimpkins]

@anthrotype OK thanks. Just confirming. I am building releases with pip3 installed build tools / Py3 so I can confirm that we have no problems with the current build on Py3.

[chrissimpkins]

I guess the drawback here is that we are asking users who don't necessarily have Python3 installed to do so in order to build Hack which is an unnecessary dependency for this project because it builds without issues under Python2 and Py2 remains the default interpreter installed on those platforms that install Python by default (including most Linux distros and macOS). Is there a project-specific benefit to Py3 here or is this a suggestion to participate in the movement to push Py3 as the default interpreter everywhere. The root of this problem is that python / pip are not mapped to the python3 interpreter and pip3 out there in the wild, not that we don't support Py3.

[chrissimpkins]

And understand I am in complete agreement with your opinion in general. Simply playing devil's advocate.

[anthrotype]

I think you're mixing up font users and devolopers.
Normally, "users" aren't expected to build anything (including fonts) from source, they are intrerested in using the end product and they can get it as part of the OS or with some package manager.

"developers" on the other end can easily install python 3 as a build requirement.

Python 3 is just python; python 2 is _legacy_ python. In a couple of years upstream will drop support and it will no longer receive bugfixes.

All linux distros that I came across, ship some version of python3 alongside python2.
I'm not impressed by the fact that the generic python executable is not a symlink to python3 everywhere (yet). This is to avoid breaking so many scripts that are written for python2 but don't say it explicitly in their shebang.

It's increasingly becoming a burden for all of us that maintain python libraries to keep writing code that is compatible with both 2 and 3.

The Hack fonts project is not a "library"; it doesn't need to support any possible python runtime. You don't get a different output if you build the fonts with PyPy, do you?

[chrissimpkins]

"developers" on the other end can easily install python 3 as a build requirement.

Agree. But this isn't a pure Python source project, though it does use Python source as part of its build tooling. It is a typeface project built around UFO source files. There may be (I hope that there are...) typeface developers (broadly defined, let's use the name tinkerer since there is reasonable Python knowledge in much of the typeface development community given that many of the typeface tools are built with Python) out there who do not develop in Python, do not need Python3, don't know about the pip/pip3 issue yet may want to modify and build fonts from the source here, whether for upstream contributions or to maintain their own downstream source with any modifications to the design that they'd like to make using whatever editor they would like to use. We want to support this IMO. My goal has been to develop a scripted build strategy that is relatively transparent across all developers/users for as many types of build requirements as we can support so that no one is forced to build via a GUI font editor and can use any font editor that supports UFO source files. We don't have a good approach for Windows users at the moment. They have to install a number of significant dependencies to build the fonts. We can only do so much :) Our entire alternate glyph build strategy through the alt-hack repository is designed around empowering users to build their own modified fonts on the command line with the build tooling workflow that we've developed here.

I am not arguing against you about the benefit of the transition to Python 3 in general. It must happen out there and I would like to help support the transition here. I suppose I am arguing against a hard change to python3 use or a failed build if that is the suggestion. Unfortunately Py3 is just not a broad reality in the wild at this stage so this means that we introduce a new dependency to the project and potentially create obstacles to development of the typeface that are not necessary for the advancement of the typeface. When/if Cosimo deprecates and drops support for Py2 in the compiler and fontTools, we will promptly make an enforced move to Py3 for all :) If there is to be an impetus for this type of move across projects that do not need Py3 features, it will come from the libraries not the users of the libraries and I suspect that the libraries are waiting for the Python3 penetration to broaden before they are prepared to make such a move. It's a problem and it is not resolving quickly.

If it would work for both of you, I think that we can work around the issue by detecting python3 and pip3 in both the dependency install scripting and the execution of Python scripts during the build process, then default to Py3 when they are installed. How does that sound?

[anthrotype]

Sure, but then the burden will be on you to support building the Hack fonts with _both_ py2 and py3. You'd have to test the output of both and make sure they produce identical output.

[chrissimpkins]

You just told me that you were testing against both ;)

[chrissimpkins]

And I am assuming that your versions follow SemVer. If not, we need to know this.

[anthrotype]

We are testing on both (and yes, we do follow semver). However coverage is not 100% everywhere, I'm afraid. And we do sometimes find small differences, mostly to do with the fact that in python3 unsorted collections like sets or dicts use a random hash seed that changes upon every run; if the code was originally written for python 2 (which is usually the case with old code) and it doesn't take that into account (e.g. when doing loops over unsorted collections, or when converting them to sequences like tuples or lists without a deterministic sorting), then you may get different (though functionally equivalent) output when running with py2 or py3.
I'm not aware of any such remaining differences myself, but we did come across a few of them.

[anthrotype]

This is mostly a concern for the libraries that we inherited, so to speak. All the new code that I merge is (should be) accompanied by its tests, and coverage cannot (ought not to) regress.

[chrissimpkins]

Why don't we set up a branch that tracks main and builds with Python3 only. We can help test that issue if that is helpful. It would be useful to know what we might expect to see as differences so that we know what to diff.

[anthrotype]

It would be useful to know what we might expect to see as differences

if I knew them, I would fix them :)
I'd watch for things that end up with a different order. This is mostly about reproducible builds rather than functionality, as sometimes the order doesn't matter, until it does.

[chrissimpkins]

If the Py3 issue is mandatory for Paride, he can use this branch in his builds as well. Might serve both issues.

[chrissimpkins]

I'd watch for things that end up with a different order. This is mostly about reproducible builds rather than functionality, as sometimes the order doesn't matter, until it does.

OK, sounds good. Would you be willing to help set up some scripts that we can use to test this branch against things that might be of importance? I'm not certain where to start. We can hook this in to our CI testing.

[chrissimpkins]

We do not currently test against glyph order in the CI tests so we will not capture any of this with our current testing approach.

[anthrotype]

I would start by making the name of the python executable in your shell scripts or makefile overrideable via environment variable, then set up a new Travis job that has PYTHON_EXE=python3 in its env or similar. You can call pip as $PYTHON_EXE -m pip instead of pip3 (it's more bullet-proof).

Then we figure out a way to diff the two outputs. Dumping to ttx and running the diff command should exit 0 (well the timestamp and checksum, but you can use SOURCE_DATE_EPOCH).

[chrissimpkins]

👍 The ttx dumps are massive. Possible to narrow down to a specific set of tables of concern?

[anthrotype]

no

[anthrotype]

you can diff the binary if you like but that'll only tell you YES or NO, not HOW different

[chrissimpkins]

ok. let's see what we get with that approach and can modify from there. I dumped ttx a few weeks ago to investigate a hinting issue and it was tens of thousands of lines with unpredictable changes in assembly for the hints, the changes in the timestamps, version number string, etc. I don't know that there will be an automated approach that tests "OK" vs. "Not OK"

[anthrotype]

the diff between py2 and py3 builds from the same commit should _not_ be massive, I actually expect no differences. Any differences is indeed a bug.

[chrissimpkins]

OK, that is helpful. Will open a new issue for it since this diverts from Paride's original report.

[paride]

@chrissimpkins it is totally not mandatory, everything works fine with Python 2. At some point the switch has to happen, and I think it's worth supporting the movement towards Python 3, but from my point of view as a downstream packager, that's it.

[chrissimpkins]

Ubuntu is taking a big step in that direction. Is Debian making the move to Py3 as the default as well?

[paride]

I don't think the switch to Py3 as the default is imminent, but:

Debian currently supports two Python stacks, one for Python 3 and one for Python 2. The long term goal for Debian is to reduce this to one stack, dropping the Python 2 stack at some time.

Packages in Debian should use Python 3 if Python 3 is supported. New packages should use Python 3 from the initial upload, new upstream versions for existing packages should use Python 3 if the new upstream version supports it.

https://www.debian.org/doc/packaging-manuals/python-policy/ch-python3.html

[chrissimpkins]

OK let's find an approach that works for you to support it. Will do the testing rec'd by @anthrotype so that we can confirm that the build does not differ between py2/3 and make the approach available.

[chrissimpkins]

I can confirm that there is no difference between Py2 and Py3 builds. See https://github.com/source-foundry/Hack/issues/398

Paride, is it helpful to have a Python 3 specific branch or does conversion to default to Py3 when installed / available address your report here?

[paride]

No need for a specific branch.

[chrissimpkins]

I believe that default macOS installs do not include pip which is part of the dependencies that we define for the build tooling installation. This means that macOS users cannot build without a Python related install with current make scripting and if they are going to install the pip application, they really should be installing this as a Python3 associated release based upon your points in the OP. Windows users always must install some version of Python and new installs should be Py3.

This may not be as significant of a barrier to builds as I thought. I was hoping to avoid creating new impediments to builds from source but they appear to already exist in some fashion on most platforms.

Let me give this some more thought. Seems like less downside to full transition than I initially thought.

[chrissimpkins]

Does Debian Py2 come with pip installed or does this require a manual install by users?

[paride]

It does not come with pip installed by default.

[chrissimpkins]

Yeah I suspect this is the case nearly everywhere. This is likely not an issue. Will revisit it after we get through an assessment of pinning versions of Python package dependencies that are currently rolling in other open threads.

[chrissimpkins]

This just happened today with Homebrew Python :)

[chrissimpkins]

You convinced me. Adding on as part of v4.000 work. Is anyone aware of a platform where python3 is not linked to the Python 3 interpreter (i.e., that it is python and python3 is not available by default)? I checked with the changes in Homebrew Python this week (see https://github.com/source-foundry/Hack/issues/397#issuecomment-370069144) and a python3 executable path remains present. Is there any documentation or set of guidelines out there about how long support through a python3 executable should be present in the wild?

Applies to pip3 vs. pip as well...

[chrissimpkins]

This is now supported in the build-scripting branch using pipenv venv's with version pinned Python build dependencies. The venv is built and defined with frozen Python interpreter and Python build dependency versions in the new build-pipenv.sh script. All future builds with the default make build approach will be defined with the Python 3 interpreter and we will be pinning the minor version of Python 3 using our Pipfile and pipenv execution of the Python executables.

More work needs to be done to address the request to use PATH defined, system installed versions of Python 3 on Linux distros (and elsewhere). As of now, the plan is to support this through different make targets and a separate set of shell scripts. Feel free to weigh in if you have any thoughts or submit a PR against the build-scripting branch if you would like to contribute to the development of this new build approach.

These changes will be part of the v4.000 release of Hack.

[chrissimpkins]

I believe that we have addressed all requests/suggestions in this thread. The work is merged into dev branch and will be included in the v4.000 release of Hack.