Emscripten: Remove support for python2

[sbc100]

We have a fair bit of complexity in the code in order to maintain python2 and python3 support at the same time. It also doubles our testing requirments (in fact we don't test both but in theory we should).

It would be great to drop support for one or the other and python2 seems like the logical choice.

Do we have any users who can't install python3 in order to use the toolchain?

[rth]

A typical thing to do in such cases could be to make a release and in the release notes indicate that in the following version support for e.g. Python 2.7 will be dropped and only Python >3.5 (or a version to be determined) will be supported. That would leave more time for users feedback. Though I'm not sure if this would be compatible with the way releases work in emscripten..

Another possibility could be to send an email to the mailing list (or some other channel used for announcements) to see if there is much opposition, before actually implementing it.

(Personally, I'm all for Py3 only).

[cclauss]

I agree with the Python porting best practices that it is better to have a single codebase that supports _both_ Python 2 and Python 3 at least for a little while to support A/B testing on a single codebase. https://docs.python.org/3/howto/pyporting.html

[sbc100]

We've been supporting both python2 and python3 several years now. Presumably that is long enough to qualify as "a little while"?

[cclauss]

Surprising because these are all syntax errors in Python 3 found and fixed in the last month.
https://github.com/emscripten-core/emscripten/pulls?q=is%3Apr+author%3Acclauss+is%3Aclosed

[cclauss]

flake8 testing of https://github.com/emscripten-core/emscripten on Python 3.7.1

$ __flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics__

./tools/shared.py:3185:22: F821 undefined name 'unicode'
    if isinstance(s, unicode):
                     ^
./tests/test_core.py:7047:29: F821 undefined name 'unicode'
      elif isinstance(data, unicode):
                            ^
./tests/test_other.py:4154:19: E999 SyntaxError: invalid syntax
          for async in [0, 1]:
                  ^
./tests/test_browser.py:839:17: E999 SyntaxError: invalid syntax
        for async in [
                ^
./tests/test_sockets.py:195:24: F821 undefined name 'unichr'
        message += str(unichr(ord('a') + (i % 26)))
                       ^
./tests/poppler/poppler/gen-unicode-tables.py:28:10: F821 undefined name 'xrange'
for u in xrange(0, UNICODE_LAST_CHAR_PART1):
         ^
./tests/poppler/poppler/gen-unicode-tables.py:31:55: F821 undefined name 'unichr'
  norm = tuple(map(ord, unicodedata.normalize("NFKD", unichr(u))))
                                                      ^
2     E999 SyntaxError: invalid syntax
5     F821 undefined name 'xrange'
7

That is confusing. We do all our testing on python3 using the version that is present in ubuntu bionic so we do know tat least the the tests all pass.

The async issue is particularly strange since one would expect that to show up every time test_other.py for example is imported. Is that a very recent change? The version of python3 on my desktop accepts async as a keyword perfectly happily.

Anyway, we might be flake8 clean in python3 but we have been running all our tests under it for a long time.

[ids1024]

Maintaining Python 2 support seems mainly important for libraries (rather than executable scripts) that may need to integrate with code that only runs under Python 2. Is that an important use case for anything in Emscripten?

Though personally I think it's about time for libraries to drop Python 2 support as well, given it will no longer be maintained in a few months:

Being the last of the 2.x series, 2.7 will have an extended period of maintenance. Specifically, 2.7 will receive bugfix support until January 1, 2020. After the last release, 2.7 will receive no support.

https://www.python.org/dev/peps/pep-0373/#release-schedule

[cclauss]

I am a MASSIVE fan of dropping support for legacy Python. However I would like to get all the Python 3 syntax errors cleared in a codebase that can be A/B tested _before_ we drop support for Python 2. Just give me a pull request or two to finish the work.

[sbc100]

Looks like bionic includes python3.6 (https://packages.ubuntu.com/bionic/python3) but the async keyword was added in python3.7 which was released just one month ago.

I'm not in super hurry to make this change. For sure we want to be flake8 clean first. Happy to wait for your PRs.

[cclauss]

Now I see #9209... That is super exciting. Let's allow that to land and let the paint dry for a few weeks before pulling the plug on legacy Python. It fixes the __async__ keyword.

Python 3.7 shipped more than a year ago according to https://devguide.python.org/#branchstatus It was 3.7__.4__ that was recently released. Python 3.8 is in late beta and is due in two months.

[dschuff]

Update: I've updated the emscripten-releases build infrastructure to be ready for python3 but haven't thrown the switch yet. Probably makes sense do that whenever we have a concrete plan to stop support for python2.

[kripken]

Why not flip the switch for emscripten-releases now? Hopefully most users are on python3, so it may be better testing.

[dschuff]

Emscripten CI runs python3 IIRC so we have coverage of that already, right?
Also I'm using python3 locally for the infra code too, so I'm not too worried
about it.

[cclauss]

42 days until Python 2 end of life.

[ledyba-z]

Congratulations!
Python2 reaches the End of Life!

Search · python2
https://github.com/emscripten-core/emscripten/search?q=python2&unscoped_q=python2

[kripken]

I'd very much like to remove python2 support in emscripten. After talking to people, some things that may block that:

• MacOS has Python2 built in, but not Python3. Python2 may be removed at some point. But meanwhile it is an option to use that instead of downloading one, which we've preferred, but should probably change.
• The emsdk ships python 2.7 for windows. I'm not sure if there's a reason for that, or if we've just not gotten around to replacing it with 3.

[sbc100]

emsdk fix for windows is here: https://github.com/emscripten-core/emsdk/pull/349

[kripken]

Is the only blocker here to start bundling python on MacOS?

[sbc100]

I'm not sure. I think recent version of macOS might even ship python3.

In any case I'm not sure bundling is the right way to go. Most mac developers would probably prefer to simply brew install python3 wouldn't they?

But yes, I think its only macOS we need to worry about at this time.

Maybe we can do this in stages:

1. Instigate the current state of macOS and come up with path forward.
2. Make an announcement on the mailing list.
3. Start producing a warning when running under python2.
4. Turn the warning into an error that can be disabled (linking to this bug).
5. Finally, remove support.

[cclauss]

https://apple.stackexchange.com/questions/376077/is-usr-bin-python3-provided-with-macos-catalina

[sbc100]

Hmm.. seems like a reasonable approach. We can probably assume that emscripten developers are OK installing dev tools from apple?

So for macOS catalina I think we are good with just assuming we can use the result of which python3.

That just leaves what to do about older versions of macOS. Do you think its reasonable to ask people to setup python3 themselves (e.g. via brew install python3) in this case. I think that bundling our own copy would be perhaps more annoying because most mac users probably want or need another copy of python3 on the system.

@juj, I remember last time we tried to make this move you were concerned about no longer being able to use the python 2 that ships as /usr/bin/python on all versions of macOS. Do you still have that concern?

[juj]

If emsdk starts to bundle python 3 for macOS, then we are all ok to migrate to python 3. We do not have any specific need for python 2, except that we need the tools to be ready out of the box.

MacOS unfortunately still does not ship python 3, like was linked above, Catalina has an automated prompt to go install it (btw that is the behavior that Windows 10 also has).

[cclauss]

It would be good to have enscripten join the club: https://python3statement.org

[sbc100]

If emsdk starts to bundle python 3 for macOS, then we are all ok to migrate to python 3. We do not have any specific need for python 2, except that we need the tools to be ready out of the box.

But do you think its better to bundle our own copy of python3 or allow the developers to follow the prompt and end up using the official version. Is there not some value is not bundling an additional copy?

(Looking into the bundling option now).

MacOS unfortunately still does not ship python 3, like was linked above, Catalina has an automated prompt to go install it (btw that is the behavior that Windows 10 also has).

[ricejasonf]

On 1.39.18 I get the following error:
/usr/bin/env: 'python3': No such file or directory

The documentation does not appear to mention that python3 is required.

[cclauss]

Python 2 died on 1/1/2020.

[kripken]

@ricejasonf What OS is that on?

In general installing Python 3 is the right solution, and the tools look for it, since as @cclauss said Python 2 is done. However, you can also run in Python 2 for now, by doing python2 emcc.py for example.

[ricejasonf]

I was using ubuntu:eoan in a Docker container.

Say, isn't Java dead too? Maybe we could get rid of that dependency. :stuck_out_tongue:

[sbc100]

If you run emcc directly, which is a shell script, it should pick whichever version of python you have installed, and use that. We currently support both 2 and 3.

[ricejasonf]

Sorry. This is occurring within embuilder.py.

Step 14/16 : RUN ./embuilder.py build ALL
 ---> Running in 7b7e83a387e6
/usr/bin/env: 'python3': No such file or directory

I'm not against installing python3. Building Binaryen requires it also.
I was just pointing out that the documentation explicitly states python 2.7.

[cclauss]

Please submit a PR to change the documentation because Python 2 is dead.

[sbc100]

We still support python2 for now. The reason is that we want to run out-of-the-box on macOS which has system python2 but not python3.

[sbc100]

We are working on fully removing python2 support as soon as we can figure out a way to bundle python3 on macOS as part of emsdk.

[sbc100]

Sorry. This is occurring within embuilder.py.

Step 14/16 : RUN ./embuilder.py build ALL
 ---> Running in 7b7e83a387e6
/usr/bin/env: 'python3': No such file or directory

I'm not against installing python3. Building Binaryen requires it also.
I was just pointing out that the documentation explicitly states python 2.7.

If you run "./embuilder" it will pick the correct version of python2. It looks like you can change your docker file to just drop the .py part. Or better still install python3 :)

[sbc100]

Once https://github.com/emscripten-core/emsdk/pull/561 lands I think we can start the process of removing python2.

@juj will you be able to use the bundled version of python macOS relatively easily?

[sbc100]

I was thinking it would be convenient to hitch the python2 removal the fastcomp removal (#11319) meaning that emscripten starting with version 2.0.0 will require python3.

[kripken]

What would the benefit be to bundling them together? I guess it might feel nicer to use all modern stuff in 2.0.0, and I can't think of a real risk to doing both at once (it's not like we'll bisect on these things).

[cclauss]

Given that Python 2 has been dead for 218 days, let’s just drop that now and wait on fastcomp.

[sbc100]

Yes, I think I just like the idea of 2.0.0 being the clean slate release.

Fastcomp removal is basically happening now (ish) .. so no need to wait :)