Chapel: can we require Python 3 for 'make docs' and start_test?

[mppf]

Python 2 is reaching EOL and many of the Python libraries used in make docs have newer versions that are Python 3 only.

This is primarily an issue for chpldoc dependencies (like sphinx) but it would probably make sense to have the same requirement for the other tools with dependencies managed by chpl-venv (start_test, chplspell).

For example

https://github.com/chapel-lang/chapel/blob/7ba0e0fe50d122f673b3f87f16b0da772fb9f724/third-party/chpl-venv/chpldoc-requirements.txt#L1-L9

Sphinx 2.0.1, 2.1.2, and 2.2.0 have been released but require Python 3.

[lydia-duncan]

This is related to Cray/chapel-private#201

[bradcray]

I don't know how rhetorical this issue's question is meant to be, but it sounds great to me if we can get away with it.

[mppf]

Does anybody know of a reason we cannot make Python 3 a required dependency for start_test and chpldoc / make docs ?

[mppf]

FWIW NERSC just announced they will default to Python 3 starting in Jan 2020. I view that as evidence we might be able to require Python 3 for everything if we wanted to...

[ram-nad]

With latest 20.04 release Ubuntu has removed support for python2. Also because of unavailability of python command by default, none of the commands will work as expected (until we download python-is-python3 package which would make python command available as python3). I have tried to compile all of them with python3 and most of the commands have worked till now as expected.

[mppf]

I just upgraded to Ubuntu 20.04 and I see:

i   python-is-python2               - symlinks /usr/bin/python to the DEPRECATED

Nonetheless the relevant part for this issue is that there's no problem using python3 in these programs as far as Ubuntu 20 is concerned.

[ram-nad]

Yes, sir. But our code will not work without installing this package (or without installing python-is-python3). But as the code works well with python3, can we change the code to use python3 everywhere?

[mppf]

Ah, I see. This issue is about making start_test and make docs support scripts require Python 3, which would probably work just fine.

But changing all of the Python scripts to use Python 3 is still challenging as there are still platforms hanging around that only have Python 2 by default (like Mac OS X). But perhaps we could have some scripting / command that detects the best python (e.g. prefer python3, then try python2, then try python) and use that in the shebang.

[ghenriks]

Perhaps worth considering that Python on macOS is deprecated by Apple in 10.15 (Catalina) with the indication that it will be removed in future versions of macOS. So keeping Python 2 for macOS may be a bit pointless if users will need to install it from a 3rd party anyway, at which point it should be Python 3.

[mppf]

It is my understanding that the current best practice for the shebang for Python3 scripts is:

#!/usr/bin/env python3

(from https://docs.python.org/3/using/unix.html#miscellaneous and https://stackoverflow.com/questions/6908143/should-i-put-shebang-in-python-scripts-and-what-form-should-it-take ).

While it appears that #!/usr/bin/env python might be acceptable for scripts that work in both Python 2 and Python 3 (such as ours), this does not seem to be supported by a number of distributions. E.g. https://chapel.discourse.group/t/mason-doesnt-work-out-of-the-brew-box-on-linux/950/5

So, I think that along with this change, we will need to update the shebangs across all of our python scripts.

Note that util/printchplenv is already a shell script and it would be relatively straightforward for that to check for python3 / python / python2 e.g. if we feel the need to do so.

[mppf]

I have started to create some detection of python3 / python / python2 for core scripts in #16560 and a start at updating other shebangs in #16561

[lydia-duncan]

I've been poking at getting chpldoc specifically to be installed with python3. I'm in pretty early stages and looking at your #16560. One thing I'm noting so far is that Python 3 seems to prefer venv over the old virtualenv, though perhaps virtualenv is still supported

[mppf]

@lydia-duncan - if you want to look at my branches I suggest combining #16560 and #16561 in your experiments. As far as I know, those PRs combined will already cause chpldoc to build with python3.

[lydia-duncan]

I think #16561 misses a python -> python3 update in the compiler itself, have you run chpldoc specifically over a test program?

[lydia-duncan]

It's this guy right here

[mppf]

I made a note to update #16561 for that line.

[lydia-duncan]

I think we can say the answer is yes and close this issue

[mppf]

Yeah. PR #16560 is a starting point at the change (and there are many follow-ups).