Esmvaltool: Timeline for dropping Python 2 support

[nielsdrost]

ESMValTool now works using Python 3. It is perhaps time to start thinking about how the migration to Python 3 will take place.

[nielsdrost]

As I recall, we only chose Python 2 instead of Python 3 because we though it would be too much work to change to Python 3. If this is now done, and it all works in Python 3, this would be an ideal moment to switch to Python 3 only.

Now, I hear you ask, why not support both Python 2 and 3? Normally, for a library (say, Iris) it makes sense to support both 2 and 3 at the same time to allow users to slowly adapt to Python 3. This is not the case for ESMValTool, as to keep Python 3 support (now we have it), we will need to disallow any Python-2-only code. So, the _only_ benefit I see of supporting 2 and 3 is the ability to run ESMValTool using Python 2 if no development of (Python) code is needed. This is not worth the trouble, I think. Especially since we strongly recommend Conda, for which the difference between Python 2 and 3 is very minor.

So, I vote for dropping Python 2 support for ESMValTool 2.0 right now. For the 1.X series this of course does not apply, as it is Python 2 only.

[mattiarighi]

We need to keep Python 2 support to be able to include external diagnostic packages in the ESMValTool. These are provided by external partners and not necessarily written in Python 3.

@axel-lauer

[nielsdrost]

He Mattia,

True, but if these do not support Python 3 they will either not work in ESMValTool 2.0, or ESMValTool 2.0 will not support Python 3. Neither sounds like an ideal situation.

Perhaps start with creating an overview of these external diagnostic packages and see where they are with Python 3 support? Are all of these in the ESMValTool 1.1 release?

[mattiarighi]

@axel-lauer knows more about this issue.

[bjoernbroetz]

Would it be an option to treat python 2 as just another language for the use with diagnostics, just like R and ncl? If someones diagnostic relies on a Python2-only library, than this person needs a separate python2 installation just like a ncl or R installation. The backend would be Python3 only.

[mattiarighi]

This would be the optimal solution, if viable.

[bouweandela]

Doing that would make the installation troublesome, because it would require you to install libraries in two versions of Python. That means either that such diagnostics would require manual installation steps, which is not user friendly, or maintaining and testing even more complicated installation scripts.

[mattiarighi]

Still. there are a number of external diagnostics, originally developed as stand-alone tools, which will be part of the ESMValTool and we need to provide to support for them. We cannot ask the developers to rewrite them in Python3.

[jvegasbsc]

Why we don delay this until the beta release is ready?

In the meantime,:

1. We encourage everyone to develop their diagnostics with Python 3 in mind
2. We collect a list of diagnostics that require Python 2 to work.

Just before release, we can go through the list and check what it's preventing them to work in Python 3 and if it is possible to fix or there are plans to be Python 3 compatible in the future.

I think we can not take a decision without knowing exactly what diagnostics are affected by it, because it could be even more productive to fix an external open-source library to work with python 3 than to develop a complex installation procedure

[nielsdrost]

Agreed we should keep everyone onboard during the transition. Let's try to come up with a list of Python2 based exteral dependances so we can "count down" to Python 3 only :-)

[valeriupredoi]

Guys, it appears that Anaconda 5.1 (python2.7 version) comes prepackaged with the real-deal python 2.7 (and not the impostor py2.7 that is in fact 3.6) -- this is untested by me but rather info I got from my office mate, so don't hang me if it's wrong (he knows his stuff so I trust it to be accurate). What I've tested myself and can confirm is that in Anaconda 3, if you create an virtual env asking for python 2.7 in the environment file then python 2.7.14 is what you get and not 3.6+. Saying this since @mattiarighi asked me to chime in here :grin:

[mattiarighi]

This is the list of python diagnostics implemented in version 1.
Some of them have been already ported to version 2 (in Python 3).

Given that the amount of code is relatively limited and that some of these diagnostics have already been ported to v2/python3, we can think about dropping Python 2 support now.

• [ ] alb_QA4ECV.py - @BenMGeo ?
• [x] AutoAssess_radiation_rms.py - Done by @valeriupredoi
• [ ] catchment_analysis_val.py - Ongoing by @tobstac #663
• [x] dummy_python_external.py - Example diagnostic
• [x] dummy_python.py - Example diagnostic
• [ ] fAPAR_QA4ECV.py - @BenMGeo ?
• [ ] fire_ESACCI.py- @BenMGeo ?
• [ ] ipcc_fig_9_14.py - @zklaus
• [ ] ipcc_fig_9_15.py - @zklaus
• [x] ipcc_fig_9_42a.py - (@schlunma #651, missing provenance)
• [ ] lc_ESACCI.py- @BenMGeo ?
• [ ] sm_ESACCI.py- @BenMGeo ?
• [ ] sm_pr_diag_nml.py - ?
• [ ] sm_pr_diag_test.py - ?
• [ ] SouthernHemisphere.py - @ledm ?
• [ ] SouthernHemisphere_scatter.py - @ledm ?
• [ ] sst_ESACCI.py - @BenMGeo
• [ ] TropicalVariability_EQ.py -
• [ ] TropicalVariability.py - @zklaus ?
• [ ] TropicalVariability_wind.py - @zklaus?
• [ ] ww09_ESMValTool.py - Done by @axel-lauer #635

[valeriupredoi]

I can't stress how happy I am to see the move to Py3, just a heads up that the other autoassess metrics that (are and will be in) ESMValTool v2 are Py3 as well

[jvegasbsc]

I will use Python3 for PRIMAVERA metrics also

[mattiarighi]

@axel-lauer @BenMGeo @zklaus @tobstac can you give us your opinion about this issue?

We would like to drop python 2 support asap, but there are still v1 diagnostics written in python 2 (see list above). Do you think it is possible to translate them to python 3 when porting them to v2?

[zklaus]

Yes, absolutely. Good riddance. If somebody wants to stay compatible with both versions (as I also generally try), I recommend taking a look at Six.

[axel-lauer]

I am fine with dropping support for Python 2 now. If needed, diagnostics can be changed accordingly when being ported from v1 to v2.

[tobstac]

I am fine with it. I ported both of my diagnostics to python 3 while implementing it in version 2. At least, I am not aware of any remaining python 2 features. The catchment_analysis diagnostics is already merged (2d02dd9112f15b8a7257ed3381145a5f87f681d0) and the landcover diagnostics is finished from my side just waiting for a final statement about the bounds issue (https://github.com/ESMValGroup/ESMValTool/pull/770#issuecomment-461797221)

[BenMGeo]

It is planned to drop Py2 in favor to Py3 for porting. No problem with that. The big thing on my to-do list is the usage of iris instead of geoval, which will be the main workload here.

Edit: Sorry for accidentally closing this issue.

[mattiarighi]

@bouweandela @valeriupredoi given the feedback, I think we can drop Python 2. :tada:

[ledm]

Linking @bjlittle from Iris dev team as he asked us about this yesterday,

[valeriupredoi]

woohoo party it's like 2019! :grin: who wants to remove Py2 testing? I can do it if there's no volunteers :beer: