Geonode: GNIP-64: Python 3 & Django 2 Upgrade

[Coop56]

NGA is proposing to accelerate the work of upgrading GeoNode’s codebase from Python 2.7 to 3.7, Django 1.11 to 2.1, and replace all Angular 1.4 with React and JavaScript code. We will coordinate this work as part of our GEOINT Services contract to make sure this work is completed and pushed back to the GeoNode community by the end of March 2020. Work related to GeoNode as part of the GEOINT Services program will be pushed back to the GeoNode community as part of our open source commitment when appropriate. All work will be performed by Boundless Spatial at the direction of NGA. The following document outlines the work required to deliver on this upgrade work.

We’ve already begun work on the Angular 1.4 to React migration in another GNIP (https://github.com/GeoNode/geonode/issues/4129). This GNIP will focus on the Python 2 to 3 and Django 1.11 to 2.1 upgrades. We propose to complete the Python and Django upgrade process in the following order. Of note, this process is subject to change as we work on these items and uncover dependencies during the upgrade process.

1. Remove unnecessary dependencies, to ensure we only spend time upgrading the dependencies that are necessary to GeoNode.
2. Improve test coverage where possible, to make it easier to spot regressions.
3. Update dependencies with a version that supports Python 3.7 or replace those dependencies with another option if they do not have a version with Python 3.7 support.
4. There are certain pieces of Python 3 syntax that are already compatible with Python 2.7. We’ll make as many of these changes as possible in separate commits and get those PR’d into master before moving onto dropping Python 2.7 support entirely.
5. In one large PR, drop support for Python 2.7 for the entire GeoNode codebase and ensure all code is Python 3.7 compatible. (this PR will have to go on a new version, as it will of course break old systems, the community can decide what version it will be).
6. In a separate PR, update django-osgeo-importer to drop support for Python 2.7 and ensure all code is Python 3.7 compatible.
7. Since Django 1.11 supports Python 3 already, we’ll be able to finish the Python 3 upgrade and then move onto upgrading Django separately. We’ll upgrade any dependencies of GeoNode to a version that supports Django 2 first or replace them with another alternative if necessary.
8. Then we’ll move onto the GeoNode codebase itself and upgrade all of the code to Django 2 compatible syntax.
9. Finally, we’ll upgrade django-osgeo-importer to Django 2 compatible syntax.

Once all of this is done on master, the community will be able to decide which version we call this, and projects can start upgrading their own codebases if they choose to do so.

Key links:

• GeoNode dependency list: https://docs.google.com/spreadsheets/d/1buUuIeGPL8zfv5xFotfGMpxHt2i8njoCEh4Y4kna3rY/edit#gid=0
• Mailing list discussion thread: http://osgeo-org.1560.x6.nabble.com/GNIP-64-Status-and-Plan-td5420981.html
• Python 3 interpreter configuration (PyCharm): https://www.jetbrains.com/help/pycharm/configuring-python-interpreter.html
• Final testing document (need to update a sheet for this GNIP when we are near release): https://docs.google.com/spreadsheets/d/13aaZ1l_2_SHzdmz9DwExEO_jxwgNuXfMa6oF5eWIv_s/edit#gid=376642658
• Task sheet: https://docs.google.com/spreadsheets/d/1gr4ISkFBarY-7i_DPi0Vt_FnA5_uF30dGUgYpcIZ-xw/edit?usp=sharing

[travislbrundage]

Looks good. +1 👍

[copaskevin]

This work has been put on pause by NGA at the moment. We are working with them to identify who/when the work will be completed. Hope to have more information by the end of April.

[frafra]

@copaskevin Any news?

[copaskevin]

@frafra No news from NGA. The rumor is that they will be funding this work outside a different contract but it will more than likely not be coming from the contract awarded to us.

[GeoTob]

Any update on this proposal? We may be able to contribute some dedicated time late in the year / early next year to help with upgrade efforts.

[afabiani]

Hi @GeoTob no particular updates up to now. GeoSolutions will probably invest some resources on this hopefully before the next year. It would be good to cooperate on this of course.

[gannebamm]

If our proposal gets funding, we will also be able to contribute. We will get more info later this year. Let´s do it together.

[coulonpascal]

Hi. SCISYS UK is sponsoring the code sprint at the up coming UK FOSS4G conference. We are using geonode in a number of uk government and private sector projects. It would be great to get some progress on this issue. Has there been any progress anywhere on this?

[gannebamm]

Regarding the mentioned proposal: If it is succesful the funding will not start before Oktober 2020. I am sorry.

[coulonpascal]

It would be great to get some help from the community at the up-coming UK FOSS4G, this is a great forum for this?

[travislbrundage]

Hey all, while I won't be at the up coming UK FOSS4G, I am going to be getting up to speed on this task and picking it up soon. The information is largely the same as described originally to my understanding, but for now the task is on my radar instead. Will keep you updated as I start making progress.

[GeoTob]

Nice, we have time set aside from late November onwards if help is still needed by then :-)

[GeoTob]

Just wondering if any one managed to make progress on this? If not I would like to propose that our team picks this up from 18 November. We would develop against a public fork of the latest stable release while keeping close to the plan outlined in the issue above.

Would obviously welcome any community contributions to our efforts.

[t-book]

@GeoTob What great news! We had this yesterday in gitter chat.
I hope it's okay to quote @travislbrundage here:

Well, I've been wrestling with that + trying to speed up the tests for GeoNode (there's some conversation before about it). So far I have only really done organization. I have a spreadsheet continued from the work coop had done with all the libraries that will need replacement or updating with Python3 (fortunately not too many). With this problem out of the way I'll start committing some updates to the unit tests to increase coverage/speed up if possible.

So it would be great if you two could get in contact and coordinate !

[afabiani]

@ all ... I guess GeoSolutions also will jump soon into this. Probably from November.

The priority now is to close the 2.10.1 release. Then we will focus on Python3 upgrade.

[GeoTob]

We have two team members free after FOSS4G Oceania through to February - sounds like this won't be a problem to get done with everyone's help:) Let's coordinate in November?

[t-book]

* Dreamteam * 👍

[travislbrundage]

This is excellent to hear! I welcome all the help I can getting this done. :)

There is not too much to update since I last posted as it took a while to get transitioned onto this task. I will reach out on the developer email list to create a more detailed post on progress and what I foresee going forward.

For now I am just trying to help with the 2.10.1 release + improve unit testing so the Python upgrade (hopefully) goes more smoothly. Looking forward to working with you all!

[travislbrundage]

Hey all, making a quick update to this. The main post will be edited with key information and links. I've made some documents to keep track of work that needs to be done so people can work in parallel. You can find more detailed information in the thread I've made on the mailing list, also linked. Thanks for all the help and feedback :)

[GeoTob]

We now have a prototype working on Python3.

For testing purposes I've merged all the currently open PRs into my own fork at https://github.com/geotob/geonode, but once the PRs are merged we should be able to run against the master branch here.

If you want to test it out, I have pasted instructions below. Next steps from our point of view are:

• Merge existing PRs
• Work out why some modules don't install from Pypi (see requirements.txt in my fork) and re-release them to Pypi for Python 3
• Integrate this into our docker images to test against Postgis and GeoServer
• Lots of manual and automated testing

GeoNode Python3 environment
(hoping this works ....)

1. Clone from https://github.com/geotob/geonode
2. Create a clean python3 virtual environment and install all requirements:

python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt

1. Install the version of pygdal compatible with your system's gdal (run gdal-config --version and pick a compatible pygdal version).
2. Run 2to3 over gsimporter and fix an issue (we're creating a PR at the moment):

NOTE: You will have to change the paths below to work with your version. You may be running a different version of Python3, and packages may be installed in the lib folder instead of lib64

2to3 -w venv/lib64/python3.7/site-packages/gsimporter
sed -i "s/ async/ asyn/g" venv/lib64/python3.7/site-packages/gsimporter/api.py

1. Run your DB migrations:

python3 manage.py migrate
SESSION_EXPIRED_CONTROL_ENABLED=False python3 manage.py runserver

1. You're now running GeoNode on Python3 backed by sqlite.

[francbartoli]

@GeoTob regarding all the submitted PRs I'm wondering what's the rationale to merge several try/catch for maintaining the compatibility with the import of python 2. I guess the problem is Travis but then I would expect the required changes to make it pass.

cc @afabiani

[GeoTob]

@francbartoli just compatibility reasons really:

• we want to avoid errors/questions for people checking out master and running that instead of stable
• others have assigned themselves to improving/changing tests and I'm not sure what version they're on
• we want to be able to test against py2 and py3 initially and make sure nothing breaks across the whole application

Once we're confident everything works on python3 it'll be trivial to remove the try..excepts

[francbartoli]

Thanks @GeoTob, as already expressed on gitter I strong believe the priority on master is to make the switch to python 3 as soon as possible rather than avoid problems for people checking out. Anybody who can help out in whatever form is welcome, including the port of Travis to python 3.
In my honest opinion, I think the strategy to make sure the tests pass against py2 and py3 in parallel is a little bit too much conservative but I can be wrong.

[simboss]

I agree with @francbartoli on this one.

[GeoTob]

@francbartoli I think we can disagree here - keeping compatibility has helped our team upgrade all apps and most dependencies to python3 and lets us rely on the existing test runners to make sure we haven't broken anything along the way.

I can see absolutely no downside in keeping master from breaking as long as possible, even if that means we have to temporarily accept a code quality issue around try..except imports. But again, it will be trivial to remove those once everything works on python 3 and the test runners have been upgraded.

So for now, compatibility helps us to work in parallel across teams and make small, incremental & testable changes .... I'm failing to see how changing this would make us go even faster ?

[simboss]

@GeoTob that is a valid point but we need to think about the cost of maintaining this.

Adding more code will increase maintenance cost, will make it harder to add unit tests, etc. etc.

[GeoTob]

I agree @simboss and sorry if it wasn't clear -- we have to drop python2 compatibility for Django 2+ in any case, so this is a temporary crutch to help us make the transition easier. We can remove all those lines once everything works in python 3.