Smokedetector: Should our CI testing default to spanning the officially released and supported Python versions (currently 3.5 though 3.8)?

[makyen]

Should we maintain our CI testing on specific Python versions, or have the CI testing be targeted at the spread of Python versions which are currently released and supported (i.e. Travis testing the oldest officially supported version, and CircleCI the most recent released version)?

At this point, Python 3.5 is the oldest officially supported version, but is scheduled for End of Life on 2020-09-13. Python 3.8 is the newest version which is officially released, but 3.9 is scheduled for release on 2020-10-05.

Obviously, this strategy may need to be modified if we find there's a feature we need/want which is only available in a release newer than the oldest which is officially supported. I believe we did this when moving from 3.4 to 3.5.

---

Note: This inquiry was prompted by a recent change to the version of Python which was being used for testing on CircleCI to move from Python 3.6 to Python 3.8. I asked @iBug about the change, because I thought I remembered a conversation back when 3.7 was released that we would not switch testing away from 3.6, for some reason I'm not remembering. I'm not finding that conversation.

[teward]

If we are capable of going and doing multiple tests, then all currently supported versions of Python that we dictate are compatible should be supported.

That is, at this time, the most major versions of the Python 3 line that are alive from 3.6 onwards:

• 3.6.11 (though I think we have some 3.6.9 boxes, it should Just Work with 3.6.11)
• 3.7.8
• 3.8.3

At least, at this point in the things.

We need to at least keep Python 3.6 compatibility in our testing while also testing out 'newer' featuresets and reverse-compatibility.

My two cents, anyways.

[iBug]

FWIW, I started running CI on GitHub Actions. I originally proposed this for whatever reasons that I can't recall now. I've just added relevant config files and tweaked them well.

For now, our GHA CI runs the same stuff as that of Travis, including flake8, pytest and coveralls. In addition, it uses a matrix to run the whole suite on both Python 3.5 and 3.8, which as apparent as the config seems, can be easily extended.

The concern behind this addition was primarily speed. Both Travis and Circle spend like more than a minute just for spinning up the environment (which was also a decisive reason I bumped CircleCI image from 3.6 to 3.8). GHA OTOH, starts almost instantly, is integrated into GitHub, and runs almost as fast as CircleCI. It even provides a hassle-free token for authentication (API & push etc.) (docs), meaning we could defer the job of updating the deploy branch from MS to GHA.

TL;DR The base problem of this issue, our Python version (or more precisely, CI test suites), is covered above.