Vega-strike-engine-source: Python 3 test fails on some platforms

[nabaco]

On some platforms CMake configuration fails due to Python dependency.

When running normaly, it expects python2, but for some reason it links against python3 and compilation fails.

On -DUSE_PYTHON_3=ON it fails, claiming to not find python. To overcome this, I have to replace in a single place from python3 to python.

I will post logs tomorrow.

[nabaco]

@ermo will you be willing to have a look at this?

[BenjamenMeyer]

Linux distros are moving over to Py3 by default. Arch and a few others jumped early; guessing that might be part of what you're running into (based on gitter chat).

FWIW - So CMake use to have a mailing list for support, which was always very helpful. Apparently they transitioned to solely using discourse - https://discourse.cmake.org/. Haven't logged in there yet, but they're devs were always helpful.

[BenjamenMeyer]

A few questions:

• [ ] what all platforms are we seeing this on?
• [ ] is Py2 installed on those platforms?
• [ ] what is the default Python version on those platforms?

[nabaco]

I'm seeing this on Arch and MacOS. For the latter you can see here:
https://travis-ci.org/github/vegastrike/Vega-Strike-Engine-Source/jobs/695768054

For Arch I still need to collect the logs and upload.

In the case of Arch, Python2 is installed alongside Python3. Regarding default, I'm not sure, need to check.

As can be seen here: https://wiki.archlinux.org/index.php/Python#Installation, I'm assuming Python3 is the default.

[BenjamenMeyer]

@nabaco Yeha, I think Arch is Py3 by default. /usr/bin/python should point to the default Python on any platform. If you want a Py2 or Py3 then use python2 or python3 respectively.

[ermo]

Could someone help me understand how, exactly, it is our job to possibly have to work around Arch not doing things like other Linux distributions in terms of detecting py2/py3 via cmake?

And for mac, it's anyone's guess why the cmake functionality doesn't work. Sounds more like a cmake issue than our issue if I'm honest, given I'm just using standard cmake functionality...

[BenjamenMeyer]

@ermo if we want to support Arch, then yes we'd need to figure it out... that said, I don't think supporting Arch is high on the list. If someone wants to take it on and generate an AUR for us then great.

Mac is a little bit different; but also, like Windows, a bit more problematic. On OSX they've historically provided various scripting languages but that's suppose to be changing - they've said they're going to start not providing various scripting languages by default, IIRC Python was among those to be dropped. That said, we require Homebrew to be installed already at least for dev, and Homebrew requires Python too IIRC so we're probably covered that way.

[nabaco]

@ermo I don't know standard cmake functionality you're using, but I say we could look at it from the other way around, that Ubuntu and such are not doing it the right way, since Python 2 is dead, and the default should be Python 3.
Either way, we may try to contact the cmake guys and ask them to help with that.

[BenjamenMeyer]

@nabaco honestly it doesn't matter that Py2 is dead. There's LTS platforms that still use Py2 so as an application we have to be mindful of that.

Arch was able to and lept into Py3 before any other distros did. Everyone else created a standard which Arch may or may not move to. Even Python has a defined standard (https://www.python.org/dev/peps/pep-0394/) on how it should be done.

Any way.... I suggest we:

• only worry about the platforms we are officially supporting
• let anyone stepping up to package for a platform worry about those platforms, especially if it's not in our official list.

I don't think Arch is in our official list right now. Not opposed to it or having an AUR, but just not in our list right now.

[nabaco]

OK, agreed about Arch. But what about MacOS?

[BenjamenMeyer]

Keep in mind were only continuing to support Py2 for a few more months. We need to get 0.6.x and 0.7.x out and then hit dropping Py2 hard to quickly follow with 0.8.x.

Until then I'd suggest we do a best effort approach and work as quickly as we can to get the tasks done to get through those releases.

[evertvorster]

Hi there...
I am actually in the process of writing the AUR for vegastrike, and I'll give you 1 guess as to why I am here.
XD

Pushing the issue further up the hill is just prolonging the inevitable. At some point other distros will also switch over to Python3 as the default. I'll go poking around on how all the other packages that require boost on Arch have circumvented this issue, and come back to you guys.

[royfalk]

I'm in favor of phasing out python support in the mid term and have the code stuff in c++ and data (missions and the like) in json, yaml or something like.
This will solve several problems including python 2 support, unit testing and provide ease of content development.

[BenjamenMeyer]

@evertvorster awesome! thanks; we're actually trying to get through to things as quickly as possible. Only limitation is time for our volunteers to get us there. Goal is to be on only Py3 by EOY, hopefully far sooner.

[evertvorster]

No problems. For now, I have resorted to just making a small patch file, just for Arch Linux.

From what I read on the web, the way that CMake detects Python and boost is very fragile, and no clear correct way is slapping me in the face right now. I'll get the packages to build & install first, then make them pretty before letting them loose on the world.

[evertvorster]

Hi there... for the people that are running Arch Linux, there are now two packages in AUR.
vegastrike-data-git, and vegastrike-git. It compiles and plays.

[nabaco]

@evertvorster could you add that info into our README please?

[evertvorster]

Hi there, nabaco.
Arch Linux seems to be the outlier here, and the way that the boost python bindings are done in the source works for most people.
The way that I made the Arch AUR packages works only for Python 3, as that is what I specified in the build instructions. It's hackish, and I agree with royfalk in phasing out Python. Of course I am not a coder, so I can't help too much with that part.
TL;DR Nothing changes for people reading the README, and in Arch it's behind the scenes/sorted

[nabaco]

@evertvorster I meant to add info about the AUR so new users will know to just yay vegastrike

[evertvorster]

Ah, I get you now.
The proper command is now: yay -S vegastrike-git

I doubt that I have write access to this project....

[nabaco]

You don't need access write to the project :) You fork the project and open a Pull Request. It's quite simple.
But if you prefer I can do the PR, no problem.

[ermo]

Arch Linux seems to be the outlier here, and the way that the boost python bindings are done in the source works for most people.
The way that I made the Arch AUR packages works only for Python 3, as that is what I specified in the build instructions. It's hackish, and I agree with royfalk in phasing out Python. Of course I am not a coder, so I can't help too much with that part.
TL;DR Nothing changes for people reading the README, and in Arch it's behind the scenes/sorted

My whole point above was that Arch seems like an outlier. Thank you for confirming that I wasn't going crazy!

Also: Thank you for patching the AUR build. Just be aware that py3 is not officially supported in VS (yet). According to the roadmap, verified py3 compatibility is what v0.8 is all about.

[evertvorster]

@ermo whether py3 is official or not, it's the only thing the AUR version of vegastrike builds against, and seems playable. How does one "verify" that everything works in py3?
Currently play ability is hampered for me due to an unrelated bug.

Of course, I am pulling from master, but IMHO it looks like VS is on track to kick py2 to the curb, maybe earlier than expected.

[ermo]

How does one "verify" that everything works in py3?

Of course, I am pulling from master, but IMHO it looks like VS is on track to kick py2 to the curb, maybe earlier than expected.

One runs the various porting suites (2to3) on every single .py file to ensure there are no hidden surprises.

And then one playtests in a way that ensures that there is full code coverage.

At least, that's how I would approach it.

[evertvorster]

How does one "verify" that everything works in py3?
Of course, I am pulling from master, but IMHO it looks like VS is on track to kick py2 to the curb, maybe earlier than expected.

One runs the various porting suites (2to3) on every single .py file to ensure there are no hidden surprises.

And then one playtests in a way that ensures that there is full code coverage.

At least, that's how I would approach it.

What are porting suites? Is there any documentation that tells me how to run them?

Thanks for your patience and help so far while I get up to speed.

[BenjamenMeyer]

@evertvorster We have to go through and verify that all our Python files work under Py3, possibly port Object relationships/classes, print '' to print(), etc. Tools like 2to3 can help. I've done a little analysis and we're probably in a better situation than we think - at least I haven't seen anything depending on stuff that is Py2 only.

FTR, I set 0.8.x as the Py2 to Py3 conversion target as we already had some things underway and I wanted to clear those out before tackling the Py3 conversion and verification. The work will be split between (a) auditing any and all Python related code (both on the C++ side and Python side) and (b) a lot of game play to verify things. We'll have to develop a plan once we move 0.7.x to a branch for release of how we're going to tackle it.

There's been some light testing with Py3, which is why I think we're a better position than we initially thought. Thanks for being patient with us in this.

[evertvorster]

@BenjamenMeyer, I have been bored again, and built a python2 version of the vegastrike engine just for testing. (It's now in aur as vegastrike-py2-engine-git)
It's still got that weird bug where things seem to jump around.

Just for kicks I unpacked the privateergold and tested both python2 & 3 engines against that as well. Python 3 confuses it heavily. Python2 is almost running, I think the resolutions available on this machine is confusing it. May just be that the config file included with the privateergold stuff is too old. It does prove that the multiple game data installs does work, and do not interfere with each other.

FYI, I'll help with the play testing with python3 enabled, once we kick that jumping around bug to the curb. It's messing up gameplay quite bad.

[evertvorster]

Coming back to this one.... I ended up with basically two patches, one for each version of Python. Now I have made a CMakelists.txt file that does the right thing for either version of Python... however I have a feeling that it will start failing on Python 3 when that gets an update.

The issue is that just looking for "Python" is ambiguous. (Thanks, Arch) Looking for Python2 or Python3 does not work either... it needs to be python27 or python38.

This was lifted from the findboost manual https://cmake.org/cmake/help/latest/module/FindBoost.html

It does not mention how to look for a version same as or later. So, here is the thing. I can make a CMakelists.txt that will work for everything now, but it will need a version bump whenever there is a new release of python/boost.
What is the best way to proceed?

[nabaco]

maybe we can use some macro checking?
something like

IF(ARCH OR MACOS)
<arch specific code>
ELSE()
<regular code>
ENDIF()

[evertvorster]

@nabaco, I don't know. That still feels hackish.
The problem stems from having two versions of python installed, which is not usual for a Unix-y OS. The proper way around this, is to specify the exact version of the libraries which you want for older stuff, and just the library name if you want just the latest version of something. So, this is probably why Arch moved "Python" to mean python3, as this fits the philosophy.

[BenjamenMeyer]

@evertvorster most distros have Py2 and py3 installed side-by-side and it's a matter of which is the system default. Arch is the odd man out.

[evertvorster]

@BenjamenMeyer sure, Just because it's the right way does not mean everybody follows it.

Of course, the best solution would be something that works in every case. Which is what I was attempting today.

Ideally one would specify the exact version of the older library that hangs around for compatibility, and just the library name that follows the newest version. This is how it works on Arch & Manjaro(and recently Gentoo), but since the rest of the world specifies the default as Python2, we now have this weird situation.

The only thing that works without ifdefs is to specify the exact version of python for each branch.
But then that falls apart in the future, when Python 3 bumps their version.

The ifdef solution will also fail some time in the future as other distrobutions move their default from Python2 to Python3

The simplest solution for everybody is to just leave things as they are, with the Arch packages patching the CMakelists.txt. At some point in the future I suppose this conversation will be revisited when more distributions move over to Python 3 as default.

[BenjamenMeyer]

@evertvorster actually most distros follow https://www.python.org/dev/peps/pep-0394/ which specifies that python2 and python3 should point to the installed python versions appropriately. It also gives the behavior of python. The PEP was specifically designed to help with the Py2 to Py3 migration.

[evertvorster]

@BenjamenMeyer I did have a quick read at that document. Originally posted in 2011, nine years ago. Also, according to that document Python 2 was EOL in January 2020 ( maybe the world was running on Python 2 and the current epidemic is somehow related? )

As I said before, where we are standing now, the current CMakelists.txt works for most distros. I don't mind leaving it as is, mainly because I am lazy, and the patches for Arch works. Sometime in the future where there is no more Python 2, or the VS engine evolves beyond python 2, then we'll come back to this, and the edit will be pretty easy.

[BenjamenMeyer]

@evertvorster yes, Py2 was suppose to be deprecated ages ago, but they kept delaying it as the community hadn't gotten Py3 compatibility across enough of the various libraries. The tipping point finally came in 2017/2018 with most Linux distros planning on their base python conversion in 2018/2019. So the EOL of 12/2020 will happen aside from LTS distros which will have to support Py2 on their own after that date. Basically any Ubuntu LTS before 20.04 and any RH LTS before RH8, so there will still be a fair number of systems using Py2 in some form, just without security updates or support from the Python Foundation.

Arch and Gentoo led the conversion process IIRC and have been ahead of the game overall. The CMakeLists.txt we have in repo works on Ubuntu and should work on any distro following PEP-394. That said, we only have to support it until we make the Py3 conversion, so having something "hacky" until then is okay as that's a near term goal - I'd really like to see 0.8.x out by EOY so really pushing to get 0.6.x out ASAP.

[nabaco]

Can anyone verify that issue was resolved with #163 which was merged into master?

[stephengtuggy]

@nabaco As of #202 , @evertvorster has confirmed that the issue is resolved on Arch. I can also confirm that Python3 builds work on Ubuntu 16.04 xenial; Ubuntu 18.04 bionic; Ubuntu 20.04 focal; and Debian 10 buster. See also https://travis-ci.org/github/vegastrike/Vega-Strike-Engine-Source/builds/713735244 .

Not sure about other distros/versions. Those that have Boost version 1.67.0 or later should be fine. Those that have earlier versions of Boost might not be.

See list of distros with their respective Boost versions here.

[BenjamenMeyer]

@evertvorster @ermo @stephengtuggy @Loki1950 @P-D-E can we confirm if this is still an issue?

[P-D-E]

@BenjamenMeyer On my Debian Unstable, Python 2 is still the default; by setting use python 3 on in ccmake, all the references point to Python 3 resources, and it builds correctly. I've just tried to switch the system default to Python 3, and it builds correctly as well, so no reproducible issue on my side. Boost is at version 1.74.0 by the way.

[BenjamenMeyer]

@P-D-E thanks; based on https://github.com/vegastrike/Vega-Strike-Engine-Source/issues/142#issuecomment-667234244 and you're comment I'm going to close this. We can re-open if another distro has issues; but I think we're fine now since we've rev'd our supported platforms a little.