Openrct2: Migrate distributed Linux builds to snap

[janisozaur]

OS: Linux

Right now the Linux builds are distributed as binary tarballs. I have investigated snaps a little and they seem to offer broader compatibility. Until recently I have feared the snap package would weigh hundreds of megabytes, but a full build of OpenRCT2 weighed only around 30MiB with symbols(!).

[rwjuk]

Just to clarify, this wouldn't stop us from distributing through distro package managers (e.g. Pacman, apt, etc.) right?

[janisozaur]

Correct. This concerns only the things we build on travis.

[rwjuk]

In that case I'm all for it.

[jansegre]

I'd suggest flatpak too. From what I understand (I may be wrong, can't find sources to back it up) it basically boils down to snap/snappy centralizing the packages on a single repository (snapcraft?) while flatpak doesn't, which mostly means a flatpak package could be attached to the github release the same way targzs are, while a snap package would have to be published to snapcraft, which is not bad per se.

Also while snappy comes by default in recent Ubuntu, other distros haven't embraced it too much (source from a couple months ago).

[janisozaur]

Interesting. I know there are at least a few contending formats: flatpak, snappy, appimage, to some extent docker, maybe more.

I have only tried snappy so far and was quite impressed by results and the ease with which I was able to produce an openrct2.snap, at least on a Ubuntu machine. The supported distros also looks OK: https://snapcraft.io/ has logos for Arch, Debian, Gentoo, Fedora, SUSE with some installation instructions (which I haven't checked).

What do you mean by "centralizing"? In my very brief experience with snap, I noticed it downloaded a "core" image, which allows reducing amount of userspace libraries I need to install in snap itself, but other than that the snap archive looked pretty self-contained.

Are you any familiar with flatpak? Can you provide more arguments for using it? How much userspace do I need to include in resulting file, what's the size of it?

[janisozaur]

https://github.com/OpenRCT2/OpenRCT2/issues/6232 should be taken into account when doing this.

[ykgmfq]

I have managed to build a Flatpak from the latest release. There are still some issues that need to be looked at before it is functional though. The code and documentation can be found here. If someone had some ideas about how to fix the problems, that would be great!

[janisozaur]

@ykgmfq

The dialogue for the game file selection does not work. It can be circumvented by setting game path in ~/.var/app/org.openrct2.openrct2/config/OpenRCT2/config.ini to the correct directory.

Does flatpak archive access native file system sandboxed one? What if user has kdialog or zenity installed, does it still not show up in the flatpak'd $PATH?

See ./openrct2 --help for how you can have it set game_path for you.

There seems to be a problem with resolving the data path. This leads to an immediate crash.

Does it _crash_ or exit with error code? If former, that's a bug and please file it. I don't know flatpaks that well, but is it feasible to embed data/ inside the archive? You'd have to copy data/ from root of the repo, do make g2 and move resulting g2.dat into data/.

The build system cannont download the title sequence because it won't resolve the host name. To get around this the download has been deactivated with a cmake option.

Likewise, you can just embed title sequences in data/title/

[ykgmfq]

My configuration has full host access with "--filesystem=host" for now.
It crashes with this:
ERROR[/run/build/openrct2/src/openrct2/localisation/LanguagePack.cpp:106 (FromFile)]: Unable to open /language/en-GB.txt: Unable to open '/language/en-GB.txt'

The language files are as expected in ../share/openrct2/langauges respective to the executable. I think platform_posix_sub_resolve_openrct_data_path() might not work correctly with Flatpak. Unfortunately I have not yet found out how to attach a debugger to a Flatpak process.

The download failure seems to be a Flatpak + cmake bug as it works when buidling directly on the host.

I will try your steps with modifying the package. I am aware of --openrct-data-path, but it is parsed by flatpak run itself!
$ flatpak run org.openrct2.openrct2 --openrct-datapath="/home/dmp/workspace/rctflat/files/share" Unknown option: --openrct-datapath

[janisozaur]

You're missing a dash there. datapath vs data-path. What does --version say, flatpak or OpenRCT2 version?

If data directory is next to executable, it will be picked up automatically too.

[ykgmfq]

Damn, that's embarassing. Yeah, version does show OpenRCT2. And with data-path the game actually works.

[ykgmfq]

The title sequences are now included in the manifest and work as expected.

[Luraktinus]

why not AppImage ?

for snap or flatpak the users have to install said tools
(snap is not centralized btw, snaps can be added to the github release page too @jansegre )

[TheAssassin]

Building an AppImage is ridiculously easy using the binary tarball you provide.

Check out the release page of the repo I created tonight: https://github.com/TheAssassin/openrct2-appimage/

Edit: the AppImage works only on distros released after Ubuntu xenial. This is due to the version of libstdc++ the tarball's binaries are linked to.

[janisozaur]

@TheAssassin that's very nice. Can you comment on snap vs flatpak vs appimage?

I have yet to test this on my Arch system, but my previous encounters with AppImage were successful, much more so than snap (which had issues obtaining GL context and required building on Ubuntu).

the AppImage works only on distros released after Ubuntu xenial. This is due to the version of libstdc++ the tarball's binaries are linked to.

Is it something dependent on the system used to build the AppImage? If I were to switch the builder to Arch, it should still work or would I need to watch out for some things? The release version still uses pre-C++-17 compiler, I think, we've updated since then and that required passing -static-libstdc++ -static-libgcc.

I still have to look into what gets provided in the embedded file system, but that looks like a viable option to me. @TheAssassin would you be interested in submitting a PR to package our develop builds into AppImages?

[TheAssassin]

@janisozaur I am an AppImage developer (that is, I develop the apps to make AppImages), therefore I am always in favor of AppImages. They're single files which can just be copied from one system to another. They don't need any root permissions to install, they "just run" (well, in 95+% of all cases). And with tools like AppImageLauncher, they're really nice to use on the Linux desktop.

Regarding snaps: I have to disagree with @jansegre. Snaps _are_ centralized. Always. And that's by design (but that's a story for another day). Even if you upload a built snap to GitHub, it still depends on files managed by a central entity (most likely the Canonical maintained snap store), e.g., the core image(s). They're not self contained in the way AppImages are.

I haven't tried Flatpak very much, but in contrary to AppImages, where it's a simple "make it executable" (unless you use AppImageLauncher), you need to perform quite a few more steps in order to get them up and running.

See also https://appimage.org, https://github.com/AppImage/AppImageUpdate for updating, and https://github.com/TheAssassin/AppImageLauncher for desktop integration.

Regarding the PR resp. the C++17: The general rule of thumb for AppImages is: build on the oldest OS you want to support. For most people, this is Ubuntu trusty at the moment. The reason is that we bundle libraries based on a blacklist, which is primarily used to save space in the AppImage by not bundling libraries which are available on all the distros according to our tests (e.g., libc, libstdc++, etc.) (or which are producing incompatibilities). If you need to build on a fairly recent system, you need to add some additional work, e.g., by using tools like https://github.com/darealshinji/AppImageKit-checkrt which allow you to bundle a libstdc++, and load it on demand on older systems.

We should be able to develop such an AppImage build script. I can't promise I'll get to it, but I'll put it on my TODO list. Maybe I have a few minutes during the next weekend.

[IntelOrca]

Other than libc, are all the other libraries such as libpng, curl, jansson etc. bundled in the app image?

[janisozaur]

@IntelOrca

/tmp/.mount_OpenRCYoiIcX
├── [  16]  AppRun -> usr/bin/openrct2
├── [ 117]  openrct2.desktop
├── [ 26K]  openrct2.png
└── [   0]  usr
    ├── [   0]  bin
    │   ├── [   0]  data
    │   │   ├── [ 89K]  g2.dat
    │   │   ├── [   0]  language
    │   │   │   ├── [219K]  ar-EG.txt
    │   │   │   ├── [221K]  ca-ES.txt
    │   │   │   ├── [324K]  cs-CZ.txt
    │   │   │   ├── [214K]  de-DE.txt
    │   │   │   ├── [222K]  en-GB.txt
    │   │   │   ├── [111K]  en-US.txt
    │   │   │   ├── [210K]  es-ES.txt
    │   │   │   ├── [160K]  fi-FI.txt
    │   │   │   ├── [217K]  fr-FR.txt
    │   │   │   ├── [ 12K]  hu-HU.txt
    │   │   │   ├── [210K]  it-IT.txt
    │   │   │   ├── [266K]  ja-JP.txt
    │   │   │   ├── [363K]  ko-KR.txt
    │   │   │   ├── [196K]  nb-NO.txt
    │   │   │   ├── [284K]  nl-NL.txt
    │   │   │   ├── [293K]  pl-PL.txt
    │   │   │   ├── [240K]  pt-BR.txt
    │   │   │   ├── [319K]  ru-RU.txt
    │   │   │   ├── [204K]  sv-SE.txt
    │   │   │   ├── [208K]  zh-CN.txt
    │   │   │   └── [200K]  zh-TW.txt
    │   │   ├── [   0]  shaders
    │   │   │   ├── [ 201]  applypalette.frag
    │   │   │   ├── [ 182]  applypalette.vert
    │   │   │   ├── [ 702]  applytransparency.frag
    │   │   │   ├── [ 182]  applytransparency.vert
    │   │   │   ├── [  95]  drawline.frag
    │   │   │   ├── [ 581]  drawline.vert
    │   │   │   ├── [1.9K]  drawrect.frag
    │   │   │   └── [1.4K]  drawrect.vert
    │   │   └── [   0]  title
    │   │       ├── [4.0M]  openrct2.parkseq
    │   │       ├── [ 972]  rct1aall.parkseq
    │   │       ├── [ 798]  rct1aa.parkseq
    │   │       ├── [ 501]  rct1.parkseq
    │   │       └── [ 264]  rct2.parkseq
    │   ├── [   0]  doc
    │   │   ├── [ 38K]  changelog.txt
    │   │   ├── [6.6K]  contributors.md
    │   │   ├── [ 34K]  licence.txt
    │   │   └── [7.1K]  readme.txt
    │   ├── [   0]  include
    │   │   ├── [ 602]  discord_register.h
    │   │   └── [2.6K]  discord_rpc.h
    │   ├── [5.2M]  openrct2
    │   └── [ 24K]  openrct2-cli
    ├── [   0]  lib
    │   ├── [685K]  libasn1.so.8
    │   ├── [ 26K]  libasyncns.so.0
    │   ├── [ 83K]  libbsd.so.0
    │   ├── [2.3M]  libcrypto.so.1.0.0
    │   ├── [451K]  libcurl.so.4
    │   ├── [320K]  libdbus-1.so.3
    │   ├── [290K]  libdiscord-rpc.so
    │   ├── [ 34K]  libffi.so.6
    │   ├── [481K]  libFLAC.so.8
    │   ├── [905K]  libgcrypt.so.20
    │   ├── [524K]  libgmp.so.10
    │   ├── [1.2M]  libgnutls.so.30
    │   ├── [306K]  libgssapi_krb5.so.2
    │   ├── [272K]  libgssapi.so.3
    │   ├── [207K]  libhcrypto.so.4
    │   ├── [ 63K]  libheimbase.so.1
    │   ├── [ 39K]  libheimntlm.so.0
    │   ├── [212K]  libhogweed.so.4
    │   ├── [316K]  libhx509.so.5
    │   ├── [207K]  libidn.so.11
    │   ├── [ 54K]  libjansson.so.4
    │   ├── [ 47K]  libjson-c.so.2
    │   ├── [188K]  libk5crypto.so.3
    │   ├── [575K]  libkrb5.so.26
    │   ├── [863K]  libkrb5.so.3
    │   ├── [ 47K]  libkrb5support.so.0
    │   ├── [ 63K]  liblber-2.4.so.2
    │   ├── [332K]  libldap_r-2.4.so.2
    │   ├── [139K]  liblzma.so.5
    │   ├── [226K]  libnettle.so.6
    │   ├── [ 38K]  libogg.so.0
    │   ├── [ 17M]  libopenrct2.so
    │   ├── [450K]  libpcre.so.3
    │   ├── [153K]  libpng12.so.0
    │   ├── [518K]  libpulsecommon-8.0.so
    │   ├── [321K]  libpulse.so.0
    │   ├── [ 93K]  libroken.so.18
    │   ├── [117K]  librtmp.so.1
    │   ├── [111K]  libsasl2.so.2
    │   ├── [1.1M]  libSDL2-2.0.so.0
    │   ├── [135K]  libselinux.so.1
    │   ├── [408K]  libsndfile.so.1
    │   ├── [ 59K]  libsndio.so.6.1
    │   ├── [ 79K]  libspeexdsp.so.1
    │   ├── [858K]  libsqlite3.so.0
    │   ├── [434K]  libssl.so.1.0.0
    │   ├── [537K]  libsystemd.so.0
    │   ├── [ 78K]  libtasn1.so.6
    │   ├── [677K]  libvorbisenc.so.2
    │   ├── [176K]  libvorbis.so.0
    │   ├── [ 63K]  libwayland-client.so.0
    │   ├── [ 34K]  libwayland-cursor.so.0
    │   ├── [9.2K]  libwayland-egl.so.1
    │   ├── [166K]  libwind.so.0
    │   ├── [ 38K]  libwrap.so.0
    │   ├── [ 17K]  libXau.so.6
    │   ├── [ 43K]  libXcursor.so.1
    │   ├── [ 26K]  libXdmcp.so.6
    │   ├── [ 76K]  libXext.so.6
    │   ├── [ 26K]  libXfixes.so.3
    │   ├── [ 13K]  libXinerama.so.1
    │   ├── [ 67K]  libXi.so.6
    │   ├── [256K]  libxkbcommon.so.0
    │   ├── [ 47K]  libXrandr.so.2
    │   ├── [ 43K]  libXrender.so.1
    │   ├── [ 18K]  libXss.so.1
    │   ├── [ 26K]  libXxf86vm.so.1
    │   └── [ 79K]  libzip.so.4
    └── [   0]  share
        ├── [   0]  applications
        │   └── [  92]  openrct2.desktop
        ├── [   0]  doc
        │   ├── [   0]  libasn1-8-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libasyncns0
        │   │   └── [1.3K]  copyright
        │   ├── [   0]  libbsd0
        │   │   └── [ 23K]  copyright
        │   ├── [   0]  libcurl3
        │   │   └── [ 11K]  copyright
        │   ├── [   0]  libdbus-1-3
        │   │   └── [ 19K]  copyright
        │   ├── [   0]  libffi6
        │   │   └── [3.4K]  copyright
        │   ├── [   0]  libflac8
        │   │   └── [8.7K]  copyright
        │   ├── [   0]  libgcrypt20
        │   │   └── [ 15K]  copyright
        │   ├── [   0]  libgmp10
        │   │   └── [2.0K]  copyright
        │   ├── [   0]  libgnutls30
        │   │   └── [ 21K]  copyright
        │   ├── [   0]  libgssapi3-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libgssapi-krb5-2
        │   │   └── [ 57K]  copyright
        │   ├── [   0]  libhcrypto4-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libheimbase1-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libheimntlm0-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libhx509-5-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libidn11
        │   │   └── [ 17K]  copyright
        │   ├── [   0]  libjansson4
        │   │   └── [1.8K]  copyright
        │   ├── [   0]  libjson-c2
        │   │   └── [1.9K]  copyright
        │   ├── [   0]  libk5crypto3
        │   │   └── [ 57K]  copyright
        │   ├── [   0]  libkrb5-26-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libkrb5-3
        │   │   └── [ 57K]  copyright
        │   ├── [   0]  libkrb5support0
        │   │   └── [ 57K]  copyright
        │   ├── [   0]  libldap-2.4-2
        │   │   └── [ 20K]  copyright
        │   ├── [   0]  liblzma5
        │   │   └── [ 15K]  copyright
        │   ├── [   0]  libnettle6
        │   │   └── [ 13K]  copyright
        │   ├── [   0]  libogg0
        │   │   └── [3.6K]  copyright
        │   ├── [   0]  libpcre3
        │   │   └── [2.7K]  copyright
        │   ├── [   0]  libpng12-0
        │   │   └── [4.6K]  copyright
        │   ├── [   0]  libpulse0
        │   │   └── [ 21K]  copyright
        │   ├── [   0]  libroken18-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  librtmp1
        │   │   └── [2.7K]  copyright
        │   ├── [   0]  libsasl2-2
        │   │   └── [3.0K]  copyright
        │   ├── [   0]  libsdl2-2.0-0
        │   │   └── [ 19K]  copyright
        │   ├── [   0]  libselinux1
        │   │   └── [3.1K]  copyright
        │   ├── [   0]  libsndfile1
        │   │   └── [4.5K]  copyright
        │   ├── [   0]  libsndio6.1
        │   │   └── [2.0K]  copyright
        │   ├── [   0]  libspeexdsp1
        │   │   └── [2.0K]  copyright
        │   ├── [   0]  libsqlite3-0
        │   │   └── [1.2K]  copyright
        │   ├── [   0]  libssl1.0.0
        │   │   └── [6.4K]  copyright
        │   ├── [   0]  libsystemd0
        │   │   └── [7.6K]  copyright
        │   ├── [   0]  libtasn1-6
        │   │   └── [3.3K]  copyright
        │   ├── [   0]  libvorbis0a
        │   │   └── [2.2K]  copyright
        │   ├── [   0]  libvorbisenc2
        │   │   └── [2.2K]  copyright
        │   ├── [   0]  libwayland-client0
        │   │   └── [1.6K]  copyright
        │   ├── [   0]  libwayland-cursor0
        │   │   └── [1.6K]  copyright
        │   ├── [   0]  libwayland-egl1-mesa
        │   │   └── [7.2K]  copyright
        │   ├── [   0]  libwind0-heimdal
        │   │   └── [8.0K]  copyright
        │   ├── [   0]  libwrap0
        │   │   └── [1.3K]  copyright
        │   ├── [   0]  libxau6
        │   │   └── [1.2K]  copyright
        │   ├── [   0]  libxcursor1
        │   │   └── [2.7K]  copyright
        │   ├── [   0]  libxdmcp6
        │   │   └── [1.2K]  copyright
        │   ├── [   0]  libxext6
        │   │   └── [ 10K]  copyright
        │   ├── [   0]  libxfixes3
        │   │   └── [2.3K]  copyright
        │   ├── [   0]  libxi6
        │   │   └── [4.3K]  copyright
        │   ├── [   0]  libxinerama1
        │   │   └── [3.6K]  copyright
        │   ├── [   0]  libxkbcommon0
        │   │   └── [3.5K]  copyright
        │   ├── [   0]  libxrandr2
        │   │   └── [3.6K]  copyright
        │   ├── [   0]  libxrender1
        │   │   └── [2.2K]  copyright
        │   ├── [   0]  libxss1
        │   │   └── [2.6K]  copyright
        │   ├── [   0]  libxxf86vm1
        │   │   └── [1.3K]  copyright
        │   └── [   0]  libzip4
        │       └── [2.1K]  copyright
        └── [   0]  icons
            └── [   0]  hicolor
                └── [   0]  256x256
                    └── [   0]  apps
                        └── [ 26K]  openrct2.png

78 directories, 178 files

[janisozaur]

Binary delta updates, e.g., for continuous builds (only download the binary diff) using AppImageUpdate

(https://github.com/AppImage/AppImageKit/wiki#for-application-developers and https://github.com/AppImage/AppImageUpdate)

This is very interesting. @TheAssassin I'm liking AppImages more and more. I will certainly look more into this whenever I have some time. Is your demo sufficitent for linuxdeployqt to enable binary diffs? Is there anything needed on the side of the host which provides the AppImages? We ship with some assets that rarely change, can this prior knowledge be utilised to further reduce diffs?

[TheAssassin]

@janisozaur you don't need to generate much, actually, our tools will just generate a single file called .zsync file which can be used to update any file to the one the .zsync file belongs to. There's no "I need to generate a binary delta file for every possible version in advance" process involved, like it's needed for other update systems. Read more on it here: https://github.com/AppImage/zsync2 https://github.com/AppImage/AppImageUpdate

What you need to provide to have our tools generate this .zsync file is so-called update information. When it is passed to appimagetool, it generates such a file, and lets you use it. Unfortunately, instead of using linuxdeployqt's -appimage option, one has to download appimagetool and pass it manually to it. But that's not really a big issue. On Travis CI, the update information can be guessed even. I'm not sure why it didn't get uploaded previously, but I'll try to fix that soon.

Edit: .zsync files are available now. You can try AppImageUpdate now with my OpenRCT2 AppImage.

[TheAssassin]

I forgot to answer your questions.

Is there anything needed on the side of the host which provides the AppImages?

That's the key feature of zsync2: All you need is an HTTP server (or whatever else system we support, e.g., GitHub releases) serving the two files (AppImage and zsync file).

We ship with some assets that rarely change, can this prior knowledge be utilised to further reduce diffs?

Not really. AppImages are merely squashfs images with an executable header. These are the compressed. But in squashfs, all files are compressed separately. Also, they're block aligned, and the block sizes of the squashfs image and the zsync file are compatible, therefore single file changes should cause zsync2 to download only the blocks that have changed, leaving the rest alone.

By the way: regarding the homepage of OpenRCT2: See https://doesmysiteneedhttps.com/.

[TheAssassin]

Oh, another point: You may visit us on IRC (#appimage on Freenode) in order to kick off the "upstream OpenRCT2 AppImage project". I can't really provide you full-time maintenance of an AppImage, but I'd rather show _you_ how to do it. That'd yield a usable result quickly, I'd say.

[janisozaur]

By the way: regarding the homepage of OpenRCT2: See https://doesmysiteneedhttps.com/.

I'm not sure what you mean. https://openrct2.io/ is the only way to reach it, and also:

$ LANG=C wget --spider http://openrct2.io/
Spider mode enabled. Check if remote file exists.
--2018-05-11 23:37:06--  http://openrct2.io/
Resolving openrct2.io (openrct2.io)... 104.27.184.176, 104.27.185.176, 2400:cb00:2048:1::681b:b8b0, ...
Connecting to openrct2.io (openrct2.io)|104.27.184.176|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://openrct2.io/ [following]
Spider mode enabled. Check if remote file exists.
--2018-05-11 23:37:06--  https://openrct2.io/
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Connecting to openrct2.io (openrct2.io)|104.27.184.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

[qwertychouskie]

openrct2.org supports HTTPS, but doesn't force it. Also, why are there two sites in the first place?

[janisozaur]

We only have one official website, https://openrct2.io/, as noted in the project description up on top here.

openrct2.org is a third-party (https://github.com/JarnoVgr/OpenRCT2.org) creation that lets us use their space to upload builds to. We got an offer from DigitalOcean for open source projects and we will probably be moving our builds there, when that gets ready.

[TheAssassin]

@janisozaur I was referring to http://openrct2.org/. They also use HTTP to distribute the files, so there's zero authenticity. That's really something you should fix.

[janisozaur]

https://github.com/JarnoVgr/OpenRCT2.org/issues/25 got filed

[janisozaur]

https://github.com/OpenRCT2/OpenRCT2/pull/7523 migrated our Ubuntu docker images from Ubuntu 16.04 to Ubuntu 18.04, our future builds will be using that as a base

[TheAssassin]

Is any of you willing to maintain an AppImage in the future? Then I could assist you with setting up the initial build scripts. As mentioned above, I can't offer to maintain it in the future, therefore I'd rather help one of you to build and understand an AppImage.

[janisozaur]

I'm up for the job, but that has lower priority than some other things right now.

[TheAssassin]

Alright. If you're too busy to do it step by step, I could perhaps write a build script that builds an AppImage on Travis CI. You already use Travis, and I think it should be possible to integrate with that.

[IntelOrca]

@TheAssassin I think if you provide the initial setup to build the AppImage we should be able to understand it from there on and continue maintaining it. Personally I think AppImage is the only thing we should provide for Linux builds.

[TheAssassin]

Working on this now in https://github.com/TheAssassin/OpenRCT2.

[tian2992]

+1 to App Image!

[TheAssassin]

I'd say, it doesn't make much more sense to try to increase backwards compatibility < Ubuntu 18.04. Let's just build an AppImage against the libraries in 18.04. You guys should just keep in mind that you should try to stay on C++17 now, ideally forever.

[IntelOrca]

You guys should just keep in mind that you should try to stay on C++17 now, ideally forever.

Why? There will always be platforms that don't support X, Y and Z so where do you draw the line? The operating system should not constrain the development tooling, this is something that annoys me about the Linux and macOS ecosystem. I'd rather bundle the entire runtime and dependencies if necessary to be able to use the latest tech.

[janisozaur]

You guys should just keep in mind that you should try to stay on C++17 now, ideally forever.

That's an odd thing to say. It's already annoying enough to support 2-year old Ubuntu version that ships obsolete on day 1, but the alternative is to update to the non-LTS releases and create more work for me, which I'm not so happy about, or just drop Ubuntu from our build array. We strive to maintain at least that, but will likely move on to the next LTS version the day it is out if there are features we would like to use*.

Let's just build an AppImage against the libraries in 18.04.

Should we expect a PR?

* We did wait long enough to make one more release and then switch, it will likely be the same next time.

[TheAssassin]

Well, do you _need_ the latest version of your programming language? I've never wanted to switch to C++17 so far for instance.

Please see https://docs.appimage.org/introduction/concepts.html#build-on-old-systems-run-on-newer-systems.

[janisozaur]

Well, do you need the latest version of your programming language?

Yes.

We also have made changes to other projects that come as packages on Ubuntu (see e.g. https://github.com/OpenRCT2/OpenRCT2/wiki/External-influences) and we are not going to backport each one of them.

[TheAssassin]

@janisozaur I'm not requesting you should revert, don't get me wrong, I just ask you to think about such upgrades twice in the future. If we build an AppImage now against what you currently established (forgetting about all the hacks I tried to somehow backport the changes to older platforms, that just didn't work), you must expect it not to work on any older systems. However, if you stay on C++17 for now, you know that the AppImage will work for anything >= Ubuntu 18.04.

CC @probonopd

[IntelOrca]

Well, do you need the latest version of your programming language?

Absolutely. Take the filesystem API for example, something that should have been in C++ since the beginning, instead we have to write implementations specific to each platform. Other good programming conventions are also desired like std::optional.

[TheAssassin]

@IntelOrca you misunderstood me, just like @janisozaur. I'm saying that this might have been a requirement _now_, but one should not upgrade in the future just for the sake of it. I guess C++17 is most likely sufficient for your project for the next years.

[janisozaur]

We will likely stay on C++17 for a while. The next standard is expected to be C++20, and probably ratified at the end of 2020, I don't expect us moving on to it at least until around that time, which would likely mean Ubuntu LTS 20.04, so in about 2 year's time – it's what we did with C++17.

[TheAssassin]

See? That's what I mean -- upgrading for the sake of upgrading. You should try to see what exactly will change, and then see how urgently you need to change. If there's no major useful change (such as filesystem), why upgrade, right?

[janisozaur]

We are discussing standard that is not ready yet. It's not worth worrying about it right now, many things will change until it is released and supported. I'm still sure there will be plenty of things that we will be interested in.

For now let's address what we have.

[janisozaur]

Can you explain how using C++17 is creating a problem for AppImage?

[TheAssassin]

Has any of you read https://docs.appimage.org/introduction/concepts.html#build-on-old-systems-run-on-newer-systems?

There is no problem for AppImage. The problem is rather that by choosing C++17, you're setting the compatibility level to distributions which _have_ it, such as Ubuntu 18.04 onwards.

Please read the section in the docs.

[TheAssassin]

I'm bringing my AppImage feature branch up to date, and will send a PR shortly. Then, we'll have an AppImage build script that builds from source and creates an AppImage which won't work < Ubuntu 18.04.

[janisozaur]

I read that part and found nothing related to C++17 or any other standard, but as C++17 was the mostly discussed issue here, I thought it was the blocking issue. Support for current LTS release of Ubuntu is enough.

[janisozaur]

It appears from your branch that wget libfuse2 are required for AppImage to work? I can add them to our base docker image, it's much faster in CI this way.

[TheAssassin]

Feel free to push to my branch, you have write access. I've sent a PR.

[jansegre]

Let me add my 2¢. From what I gather from https://docs.appimage.org/introduction/concepts.html#build-on-old-systems-run-on-newer-systems, building on other systems is considered best _practice_ because it lowers the chance for an AppImage to break on older systems. Some cases require libs to go to an exclude list, so that the AppImage can use the lib provided by the system (distro) instead of bundling it.

That means that if C++17 (or whatever version) can bundle its dependencies (mostly libstdc++ or libc++ I think), and those are not on the exclude list, there should be no obvious reason for the image not to work on older distros. I see that libstdc++.so.6 is on the exclude list, but I'm not sure what version of the lib C++17 is built with, either way libc++ is not on the exclude list, so if building with that (Clang only?), it will get bundled and most likely work on older distros.

So, theoretically an AppImage for OpenRTC2 built with libc++ should work on Ubuntu 16.04 (and probably previous versions too). This sounds great. The original link only refers to this as a best practice, probably to maximize compatibility.

So, support for older distros can be unofficial/best-effort? I really don't see what's wrong with wanting to jump to C++20 when it's out, even if older distros cannot _compile it_, they can most likely _run it_.

[janisozaur]

Building with clang does not imply using libc++, at least not on most Linuxes.

[jansegre]

I mean the other way, using libc++ requires building with Clang. I'm not really sure though.

[TheAssassin]

@jansegre well it's not as easy to "bundle everything". That document in the docs is still a bit WIP, so here's the short version:

Bundling libraries like libc or libstdc++ is not as easy as it might seem at first. Some libraries just cannot be bundled but depend on these. Graphics drivers are an example for that. If a system graphics driver links to an older version of libstdc++, you can not combine it with the newer version, as changes are really high that the ABI of the newer libstdc++ has changed in an incompatible way.

Therefore we recommend to try to target the oldest system you want to support. You chose to ignore half of the Linux users (< Ubuntu 18.04), and that's okay, it's your choice. However, we consider it an anti-pattern to always try to use the most recent versions of $stuff, unless you _really_ need to use that. Big projects such as LibreOffice or Firefox have recognized that, and target really old systems (CentOS 6 resp. 5). Their binaries work on virtually _any_ Linux released after 2011 therefore.

It's not even as hard as it might seem at first. For instance, I am using boost::filesystem in linuxdeploy, which really helps in working with the filesystem. I wouldn't want to miss it. By using it, I can safely keep using C++11, linking boost::filesystem and the dependencies statically into our official binaries. Our binaries are built on trusty, and work for almost all AppImage builds.

Developers should _really_ try to avoid breaking backwards compatibility in general. As said, it's completely your choice, but please beware of the consequences.

Again, as some don't seem to have understood it yet: AppImage does not have a problem with C++17. But your AppImage won't work for a majority of users on older systems.

Edit: This "bundle everything" stuff is being looked into by us constantly. For non-GUI applications, it's even doable as of now (we're even thinking about a linuxdeploy plugin for the purpose, see https://github.com/linuxdeploy/linuxdeploy/issues/5). But in some cases, it's just not possible. We don't recommend it even if it works, as it adds a lot of bloat. AppImages aren't supposed to ship an entire distribution.

[janisozaur]

But your AppImage won't work for a majority of users on older systems.

We won't reach an agreement here. If a user stays on an outdated system, they should not expect to get all the shiny new stuff and I feel no obligation to provide it for them, especially when it costs us limiting ourselves to old versions of software, whether compiler or libraries.

Edit: I see the point you're making regarding Firefox or LibreOffice, but I would consider them vastly more important compared to our project and they make different choices from ours, with more resources to spend on their DevOps. I don't see it as fair comparison.

[TheAssassin]

@janisozaur again: I don't care _what_ you do, as long as you are aware of the consequences. I'm not requesting any sort of backport, just trying to make you keep this in mind for the next time a new release of C++ is available.

And define "outdated". I disagree these people use outdated systems. I mean, their systems are still supported.

[pizza2004]

Personally, I feel that any time a new OS version releases that's free and someone stays on an old version past the first month of waiting for bug fixes or whatever despite being /able/ to upgrade is using outdated software and shouldn't expect to get any updates to anything. It's nice to be able to support older machines that can't run new software, like supporting a few years worth of macOS, but it's another thing entirely to do this weird nonsense of just sitting on half a decade old versions of libraries because people can't be bothered to update.

The only time I ever didn't update was when they removed RSS from the Mail app on OS X and I spent a year or two putting it off since I rarely used that machine anymore (it was my iMac and I typically used my MacBook Pro) and I used RSS in Mail basically daily. That should be even less of a problem in Linux though because an open platform means that it should be possible to add back any lost functionality yourself.

[TheAssassin]

Personally, I feel that any time a new OS version releases that's free and someone stays on an old version past the first month of waiting for bug fixes or whatever despite being /able/ to upgrade is using outdated software and shouldn't expect to get any updates to anything.

That's a quite ignorant statement. Look at what e.g., Ubuntu do. They stabilize the upgrade process, and publish it together with the first point release. That's ~5 months of waiting period. Telling these people "it's your fault" is unfair.

Then there's university labs, workspaces etc., where older, still supported distros are used. Telling these people "your own fault, you can upgrade your computer" is ignorant.

You're quite arrogant if you treat your users like that, @pizza2004.

[TheAssassin]

Anyway, all of this is off-topic, and I don't want to discuss it in here.

[probonopd]

Corporate and institutional users stay on LTS releases for a _long_ time. They want to use a mature operating system that doesn't change (and needs to be re-validated) every year.

But the users using those systems still want the latest applications.

[janisozaur]

Corporate and institutional users stay on LTS releases for a long time.

I get that and I already said it makes sense in context of projects such as Firefox or LibreOffice, but _corporate and institutional users_ are not a target of OpenRCT2, a game engine. If the project was of other nature, I would probably weigh the arguments at hand differently.

But the users using those systems still want the latest applications.

But you just said they (whether them personally or through some higher entity) deliberately stay on LTS, which means _support_, not necessarily _features_. See the example you provided: Firefox ESR ships security patches for extended periods of time, but it doesn't add new features.

They want to use a mature operating system that doesn't change

This is in direct contradiction to the previous quote.

"Still supported" does not mean "not outdated". We are free to decide that maintaining support is too burdensome for us, but as we are an open source project, users have the same freedom to backport all the new features to their outdated system so they can use our project, but it is simply too much work _for us_.

[probonopd]

Operating system (stable, LTS) != applications running on top of it (latest releases). Think Windows 7 with the latest apps. But anyhow, I take your point that you don't intend to target all users of still-supported distributions.

[IntelOrca]

For Windows we statically link all dependencies including the C++ runtime, this maximises the compatibility and does not require the user to install any redistributable. I thought that Snap or App Image would allow the same thing for Linux?

[Ryder17z]

I've seen linking to .so files but I don't know what good it is in this case. (windows dll equivalent)

[TheAssassin]

@IntelOrca if you manage to link to libstdc++ statically using your build system, it should work fine. But I suspect unless you explicitly link to an older libc at the same time, at least on Linux, there will be issues.

I could imagine you build your own compiler and libstdc++ on e.g., trusty, where an older libc is the default, link libstdc++ statically, and hope for the best.

But, again, let's ask: is it worth the effort? I was already working on that, and decided it's not. Please let's stop discussing hypothetical issues, and work on the AppImage. I've sent you a build script in a PR a couple of days ago. All you need to do is integrate it into your own CI/CD system, and maybe optimize it a bit.

[probonopd]

For Windows we statically link all dependencies including the C++ runtime, this maximises the compatibility and does not require the user to install any redistributable. I thought that Snap or App Image would allow the same thing for Linux?

For AppImage, you can bundle all libraries with the application that you can't expect to be "there" in the default installation of all still-supported Linux distributions in a recent enough version. The only rule is to compile on the oldest system (Linux distribution) you are still targeting (recommendation: target all still-supported distributions).

[TheAssassin]

So... will you ever release AppImages (even if they won't work on Ubuntu <18.04)...?

[IntelOrca]

@TheAssassin If I can get them working we will add to release and app image repo. See #7957.

[TheAssassin]

Did you manage to integrate AppImage builds in your release toolchain?

[janisozaur]

There was no effort spent on that.

[probonopd]

What would it take to make you spending effort on that @IntelOrca @Gymnasiast? :100:

[Gymnasiast]

Time, mostly. We have a limited amount of it, so we have to set priorities.

[TheAssassin]

The script still "just runs" (it's standalone), just needs to be called somewhere.

[baryluk]

Yeah, snap or AppImage would be a great for distributing via webpage. I personally like AppImage for it being dead easy and requiring zero dependencies in the host system basically, but snap could be an option too probably. The default build system should still be made to make distros and packaging easy without them tho.

[takluyver]

I've put together an initial manifest for a Flatpak - it launches, but I can't test it fully until I retrieve my RCT CD to get the game files. Would anyone else be interested in co-maintaining it?

https://github.com/flathub/flathub/pull/1694

[Gymnasiast]

@takluyver I am willing to help test if you can give me some instructions. I haven't done anything with Flatpak before, but I would still like our users to have the option - I expect it to be beneficial for quite a sizeable amount of Linux users (mainly thinking about Fedora; Ubuntu users can already use the PPA and Arch users are also served, IIRC).

[takluyver]

Yup, I'm on Fedora, and Flatpak is a pretty nice way to get up-to-date applications. The sandboxing is also at its best with games, as they generally don't need many permissions.

To test:

1. Get flatpak and flatpak-builder installed. I think they're in the repos of most major distros now - see https://flatpak.org/setup/ (that only gives instructions for flatpak, but hopefully flatpak-builder is just another package).
2. Clone the branch of my Flathub PR (https://github.com/flathub/flathub/pull/1694). I can give detailed instructions if you like, but I guess you know git well enough already. It needs a git submodule, which may involve an extra step or two.
3. Install the SDK it will build with: flatpak install org.freedesktop.Sdk//19.08
4. Build and install it: flatpak-builder --user --install build-dir io.openrct2.OpenRCT2.yaml

If that all works correctly, it should create a launcher/menu entry for OpenRCT2. You can also run it from the command line with flatpak run io.openrct2.OpenRCT2.

Apart from making sure it works, we should also figure what sandbox permissions it needs - I've guessed some common ones (see finish-args in io.openrct2.OpenRCT2.yaml), but I may well have missed some it needs or included some it doesn't. It probably shouldn't have R/W access to the whole home directory - we can either make that read-only or remove it altogether, depending on how getting the data files works.

Other notes:

• To remove it once you're done, run flatpak remove io.openrct2.OpenRCT2.
• The standard cache, config & data directories live in ~/.var/app/io.openrct2.OpenRCT2 (XDG_* environment variables are set for the application by Flatpak).
• Inside the application container, the application & bundled library files are under /app.

[takluyver]

OpenRCT2 is now on Flathub. Currently 0.2.6, but @rmnvgr has already made a PR to update it to 0.3.0. If anyone else wants to contribute, the recipe is at https://github.com/flathub/io.openrct2.OpenRCT2

[felipehw]

@takluyver

I tried to play without Flatpak and fail.
Now, with Flatpak, everything works here :D
Thanks!