Terasology: Improving packaging of the whole game

[qwc]

Due to the recent talk at slack about packaging I thought about writing down my thoughts on packaging of terasology.
Forum discussion thread

Terasology Variants

First of all I would see 3 package variants of terasology

1. The Launcher - the universal desktop approach, easiest for beginners too
2. Latest Stable (Omega release)
3. Latest Unstable (Omega release)

2/3 Would be best for setting up servers without docker.
On Linux if the user wants to get updates through the distribution and not with some manual clicks inside the launcher. (May also be good for ppl with slow internet connections to get the latest version with the distribution update instead of when they just want to play and realize they have to download first to play)

Package signing

I think it would be good to also implement package signing so that downloaded packages can be verified.

Things to discuss here are

• pgp/gpg signing
  
  
  
  • how to achieve automatic signing with high enough security for the key
  
  
  • should multiple ppl know/have the private key?
  
  
  • can be done 'offshore', and maybe should be done on a separate system (private key security)
  
  
• md5/sha2/sha512 etc. should be standard and no problem to implement.
  
  
  
  • already implemented on terasology jenkins, md5sums.txt and sha256sums.txt are included into the artifacts
  
  

Linux distributions to target

• debian based (.deb)
  
  
  
  • [easy going, already did some tests with fpm]
  
  
  • ubuntu [already existing, got permission to use!]
  
  
  • Mint
  
  
  • alot more!
  
  
• rpm based (fedora, SuSE, etc.) (.rpm)
  
  
  
  • [problems with packaging due to not running on a rpms based distro]
  
  
• Arch Linux based (.tar.xz) [already existing in aur, got permission to use!]
  
  
  
  • [did binary packaging with fpm, AUR still open]
  
  
• gentoo based

Did I forget a distribution type with reasonable user base? O.o

@qwc is working on these.

Linux distribution independent packaging

• snap
• flatpak
• appimage [doesn't support dependencies... afaik]

@qwc is working on these also

Windows packaging

• Chocolatey?
• Installer?
• downloadable archive?
• downloadable executable?

@gianluca-nitti is working on this.

Apple MacOS

Well, here I have quite a bit of a lack of knowledge. I know there are at least four things to consider:

• .dmg Apps
• MacPorts, if that is still an approach to install software on Apple's Darwin Unix system...
• homebrew
• Apple's AppStore ... There I have fears to get the approval from the Apple guys and how that works in general...

Expert knowledge needed!
@skaldarnar put homebrew on his list.

docker

Finally the docker solution for servers and hosting platforms, provided by my docker-terasology project

• Stable
• Unstable

Maybe with predefined configurations and more customization in the future, needs to be discussed.

Important: automation

Everything mentioned must run through an automated job on a jenkins instance of the project (I'm so brash to include my own instance here).

Final words...

So far my thoughts to how the future variants could look and which package formats to target.

Human resources thoughts and claims for this tasks:
So, come what may, but I'll at minimum claim the docker kill, that was my first contribution for terasology and I'll keep that baby. ;)
I am familiar with all mentioned linux distributions and already packaged software for all of these (although it may be some time since then...) so I can do that basically, or may mentor ppl who want to do them. snap and flatpak are new to me, but I know basically what those projects want to achieve so it's not that hard to understand those formats.

On Windows - I'm almost gone, but already wrote some installers (WiX, and an other I cant remember...).
On Mac - I'm completly gone.

An own repository for the packaging scripts may be needed, at least for all those linux distros I would wish for one. One for Win and Mac and we're done.

[gianluca-nitti]

I'm a Windows and Chocolatey user and if it helps I can tell that Chocolatey packages are almost always very simple scripts which download an installer (exe or msi) from an external URL (usually from the software's official site) and execute it in "silent mode" (without displaying windows / asking for confirmation), so I'd suggest to build an installer for the launcher and publish it somewhere, perhaps on the launcher repository releases so that it can be downloaded either from a website link or from Chocolatey (building a package is quite simple).
It is also possible to create a package which actually contains the software files without downloading anything (self-contained) however I personally don't see big advantages in doing this if we consider that most Windows end-users don't use Chocolatey and thus a "normal" installer is probably important anyway.
To build the installer I'd suggest NSIS , you configure the installer with a simple scripting language and build it with makensis, which also runs on Linux (popular distributions have it in their repos) so should be easy to integrate in the Jenkins build.
Generating the Chocolatey package on Linux should be feasible too (choco is based on the .NET framework and Mono should be able to run it) but I've not tried personally.

As for Linux I'd like to mention fpm, not sure how it compares to the other solutions you suggest but I used it to generate deb and rpm packages and I found it a useful tool.

[qwc]

@gianluca-nitti Thanks for the clarification on Windows and Chocolatey.
And a second thanks for pointing out fpm, I can remember reading about it, but forgot it until now.

Regarding windows, I would gladly give away the task to a Windows crack, instead of setting all up from scratch just to build an installer. :)

[pollend]

Have you looked at Appimages. An Appimage is a single file executable for linux. It basically mounts the file like a drive and executes the application. I think it would work fairly well for the launcher assuming that most people have java installed. It's possible to throw an error if java is not installed.

http://appimage.org/

[skaldarnar]

I can put homebrew somewhere on my todo list in the hope I'll find time to look into it. Brew is the only source of applications for me running my Mac, so I cannot say anything about other packaging variants (in addition, I'm new to Mac and have been using it for a few month now).
In other notes: we had the idea of bundling a JRE with the game (and/or the launcher) to make sure everything is available to run the game properly. Can we make use of package managers and package repositories to model dependencies without bundling them (at least on Linux)?

[gianluca-nitti]

Can we make use of package managers and package repositories to model dependencies without bundling them (at least on Linux)?

+1 for this, IMHO bundling increases the size of the package without adding much real benefits. It can be done on Windows too, Chocolatey supports dependency management and we could make the Terasology package dependent on JRE 8+. The installer I'm working on can detect the JRE, so users which just download the installer's exe without using Chocolatey would be asked to download it from the official Java site before proceeding (screenshots here).

[qwc]

Participation, YAY! :)

Can we make use of package managers and package repositories to model dependencies without bundling them (at least on Linux)?

That is exactly, what I were aiming for!
I've currently the linux packaging for all major distributions on my agenda. And of course improvement of the docker image.

@pollend Thanks for adding appimage to the list! Will try to include that aswell.
@skaldarnar Yay! A Mac user, thanks for taking over at the Mac part. ;D

My goal would be to NEVER bundle the JRE, but adding always a dependency.
On Linux that's easy.
On Windows it has to be checked, thanks to @gianluca-nitti we'll have a nice installer doing this.
On Mac, it's still unix so I can think of possibilities but in the end regarding the Apple way, I have no idea.

[gianluca-nitti]

I have another consideration, I'm not sure if I should post it here, on the forum thread or on my installer's PR. Writing here since I think this can be relevant for every platform.

As @qwc says in the opening, for desktop clients the launcher is what should be packaged for desktop clients (opposed to headless servers) since it allows to switch between versions easily. I just found out, however, that the launcher has an auto-update system which is used to self-update the launcher files. However, in most setups these would probably be put in system-wide locations (e.g. I imagine something like /opt in Linux, and my Windows installer currently uses %programfiles% which expands to C:\Program Files; no idea about macOS) which are usually read-only for a normal user; thus, the self-update wouldn't work. In my opinion the possible solutions are:

• Disable the self-update in the launcher when it is built for packaging (i.e. for Linux packages or for the Windows installer) while leaving it enabled when building zip/tar distributions which the user would likely put somewhere he has write access to. The packaged versions could simply show a message suggesting the user to update through apt/pacman/chocolatey/whatever or to download the new installer and run it; after all, probably launcher updates shouldn't be very frequent.
• Keep the self-updater and somehow ask for the superuser rights when it needs to run (which would require the user to enter the password); there is a way to do it in Windows (a separate small exe with a special manifest, if I recall correctly) and there are various options on Linux like GKSu and PolKit.

The second one feels weird in my opinion, updating a package is usually done by package managers or installers.

[qwc]

@gianluca-nitti Good point.
I think I would go for 1. because, as you already wrote, the package update is managed through apt/pacman/emerge/yum/yast/whatever. And for the non packagemanager variants just enabling the autoupdate method.

[skaldarnar]

Disable the self-update in the launcher when it is built for packaging [...]

@gianluca-nitti That sounds sensible, although I think it is more feasible for package managers than for standard Windows installations. You usually end up with users having pretty old versions of the launcher because they think they don't need a newer release right now, and then they run into problems. We definitely should point them to the updated version on startup, if present.

@qwc The launcher has some "special" dependencies to take into account. The launcher does not only require a specific Java version, but also the JavaFX runtime lib. It is bundled with Oracle JRE, but comes separate for OpenJDK (see https://www.archlinux.org/packages/extra/i686/java-openjfx/).

[qwc]

My ongoing work can be found at https://github.com/qwc/terasology-packaging
These scripts need docker images I am currently providing manually, I'll include the Dockerfiles in the repo when they are stable. The images bring everything to package the game, multiple images are needed because it looks like I can't do everything with *buntu due to the background checking of rpm. That's also a reason the images are not finalized.
The images will be able to be used as a jenkins slave (I've added a headless jre inside) and I am already providing one of these docker images as a jenkins slave to my instance.

[qwc]

Drawback on fedora(rpms): openjfx is currently not included into the standard repo, so I'll just package the game for this distro, leaving the launcher aside until this is solved somehow.

Looks like I have to put up a fedora instance for testing some time how this could be solved through another approach.

[qwc]

With the current progress (see forum thread) it's already possible to test the debian and rpm packages.
Download here: https://jenkins.mmo.to/view/Packaging/

So everyone using a debian variant (Ubuntu, Mint, etc) or Fedora may now test the packages, please report any issues to https://github.com/qwc/terasology-packaging thanks!

[qwc]

Simple debian repository is up: http://terasology.mmo.to/deb/