Osu: Linux distribution via distribution agnostic format

[lionirdeadman]

It would be nice if osu were distributed on Linux through one of the many distro agnostic formats such as Snap, Flatpak or AppImage.

(Reposting this from https://github.com/ppy/osu-framework/issues/1650, I created the issue on the wrong repository, sorry about that.)

[probonopd]

Providing an AppImage would have, among others, these advantages:

• Applications packaged as an AppImage can run on many distributions (including Ubuntu, Fedora, openSUSE, CentOS, elementaryOS, Linux Mint, and others)
• One app = one file = super simple for users: just download one AppImage file, make it executable, and run
• No unpacking or installation necessary
• No root needed
• No system libraries changed
• Works out of the box, no installation of runtimes needed
• Optional desktop integration with appimaged
• Optional binary delta updates, e.g., for continuous builds (only download the binary diff) using AppImageUpdate
• Can optionally GPG2-sign your AppImages (inside the file)
• Works on Live ISOs
• Can use the same AppImages when dual-booting multiple distributions
• Can be listed in the AppImageHub central directory of available AppImages
• Can double as a self-extracting compressed archive with the --appimage-extract parameter

Here is an overview of projects that are already distributing upstream-provided, official AppImages.

If you have questions, AppImage developers are on #AppImage on irc.freenode.net.

[peppy]

Is it feasible to create AppImages from a non-linux system? If someone makes me a script to do this, I'm happy to manually create them for each release for the time being.

script should

• run on macOS
• create a signed package from the output of dotnet publish

for reference, this is what i'm using for mac builds currently:

cd ~/Projects/osu
dotnet publish -f netcoreapp2.1 -r osx-x64 osu.Desktop --configuration Release
rm -r /Applications/osu\!.app/Contents/MacOS/*
cp -r osu.Desktop/bin/Release/netcoreapp2.1/osx-x64/publish/* /Applications/osu\!.app/Contents/MacOS
codesign --deep --force --verify --verbose --sign "$DEPLOY_CERT" /Applications/osu\!.app
ditto -ck --rsrc --keepParent --sequesterRsrc /Applications/osu\!.app osu\!.app.zip

# check code signing
spctl --assess -vvvv /Applications/osu\!.app

[probonopd]

Is it feasible to create AppImages from a non-linux system?

While this is feasible (e.g., electron-builder is producing AppImages for Linux on macOS systems, I am not aware of tools to do this for non-Electron applications. I also think this wouldn't be worth the effort, since you'll need a Linux system for _testing_ anyway.
Also, using a server-based build process (on something like Travis CI) is generally preferable to local developer machine builds.

[peppy]

Builds must be able to be produced on any platform. This is a personal requirement.

[user5145]

If wine is ok then there is a project which wraps games into wine and flatpak, It's winepak/applications. Osu's an issue number 25. Games can be installed by e.g. gnome-software, if repository was added.

there is a good comparison.
https://askubuntu.com/questions/866511/what-are-the-differences-between-snaps-appimage-flatpak-and-others

And if I can say i prefer flatpak, like this flatpak > snap > appimage, and it seems to be the same for major linux distributions, except ubuntu. Flatpak is supported in gnome-software and chromium OS and is being tested with steam. Appimage has probably never been liked :<

I'm just porting an application to flatpak and it's really easy. it's basically copy-paste of a game and necessary libraries and it just works.

And thanks for this beautiful game :)
Edit: grammar :(

[peppy]

This is lazer, which does not require wine.

[user5145]

^^ It makes it even easier.
Thanks for the info, I haven't known about lazer.

edit: typo

[lionirdeadman]

I'll try building a flatpak since I don't think there'll be the possibility of any of those package managers being built on windows. I'll post updates here. I hope you don't mind, @peppy

[peppy]

Go for it, just don't publish it on a package manager. Optimally we're looking to have a packaging script PR'd so we can do this ourselves.

[lionirdeadman]

Yes, I'd PR that script. We'd probably build it with Travis-CI since none of the formats mentioned here can easily be built on all platform. It also makes it easier for you.

[probonopd]

If you build on Travis CI then it is easy to build an AppImage as well.

[peppy]

As food for thought, appveyor also supports linux runs too.

[lionirdeadman]

I will consider AppVeyor but will prioritise Travis-CI since that's something that seems to be more standard.

Also, yeah, Travis CI could also easily build an AppImage but I personally prefer Flatpak because of the repositories, it makes it easier to discover software.

Since I decided this wouldn't be used in favour of flatpak bundles until osu becomes more stable, it makes that argument a bit less valuable other than making it easier in the future to publish on Flathub.

I'm already working on a flatpak with someone (they're doing most of the work honestly) so let's not make the same effort in different ways. Thanks.

(Sidenote: I don't know if it was an accident but you edited probonopd's comment)

[peppy]

The host is less important.

Also for what it's worth, you will likely want to commit this to ppy/osu-deploy

Your PR should not be dependent on any CI engine. It should just create the packages.

[lionirdeadman]

That script will not be dependent on any CI engine then, I'll leave that up to you.

I was simply mentioning CIs because you can't create a flatpak on Windows or Mac (although it could be done through WSL or simply a VM but that just sounds like a pain).

[peppy]

See osu-deploy. There's already platform specific code for macOS.

[lionirdeadman]

I think you're misunderstanding.

Flatpak does not work on MacOS or Windows, it is Linux exclusive and can only be built on Linux. My script will not use any of the code in that repository.

I will simply make a PR to that repository to add the files required for flatpaks. (YML for build, .desktop for desktop shortcut, icon and the Appstream.xml to allow viewing of general package information inside gnome-software or similar package managers)

[peppy]

Yes, so add that to osu-deploy in a similar way to the macOS specific deploy (which only runs on macOS).

[lionirdeadman]

Ah, sorry. My bad, I'll look at it. I feel dumb, haha.

[probonopd]

Also, yeah, Travis CI could also easily build an AppImage but I personally prefer Flatpak because of the repositories, it makes it easier to discover software.

Why not have both :-) they address different needs.

[lionirdeadman]

Also, yeah, Travis CI could also easily build an AppImage but I personally prefer Flatpak because of the repositories, it makes it easier to discover software.

Why not have both :-) they address different needs.

Having two builds for linux would be both redundant, require more maintenance and would be counter-productive to saving developers' time.

[probonopd]

Having two builds for linux would be both redundant,

No, as they serve different goals. AppImages are single-file portable apps whereas the others are not.

require more maintenance and would be counter-productive to saving developers' time.

That's true though ;-)

[sr229]

@probonopd we can always try Flatpak and Snaps

[refi64]

Don't Flatpak and Snap overlap a lot more than Flatpak and AppImage?

As a side note, I'm currently working on an osu!lazer Flatpak, so Coming soon to a repository near you.

[probonopd]

Don't Flatpak and Snap overlap a lot more than Flatpak and AppImage?

Indeed. AppImage uses a no-repository, no-install, portable single-file approach.

[sr229]

@probonopd @kirbyfan64 Flatpaks and Snaps can come in a no-repository distribution FYI, though AppImage is still winning on this, tbf

[lionirdeadman]

The no-repository model of AppImage is also it's biggest disadvantage. The advantage of Flatpak (can't talk for snap) is that the environment is fully-reproduceable since they often rely and run on standard platforms which is not something you can say about AppImage which will undoubtedly mean distro-specific bugs..

In any case, I think one format is enough, let's not start a format debate and feel free to PR your own format if @peppy wants more than one. Thanks @kirbyfan64 :heart:

[sr229]

I can work on the AppImage distribution, since its easy to do.

Flatpaks might be needed to be uploaded to FlatHub.

[lionirdeadman]

Whether we distribute through Flathub or not will be @peppy's decision in the end, we can always just do flatpak-bundles.

[sr229]

Setting this on my TODO right now: you can track it over at sr229/osu#1

[peppy]

As stated above, please do not add travis as it is unnecessary.

[sr229]

Initial PR now done on ppy/osu-deploy#1 - Will now package osu! into a GPG-signed AppImage (current cert will not work)

[quantumgc]

What's the status on ppy/osu-deploy#1? I notice it's been left untouched for quite some time.

[peppy]

Hasn't been tested yet.

[probonopd]

Can the AppImage be downloaded somewhere for testing? Thanks.

[peppy]

The idea would be running the osu.Deploy application to generate it to test.

[SlawekNowy]

Forked osu.Deploy and modified it to create AppImages (still requires few files for AppDir though).

https://github.com/SlawekNowy/osu-deploy

Post those on osu or osu-deploy repo?

On a diffrent note, AppInage accepts only GPG keys.

[peppy]

Rather than forking, are you able to PR your changes? Also please check https://github.com/ppy/osu-deploy/pull/1, which is supposedly doing the same thing.

[SlawekNowy]

Yes, I based it on sr229's work. It's mostly the same, only that I changed last lines and added AppDir stuff.

Appimage now wants env variable ARCH to be set, and --comp argument doesn't work right now.

And idk if I want to generate AppDir stuff from code or supply one with osu-deploy.

[sr229]

@SlawekNowy recommend you merge your work with mine and let's work together to get osu.Deploy to work properly on AppImage should you wish.

[SlawekNowy]

@sr229 I sent pull request to your fork of osu-deploy.

[jai-x]

Just to provide an additional opinion, I disagree with using an AppImage or any other pre-packaged format for distributing on Linux.

The normal distrubution format for Linux is simply a .tar.gz tarball containing the program executable and any additional config files and program directories. This allows for better integrations with Linux package managers and using the system package manager should be the preferred method of keeping the program updated.

Supplying shared object program libraries with the program should also not be a responsibility of the program package and should delegated to the system package manager. In the case of libbass and other non-standard libraries, it may be acceptable to bundle them.

[lionirdeadman]

The normal distrubution format for Linux is simply a .tar.gz tarball containing the program executable and any additional config files and program directories.

Which doesn't work on all linux distributions, doesn't integrate in the desktop environment without effort, doesn't auto-update and is just kinda bad. There's a reason people wanna move away from that.

This allows for better integrations with Linux package managers and using the system package manager should be the preferred method of keeping the program updated.

Should a game be part of the system though? Should osu! have to wait 10years for Debian to update it? No, that's why these agnostic formats were created, one source, updated regardless of system, works everywhere. Although... AppImage doesn't provide that which is really a bummer.

Supplying shared object program libraries with the program should also not be a responsibility of the program package and should delegated to the system package manager.

In the case of AppImage, it is osu!'s responsability to update them but in the case of Flatpak or Snap, most libraries are provided by their respective runtime which means osu! developers don't have to care about them.

[jai-x]

@lionirdeadman I understand the relative advantages and disadvantages that both approaches have. I was just stating my preference for the record for how I think osu!lazer should be packaged for Linux systems. I don't expect anyone to change their mind based on my comments.

[probonopd]

Should a game be part of the system though? Should osu! have to wait 10years for Debian to update it? No, that's why these agnostic formats were created, one source, updated regardless of system, works everywhere. Although... AppImage doesn't provide that which is really a bummer.

AppImage is a self-mounting disk image that executes whatever the developer has put inside it. It can't magically fix the issues the "Linux desktop" is suffering from (fragmentation, lack of clear long-term compatibility guarantees). A well-built AppImage can run on all current mainstream distributions; if that is not sufficient one would have to bundle everything (down to glibc and ld-linux) as a private copy and use nothing (besides the kernel and Xorg) from the system. Can be done with AppImage, too - although it's generally not recommended since it may be perceived as "bloated".

[sr229]

I want to disagree with some of the contradictions here, while I agree library management should not be our problem, a game should NEVER be part of the system. Take a look at Steam's game handling for example. The only time you want it to be part of anything of the system if its from a package manager, or its just an updater for the game (which makes more sense). I also want to disagree using .tar.gz as a de facto method since that's not VERSIONED properly by the package manager because it was never part of the database to update the thing.

[lionirdeadman]

It can't magically fix the issues the "Linux desktop" is suffering from (fragmentation, lack of clear long-term compatibility guarantees).

I think trying to make the free desktop non-breaking is a respectable goal but I believe that trying to make the free desktop standardized in the ways needed for platform agnosticism would be near impossible or restrict distributions to do what they want because of other people's decisions unless they want to forfeit applications being usable on their desktop which would simply be horrible. It would no longer give the freedoms that it once gave.

A well-built AppImage can run on all current mainstream distributions; if that is not sufficient one would have to bundle everything (down to glibc and ld-linux) as a private copy and use nothing (besides the kernel and Xorg) from the system. Can be done with AppImage, too - although it's generally not recommended since it may be perceived as "bloated".

And that's exactly why I think it's the wrong way of doing things.

If I truly want AppImage to be available for everything like it wants to be, I need to bundle the whole desktop basically which is just... a terrible idea on the storage-side for one since there is no deduplication of any kind between AppImages and even more so on the security-side, I really don't want to leave a whole desktop stack in the hands of application developers no matter how good they are, it's just much harder that way and will result in problems.

[Joehuu]

Resolved in https://github.com/ppy/osu-deploy/pull/36.