Appimagekit: Upstream AppImage patches for squashfuse and squashfs-tools

[TheAssassin]

We should try to send the AppImage patches to https://github.com/vasi/squashfuse and https://github.com/plougher/squashfs-tools to be able to drop these patches and not have to build and ship our own libraries. This would also make packaging for distros easier.

Patches to send:

• [x] https://github.com/AppImage/AppImageKit/blob/appimagetool/master/src/mksquashfs-mkfs-fixed-timestamp.patch
• [ ] complex squashfuse patches: https://github.com/AppImage/AppImageKit/blob/appimagetool/master/src/patch-squashfuse.sh.in

The squashfuse patches are pretty complicated, and we'll see how this will turn out.

[TheAssassin]

First patch sent to upstream: https://github.com/plougher/squashfs-tools/pull/51

[TheAssassin]

I had an idea that'd allow us to get rid of the squashfuse patches entirely. We just build squashfuse as a shared library, and bundle it. Then, appimagetool just dlopen()s this bundled library. If that doesn't succeed, we can probably assume FUSE support is not available, libfuse2 is missing, etc. (perhaps there's even some kind of errno we can check).

I'll try that before trying to get our patches into squashfuse. If the idea doesn't work, I'd first try to find out whether a lib can find out it's dlopen()ed, and then adjust the patches in a way to be able to move the error handling (i.e., the error message) into AppImageKit's code. Ideally, the squashfuse people just accept our patch, and use dlopen() by default then.

[probonopd]

Sounds like additional work and a source of additional complexity and bugs in an area that is already working for us as of today. I don't see messing with this as a high priority. You know my mantra - never touch a running system. While _in theory_ upstreaming the patches would be nice, _in practice_ I'd not invest time into it, at least not until appimaged still copies in the desktop files all the time, appimageupdatetool can't use GitLab, etc - things that are _really bothering_ us.

[TheAssassin]

@probonopd I really want to get rid of those patches for multiple reasons. Most prominently, it'd allow us to let users link to prebuilt libraries. That's also a core requirement for distro packaging.

[probonopd]

I see, especially Debian packaging is of course a very worthy goal. (I'm just a bit clueless how this will work in practice - since we will want to use Debian oldstable, but the latest and greatest AppImage tools on it.)

[TheAssassin]

@probonopd we can still recommend our users to use the most recent versions of AppImageKit. However, stuff like libappimage should be available in the distro repos.

[probonopd]

My fear is that it will be ever-outdated. I don't feel like answering questions from users having a two year old *appimage*... I fear we might end up in the same boat as the Krita lead developer has requested Debian cease distributing his software, if we don't find a good way to handle this (e.g., by providing backports of the latest *appimage* to all still-suported versions of Debian, _without_ having it in the normal (non-backports) repo. Don't know if this is even allowable as per Debian policy.

[TheAssassin]

Well, that doesn't mean we should not remove annoyances that also prevent packaging. The point is, some software (like AppImageLauncher) is meant to be included into distros, but it can't because it depends on libappimage. We could at least allow building libappimage.

Also, even Debian announced a rolling release distro now, based on sid. They're making progress in providing a more recent distro for desktop users.

[probonopd]

Agree

[TheAssassin]

We might be able to generate a wrapper library using https://github.com/yugr/Implib.so that implicitly loads the squashfuse libraries on first call.

Thinking about it, however, we use the squashfs functionality a lot for non-FUSE operations as well, and if we'd rely on the fact that when dlopening squashfuse fails means FUSE is not available, we'd need to separate all the other squashfs related calls. Therefore I think it'll be best to implement dlopen based solution right in squashfuse, in a way that allows for some error handling (i.e., some sort of struct that keeps the references to the loaded fuse library that can be managed by users of squashfuse).

[probonopd]

I fear we are complicating it too much here - while making it easier for distributions to package our software is a noble goal, it is not one that we should sacrifice simplicity of how our software works for.

[TheAssassin]

I actually propose to move this _out_ of our maintenance scope, and have squashfuse worry about it instead of us.

[probonopd]

As long as the moving doesn't create work for us. This issue is again an example of where I think things are already working well enough that I don't see it's a good investment of time.

For me personally, I am not willing to invest time to change anything so that it works in the way Debian thinks it should work. Because in the end, this just creates work for me, and as the result, we will have outdated and unmaintained versions of our software in Debian, which will only create additional work, and we will have to tell users "use the continous builds from this website instead, because everything else is bit-rotten".

Let's focus on things we don't have yet, or things that are not working yet.

[TheAssassin]

The idea is to invest some time _now_ and move the dlopen stuff into squashfuse's codebase, given they accept it. Then, any recent squashfuse binary will work for libappimage. That's a big plus IMO.

[probonopd]

I see. Let's ask the squashfuse team then what they think about it.

[TheAssassin]

@probonopd that's the plan, but it's scheduled for tomorrow.

[azubieta]

It seems that the squashfuse-tools project where the patch was purposed is not quite active. Also I found in the tread a reference to use another project instead, see https://reproducible-builds.org/docs/system-images/#squashfs-metadata--compression

[TheAssassin]

@azubieta are you referring to squashfuse or squashfs-tools (that's two separate projects)?

https://github.com/squashfskit/squashfskit is the reference you mean, I guess. Doesn't seem much more active either, I'd say.

[bignaux]

For what it worth, when i was contributing to appimage, i sent 2 patchs on squashfuse and they were merged quickly ( https://github.com/vasi/squashfuse/pulls?q=is%3Apr+author%3Abignaux+is%3Aclosed ) .

[TheAssassin]

We'll probably be settling on squashfskit in the future anyway, they already merged the required patches.

[TheAssassin]

squashfs-tools now have their own reproducibility stuff, my PR was closed. See https://github.com/plougher/squashfs-tools/pull/51#issuecomment-519349152.

[probonopd]

We'll probably be settling on squashfskit in the future anyway, they already merged the required patches.

Well... that project seems to have its own set of issues... https://github.com/squashfskit/squashfskit/issues/4

[TheAssassin]

License wise that's fine. An author cannot insist on a specific way of attribution. They just need to put it somewhere else.

But after all that's all off topic. Seems like squashfs-tools have come up with something satisfying. So we should move on to the latest version at some point and change our code to use these new variables they set up.