Appimagekit: Is AppImage a reasonable substitute for executable Docker containers?

[anderspitman]

I'm working on a web application where the server executes 3rd party tools and pipes the output to the client. The tools vary from simple statically linked binaries to ancient Perl contraptions with lots of messy dependencies. My desire is to isolate each tool into an environment it can be executed from, without having to install the runtimes and dependencies for each tool globally. I first tried Docker, using commands such as this:

cat datafile | docker run --rm -i anderspitman/samtools

However, I found that the performance of Docker isn't great in these situations (especially using udocker, which is a requirement since the final deployment will be on a user account that doesn't have root access). Rather than trying to debug something as complex as Docker, I realized my use case is much simpler. I don't need all the networking and other things that Docker provides; just the dependency isolation.

That brought me to AppImage. I spent a few hours yesterday playing with it, and managed to get a simple image built for one of my tools.

My current problem is that apparently AppImage does not automatically handle copying over dependent dynamic libraries, and magically making sure everything points at the right .so files. There does appear to be some documentation for working around that, but at this point I'm faced with the decision of debugging AppImage, or going back to debugging Docker performance, since with Docker you get the full-system isolation (including dynamic libs) for free.

Before moving forward I wanted to ask here and get opinions on whether I'm trying to fit a square peg in a round hole by using AppImage here, and maybe see if anyone could suggest alternative tools.

[Calinou]

What about using Exodus? It should work well with command-line tools at least.

[anderspitman]

Wow exodus is awesome! And very easy to use. I ran it on my samtools binary and it just worked. So it seems like the way to go would be to run exodus on a dynamically linked binary, then package the exodus bundle into an AppImage for easy distribution (and perhaps versioning)?

[Calinou]

You could compress and distribute the binary generated by Exodus directly, but if you want better desktop integration, you'll want to pack it into an AppImage (which may or may not work, I haven't tested).

[anderspitman]

I'm upgrading my opinion of Exodus from "really awesome" to "complete black magic". Not only is it trivial to integrate with AppImage (make tarball with exodus -t, extract to "exodus" dir, mv exodus into AppDir, symlink exodus/bin/binaryName to AppDir/AppRun), I also just threw my system's node binary at it and it packaged it no problem.

There does seem to be ~200ms of startup delay for the binaries. Not sure if there's anything that can be done about that.

[Calinou]

There does seem to be ~200ms of startup delay for the binaries. Not sure if there's anything that can be done about that.

You should install a lightweight libc such as musl to build optimized launchers; see Optional/Recommended Dependencies in the Exodus README.

[anderspitman]

Yeah the delay is with using musl.

[anderspitman]

So when running exodus/bin/binaryName directly, I can't perceive any delay. Apparently it's coming from AppImage.

[TheAssassin]

since with Docker you get the full-system isolation (including dynamic libs) for free.

Though, that's not advertised as secure for good reasons. When tools are run as root, there is little to no protection for tools to escape the containers. Gets better with non-root users, but it's not overly secure either.

But let's leave the security bit out for the moment. AppImages are not sandboxed or anything at all by default.

AppImages are self-contained bundles of tools following the rule one app = one file. That concept fits your demands quite well, I'd say: it's easy to handle a single file (not unique to AppImage, though; I guess it's not very much more complex to move a Docker image from A to B, a few more steps involved but overall still the same set of steps).

My current problem is that apparently AppImage does not automatically handle copying over dependent dynamic libraries, and magically making sure everything points at the right .so files.

Well, you didn't seem to read the documentation, it seems. There's tools that inspect your binaries and copy all libraries needed into an AppDir, most notably linuxdeploy. That works best for native binaries, but we also maintain plugins to bundle non-binary dependencies such as Qt plugins or even dependencies for interpreted languages like Python. We can't provide plugins for every possible language yet, though. You mentioned Perl for instance, there's no plugin for that yet (mostly because noone really uses that any more...). But we'd be very happy about contributors who want to write new plugins and will try to help them as good as possible.

The documentation is linked to from our homepage: https://docs.appimage.org/
There's a large packaging guide with a dedicated section on how to build and bundle software from source. If you have binaries already, check out the linuxdeploy user guide.

You should join our IRC channel #appimage on Freenode, there you'll find a large community of AppImage users and creators and also all the developers of the software.

P.S.: Didn't know about this exodus yet. Might plug in well into linuxdeploy. Writing linuxdeploy plugins is very simple since they're all standalone programs, even bash scripts can be plugins. If you're interested in this, please get in touch.

P.P.S.: Of course, running an AppImage requires some setup time. That's due to how AppImages work. You can find more information here: https://docs.appimage.org/reference/architecture.html

Edit: for many subsequent runs, it might help to extract an AppImage once and then run it over and over again that way. It's not very clean and breaks this whole "one app = one file" approach. I guess a better alternative would be to write a small manager to mount AppImages once, which should avoid any further delay. If you're interested in that, we can brainstorm a bit on the topic.

[anderspitman]

Thanks for the detailed reply @TheAssassin. I agree I'd much rather get AppImage working than use Docker. At this point, even with the slight delay, I think AppImage + exodus is the way forward for my use case.

I did read the documentation. I read the linuxdeploy user guide twice. After running linuxdeploy on my AppDir (which includes my compiled binary), and mounting the resulting image, running ldd on the final binary in /tmp/mount* shows it as still linking to hard-coded /usr/* libraries. Additionally, copying to AppImage to a different machine and attempting to run it fails due to dynamic loading. It's certainly possible I did something wrong, or I'm misunderstanding what's intending to happen when linuxdeploy runs.

If linuxdeploy isn't already doing what exodus does, I highly recommend integrating it. Since I'm going to be bundling at least several more tools this way, I'll look into writing a linuxdeploy plugin and let you know how that goes, or if I have any questions.

Thanks all!

[TheAssassin]

Please note there's no "plugin development guide" yet, so I highly recommend you to show up on IRC where I can help you. Or you open an issue over at linuxdeploy's tracker. Either way, we'll get you started.

Here's something for you to read: https://github.com/linuxdeploy/linuxdeploy/wiki/Plugin-system

[anderspitman]

@TheAssassin I'm back to working on this today. Before I go through the effort of integrating Exodus into my AppImage workflow, I want to clarify. There is currently no functionality built in to AppImageKit and/or linuxdeploy to automatically detect which .so libs a binary depends on, and bundle them into the AppImage. Is that correct? Because that's what the point of Exodus is.

[anderspitman]

So for example if I have a samtools binary somewhere on my system, which has ~15 dynamic lib dependencies, I want to be able to run something like:

linuxdeploy samtools

And have it spit out a samtools-x86.AppImage binary that includes everything I need to copy it to another machine and run it, including all of the .so files.

I'm confident I can build a plugin to do this with Exodus. I just want to make sure this doesn't already exist, since it seems like it would be a nice feature.

[TheAssassin]

Of course linuxdeploy bundles dynamically loaded shared objects. Doesn't mean that's perfect, therefore the idea to perhaps combine it with exodus. You should try it, it does what you want, you just need the right syntax. https://docs.appimage.org/packaging-guide/from-source/linuxdeploy-user-guide.html https://docs.appimage.org/packaging-guide/from-source/index.html

[anderspitman]

The reason I ask is because I did try (by following the documentation) and it didn't seem to work, as I mentioned in a previous comment. What would the proper command look like for the example I just outlined? Lets say you had a dynamically linked C++ program stored in /usr/bin/myProgram, what command would you run with linuxdeploy to bundle a static AppImage? I'm sure I just missed some important step or flag or something.

[anderspitman]

These are the steps I originally followed: https://docs.appimage.org/packaging-guide/from-source/native-binaries.html#using-linuxdeploy-for-building-appimages

[TheAssassin]

./linuxdeploy... --appdir AppDir -b /usr/bin/myProgram -d myProgram.desktop -d myProgram.{svg,png,...}, provided you write a desktop file and have a proper icon.

[anderspitman]

Glad I asked! That does indeed work (only small change was using -e instead of -b). I think my problem when I originally tried it was that I partially built the AppDir myself by copying the binary inside. Apparently when you include it via -e, linuxdeploy takes care of figuring out the dynamic dependencies. In retrospect this makes sense. Looks like there's a good chance I wont need Exodus after all.

Thanks @TheAssassin!

[TheAssassin]

JFYI It also does when you put it in into usr/bin/....