Appimagekit: Payload software may hang when not intercepting SIGCHLD properly

[dimbleby]

As originally reported at neovim/neovim#8104

To reproduce

• Get neovim appimage: curl -LO https://github.com/neovim/neovim/releases/download/nightly/nvim.appimage
• chmod +x nvim.appimage
• ./nvim.appimage -u NORC
• :term (opens a terminal inside neovim)
• i (enter insert mode)
• <C-d> (send EOF)

At this point the process hangs. (If it wasn't hung, you'd be able to type <Esc>:q<CR> to exit.)

This hang doesn't happen in the unpacked appimage (./nvim.appimage --appimage-extract; ./squashfs-root/AppRun -u NORC etc)

Over in that issue the suggestion was to use strace to debug - but as noted over there, strace -f ... also hangs on the packed appimage. Since that -f is all about following child processes I wonder whether this isn't in fact somehow the same bug? But that might be too much of a leap.

[probonopd]

Does this also happen with sudo strace -f ...AppImage?

[dimbleby]

I think you're drawing attention to the sudo? Yes, the strace command still hangs when run as root.

(Does this not reproduce for you? As and when you are able to look at this, it'll be much more efficient to answer such questions yourself than by having me act as a slow terminal, and one prone to misunderstandings at that!

Of course if my steps above do not reproduce the problem for you then I'll be glad to help try and figure out the difference).

[probonopd]

Not in front of a Linux machine right now, but can you confirm that the AppImage is still mounted at the point when it hangs? Or may it prematurely have exited?

[dimbleby]

Not in front of a Linux machine right now ...

I very much appreciate your interest in this issue - but I really think that it will be a better use of both our time for the debugging to wait until you are able to drive it yourself!

But OK, let's do another round. Yes, the appimage is still mounted at the time it hangs, and has not exited:

$ mount | grep appimage
/data/dch/nvim.appimage on /tmp/.mount_nvim.ahms0iU type fuse.nvim.appimage (ro,nosuid,nodev,relatime,user_id=20170,group_id=20002)

$ ps -ef | grep appimage
dch      11749 11528  0 21:21 pts/1    00:00:00 ./nvim.appimage -u NORC
dch      11753     1  0 21:21 ?        00:00:00 ./nvim.appimage -u NORC
dch      11814 11589  0 21:21 pts/2    00:00:00 grep --color=auto appimage

[TheAssassin]

@dimbleby can you provide an strace log? It'd be interesting to see which syscall makes the AppImage hang, as the fact that the strace process hangs is not very surprising.

[dimbleby]

Once again I'd encourage interested parties to set up their own repro - or let me know that the steps I gave do not reproduce the problem for them. That'll be much faster than routing all questions through me.

But anyway: for this one, we already have what you're asking for from the original neovim issue - here.

[TheAssassin]

@dimbleby as you reported the issue here, you must expect that you are contact person for the upstream project. You should probably make the participants of the other issue contact us here if you're not personally interested in this issue, then we'll talk to them.

[dimbleby]

Very happy to handle non-trivial questions or, for instance, to help you set up your own repro in case my steps are not sufficiently explicit or just don't work for you.

Less happy - but, per this thread, willing! - to handle trivial questions that could easily be answered by y'all setting up your own repro. It just seems like it would be more efficient for both of us to skip the middle man.

[TheAssassin]

@dimbleby the problem is the limited amount of time available to maintain not only AppImageKit but all the AppImage related projects. Therefore we need to delegate some things to the reporters.

Regarding the strace output, there's no useful information in them, so yes, the only way to debug this is probably to try to reproduce this issue in an isolated environment. I'm not 100% sure what this -u NORC is supposed to do, but from the strace output it seems like it's trying to write something into the mounted AppDir, which is not possible because AppImages are mounted read-only. Perhaps nvim's error handling isn't very good in this scenario. But that's just my best guess right now. A link a link on those CLI parameters would probably help.

[dimbleby]

Yes, well, we all are busy...

-u NORC tells neovim not to load vimrc files. which typically contain a bunch of user customisation - it's standard when hitting vim or neovim problems to do this so as to rule out the problem being something specific to the user environment. See here.

I'd guess that this is unlikely to cause neovim to try and write anything - though that doesn't rule out that it might be trying to write something anyway.

However, I am very not familiar with neovim internals. (My involvement in this issue is really just as someone who has hit the problem and would like to see it fixed). If something like that does look like being the problem - we'll likely need to ask one of the maintainers.

[TheAssassin]

I see, NORC is probably harmless. I can confirm that nvim starts to hang, but I'm not sure how this is related to the AppImage runtime. I am 99.9% sure this isn't caused by the AppImage runtime, but probably because nvim tries to perform any tasks which aren't possible when being run from the AppImage (e.g., it doesn't recognize that the contents aren't writable, etc.).

When running the extracted contents (i.e., call the AppImage with --appimage-extract, then cd squashfs-root and call ./AppRun), it works perfectly fine. To make sure it's not some AppImage specific code, I exported the $APPIMAGE environment variable to simulate an AppImage runtime environment, and it still works.

My conclusion: this is not caused by the AppImage runtime, but an issue in the payload software.

P.S.: Pressing Ctrl+D apparently has the same effect on the shell, causing nvim to get stuck.

[TheAssassin]

Closing issue as invalid. Please reopen if you disagree.

[dimbleby]

I'm not seeing where neovim writes inside the appimage. Here are all of the files that are ever opened in that strace:

$ grep "open(" unnamed.txt
6384  open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
6384  open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
6384  open("/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
6384  open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
6384  open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
6384  open("/proc/self/exe", O_RDONLY)  = 3
6384  open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
6384  open("/lib/x86_64-linux-gnu/libfuse.so.2", O_RDONLY|O_CLOEXEC) = 3
6385  open("/home/vagrant/t/nvim.appimage", O_RDONLY) = 3
6385  open("/dev/null", O_RDWR)         = 5
6385  open("/tmp/.mount_nvim.aMVe9H7", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
6385  open("/dev/fuse", O_RDWR)         = 5
6386  open("/dev/null", O_RDWR)         = 8
6384  open("/tmp/.mount_nvim.aMVe9H7", O_RDONLY <unfinished ...>
6384  open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC <unfinished ...>
6384  open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC <unfinished ...>
6384  open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC <unfinished ...>
6384  open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC <unfinished ...>
6384  open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC <unfinished ...>
6384  open("/lib/x86_64-linux-gnu/libutil.so.1", O_RDONLY|O_CLOEXEC <unfinished ...>
6384  open("/lib/x86_64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC <unfinished ...>
6384  open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC <unfinished ...>

Do any of these look suspicious?

What other possibilities might "etc" cover? ie what other tasks are possible in general, possible in an unpacked appimage, but disallowed in a packed appimage?

[TheAssassin]

@dimbleby no, this doesn't look suspicious, I was just guessing since nvim tries to write to a file descriptor 4. This is not one of the standard file descriptors (0-2), and could be a file, or maybe a pipe. At first I thought it might be the terminal's input pipe to which it tries to write which gets filled up and then blocks the process, but as the AppImage's contents work when extracted, I assumed that it were writes to the contents, i.e., a file. But only the nvim devs can tell you what FD 4 is in this case.

Any sort of modification to the files in an AppImage is not possible. Files may not be written to or moved or copied or whatever.

Edit: it seems like FD 4 is /proc/self (https://gitlab.com/justinmk/gist/snippets/1725258#L93). I have no clue what nvim does there. forget what I wrote, it's wrong. Please ask the nvim team. I'm still 99% sure this is not an issue within the AppImage software.

[dimbleby]

Line 93 is before neovim has even been exec'd - that happens at line 193. If that's really what's happening, then this is surely back in appimage's court?

Edit: file descriptor 4 is later re-used, several times. However those write(4, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", 32 <unfinished ...> start happening before neovim is started - so if they're the suspect ones, then I think they're not neovim's.

[TheAssassin]

See https://gitlab.com/justinmk/gist/snippets/1725258#L103-111. An FD 4 doesn't even exist, but it's written to. I'm not sure how strace behaves when an exec*(...) call is made, I think we'd need an strace of the nvim binary in the AppImage. But still, I don't see the relation to the AppImage runtime itself. All it does is mount it and execute. It's not that complicated, actually.

IIRC the runtime actually runs the payload software as a subprocess. Perhaps this construct doesn't like you close its input pipe? That's just a guess, though, and not at all very likely...

[dimbleby]

I also don't understand what's happening with that FD 4. But do we agree that it's happening before the neovim app has even started? Shall we re-open this issue?

[TheAssassin]

I am still quite sure this doesn't have anything to do with the runtime but is a neovim specific issue. I'm missing some neovim devs here. We can reopen this issue, but I won't be able to invest much more time into this for now. Perhaps a neovim dev can explain some of the behavior.

[probonopd]

Agree, we need someone who knows what neovim is doing there.

[dimbleby]

We can certainly ask someone, but what questions do we have?

(In the trail above we've focused on "what's going on with that FD 4?" - but, while we don't have the answer, we have established that this is nothing to do with neovim.)

[TheAssassin]

we have established that this is nothing to do with neovim

No, we have not, that's the point. It's _neovim_ doing weird writes here.

[dimbleby]

As above, these writes start happening before neovim has even been exec'd.

• The first is at line 165 (also 169, 173, etc)
• neovim is exec'd at line 193

What am I missing?

[TheAssassin]

I assume the strace log is broken due to the point nvim is run as a subprocess (i.e., fork and exec).

An FD 4 doesn't even exist, but it's written to. I'm not sure how strace behaves when an exec*(...) call is made, I think we'd need an strace of the nvim binary in the AppImage.

https://github.com/AppImage/AppImageKit/issues/812#issuecomment-404121910

[dimbleby]

strace and neovim both being broken is an uneconomical explanation.

Let's knock this FD 4 thing on the head. I built an appimage containing just a 'Hello World' executable. The same mysterious writes happen. Here's the full trace. They are definitely not coming from the app.

Also of note, of course, is that strace'ing this appimage did not hang! So certainly there's something about neovim's behaviour which is contributing to the hang. But it's not those writes.

[TheAssassin]

Thanks for testing this. I did already mention this issue is very limited to neovim's specific AppImage.

I have no clue how to continue debugging this issue. We need some neovim folks, I guess.

Perhaps it's time to pick up the other idea I had:

IIRC the runtime actually runs the payload software as a subprocess. Perhaps this construct doesn't like you close its input pipe? That's just a guess, though, and not at all very likely...

[dimbleby]

Looks like I have a fix for this in neovim - neovim/neovim#8737. The issue seems to be a waitpid(-1, ...) catching SIGCHLDs that weren't really wanted - presumably stopping them from going to someone else.

I guess this is a classic interaction bug: neovim was just fine using this general waitpid() until it was put into the appimage environment - but I imagine you'll consider that appimage isn't doing much wrong either!

However, here's another very similar issue from another project: https://patchwork.kernel.org/patch/9949491/

Suggest: it would be nice to find a place to document that to be appimage-compatible, apps must not catch more SIGCHLD than they need. Maybe future travellers will find the road easier.

Thanks for your patience; I'll leave it to you to close this out when you're happy you've done all that you want to do with it.

[TheAssassin]

Big thanks for the detailed analysis, really a great job, I wouldn't ever have thought of that. Well, as you say "must not catch more than they need", it seems like a "meant well isn't done well" issue. nvim clearly tries to do more than necessary, and they apparently didn't expect to ever be run as a subprocess like the AppImage runtime calls them.

Now, I agree that this is AppImageKit's fault to some extent. I guess if we improved the way the subprocess is created (i.e., more shell-ish, because shells are processes and nvim will be a subprocess of those processes, too), we could make sure this won't happen again.

I'll leave the issue open, but will try to find a better title. Thanks for the link, too, I'll check it out.

[probonopd]

I guess if we improved the way the subprocess is created (i.e., more shell-ish, because shells are processes and nvim will be a subprocess of those processes, too), we could make sure this won't happen again.

Do we want this though? When writing runtime.c I intentionally avoided launching shells to minimize the number of running processes on the system.

[probonopd]

Suggest: it would be nice to find a place to document that to be appimage-compatible, apps must not catch more SIGCHLD than they need. Maybe future travellers will find the road easier.

Thanks @dimbleby, added a section to https://github.com/AppImage/AppImageKit/wiki/Creating-AppImages#troubleshooting

[TheAssassin]

@probonopd I don't want to start a shell, but I want to fix the tty and signal handling so that such situations can be avoided in the future. Apparently, a shell can handle the issue @dimbleby described.

Don't get me wrong, using a shell wrapper might actually solve such issues, but we don't know for sure whether that'd fix the bug, and shells are always risky (it's possible to use them securely, e.g., Python implements the sort of escaping required for that purpose).

This is not a priority at all, this specific bug is _really_ rare, I'd say.

[probonopd]

I'd say, its so rare that it should be handled in a custom AppRun script/executable for those few apps that might need it. I think it's outside the scope of runtime.c which we intentionally want to keep as small and generic as possible.

[TheAssassin]

Making the runtime behave more transparently and also robust is _really_ worth investing time. It won't add any real complexity if written properly.

[probonopd]

And it won't increase the number of running processes on the system, hence increasing clutter in the output of ps ax?

I still think the issue discussed here is an issue of the payload application and not of AppImage and can be fixed by changing the payload application. Similar to payload applications that use hardcoded paths in /usr rather than paths relative to the main binary. Things we _can_ hack around but _shouldn't_. Hacking around this sort of things was something we had to do when AppImage was targeting _users_ rather than _application developers_. That model did not work, and now that we are working with _application developers_ we can assume that they work their part, too, to make AppImages great.

[TheAssassin]

@probonopd _again_, I don't want to add a shell there but fix the subprocess handling code (stuff like TTYs and pipes and signal handling/forwarding and such stuff). And actually I don't want to do that myself, please see the labels which say "deprioritized" and "pr-welcome".

[dimbleby]

So looking back now that we know more about the broken case, I see that with 'bad' neovim we get a process tree like this:

$ ps -ef | grep appimage
dch      14288 13974  0 22:12 pts/6    00:00:00 ./nvim.appimage
dch      14289 14288  0 22:12 pts/6    00:00:00 [nvim.appimage] <defunct>
dch      14291     1  0 22:12 ?        00:00:00 ./nvim.appimage

The defunct process is the one that later caused neovim to hang: neovim does a waitpid(-1) and instead of learning about a child that it cares about it learns about one that it doesn't care about - and then never asks again.

I think that the defunct process comes from this fork().

I wonder if the fix is as simple as adding waitpid(pid, NULL, 0) here, making AppImage explicitly responsible for reaping its own child.

(I confess I'm confused about what the keepalive pipe is doing in that code!)

[TheAssassin]

@dimbleby I honestly don't know, I didn't write that code. But since the waitpid call seems like a really good idea. Would you be willing to send a PR? Then, we can simply test it with nvim, by building a new AppImage with the new runtime.

[dimbleby]

Sorry, I think I'm going to duck out now. But I hope that the above suggestion works out!

[TheAssassin]

Sure, thanks for all the information.

[probonopd]

I think that the defunct process comes from this fork().

I wonder if the fix is as simple as adding waitpid(pid, NULL, 0) here, making AppImage explicitly responsible for reaping its own child.

Let's try it then.

[dimbleby]

Well I was running out of enthusiasm last night but this morning I decided that I might as well see this through - and I'm delighted to report that the proposed fix does indeed produce an appimage that works well even with 'bad' neovim.

So I guess this was an appimage bug all along! (Though I'm also happy to report that neovim has accepted my pull request making them more resilient to it).

Pull request coming shortly.

[TheAssassin]

Heh, well, let's say both nvim and AppImageKit had their bugs. Thanks for the PR, I'll merge it after some testing.

[probonopd]

Thank you very much. Glad the solution was just one line.