Elvish: SIGPIPE should not be considered a failure for data-producers in a pipeline

[zakukai]

This is debatable perhaps, and one could argue that it's a bug in most Unix commands that currently exist: But I think SIGPIPE should not be considered an error.

The rationale is this: It's not uncommon to create pipelines that will terminate one or more of their commands with SIGPIPE - and typically this does not mean that anything has failed, but rather that one of the processes in the pipeline has stopped consuming values, and the program that was producing values for that process terminated as a result. I think it's fair to argue that these producers should not return an error status in this case, but the usual (default, in fact!) behavior is to terminate on an unhandled SIGPIPE, or catch the signal and terminate with the equivalent error code.

So for instance, this produces a SIGPIPE:
e:sort --random-sort /usr/share/dict/american-english | e:head -n 10 (Produce a list of random words)

"head" reads the first 10 lines of inputs and then terminates, breaking the pipe. "sort" then receives SIGPIPE (or possibly, equivalent information via another means, depending on implementation) next time it writes to its output. Being unable to write out more data, it terminates. This is not an "exception"al condition, but rather a fundamental part of how stream processing works in shell pipelines.

On the flip side of this argument: a "graceful" shutdown of a pipeline is not the only condition that can produce a SIGPIPE. A program could fail with SIGPIPE due to a purely internal error, or due to a connection loss, etc. This is why I consider my argument "debatable" and say "it could be considered a bug in the programs called by the shell" - If the SIGPIPE occurs in a scenario where it should not be treated as an error, then arguably "e:sort" and so on should not terminate with a non-zero exit code. I think it's a fair argument that people should simply recognize this and capture errors or use try/catch when running a pipeline that could reasonably be expected to SIGPIPE. But it's very typical for SIGPIPE to simply indicate that a pipeline has shut down. I don't think there is a set of criteria that can be applied to reliably distinguish between a "pipeline SIGPIPE" and an "internal error SIGPIPE" - the sequence in which processes terminate isn't a reliable indicator because SIGPIPE is triggered by the consumer closing its input, which could happen before termination - and if we said "SIGPIPEs aren't exceptions if they're generated by producers in a pipeline" there's always the chance that we're suppressing some true internal failure.

(I think treating process exit codes as exceptions is a good idea, though a challenging one to resolve against a tradition in which exit codes mostly don't matter...)

[krader1961]

As I wrote in issue #951:

The other example involving an elvish block on the pipe LHS and an external comment on the RHS should definitely be changed so the block terminates when the pipe is no longer writable. The devil is in the details. Given the design of Elvish it should presumably involve the echo throwing an exception.

The key question is whether a SIGPIPE that results from executing a builtin such as echo should silently terminate the enclosing block of code or result in a visible exception. That most external commands simply terminate when SIGPIPE occurs is not a strong argument that elvish builtins should behave the same way. Especially since they are builtins and we probably don't want the entire elvish session to terminate on receipt of SIGPIPE. Certainly not if it is an interactive session. It seems to me that given the design of Elvish throwing an exception that the user can trap is preferable. If the program doesn't trap the exception, and it's not an interactive session, that will have the effect of terminating the elvish process; albeit with an exception written to stderr.

[zakukai]

I don't know why you're talking about builtins. I even prefixed "sort" and "head " with "e:" so there would be no question in my example that I'm talking about external commands. The question is strictly whether the shell should raise an exception for this condition, which IMO is a very common, non-failure case in existing shell utilities. Was I not clear about the situation I'm describing? Let me try again:

If you have a Unix pipeline like this:
cmd1 | cmd2
The data exchange between the two processes is almost unidirectional, from cmd1 to cmd2. But in fact some information from cmd2 does make its way back to cmd1: if cmd2 is not consuming the data fast enough, the pipe buffer will fill up. If cmd2 closes the pipe, cmd1 will see that the pipe is broken and can respond by shutting itself down.

And SIGPIPE actually makes this behavior the default, effectively: because if a program tries to write to a pipe whose read end is closed, it will receive SIGPIPE and terminate.

The issue here is that this default behavior is found in all sorts of Unix utilities, and it's not an error condition. But when you run a pipeline like that in elvish, when the pipe breaks and the data producer terminates with SIGPIPE, elvish sees that the program exited with a non-zero result code, and raises an exception:

> find ~ | head -n 2
/home/tetsujin
/home/tetsujin/quarter_home
Exception: find killed by signal broken pipe
[tty], line 1: find ~ | head -n 2

While I think it makes sense to treat non-zero exit codes from external programs as exceptions, SIGPIPE is an exception, IMO, because it usually does not indicate a failure. I understand this creates some disparity in the handling of exit codes, but overall I think it probably makes more sense, when an external tool terminates with SIGPIPE, to not treat it as an error.

[krader1961]

I don't know why you're talking about builtins.

Because how to handle this for a builtin, or function, that produces output into a pipe where the RHS terminates before consuming all the output is the more difficult question.

How to handle an external command that terminates due to a SIGPIPE is also far from obvious given the design of Elvish. When an external command is killed by SIGTERM it results in elvish reporting an exception. Bash silently ignores it. Fish also reports the equivalent of the elvish exception.

SIGPIPE is an exception, IMO, because it usually does not indicate a failure.

Actually, for programs where that is true it is generally expected that the program capture the SIGPIPE and exit with a zero status if it really doesn't matter. It's not obvious that elvish should special-case programs that don't do that. Why is SIGPIPE special but SIGTERM not?

[zakukai]

As far as builtins and user-defined functions are concerned: usually the distinction would be whether the code in question is functional in nature, or procedural. If it's functional in nature, the correct answer is to simply stop running when there's a pipe break, because it's just generating data for consumption by that next process in the pipeline. The condition should be detectable, perhaps, but it should not be treated as an error. Builtins like "range", "splits", "joins", "keys", etc. would fall into this category IMO. If they're piping to something that doesn't want the whole result, it shouldn't be an error.

If the code is procedural in nature, the correct answer is usually to keep going, because the side-effects of the procedure are probably of at least equal interest to the output stream. Since the shell can't be expected to discern between procedural and functional code, the implementer of the function must have the power to choose which behavior suits their program - for instance, by ignoring I/O failures and continuing the work.

So, OK, I can see where this creates a disparity between my proposed behavior for external commands, and how I believe things would have to play out for builtins. And I won't dismiss a disparity like that as if it were a non-issue. I think it's something that needs to be resolved regardless of where the chips fall on this SIGPIPE thing: I think as a programming paradigm, pipes are very functional in nature and this case, where a consumer doesn't need the entirety of a producer's output, is a natural way of expressing ideas in that paradigm, and as such it shouldn't be an exception. But it's not a purely functional language, so there are cases where quietly terminating is not the correct response to a broken output pipe. And I acknowledge the value of making internal and external tools behave analogously when possible.

You say the expectation is that Unix tools should catch SIGPIPE and return a zero error status - I agree. That would probably be a better choice. (Though one could argue that it's worthwhile to report the reason the process terminated even when it's not an actual failure condition.) But, these tools do what they do, it's kind of meaningless to talk about what they "should do". In their GNU and busybox versions at least it seems most of these line-oriented text processing tools (like grep, head, sed, awk, tr, find, etc.) exit with SIGPIPE when they try to write to a broken pipe. If you have examples of other common implementations of these tools that behave differently - I actually would be interested to learn about such cases.

I understand that elvish is kind of setting its own course, and I respect that. But it's built to work with existing command-line tools as well, so it needs to work with those tools as they are, and in ways that make sense.

SIGPIPE is "special" because it is a non-failure, automatic termination case that emerges from the normal operation of pipelines. The consumer stops consuming, so the producer stops producing. When a process in a pipeline exits on SIGPIPE it usually just means it stopped producing output because no more was required. Effectively, it stopped because its job was done. I understand that "usually" is not 100%, but I believe it to be the most common case, and as such the best fit for integrating with existing tools.

By contrast, SIGTERM would not be expected to emerge from the normal operation of pipelines. It usually means something external to that process asked it to stop. The reason for terminating the program probably had nothing to do with the normal operation of the pipeline. (It's possible a process in the pipeline, having finished its work, could SIGTERM its whole process group, but that is not at all a common behavior in my experience.)

[xiaq]

Replying to @krader1961's comments

Actually, for programs where that is true it is generally expected that the program capture the SIGPIPE and exit with a zero status if it really doesn't matter.

Pretty much none of the standard Unix tools trap SIGPIPE. Try this:

cat /dev/random | base64 | head -n1

Both cat and base64 will be killed by SIGPIPE.

A simpler case is:

seq 10000 | head -n1

Because how to handle this for a builtin, or function, that produces output into a pipe where the RHS terminates before consuming all the output is the more difficult question.

In fact it's a somewhat simpler problem to handle builtins and functions, because Elvish actually controls the behavior of builtins and functions.

Currently their behavior is to keep going. I didn't program this explicitly, it's a consequence of the following factors:

• Go's runtime traps SIGPIPE
• Most of the stdout-writing builtin functions in Elvish do not check write errors. For example, this part of the implementation of print: https://github.com/elves/elvish/blob/730bd969d819b33f9d2134187dd198bff7fb62f2/pkg/eval/builtin_fn_io.go#L398

[xiaq]

Replying to the issue per se:

In general, I agree that SIGPIPE received by the writing-end command is an expected condition and should not be treated as an exception. However, it's impossible to distinguish SIGPIPE raised under this condition from SIGPIPE raised under other conditions, so it's pretty much infeasible.

There is another way to solve this problem. It's a bad idea, I am just describing it because it's possible: we can make the writing-end commands adopt the "keep going" semantics when the the reading-end command exits. Unix FDs actually implement a reference counting system, and the close syscall merely decrements the refcount; the FD is only closed when its refcount reaches 0. When Elvish executes any command in a pipeline, it retains a reference of all the pipes.

Now consider the pipeline seq 10000 | head -n1: when head -n1 exits and closes its stdin (which is the reading end of the pipe), the reading end of pipe is not actually closed. It is only closed when Elvish notices that head -n1 has exited and closes its reference to the reading end. So instead of closing the reading end of the pipe, Elvish can continue reading it, until there is no more data.

As I said, this is likely a bad idea, because now commands like the following won't terminate:

cat /dev/random | base64 | head -n1

Nonetheless, this is how Elvish treats value pipes. The code to drain the value pipe is here: https://github.com/elves/elvish/blob/730bd969d819b33f9d2134187dd198bff7fb62f2/pkg/eval/compile_effect.go#L140-L147

This is an inconsistency, which should be fixed.

[xiaq]

To get back to the original problem of handling SIGPIPE from external commands: a good compromise is perhaps to make it easy to ignore SIGPIPE from individual commands. For example, if a swallow-sigpipe command is available, you can write code like the following:

swallow-sigpipe { cat /dev/random | base64 } | head -n1

An extra note: We want to make it possible to implement swallow-sigpipe as a function. Currently exceptions in Elvish are quite opaque, so today you can't do that without resorting to hacky string match. We want make it easy to tell if an exception was caused by a SIGPIPE.

[krader1961]

Pretty much none of the standard Unix tools trap SIGPIPE.... Both cat and base64 will be killed by SIGPIPE.

They need to terminate on receipt of SIGPIPE. It's not a question of whether they trap it but whether they do so and perform whatever cleanup is appropriate before exiting. The problem, as you note, is that most programs don't trap SIGPIPE and perform a clean exit. Which is an unfortunate reality Elvish needs to deal with.

To get back to the original problem of handling SIGPIPE from external commands: a good compromise is perhaps to make it easy to ignore SIGPIPE from individual commands.

I am tentatively leaning towards such a solution because I am extremely wary of unconditionally ignoring this condition in elvish. Consider that so far elvish has worked well for people without needing to always ignore an external command terminating due to SIGPIPE. The vast majority of the time an external program being killed due to SIGPIPE indicates an unexpected condition.

[zakukai]

"it's impossible to distinguish SIGPIPE raised under this condition from SIGPIPE raised under other conditions, so it's pretty much infeasible."

I agree, and it's an unavoidable problem when dealing with these pre-existing tools that behave this way. I think the best one can do is look at the existing tools out there and see how they behave, see if there really are cases of tools that SIGPIPE-exit in actual failure conditions.

I would assume (and yes, assumptions and all that: this is something that should probably be investigated) that the programs which exit with SIGPIPE are mostly those that don't change or disable signal handling for SIGPIPE: they simply accept the default behavior and are killed by the signal. If a program is opening network connections on its own, etc. - one would hope that it's simply disabling SIGPIPE, in which case it's unlikely that it would exit with SIGPIPE - in the case of a broken connection it'd be more likely to detect that itself and error out. A lot of assumptions there but if it's true then that simplifies things: it means we're unlikely to get SIGPIPE from cases other than a pipeline where the consumer closed its input - and therefore SIGPIPE would mostly indicate these cases of termination due to a pipe break.

"Consider that so far elvish has worked well for people without needing to always ignore an external command terminating due to SIGPIPE."

That doesn't necessarily mean much. Elvish still has a fair deal of bugs and missing features. That's just where it is in terms of its development. If people are using it productively in that state, does that mean those things don't need to be fixed? No, of course not.

"The vast majority of the time an external program being killed due to SIGPIPE indicates an unexpected condition."

You _could_ be right. Or you could be wrong. On what do you base this assertion? (To be fair the same applies to me - I made the opposite assertion. I believe I am right, but it is possible I am wrong...)

"In fact it's a somewhat simpler problem to handle builtins and functions"

In a sense, yes, because as you say you have total control over how builtins behave and can decide what they should do. But where I see that getting complicated is in making that behavior both sensible and consistent.

Basically, there are builtins like "range" that I see as functional in nature (there is no reason, therefore, to care if its entire output is not consumed) and others like "put" or "echo" which I see as more fundamental I/O operations - and code that is built around them should have the ability to choose whether to treat a pipe break (or its value-oriented equivalent) as an error condition or not. But the trick is that actually both functions perform the same kind of I/O. When creating a function that produces output, there's no need to use the dedicated "output primitive" functions, as anything that generates values will generate output, and that output becomes part of the function's output.

I think it could make sense to change that: that is, code run from within a function no longer gets access to the function's output streams by default: it contributes to the function's output only if you explicitly pass its output to the function's output, using redirection syntax perhaps. This would obviously be a huge change for people used to the way shell functions typically work, but I think it could be worthwhile for preventing cases where a function calls some utility (and the programmer neglects to redirect it to /dev/null or whatever) and in some rare case that utility produces output - which winds up becoming part of the function's output. Forcing the programmer to declare intent to include a particular command as part of the function's output means that rather than excluding the things they don't want in the function's output, they just have to include the ones they do want. (Same with input...) And then if a pipe break occurs when a command is writing to the function's output - then that is worth an exception. I could see a move like this being a tough pill to swallow, arguably as bad or worse than the problem I'm trying to solve... But I think it could be a good thing, too.

"a good compromise is perhaps to make it easy to ignore SIGPIPE from individual commands"

Perhaps. My concern there is that, personally, at least, this is behavior I would expect in virtually any pipeline I create. If it's (almost?) always the right choice, it should just be the default.

I have been giving this some thought, trying to come up with a good answer. Honestly I'm not sure I have one. I think the right answer is that when a command terminates due to a broken pipe, that should be a condition that is detectable, but it should not be considered an error. (From that perspective I believe it's appropriate that these tools exit with a status code that indicates they got SIGPIPE'd - and the problem is really that all nonzero exit status is treated as a failure) The problem is how to do that in a way that serves the other needs of the language.

[krader1961]

You could be right. Or you could be wrong. On what do you base this assertion?

40+ years of experience, 35 of which have been on various flavors of UNIX, and 15 years as a level three support engineer for Sequent Computer Systems and IBM. I agree there are situations where a pipe RHS is expected to not consume all its input causing the LHS to (maybe) receive a SIGPIPE. But that is not the common case. That is why bash introduced the pipefail option. So that bash scripts can reliably detect if a pipe LHS dies unexpectedly.

I think the right answer is that when a command terminates due to a broken pipe, that should be a condition that is detectable, but it should not be considered an error.

And if a command terminates for any other reason, including a simple (not due to termination by signal) non-zero exit status? I'm always leery of special-cases. In this case treating SIGPIPE different from every other termination status other than zero. If elvish exception objects contained more structured info it would be easy to create a function to wrap a command and selectively ignore termination due to SIGPIPE (or any other reason).

[zakukai]

Citing your years of experience inspires confidence that you know what you are talking about - but it doesn't answer the question. Can you give examples, perhaps, of cases where you've seen a program terminate on SIGPIPE in a case other than a pipeline shutting down? I know such cases are possible, but I don't believe they are common. So enlighten me.

I agree with you about special cases: but unless elvish is going to just run elvish things, it needs to work with existing tools, and they don't fit neatly into this model where non-zero exit status is treated as an exception. Different programs use the exit status in different ways. It's common enough for a non-zero exit to indicate failure that I believe it makes sense to translate that into an exception within the shell - but it's not universal. "test" or "grep" give non-zero exit status to indicate when the test condition wasn't matched, or the pattern wasn't found - but that doesn't mean the tool failed, it just happens that the exit status code is a convenient way to pass that kind of information back to the shell. Solutions like this are always messy, because these tools were built around a more relaxed set of rules. To really get it right would mean providing wrappers for every tool (which I believe actually is on the agenda - though it's another messy solution IMO, mostly for maintenance reasons.)

My argument for treating SIGPIPE differently is that it has a specific meaning, and that it's not generally a failure case. Exceptions to that rule exist, just like not every non-zero exit status is a failure - but there's no single set of rules that gets it right for every situation and every tool. It's always going to be a compromise because these programs were not designed around the idea that exit status would have a consistent meaning across tools.

[xiaq]

@zakukai

I acknowledge your concern, but as you said making the right call is hard. Realistically there are only two choices: always treat SIGPIPE as an exception (status quo), or never treat SIGPIPE as an exception. Attempts to make a smart decision dynamically will almost surely backfire.

From a theoretical point of view, neither choice is good, but one of them may be more practical than the other, but we don't have enough evidence today:

• Getting an exception every time you run cat a-long-file | head -n10 is surely an annoyance, but it's unclear how often this happens in reusable modules and complex scripts, where unsurprising exception exception is more essential.

• There isn't much experience to draw from traditional shells either: their default behavior is to silently ignore non-zero exit values [1] and even with set -e only the last command's exit value is examined (which is not affected by the kind of SIGPIPE discussed here). Only with Bash's set -o pipefail is the behavior similar to Elvish's, but that is perhaps far from a common practice (I'd be happy to be proven wrong).

I hope that when enough Elvish code is written - especially reusable modules and complex scripts - it will become clearer which choice is more practical, and we are not there yet.

[1] With limited exceptions, such as when used as the condition to if, or being the last command that was run

[krader1961]

Another option is to keep the current behavior, termination from SIGPIPE causes an exception, but provide a means for the user to explicitly indicate they want that exit reason to be ignored. For example, long ago when I started playing with elvish I wrote an nx function. It takes a list of exit statuses that do not indicate an error condition and an external command (and its args) to be run. The list can be empty in which case every exit status is non-fatal. For example, nx [1] grep something /some/file will treat the case where grep matched no lines as a successful exit. At the moment it doesn't support trapping only sigpipe exits because that information isn't encoded in the exception object. But adding more structured information about external command exits to the exception object would allow that. So you could do something like nx [pipe] cat /big/file | head -n1.

[xiaq]

@krader1961 yeah, that's definitely something worth doing regardless of the choice here; I have a similar proposal in https://github.com/elves/elvish/issues/952#issuecomment-607560829 and should be enabled by resolution of #208.

[krader1961]

Addendum to my previous comment... The SIGPIPE exception object is ?(fail 'cat killed by signal broken pipe') which means my nx function could be made to handle this case. But it would be better if the exception object had attributes other than cause which could be tested for whether the process terminated due to a signal and which signal caused the exit.

[zakukai]

"never treat SIGPIPE as an exception"

Yeah, that one. :)

I agree that attempts to split the difference, or "figure out" whether SIGPIPE should be an error are doomed to fail. In my opinion, if a program exits on SIGPIPE that simply is not an error.

[krader1961]

This behavior just bit me. To eliminate the dependency on other shells I need to be able to run this inside my ~/.elvish/rc.elv to detect if a daemon is running:

ps waux | grep -q '[p]ostfix/master'

Annoyingly it will sometimes fail with Exception: ps killed by signal broken pipe depending on the amount of data buffered by the pipe and where the postfix/master process appears in the ps output. Prefixing ps with my nx function causes that "failure" to be ignored. Which is a trivial fix for someone knowledgeable about this scenario. Such as most of the people using Elvish as I write this.

Despite everything I wrote above I'm now inclined to think the current SIGPIPE behavior, even with a trivial way to handle it, is a bridge too far for new users. Notwithstanding all the other ways Elvish differs from POSIX shells.

[krader1961]

Consider executing something like git log | less and exiting less after seeing the first page of information. That will likely result in a sigpipe exception:

 Exception: git killed by signal broken pipe
[tty 51], line 1: git log --stat -100

I mention this because the issue isn't restricted to "Elvish scripts". It seems unlikely an interactive user is going to find that information useful. After all, they (presumably deliberately) exited the less pager on the RHS of that pipeline and thus don't care that the LHS (the git log) continued to produce output.

[xiaq]

@krader1961 thanks for sharing your experience, but I don't think that has changed the argument here in a significant way. The examples and observations you added were quite similar to what @zakukai originally put forward. I have been well aware of those.

[xiaq]

Rethinking this, the sensible solution forward seems to be another layer of indirection: in the context of scripts and modules, external commands should always be used via a wrapper function, never directly. The wrapper function takes care of handling SIGPIPE. For example, the wrapper of cat will always ignore SIGPIPE.

Some notes about the general idea:

• The pattern of using wrapper functions can be enforced by #978, which makes it impossible to use an external command implicitly.

• The wrapper function can do other things. For example, when the wrapper function is created, the code can check the version (BSD vs GNU, and a minimal version number) of the external command, and then do things such as

  • bridge over some small incompatibility between GNU and BSD versions. For example, ls -G vs ls --color (larger incompatibilities are harder to bridge)
  • abort the script or module early, if a minimal version requirement is not met

• Elvish may come bundled with a module containing wrapper scripts for commonly used commands (for example, all the classical Unix commands). Wrapper functions for less commonly used commands can still be reused by creating third-party modules.

• The wrapper scripts might provide a more usable API than the original external command, possibly by taking advantage of Elvish's language features. For example, since in Elvish options and arguments are syntactically separate and can never be confused, the wrapper may take flags using Elvish options; cat &v -filename (cat here being a wrapper) translates to cat -v -- -filename.

  However, it's not clear yet whether a more usable API provides enough benefit to justify straying away from the familiar forms of these commands. Maybe there is space for both "more Elvish wrappers" and "high-fidelity wrappers".

[xiaq]

Note: the pattern of using wrapper functions, as propose above, are mainly concerned with modules and scripts. In interactive usage it still makes sense to call external commands directly. Maybe it makes sense to generalize the pattern to interactive use, but I haven't thought much about that.

That means that interactive uses may not benefit from the wrapper scripts, and as a result SIGPIPE will still result in some noise. I'm not too concerned about that; it's an annoyance at worst. On the other hand, in modules and scripts it's much more important for have simple, predictable behavior when using external commands, and this is where wrapper functions are useful (even essential).

[krader1961]

Rethinking this, the sensible solution forward seems to be another layer of indirection....

No, that does not solve this problem. Wrapper functions are useful and a straightforward way to define them would be useful. It's why I opened issue #1133. But wrapper functions do not solve this problem.

in modules and scripts it's much more important for have simple, predictable behavior when using external commands, and this is where wrapper functions are useful

Except that wrapper functions as you described the mechanism does not provide predictable behavior. Any external command I run that does not have a wrapper script can still result in a SIGPIPE exception. And that includes commands that have a slightly different spelling. For example, if you're using Homebrew on macOS to install GNU versions of utilities you're probably using the "g" prefix that is the default naming convention. Such as gsort rather than sort so there is no confusion which sort command is being used. Are we going to provide a gsort wrapper that eats SIGPIPE?

Note that bash 3.1, released in 2005, changed the default behavior to no longer report "broken pipe" when a command is terminated by SIGPIPE. I'm pretty confident that change has caused zero problems, or so close to zero as not to matter. On the other hand if you google "bash broken pipe" you'll find a huge number of questions about what that means and how to suppress it.

@zakukai was right in proposing ignoring SIGPIPE. Note too that elvish builtins appear to implicitly ignore SIGPIPE. For example, this should produce a SIGPIPE exception but it doesn't:

> { sleep .1; echo yes } | true

Whereas the equivalent external command does so:

> bash -c 'sleep .1; echo yes' | true
Exception: bash killed by signal broken pipe
[tty 25], line 1: bash -c 'sleep .1; echo yes' | true

[zakukai]

Well, I can see the value in taking the hard line that a SIGPIPE termination is a failure like any other signal exit. And I think it's a fair point that it may amount to little more than a nuisance in the end. I think my main reservation is simply that while Elvish is taking its own direction, it's still aiming to fill that "Unix shell" role as well - and as such, somehow, to some extent, it must play well with that existing software. I think if there's too much friction there, if it's too awkward or complicated to work with regular Unix tools in Elvish, that would be a real problem. I still feel like erroring on SIGPIPE is at the very least an unfortunate bit of friction, because causing SIGPIPEs is practically idiomatic in Unix shell, and it's very common for programs to simply let themselves be terminated by the signal.

I don't know that I have a good solution for how to treat SIGPIPE as an error and work and play well with pre-existing software. I think it's a complicated problem that stems, quite simply, from making a Unix shell that's very unlike a Unix shell. And I 100% support that mission, it's something I've been interested in as well, for a long time. The way I see it, a new shell needs to interface well with regular, existing programs, or people just won't use it - and that always seems to limit where I can go with a design. There's always a compromise somewhere. But as much as it frustrates me, when I try to think about what the shell could be - I think finding a way to work well with existing stuff is absolutely vital, even if it's at the cost of other design goals.

[lukeschlather]

• Only with Bash's set -o pipefail is the behavior similar to Elvish's, but that is perhaps far from a common practice (I'd be happy to be proven wrong).

Generally speaking, my policy for bash scripting is that all scripts should begin with this:

set -o pipefail
set -o errexit
set -o nounset

And this isn't something I invented... I've read a variety of bash style guides and most (all?) of them include it. So actually looking at elvish as a newcomer, one of the things that attracts me to it is that I don't need to invoke my favored cryptic "bash strict mode" but it just "does the right thing."

How does elvish's current behavior differ from bash's pipefail? In my experience I don't think I've had an issue with pipefail.

As an aside, I came here looking for where to give some feedback on this line in the semantics doc, where there is a typo (aknin), and also it says elvish essentially implements errexit but it's unclear if it implements pipefail.

If you come from POSIX shell, this is aknin to set -e or having a && between each pair of pipelines. In a sense, && is written as ; (or more commonly, a newline) in Elvish.

[krader1961]

It's important to note that using bash's set -o pipefail does not cause it to report an error if the LHS of a pipe fails due to SIGPIPE. It only causes it to report a failure for any non-zero status other than termination due to SIGPIPE. Elvish is different from every other shell with respect to treating SIGPIPE as an error.

...it says elvish essentially implements errexit but it's unclear if it implements pipefail.

Elvish does implement pipefail semantics. You can see this with a trivial experiment:

> false | true
Exception: false exited with 1
[tty 27], line 1: false | true

[hanche]

@krader1961 wrote in chat:

[…] 99.999% of the time the SIGPIPE termination of the LHS process is not an error

Probably true, although it is surely not an actual statistic. But since the vast majority of cases would arise from interactive use, with the RHS process typically being less or head, I'll buy it. Still, I tend to worry about that final 0.001%, or whatever it may be.

I agree totally that it is much better to _never_ tread SIGPIPE as an error than _always_ to do so, but I would prefer a way to be able to detect it in the remaining few cases. One solution, briefly aired in chat, is to allow options to the running of external commands. These options would never be passed to the command (as there is no mechanism for doing so), but would be used by elvish to modify how the command is to be treated. Specifically, there could be an option &pipe-err to case a SIGPIPE, in that particular process, to be treated as an error by elvish. So long as we don't allow the number of possible options to get out of hand, the implementation cost ought to be modest.

[zakukai]

Do you think it would be useful to begin compiling a set of known or potential scenarios that result in SIGPIPE?

One that comes to mind apart from interactive "less" and "head" is "grep -q": the user wants to know if a certain pattern appears in the output of a producer. But they don't actually need to see the instance (or instances) of that pattern, they just need to know if there is at least one such instance. And so the implementation doesn't continue looking for more instances after it finds the first one - so it closes input and the producer potentially gets SIGPIPE'd.

Similarly "grep" with "--max-count" would do the same. In this case the caller actually does want to capture the matching text, but as with "-q", "grep" reaches a point where it needs no more input to fulfill its request, so it terminates and the producer may SIGPIPE as a result.

I'd be curious to know what other examples people can offer - specific cases where SIGPIPE really should be treated as an error, other consumers that close their input (possibly triggering a SIGPIPE), etc. - to get a better look at the whole situation.

[krader1961]

There seems to be some misunderstandings about when and why the kernel sends SIGPIPE to a process. Perhaps the most important thing to realize is that just because the LHS process is not killed by SIGPIPE does not mean all the data it wrote was read.

Consider a command that writes 8KB or less of data before it terminates. Let's call it cmd1. Why 8KB? Because I'm not aware of any UNIX like OS that buffers less than 8KB -- going all the way back to the early 1980's. If you do cmd1 | cmd2 is cmd1 guaranteed not to receive SIGPIPE? Even though all the data it writes fits in the pipe buffer? No! There are four scenarios:

1) cmd2 reads all the data before it exits. No SIGPIPE is sent to cmd1. This is the usual case.

2) cmd2 reads some, but not all, of the data before it exits. No SIGPIPE is sent to cmd1. This is the second most common case.

3) cmd2 reads none of the data and it exits, closing the read side of the pipe, after cmd1 has written its data and exited. No SIGPIPE is sent to cmd1. All the data written by cmd1 is simply discarded. This is rare but not unheard of.

4) cmd2 reads nothing (or a fraction of the data) and exits before cmd1 writes all its data. SIGPIPE is sent to cmd1. Any data already written by cmd1 but not read by cmd2 is discarded.

It should be obvious it doesn't matter how much data is buffered by the pipe. SysVR4 STREAMS based pipes, for example, usually buffer 128KB. A larger buffer simply makes it less likely the cmd1 process will receive SIGPIPE. It does not change the probability that some (or all) of the data written by cmd1 will never be read and silently discarded by the kernel.

Consider the ps waux | grep -q pattern example that caused me to change my mind. I don't care if all of the ps waux output is read. More importantly, I don't care if the ps process is killed by SIGPIPE because the grep found a match and exited before ps had written all its data.

Consider a canonical useless-use-of-cat example: cat a_file | cmd2. Here, again, cmd2 only reads some of the data. We don't care that the cat might receive SIGPIPE depending on how much data is in a_file, how much data the pipe buffers, and how much data cmd2 reads before exiting. The example is equivalent to cmd2 < a_file. We only care about the exit status of cmd2.

I've been thinking about this a lot. In the thirty-five years I've been using, and supporting, UNIX I can't think of a single instance where it was necessary for the shell to report that a pipe LHS process died from SIGPIPE. There have been some cases where the LHS process needed to cleanup after receiving a SIGPIPE but obviously the process has to install a SIGPIPE signal handler. That signal handler either exits with a zero or non-zero status as appropriate for the situation. But note that the shell is then treating the explicit status as success or failure using the usual rules. The shell never even knows that the process handled SIGPIPE in that case.

[zakukai]

Not sure about the confusion - personally I think I've been pretty consistent on that point: when I say a consumer stops consuming, I say the producer may get SIGPIPE. As you say it won't always happen.

Generally I still lean in the direction that SIGPIPE should not be considered an error - because by convention, in many common cases it is not. Terminating on SIGPIPE is the default behavior and many programs do not override that. I also believe a shell needs to work with the programs it will be running. If it starts feeling like it's working against them I think that's a problem. And from that perspective, responding to a common Unix idiom by reporting an unhandled exception is not ideal to say the least.

But at the same time, I understand why Elvish would treat this condition as an exception, and why it might even be worth causing a little friction in order to push for something better. If a process exits in response to a signal, that means it got the signal and didn't handle it, and as a result the default behavior kicked in - which for that signal was to terminate the process. If you make the integrity of your scripting environment a priority, then for a program to get halted by a signal is kind of a big deal. So from that perspective I can appreciate the argument that an uncaught exception should be treated as an error - and I think every program should either catch or disable SIGPIPE and deal with broken connections through its own internal logic - so that as you say, if a program did fail due to a broken pipe, the shell wouldn't get a SIGPIPE status from it - and the only time the shell would ever see a process exiting with SIGPIPE status was if that program were broken - because we now define "falling back to the default handling of SIGPIPE and terminating on that signal" to mean the program is "broken" - because that kind of exit gives us no assurance that the program actually terminated gracefully and made a considered decision on whether it should return with an error status.

The problem is that's not the reality we're dealing with here. There's loads of software out there that's coded to the default and exits on SIGPIPE. There are loads of programmers out there who will stubbornly refuse to change the behavior of their programs because they want to uphold Unix tradition or believe the Elvish approach to be overly pedantic. And there are users who won't have patience for a shell that reports errors on a regular basis where there seemingly are none. To some extent I think the shell needs to work with software as it exists rather than pushing for it to work a different way - and from that perspective I'd say the pragmatic reality of the situation is that SIGPIPE is not generally an error.

I guess I'm repeating myself here and I don't mean to, I'm just trying to think through all this.

[krader1961]

personally I think I've been pretty consistent on that point

You have been consistent, @zakukai. But other commenters here and in IM seem to believe that SIGPIPE indicates an error in the same way an explicit non-zero exit status does. Too, while SIGPIPE terminates a program by default that is not at all like a termination due to SIGKILL, SIGSEGV, etc. As you and I have pointed out several times now SIGPIPE (almost?) never indicates an error. The only programs I can recall that explicitly handle SIGPIPE do so in order to remove temporary files, cleanly disconnect from a database, or perform similar cleanup. As best I recall they all then proceed to exit with a zero (success) status.

Note that the Fish shell, probably the main Elvish competitor, also ignores SIGPIPE:

https://github.com/fish-shell/fish-shell/blob/8cf389baf24d662c59dfa24d406243c930e0657f/src/proc.cpp#L559-L562

[hanche]

@krader1961

Perhaps the most important thing to realize is that just because the LHS process is not killed by SIGPIPE does not mean all the data it wrote was read.

That was illuminating. It changed my mind; I have no further objections.

[zakukai]

As you and I have pointed out several times now SIGPIPE (almost?) never indicates an error. The only programs I can recall that explicitly handle SIGPIPE do so in order to remove temporary files, cleanly disconnect from a database, or perform similar cleanup.

There are a lot of relatively simple tools for which the default SIGPIPE handling actually makes sense - a fundamental assumption of the program (the ability to write to stdout) is no longer valid, and therefore the program should terminate. By convention this isn't an error - and generally I feel like a shell needs to work with conventions like this, not against them, or else things will get frustrating.

But for almost anything more complicated than these simple command-line Producers I think allowing SIGPIPE to terminate the process is probably a bug, because it means the process (probably*) didn't terminate cleanly. The default that was beneficial to those simple programs becomes a kind of "gotcha" - a program could be written to connect to a network service, for instance, and send it some data - and respond to failure codes from write() etc. - but if the programmer forgets to ignore SIGPIPE (or use MSG_NOSIGNAL or SO_SIGNOPIPE, etc.) they'll never get the return code from that write() and the program will just terminate on the signal. From that perspective I can see the value in treating SIGPIPE as an error - it's a case where things are potentially going wrong, invisibly. So it's not really a question of what programs do this - any program could (it's just probably a bug in the program IMO).

(* "probably" as in, it is possible to handle SIGPIPE, shut down cleanly, and then terminate on SIGPIPE, for instance by clearing the signal handler and re-issuing the signal. But I don't think that's common)

In other words I think the value in treating SIGPIPE termination as an error is not so much recognizing that the program couldn't write out all its output, as it is recognizing that the program got terminated by a default signal handler as a result. Looking at it that way I think it does make sense to regard it as an error - just doing so also pits the shell against code it ought to be working with. At this point I don't know that there's a solution I'm entirely happy with. Treating SIGPIPE as an error throws a wrench in a bunch of existing tools and shell idioms. Not treating it as an error ignores a situation that should be of concern if it happens in a program that should shut down gracefully. Switching behaviors through shell options or wrapper scripts makes things inconsistent, and thus confusing. Treating SIGPIPE as an error and then wrapping invocations of old shell programs in exception handler code becomes a discipline programmers have to adhere to or risk their pipeline command failing someday because it SIGPIPE's on rare occasion. I don't know that there's a solution that serves all the competing goals well unfortunately.

[hanche]

Well, if we do sometimes, but not usually, want to detect that a process died on SIGPIPE, I think a clean solution is to allow elvish options to external commands. Those options are to be interpreted by elvish, of course, and not passed to the command. So you could write foo &pipe-err to run external command foo, with the proviso that elvish should treat foo being killed by SIGPIPE as an error.

[krader1961]

Well, if we do sometimes, but not usually, want to detect that a process died on SIGPIPE, I think a clean solution is to allow elvish options to external commands.

That doesn't work. Consider the humble echo command. If i explicitly invoke the external, rather than builtin, via e:echo &pipe-err I expect &pipe-err to be echoed. Elvish cannot interpret arbitrary arguments to external commands looking for something that might look like an Elvish option. There are similar problems with that proposal where builtins or Elvish functions are involved.

Also, "if we do sometimes, but not usually" is really "if we might theoretically want to, but never in practice" want to detect that a process died from SIGPIPE. No other UNIX shell provides a way to treat SIGPIPE terminations as an error akin to the program doing exit(1). Guess what? All the scripts (i.e., programs) written for those shells seem to behave just fine with those shells treating a SIGPIPE termination as success. Including bash scripts at companies like Google (I'm an ex-Googler so I have some first-hand knowledge).

I am firmly of the opinion that the status quo of treating a SIGPIPE termination as an error all of the time, thus requiring wrapping most Elvish pipelines in a try block, is a non-starter as a solution to this issue. I didn't fully appreciate this point until I started writing Elvish programs that used pipelines outside of an interactive context (where reporting the SIGPIPE error is usually just an annoyance).

[zakukai]

Well, if we do sometimes, but not usually, want to detect that a process died on SIGPIPE, I think a clean solution is to allow elvish options to external commands.

That doesn't work. Consider the humble echo command. If i explicitly invoke the external, rather than builtin, via e:echo &pipe-err I expect &pipe-err to be echoed.

Elvish treats "options" prefixed with ampersand as syntax. So presently if you do e:echo &pipe-err you'll get an unknown option error. The way to print &pipe-err is e:echo "&pipe-err"

[krader1961]

Elvish treats "options" prefixed with ampersand as syntax....

Yes, and no. Also, irrelevant. The presumption behind the proposal that @hanche referred to is that either

1) every external command that might be used in the LHS of a pipeline has a wrapper that defines that option (which is a non-starter), or

2) the option is special; i.e., recognized by every command including builtins as a way to modify their behavior.

Also, requiring every argument that begins with an & char to be quoted if it isn't meant to be interpreted as an option is a bridge too far with respect to backward compatibility. Even if limited to just external commands rather than Elvish functions or builtins.

[hanche]

@krader1961 Sorry, I do not understand what you call the presumption behind my proposal.

1. Providing a wrapper for “every external command that might be used in the LHS of a pipeline” is a non-starter indeed. I am assuming here that when elvish builds and runs a pipeline of commands, it keeps track of each such command in some data structure, to be used among other things for providing meaningful error messages should the command fail. This data structure could contain a flag to tell what it do in the event of a SIGPIPE. So if elvish encounters something like find &pipe-err . -name *.c, it would set that flag and run find . -name *.c. Now when that particular command exits, elvish will in any case have to figure out if it was an ordinary exit or a signal and act accordingly. In the special case that it was a SIGPIPE killing the process, elvish looks at that flag and sets the status to either $ok or an error, respectively. There are no “wrappers” involved other than what elvish will have to do in any case. Unless, of course, I have totally misunderstood how things work under the hood.

2. I do not understand why the special-ness of the option means it has to be understood by every command including builtins. When elvish resolves a command, it has to find out whether it is a user defined function, a builtin, or an external command. Only in the latter case does it have to deal with this particular option.

Also, requiring every argument that begins with an & char to be quoted if it isn't meant to be interpreted as an option is a bridge too far with respect to backward compatibility. Even if limited to just external commands rather than Elvish functions or builtins.

Huh? Isn't that how elvish currently works?

⬥ echo &foo
Exception: unknown option foo
[tty 90], line 1: echo &foo
⬥ e:echo &foo
Exception: external commands don't accept elvish options
[tty 91], line 1: e:echo &foo

This is as it should be. Except, if my proposal were implemented, the latter exception would instead complain that &foo is not a recognized option for external commands.

PS. I do not think this question needs to be resolved quickly. Rather, let's just stop treating SIGPIPE as an error for now, and then return to the option proposal at our leisure. I am sure it will wind up quite some distance down the stack of priorities. If and when the present issue is closed (soon I hope), we can open a new issue to keep track of my proposal, with a link to this discussion so we don't lose track of it. (And if it then gets closed with “will not implement”, at least that is on the record as a design decision.)

[hanche]

I went ahead and created issue #1151 for elvish options to external commands. So please let the discussion continue there. I think @xiaq ought to merge the pull request from @krader1961 and close this issue; and then we can continue discussing my options proposal at a (much) more leisurely pace.

[krader1961]

@hanche, I had not noticed that even external commands do not allow bare strings that look like Elvish options. It's not clear whether that is a feature or misfeature. Too, even if the ampersand is not the first char of a token it is still recognized:

range 1&step=2 6
▶ (float64 1)
▶ (float64 3)
▶ (float64 5)

Should that even be allowed?

Only in the latter case does it have to deal with this particular option.

I don't think that is true. Doing a_elvish_builitin_or_function | e:cmd where the external command exited before all the data was written by the builtin/function should at least simulate a SIGPIPE failure or otherwise raise an exception. The fact that doesn't currently happen would seem to be a bug. Try this pipieline: range 1 1111111 | to-lines | e:true. I would expect it to report a failure from the to-lines builtin.

[krader1961]

I would expect it to report a failure from the to-lines builtin.

The fact that doesn't happen suggests that just because the LHS of the pipeline is an external command, rather than a builtin or function, its termination due to SIGPIPE should also not be treated as an exception worthy event.

[krader1961]

@xiaq, In light of the fact that

a) builtins that write bytes to a pipe never result in a SIGPIPE exception, and
b) the consensus appears to be that SIGPIPE should not be considered an error

I would like to resubmit my P.R. that implements the desired, more consistent, behavior. I've been using an elvish program built with that change for two months without noting any problems. On the contrary, the fact that I no longer have to explicitly think about the SIGPIPE scenario has meant seeing no unexpected pipeline exceptions and having to wrap them in try blocks.

[xiaq]

Revisiting this, I realized it's possible to detect whether a SIGPIPE is caused by the next command in a pipeline exiting (and thus closing the read end of the pipe). In

yes | head -n10

Elvish knows that

1. The output of yes is connected to a pipe (Elvish created that pipe)
2. The read end of the pipe (the head command) exited earlier than the yes command

So when Elvish sees that yes exited with SIGPIPE, we can be quite confident that this is not a genuine issue and the SIGPIPE can be suppressed.

This is not going to be 100% accurate; in the general case of foo | bar, the foo command could just happen to exit with SIGPIPE for an unrelated reason after bar has exited. But at this point I'm content to declare that such cases should be quite rare and I'm not concerned about being incorrect in those rare cases.

[krader1961]

That's going to be fragile for two additional reasons. First, I don't think there is any guarantee that elvish will be notified the RHS exited before the the LHS. Second, if the RHS closes its end of the pipe but doesn't exit the LHS will still die from a SIGPIPE. I don't think it's worth trying to be clever here. A SIGPIPE exit status should be taken at face value; just like we would assume SIGSEGV was due to an invalid memory access rather than someone doing kill -SEGV.

[xiaq]

First, I don't think there is any guarantee that elvish will be notified the RHS exited before the the LHS.

You're probably referring to the fact that when Elvish waits for the yes and head processes, the wait syscall for head is not guaranteed to return before the wait syscall for yes.

However, Elvish also knows when exactly the read end of the pipe is closed, because Elvish retains a reference of that read end, and it is closed by Elvish after the wait syscall returns. Instead of tracking whether yes exited after head exited, Elvish could just track whether yes exited after the read end of the pipe is closed.

A SIGPIPE exit status should be taken at face value

That face value is "this command exited because it is trying to write to any pipe or socket whose read end has been closed". It is not uncommon for networking programs to crash with SIGPIPE.

Always equating SIGPIPE as "this command exited because its standard output was closed" is already a heuristic. When one has to resort to heuristics, it's better to use more accurate ones.

[andychu]

Hm I think this is an interesting idea.

At first I thought it's similar to just signal(SIGPIPE, SIG_IGN) on every process involved in a pipeline (before the exec() call). But then I read over this thread:

https://stackoverflow.com/questions/8369506/why-does-sigpipe-exist

I think the logic there is right -- you basically just want to TERMINATE yes early when head exits and closes the pipe.

In the old days of Unix, people didn't check error codes from write(), so it was easy just to add a signal so all programs would behave consistently.

---

I don't see the ordering issue, at least in a POSIX shell. The exit code of the entire pipeline is nonzero if any process in the pipeline exited nonzero. $? refers to the entire pipeline in POSIX, or you can use ${PIPESTATUS[@]} in bash to pick out the exit codes. So either way there's no nondeterminism.

Personally I don't see any use for SIGPIPE and I've been using shell for a long time. The only time I see it is when I pipe to head, and I don't care about that. I think of it like this:

• As long as head exits zero, then the whole pipeline should be successful.
• If there is an error, then head should exit with one, and the whole pipeline will fail. I don't see the need for an additional failure from yes.

I think this question could be answered empirically by a survey of some shell code, and I'd be surprised if anyone really wants the SIGPIPE failure.

---

I think it's possible that Oil could have shopt --set sigpipe_ignore and that would be on by default in the oil:basic option group. Right now Oil has a rather unwieldy run --status-ok SIGPIPE yes | head idiom to explicitly opt in.

But I don't think anyone really wants the SIGPIPE error, and if they do they can just shopt --unset sigpipe_ignore or use the POSIX compatible defaults.

[krader1961]

Always equating SIGPIPE as "this command exited because its standard output was closed" is already a heuristic. When one has to resort to heuristics, it's better to use more accurate ones.

I would argue it's a "simple rule". You can argue that it is too simplistic but it's not a heuristic as I understand the term. Yes, I'm parsing hairs here but I think it's important because you're proposing replacing a simple rule with a complex, fragile, heuristic.

It is not uncommon for networking programs to crash with SIGPIPE.

In my 3+ decades of using UNIX (including a decade as a level 3 support engineer at Sequent then IBM) I can't recall a single instance where a "networking program" died due to a SIGPIPE from writing to a SOCK_STREAM socket whose remote end has closed its side of the connection. At least not where such a SIGPIPE termination in a shell pipeline caused a problem because the SIGPIPE was ignored. Arguments from personal experience are always suspect but in this instance I would have expected to recall at least a handful of instances where the scenario you describe was the core problem. That is, a "networking program" that does not correctly handle SIGPIPE and is terminated because of that mistake.

Note: I am not proposing that SIGPIPE always be ignored. It should only be ignored when the process is the LHS of a pipe. Without checking I don't recall if my proposed change made that distinction. Also, shells like bash ignore the SIGPIPE when the process is in a pipeline but not when it is a standalone process. While there are many reasons to dislike the POSIX shell standard this is one aspect of that standard which Elvish should probably emulate.

[andychu]

I went back and skimmed through the whole thread -- sorry my original comment didn't reflect all of that. I think we're still missing this from April:

Can you give examples, perhaps, of cases where you've seen a program terminate on SIGPIPE in a case other than a pipeline shutting down? I know such cases are possible, but I don't believe they are common.

Does everyone agree that there are no real uses of SIGPIPE other than a pipeline shutting down like yes | head? It doesn't appear that any counterexamples were presented, other than someone doing kill -SIGPIPE $pid, and I would say that "doesn't count".

If not, then what are the counterexamples? (maybe I missed them) Otherwise it seems like everyone is saying similar things, or maybe I don't fully understand because I don't know exactly how Elvish works.

---

Just to clarify the way POSIX/bash works:

• by default, set -e / errexit is off. So ALL errors are ignored, not just the SIGPIPE (141) status. You can of course check $? manually, but I highly doubt anyone does that for SIGPIPE.
• Even if errexit is ON, the exit status of the pipeline is the exit status of the LAST element. So in yes|head, even though yes fails, the pipeline succeeds:

$ bash -c ' yes|head -n 1 ; echo status=$? pipestatus=${PIPESTATUS[@]}'
y
status=0 pipestatus=141 0  # you would have to check PIPESTATUS manually to see any error, nobody does this

set -o pipefail makes the status of the pipeline take into account ALL the processes, not just the last one.

$ bash -c 'set -o pipefail; yes|head -n 1 ; echo status=$? pipestatus=${PIPESTATUS[@]}'
y
status=141 pipestatus=141 0   # status is 141 now, so this will FAIL with errexit

---

By default, Oil does what POSIX shell/bash does. But the recommended usage is to set shopt --set oil:basic, which turns ON both errexit and pipefail.

So Oil has the same problem that Elvish has: The SIGPIPE errors are distracting.

I added the run --status-ok SIGPIPE yes thing, but now I think that is too awkward, and we should just have shopt --unset ignore_sigpipe_status.

So in the legacy mode, Oil will ignore SIGPIPE errors just because of POSIX shell's bad error handling. In the recommended mode, Oil will also ignore SIGPIPE, but you can opt into respecting it if you want.

Maybe Elvish doesn't have global options like sh/Oil, but here it seems to be useful.

tl;dr I'm considering changing Oil's handling of SIGPIPE from opt-in ignore to opt-out. I don't think there's any real problem with figuring out which cases should be ignored; maybe this is more complicated in Elvish

[krader1961]

@andychu: Thanks for your thoughts on this topic. Before I read your latest comment I came to a similar conclusion appropriate for Elvish. Specifically, SIGPIPE from a command that is the LHS of a pipe is ignored by default but there should be a way to annotate the command to treat SIGPIPE as an exception (todays default behavior). Changing the default behavior to ignore the SIGPIPE is what we want the vast majority of the time. When I do something like ps waux | grep -q '[e]lvish' I shouldn't have to wrap that in a try block just because the grep might exit before ps has written all its output. If someone really wants to know when the commands exits due to SIGPIPE they can annotate it.

What would such an annotation look like? I have no idea. When I started using Elvish I wrote a nx, short for "no exceptions", function so I could do things like nx [p] ps waux | grep.... The [p] argument in that example is a list of non-zero exit statuses to ignore with "p" meaning ignore SIGPIPE but it could also be [1 2 3], etc. An analog of that function which does propagate SIGPIPE is one way to do it but anything of this nature needs a lot of discussion.

P.S., @andychu wrote:

Maybe Elvish doesn't have global options like sh/Oil, but here it seems to be useful.

A strong NAK on that idea. Global options are probably the second biggest problem with POSIX like shells (the first being the IFS var). Global options make it easy to break dependencies; i.e., shell functions you invoke which expect a different option setting.

[andychu]

Yes I think the design dimensions are:

1. opt in to fatal SIGPIPE error, or opt out ?
   
   
   
   • Oil was opt out with run --status-ok, but now I think it should be opt in.
   
   
   • (POSIX shell is also opt in, but it ignores tons of other errors too)
   
   
2. Which errors should be ignored (regardless of opt in or opt out)? Again I don't see a big difficulty here, but I haven't implemented it. Maybe it's different in Elvish.
3. Annotation or global option?
   
   
   
   • Again Oil uses the flag to run which is basically like nx for SIGPIPE.
   
   

The reason I think changing it to a global option is OK is that in Oil adds an option stack, so you can do:

shopt --unset sigpipe_status_ok {
  echo foo
  yes | head    # now fails
  echo bar
}
# original setting restored here
yes | head

The current idiom is

run --status-ok SIGPIPE yes | head

which just feels too unwieldy IMO (in addition to having the wrong default).

[xiaq]

Getting back to this again.

First let me respond to the comments that are most relevant for the resolution of the issue.

Can you give examples, perhaps, of cases where you've seen a program terminate on SIGPIPE in a case other than a pipeline shutting down? I know such cases are possible, but I don't believe they are common.

I claimed that "it is not uncommon for networking programs to crash with SIGPIPE", so I went digging a bit. I thought that this was possible for the openssh client and curl, since I've seen both printing "broken pipe" and exiting when the network connection drops. But it turns out that both programs actually ignore SIGPIPE (openssh, curl). What's responsible for the error message is most likely the return value from write being EPIPE (and the canonical string message for EPIPE in GNU libc is "broken pipe").

I checked a few more well-known programs. OpenBSD netcat, socat, GNU wget all ignore SIGPIPE, so does BIND 9 and its utilities (nsloopup, dig, etc.).

However, busybox's wget does seem to not ignore SIGPIPE, and it can plausibly be used in a pipeline, for example to download a file and extract it without saving it to a temporary file:

busybox wget -O - https://dl.elv.sh/linux-arm64/elvish-HEAD.tar.gz | tar xz

The busybox wget command can be killed by SIGPIPE eithe because of a network error, or because of the next command in the pipeline exits earlier than it. This is where an additional check on whether the read end of the pipe is actually closed makes a difference: it can distinguish the two cases.

In fact, searching "SIGPIPE" in busybox's /networking directory indicates that it's a measured choice, rather than an oversight, to not ignore SIGPIPE in its wget implementation. Almost all the server programs do ignore SIGPIPE because they don't want a single client to crash the whole server: the only exception is dnsd, and that's because busybox's dnsd only supports UDP, and only TCP sockets can generate SIGPIPE. On the other hand, all the client programs do not ignore SIGPIPE because in this case, the default behavior of SIGPIPE (terminating the program) is a reasonable strategy to handle a closed connection.

In my 3+ decades of using UNIX (including a decade as a level 3 support engineer at Sequent then IBM) I can't recall a single instance where a "networking program" died due to a SIGPIPE from writing to a SOCK_STREAM socket whose remote end has closed its side of the connection. At least not where such a SIGPIPE termination in a shell pipeline caused a problem because the SIGPIPE was ignored. Arguments from personal experience are always suspect but in this instance I would have expected to recall at least a handful of instances where the scenario you describe was the core problem. That is, a "networking program" that does not correctly handle SIGPIPE and is terminated because of that mistake.

I don't question your credentials here, but I'd take your conclusion with a grain of salt.

Scripts in traditional shells tend to have a rather "meh" approach towards any kind of error handling. A "casual" automation scripts probably doesn't run with set -e or set -o pipefail, neither does it religiously check the exit code of every command. The default in traditional shell scripting is, really, just keep going regardless of what has happened. If the script doesn't do what is intended, eventually the sysadmin would notice and then debug it. A typical debug experience usually involves examining the stderr of the script, rarely the exit code of each command. The latter is not made easy by traditional shells because (1) the exit code is kept in a variable $? that gets overwritten immediately by the next command (2) unless you use bash which has $PIPESTATUS, only the exit status of the last command in a pipeline is available. Even if there was a case where a networking program did exit with SIGPIPE, how would you notice?

I am speaking from my experience of course, and my assumptions of how UNIX sysadmins operate may not apply to your experience. If your experience does involve more rigorous approaches towards error handling in traditional shell scripts, I'd be very curious to hear about it and happy to be proven wrong.

There is another reason I'd take your conclusion with a grain of salt: IIUC your experience was mostly in dealing with UNIX-like systems running in enterprise environments. I can only guess about your work environment, but enterprise setups usually mean stable network environments, and when the network does go down, one probably has more to worry about than a few shell scripts misbehaving.

I am not proposing that SIGPIPE always be ignored. It should only be ignored when the process is the LHS of a pipe. Without checking I don't recall if my proposed change made that distinction. Also, shells like bash ignore the SIGPIPE when the process is in a pipeline but not when it is a standalone process. While there are many reasons to dislike the POSIX shell standard this is one aspect of that standard which Elvish should probably emulate.

Well, POSIX-like shells ignore any error from commands that are not the last in a pipeline, and the ignoring of SIGPIPE is a logical consequence of that. To emulate POSIX-like shells would be to suppress exceptions raised in any command that is not the last in the pipeline, and that's a bad idea. Suppressing SIGPIPE only is an innovation.

---

So I maintain that the suppression of SIGPIPE should only happen when Elvish detects that the read end of the output pipe has indeed been closed. In fact, I claim that this is not just a better heuristic, it is a near-optimal heuristic.

To see this, let's consider the pipeline network-cmd | filter-cmd, where network-cmd is a networking program that can be killed by SIGPIPE either due to a closed TCP connection or due to filter-cmd closing the read end of its output. Now, when Elvish detects that it was killed by SIGPIPE, there are several possible situations:

1. The read end of the output has not been closed. Elvish does not suppress the SIGPIPE (and turns it into an exception). Because Elvish knows that the read end of the output has not been closed, the SIGPIPE must be caused by something else; this is always the correct behavior.

2. The read end of the output has been closed, meaning that filter-cmd has exited without consuming all of the output of network-cmd. Elvish suppresses the SIGPIPE, as if the command exited normally. There are two possibilities regarding what actually happened to network-cmd:

   2a. It is likely that the program was indeed killed by writing to the closed pipe. In this case, the behavior of Elvish is correct.

   2b. It is still possible that the SIGPIPE was actually caused by a closed TCP connection. However, in this case, it's still true that filter-cmd has exited, so the output of the whole pipeline is already finalized and would not be impacted by whatever network-cmd will write out later. So even if network-cmd did run into some network error, it is safe to ignore that error since it's irrelevant.

There are some remaining edge case in 2b where Elvish's handling (suppressing the SIGPIPE exit code) is not ideal. The network-cmd may have some additional side effect besides writing to stdout - for example, logging the request and/or response headers to an additional file. When there was a network error, such side effects may be incomplete, and argubly an exception should be raised. However, it's not possible to detect such side effects in general, and I'm happy to live with the false negative in such edge cases.

---

Finally, I want to solve the problem in the context of not just UNIX external programs, but also builtin commands, which currently lack a SIGPIPE-like mechanism for terminating writers whose reader has gone away (#923 is relevant here).

Here's the design I plan to implement:

• Introduce a new error type, say ErrOutputReaderGone, to denote that the reader of the output is gone;

• Change builtin output commands (echo, put, etc.) to raise ErrOutputReaderGone when it detects that the reader of the output is gone;

• When an external command gets terminated by SIGPIPE, check whether the read end of its output has indeed been closed, and if so, turn that into ErrOutputReaderGone;

• In a pipeline, if any command other than the last one raises ErrOutputReaderGone, silently swallow the exception.

[krader1961]

I want to solve the problem in the context of not just UNIX external programs, but also builtin commands

Good. The current inconsistency between internal and external commands is problematic as I noted in this early comment.

I still don't see how you intend to make this work reliably(ish). You wrote earlier (and this is implied by your latest comment):

However, Elvish also knows when exactly the read end of the pipe is closed, because Elvish retains a reference of that read end, and it is closed by Elvish after the wait syscall returns. Instead of tracking whether yes exited after head exited, Elvish could just track whether yes exited after the read end of the pipe is closed.

I don't see how that addresses my point you were replying to about the heuristic you're proposing. You're still relying on Elvish handling the termination of the RHS before handling the termination of the LHS. Why isn't this a race condition?

Also, with respect to internal commands it is unclear how your proposal applies to the value stream. If I write the following will it result in a SIGPIPE like exception?

range 99999999 | []{ take 1; put DONE! }

At the moment it doesn't do so because take consumes all value input, but if that were changed what would it mean for the example above?

[xiaq]

I still don't see how you intend to make this work reliably(ish). You wrote earlier (and this is implied by your latest comment):

However, Elvish also knows when exactly the read end of the pipe is closed, because Elvish retains a reference of that read end, and it is closed by Elvish after the wait syscall returns. Instead of tracking whether yes exited after head exited, Elvish could just track whether yes exited after the read end of the pipe is closed.

I don't see how that addresses my point you were replying to about the heuristic you're proposing. You're still relying on Elvish handling the termination of the RHS before handling the termination of the LHS. Why isn't this a race condition?

The important thing to understand here is that command termination itself doesn't affect other commands in a pipeline, it is the closing of files that does. Since Elvish is responsible for closing the files (actually the last remaining reference to the files), it is capable of fully synchronizing the sequence of events.

It's useful to walk through the sequence of events in two different scenarios. In a pipeline a | b, either (1) a exits early and causes b to exit due to an EOF read, or (2) b could exit early and cause a to exit due to SIGPIPE.

Scenario 1:

1. Command a exits.
2. Elvish's wait(1) on command a returns.
3. Elvish closes the write end of the pipe.
4. Command b calls read(1), gets EOF, and exits.
5. Elvish's wait(1) on command b returns.
6. Elvish closes the read end of the pipe.

The new "is reader gone" check is inserted between step 2 and 3, at which point command b is still running and the read end of the pipe is still open.

Scenario 2:

1. Command b exits.
2. Elvish's wait(1) on command b returns.
3. Elvish closes the read end of the pipe.
4. Command a calls write(1), and gets killed by SIGPIPE.
5. Elvish's wait(1) on command a returns.
6. Elvish closes the write end of the pipe.

The new "is reader gone" check is inserted between step 5 and 6, at which point command b has exited and the read end of the pipe has been closed.

Also, with respect to internal commands it is unclear how your proposal applies to the value stream. If I write the following will it result in a SIGPIPE like exception?

range 99999999 | []{ take 1; put DONE! }

At the moment it doesn't do so because take consumes all value input, but if that were changed what would it mean for the example above?

The range command will exit with ErrOutputReaderGone when trying to write the second value, and the error is subsequently swallowed because it is thrown by a non-last command in the pipeline.

[zakukai]

The important thing to understand here is that _command termination_ itself doesn't affect other commands in a pipeline, it is the closing of files that does. Since Elvish is responsible for closing the files (actually the last remaining reference to the files), it is capable of fully synchronizing the sequence of events.

It's useful to walk through the sequence of events in two different scenarios. In a pipeline a | b, either (1) a exits early and causes b to exit due to an EOF read, or (2) b could exit early and cause a to exit due to SIGPIPE.

Scenario 2:

1. Command b exits.
2. Elvish's wait(1) on command b returns.
3. Elvish closes the read end of the pipe.
4. Command a calls write(1), and gets killed by SIGPIPE.
5. Elvish's wait(1) on command a returns.
6. Elvish closes the write end of the pipe.

It took me a few iterations of reading and re-reading this to really get it. Basically because the shell holds on to copies of the pipe ends, it can enforce strict sequentiality in the steps that lead to pipeline termination as a result of broken-output or end-of-input, and therefore can detect the broken-output condition - at least under certain circumstances.

But programs can close their input or output streams before they terminate. This could lead to a deadlock (i.e. the pipe isn't broken until the shell detects a process terminating, but processes termination potentially occurs only in response to pipe break)

Programs could also hand off their input or output streams to another process and then terminate - which I think is less of a problem here but it does foul up the attempt to determine whether a SIGPIPE occurred in response to the termination of the pipe consumer.

[xiaq]

But programs can close their input or output streams before they terminate. This could lead to a deadlock (i.e. the pipe isn't broken until the shell detects a process terminating, but processes termination potentially occurs only in response to pipe break)

Elvish's retention of the stdin and stdout FDs of their subprocesses may be unusual in the case of pipes, but it is not unusual when you consider programs that run directly in the terminal - in such cases all shells retain copies of their stdin and stdout. So such programs would already be deadlock when invoked without any redirection.

Also, Elvish has always had this behaviour of retaining copies of the pipe FDs, and it has not been problematic in the past.

[zakukai]

But programs can close their input or output streams before they terminate. This could lead to a deadlock (i.e. the pipe isn't broken until the shell detects a process terminating, but processes termination potentially occurs only in response to pipe break)

Elvish's retention of the stdin and stdout FDs of their subprocesses may be unusual in the case of pipes, but it is not unusual when you consider programs that run directly in the terminal - in such cases all shells retain copies of their stdin and stdout. So such programs would already be deadlock when invoked without any redirection.

The analogy there is like "sed doesn't terminate when you run it without redirecting its I/O" - and yeah, that's exactly what happens. It tripped me up more than once when I was first learning Unix (i.e. I run a command, and it just sits there, waiting for input...) But you can terminate or suspend such a process with job control, or (in the case of a process that's waiting for input from the TTY) you can give it the end-of-file condition via CTRL-D. So it's not really a deadlock in that case.

Unix processes that continue running until their input EOFs or until their output breaks are not broken, this is a standard Unix command-line paradigm. The deadlock in Elvish would occur when two such programs are connected via a pipe, and one of them closes its end of the pipe without terminating right away. (That is the case in which Elvish would hold the pipe open waiting for a command to terminate.) That is a bit uncommon perhaps but it's a valid part of the shell paradigm. Consumers can stop consuming, and Producers can stop producing, even if they still have other work to do.

Also, Elvish has always had this behaviour of retaining copies of the pipe FDs, and it has not been problematic in the past.

While I respect the foundation on which Elvish is built and the capabilities this affords, I would not exactly say it's at a point where its history can be used to justify anything. I would make the opposite argument: Quirky behavior is not validated by the shell's history, rather the existence of some of this behavior (like "take" consuming all its input, value streams not being able to cope with a broken pipe, and this whole question of how to handle SIGPIPE in relation to normal Unix tools) is a testament to the fact that it's still a new and incomplete shell.

Why exactly does Elvish hold on to the pipe ends anyway?

[xiaq]

Unix processes that continue running until their input EOFs or until their output breaks are not broken, this is a standard Unix command-line paradigm. The deadlock in Elvish would occur when two such programs are connected via a pipe, and one of them closes its end of the pipe without terminating right away. (That is the case in which Elvish would hold the pipe open waiting for a command to terminate.) That is a bit uncommon perhaps but it's a valid part of the shell paradigm. Consumers can stop consuming, and Producers can stop producing, even if they still have other work to do.

I don't share your intuition that such scenarios can lead to deadlocks. It seems that this should only lead to a delay in when the pipeline gets cleaned up. A deadlock usually requires some kind of bidirectional communication, but in pipelines communication is strictly unidirectional (ignoring other out-of-band communication channels liked named FIFOs, of course).

An example involving real-world commands would be illuminating.

While I respect the foundation on which Elvish is built and the capabilities this affords, I would not exactly say it's at a point where its history can be used to justify anything. I would make the opposite argument: Quirky behavior is not validated by the shell's history, rather the existence of some of this behavior (like "take" consuming all its input, value streams not being able to cope with a broken pipe, and this whole question of how to handle SIGPIPE in relation to normal Unix tools) is a testament to the fact that it's still a new and incomplete shell.

Certainly. But this provides useful data points that such deadlocks at least don't come up in everyday interactive use.

Why exactly does Elvish hold on to the pipe ends anyway?

It's not a deliberate decision, just a consequence of the fact that pipelines and external commands are entirely orthogonal constructs in Elvish. Cleanup of the pipe files is part of the pipeline logic, decoupled from the invocation of external commands.

The orthogonality is useful when you have more complex pipelines, for example:

{ cat a.txt; ls } | less

Here, Elvish keeps a reference of the write end of the pipe, uses it for the stdout of both the cat and ls call, and closes it when the entire block terminates.

In theory Elvish could know that it could close its reference after executing the last external command (ls in this case), but that would lead to some unnecessary coupling between pipelines and external commands.

[zakukai]

I don't share your intuition that such scenarios can lead to deadlocks. It seems that this should only lead to a delay in when the pipeline gets cleaned up. A deadlock usually requires some kind of bidirectional communication, but in pipelines communication is strictly unidirectional (ignoring other out-of-band communication channels liked named FIFOs, of course).

Pipes do have bidirectional communication. Apart from the byte stream (communication from Producer to Consumer) there is pipe status information which communicates changes in the Consumer (i.e. the Consumer closing its end of the pipe) back to the Producer. But as things stand, Elvish impedes this notification from going through until the Consumer terminates.

An example involving real-world commands would be illuminating.

Haven't got one. But writing such a tool (whether as a compiled tool or as a script) and expecting programs on either end of the pipe to be able to detect pipe break even if the program doesn't terminate is a very reasonable thing IMO. Programs run by the shell should be able to detect pipe break, even if the process that had been holding on to the other end of the pipe hasn't terminated yet. It's a piece of functionality that should not be broken without a very good reason IMO (which I don't believe exists at this point)

In theory Elvish could know that it could close its reference after executing the last external command (ls in this case), but that would lead to some unnecessary coupling between pipelines and external commands.

I think the key is that an object representing a "pipeline command" hands off custody of the pipe ends to the handlers for the individual commands that will be using them, rather than retaining custody itself. In an example like what you described:

{ producer-1; producer-2 } | consumer-1

The code which handles "pipeline commands" creates the pipe and initially has custody over it. It then delegates responsibility over the individual pipe ends to individual command handlers - one to run { producer-1; producer-2; } and the other to run consumer-1. The command handler for consumer-1 runs the Consumer process and then can immediately release its custody over the read end of the pipe (which in this case causes the read end of the pipe to be closed). The command handler for { producer-1; producer-2; } is given custody over the write end of the pipe, which it holds long enough to run the commands in sequence until it reaches the last one. As soon as it starts command-2 it releases the write end of the pipe. As the commands terminate, they pass termination status back up the line.

...Of course this is incompatible with the idea of trying to determine what caused a SIGPIPE to decide whether to treat SIGPIPE termination as an error. Personally I think the idea of the shell trying to make that kind of determination is kind of dodgy anyway, because it can't actually know what caused a SIGPIPE termination and whether that represents an exception-worthy error condition, it can only guess.

At this point I'm honestly not entirely sure what the best solution is for SIGPIPE in general. I think the practice of treating signal termination as an error is good, but it doesn't work and play nicely with existing tools which (by convention) treat it as an ordinary termination condition. I feel like no matter which way you cut it there's some compromise. Either working with regular Unix tools is more of a pain, or you give up on the error handling, or you muddle the issue by making it a switchable (and thus inconsistent) behavior.

[xiaq]

I don't share your intuition that such scenarios can lead to deadlocks. It seems that this should only lead to a delay in when the pipeline gets cleaned up. A deadlock usually requires some kind of bidirectional communication, but in pipelines communication is strictly unidirectional (ignoring other out-of-band communication channels liked named FIFOs, of course).

Pipes do have bidirectional communication. Apart from the byte stream (communication from Producer to Consumer) there is pipe status information which communicates changes in the Consumer (i.e. the Consumer closing its end of the pipe) back to the Producer. But as things stand, Elvish impedes this notification from going through until the Consumer terminates.

Sure, but I still don't see how this would result in a deadlock. To have a deadlock like this you need two things:

• On the producer side, it has to close its output, and then wait for the consumer to close the input before terminating.
• On the consumer side, it has to close its input, and then wait for the producer to close the output before terminating.

However, I don't think the "and then wait" part is possible at all with standard UNIX APIs. Once you close a FD, you can no longe gain any information from it.

An example involving real-world commands would be illuminating.

Haven't got one. But writing such a tool (whether as a compiled tool or as a script) and expecting programs on either end of the pipe to be able to detect pipe break even if the program doesn't terminate is a very reasonable thing IMO. Programs run by the shell should be able to detect pipe break, even if the process that had been holding on to the other end of the pipe hasn't terminated yet. It's a piece of functionality that should not be broken without a very good reason IMO (which I don't believe exists at this point)

Well, I'll be convinced if/when you do have an example. Deadlocks are delicate matters and depend strongly on the exact sequence of events, so it doesn't make much sense to argue in the abstract.

---

Overall, I feel that one shouldn't overthink pipelines. Pipelines satisfy a specific type of concurrency scenario: when you have a series of data-producing and data-consuming tasks that are agnostic to each other and can thus be composed orthogonally. If you have concurrent tasks that need to be aware of each other, pipelines are probably not the correct tool.

This is where I'm comfortable making Elvish opinionated. If there are other concurrency scenarios, I can just add more mechanisms to Elvish to make it unnecessary to abuse pipelines for things pipelines are not good at.

[zakukai]

True, I guess I got that wrong and it's not a deadlock. If the Consumer closes its input but continues working, the result is that the Producer continues running (probably filling the pipe buffer and then blocking on output at some point) until the Consumer terminates. The Consumer no longer has any basis for waiting for the Producer to terminate unless there is a second communication channel between them.

I still think it's a poor idea for the shell to hold the pipe open, though, given that there is apparently no real reason for doing so, and it's just an artifact of the implementation. Why make pipelines even just a little bit worse for (what appears to be) no good reason?

[xiaq]

I still think it's a poor idea for the shell to hold the pipe open, though, given that there is apparently no real reason for doing so, and it's just an artifact of the implementation. Why make pipelines even just a little bit worse for (what appears to be) no good reason?

Well, it is an implementation artifact, but not an accidental one; as I said earlier, it is a consequence of pipelines and external commands being orthogonal constructs. Why complicate the implementation when there is no real benefit?

And in future, this will be a prerequisite for the detection mechanism I described before, so that is the real benefit here.

At this point I think I have sufficiently articulated the rationale for my design choices. I'm aware that it's a tradeoff not everyone will be comfortable with. But it is a tradeoff that I am comfortable with :)