Rust: Convert all `sh` scripts to `bash` scripts

[brson]

We've long stuck to posix shell for our bash scripts with the rationale that posix shell is the most compatible. Sometimes this has been painful. Most recently, IllumOS _doesn't have a working posix shell_, but it _does_ have bash. Conversely, I have no real-world examples of platforms people are using Rust on that only have sh.

I'm near the breaking point of my sympathy for sh now.

This must be done in-tree, as well as in rustup.sh and rust-installer.

This issue has been assigned to @Daniel-Worrall via this comment.

[brson]

cc @steveklabnik

[brson]

cc @rust-lang/tools

[brson]

The obvious place I can imagine this matter is installing rustc on busybox platforms, perhaps Android based.

[wycats]

For what it's worth, I've been playing with getting rust installed on non-standard platforms like aboriginal Linux and unrooted android recently and I can confirm that this is one of the biggest issues.

That said, the script itself is not perfectly compatible with busybox bash, which would be good to fix :)

[steveklabnik]

I'm fine with this :+1:

[alexcrichton]

I was poking around in FreeBSD and OpenBSD recently and it looks like neither have bash installed by default, but both have bash packages (just a point of note)

[brson]

@wycats That sounds like a vote for sticking with sh and fixing busybox compatibility?

[wycats]

@brson I think so yes.

[nodakai]

I wonder if we can do something like /non/standard/path/sh configure.sh so that users can choose whichever installation they prefer.

[raphaelcohn]

Just found this thread and I've got a quick observation on sh on BusyBox (I'm the main developer of shellfire).

There's two variants of sh on BusyBox: an ash (Almquist) derived one, and hush. hush is a bit... weird, but is used by some folks because, IIRC, it works on MMU-less systems, and BusyBox's ash doesn't. In addition, BusyBox's ash has a few small patches to accept a few small bashisms. In practice, apart from useful changes to read for scripts using socat, etc, you shouldn't need to rely on any. It's also worth pointing out that ksh _is_ POSIX compliant in its ksh88 form, but isn't in ksh93 form, and its idea of local is at odds with all other shells... it's a completely different feature.

I, and a number of others in the devops space are moving off bash because of its code smells, bug history and brittleness. There have been just too many problems over the years. And it's not commonly installed on non-Linux setups, etc.

I'd strongly suggest writing scripts and testing with pdksh. If they work with that, they'll work with mksh, dash, bash (running as sh, which is actually uncommon now, as it's often weird), ksh88, OpenBSD's variants, etc. I don't have much availability at the moment (legal shenanigans) but I'm happy to try to help out over email with script issues if you can get me access to a system. Shell script debugging is ...

[Mark-Simulacrum]

@brson Do we want to go with the pdksh approach suggested above? That seems to be the best option based on discussion above.

[Mark-Simulacrum]

Nominating for infra team so we can come to a decision on this.

[Mark-Simulacrum]

Discussed in meeting: We want to survey the current state of the world to know what's still written in shell, and come to decisions based on what it is how much of it needs to be in shell. Marking as help wanted, if you want to look across rust-lang/cargo, rust-lang/rust-installer, rust-lang/rust and find shell scripts and note them here, please do so. cc @alexcrichton -- feel free to edit with additional repos.

[nodakai]

@Mark-Simulacrum rust-installer is a serious shell script project which is not only written in shell but also generates a shell script. But I don't know if it will remain in a high demand, especially after rust-lang-nursery/rustup.rs#313 is resolved.

[kzys]

There are 48 shell scripts, but most of them are under src/ci/docker.

% git grep '#!/.*sh'
configure:1:#!/bin/sh
src/ci/docker/disabled/dist-x86_64-dragonfly/build-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/disabled/dist-x86_64-haiku/build-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/disabled/dist-x86_64-haiku/fetch-packages.sh:1:#!/usr/bin/env bash
src/ci/docker/disabled/dist-x86_64-haiku/llvm-config.sh:1:#!/bin/sh
src/ci/docker/disabled/wasm32-exp/node.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-aarch64-linux/build-toolchains.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-arm-linux/build-toolchains.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-armhf-linux/build-toolchains.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-armv7-linux/build-toolchains.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-powerpc-linux/build-powerpc-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-powerpc64-linux/build-powerpc64-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-powerpc64le-linux/build-powerpc64le-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-s390x-linux/build-s390x-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-various-1/build-rumprun.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-various-1/install-x86_64-redox.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-various-2/build-cloudabi-toolchain.sh:1:#!/bin/bash
src/ci/docker/dist-various-2/build-fuchsia-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-various-2/build-fuchsia-toolchain.sh:59:#!/bin/sh
src/ci/docker/dist-various-2/build-solaris-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-binutils.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-clang.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-cmake.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-curl.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-gcc.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-git.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-headers.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-openssl.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-linux/build-python.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-netbsd/build-netbsd-toolchain.sh:1:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-netbsd/build-netbsd-toolchain.sh:67:#!/usr/bin/env bash
src/ci/docker/dist-x86_64-netbsd/build-netbsd-toolchain.sh:72:#!/usr/bin/env bash
src/ci/docker/run.sh:1:#!/usr/bin/env bash
src/ci/docker/scripts/android-start-emulator.sh:1:#!/bin/sh
src/ci/docker/scripts/freebsd-toolchain.sh:1:#!/bin/bash
src/ci/docker/scripts/freebsd-toolchain.sh:99:#!/bin/sh
src/ci/docker/scripts/qemu-bare-bones-rcS:1:#!/bin/sh
src/ci/docker/x86_64-gnu-tools/checktools.sh:1:#!/bin/sh
src/ci/docker/x86_64-gnu-tools/repo.sh:1:#!/bin/sh
src/ci/init_repo.sh:1:#!/usr/bin/env bash
src/ci/run.sh:1:#!/usr/bin/env bash
src/etc/cat-and-grep.sh:1:#!/bin/sh
src/etc/installer/pkg/postinstall:1:#!/bin/sh
src/etc/rust-gdb:1:#!/bin/sh
src/etc/rust-gdbgui:1:#!/bin/sh
src/etc/rust-lldb:1:#!/bin/sh
src/test/run-make/git_clone_sha1.sh:1:#!/bin/bash -x
src/tools/build-manifest/README.md:13:#!/bin/bash
% git grep '#!/.*sh' | wc -l
      48
%

[Daniel-Worrall]

@rustbot claim