Toolbox: RFE: Migration path for users of coreos/toolbox

[miabbott]

Is your feature request related to a problem? Please describe.
In RHCOS, we are shipping https://github.com/coreos/toolbox in the OS and we want to move to shipping containers/toolbox in its place. In order to do that, we need to ensure that the upgrade path is smooth, as toolbox is heavily utilized by the Red Hat Support organization for gathering debug information.

Describe the solution you'd like
I see two requirements for RHCOS users:

• support for .toolboxrc files
• retention of the same behavior from coreos/toolbox

Describe alternatives you've considered
The CoreOS team is currently maintaining the bash script implementation over at coreos/toolbox and we could continue to do so.

Additional context
FWIW, I don't expect the containers/toolbox team to work on this alone. Please reach out to the CoreOS team to help with the work.

[debarshiray]

We have from time to time discussed some sort of configuration file for Toolbox. It could either be used to tweak the behaviour of toolbox(1) itself, or to customize the internal set-up of the container by offering some extra bind mounts, etc.. That's related to ~/.toolboxrc.

I see that ~/.toolboxrc is only about changing the name of the default container and the image, and is a shell snippet that's easily sourced into another shell script.

Even if we come up with a different format for the configuration file going forward, I could imagine some migration path where if only ~/.toolboxrc is present, then it's used, but if the new-style file is present then it's preferred. Maybe we could also add a deprecation warning to urge users to migrate.

[debarshiray]

What did you mean by retaining the same behaviour as coreos/toolbox? I suppose, it's about the command line UI, right? Our solution might depend on how smooth we want the migration to be.

It will be helpful if someone who had been extensively using coreos/toolbox so far, tried out containers/toolbox to see if there's something that's actively preventing any of the previous use-cases. It will help us understand the scope of the work better.

There could be different ways in which we can ensure compatibility with the command line UI. We could either teach containers/toolbox to accept a different UI, or we could ship a wrapper shell script in RHCOS that accepts the old command line UI and translates it into container/toolbox calls. Or something else.

[HarryMichal]

I second the questions raised by @debarshiray. I'm just wondering if adding support for ~/.toolboxrc is really needed. The config file is used to set the registry, from which the image is pulled, the image itself and the name of the container, right? All these options can be set through the Toolbox CLI. Example:

$ toolbox create -i registry.redhat.io/rhel8/support-tools my-toolbox-name

I'm aware of the fact that it is not wise to just remove a feature (even though it is replaced by an adequate substitution) without giving any deprecation warning to users.

I just remembered one thing that will most likely need a mention in the documentation. I believe coreos/toolbox calls podman with sudo under the hood by default, right? Well, Toolbox works by default unprivileged. It can be run in a "privileged mode" when executed with sudo. This needs to be clear to the users because the difference is quite big when it comes to debugging of the host system.

I'll return again to the first paragraph. Even though I don't think we don't need a configuration file for setting those 3 environmental variables, I think we do need a configuration file. I already have an idea how can that be implemented but that is for a different ticket.

[miabbott]

What did you mean by _retaining the same behaviour as coreos/toolbox_? I suppose, it's about the command line UI, right? Our solution might depend on how smooth we want the migration to be.

There could be different ways in which we can ensure compatibility with the command line UI. We could either teach containers/toolbox to accept a different UI, or we could ship a wrapper shell script in RHCOS that accepts the old command line UI and translates it into container/toolbox calls. Or something else.

Maybe we could also add a deprecation warning to urge users to migrate.

My main thought was around the various volume mounts and options that are passed to podman by the coreos/toolbox script - https://github.com/coreos/toolbox/blob/master/rhcos-toolbox#L88-L111. Though retaining a familiar CLI UX would be nice.

I think the idea of a wrapper script that calls out to containers/toobox with the correct options might be the smoothest path forward; we could even have it print a message about deprecation.

An example user story:

I am a customer that installed OCP 4.5. During the operation of my cluster, I have customized my toolbox experience via the .toolboxrc file. I've upgraded my OCP cluster to 4.8, where containers/toolbox is now provided. I want my experience with toolbox to seem as familiar as possible and reduce any intervention that is needed to preserve my existing experience.

I just remembered one thing that will most likely need a mention in the documentation. I believe coreos/toolbox calls podman with sudo under the hood by default, right? Well, Toolbox works by default unprivileged. It can be run in a "privileged mode" when executed with sudo. This needs to be clear to the users because the difference is quite big when it comes to debugging of the host system.

Right, this gets to the crux of my concern about moving RHCOS to include containers/toolbox. Users of coreos/toolbox are typically running things like strace, tcpdump, sosreport, etc trying to gather debug information about processes running on the host. I want to try to make the transition to containers/toolbox as smooth as possible for these existing users.

[miabbott]

cc: @travier @control-d

[bgilbert]

It's worth noting that containers/toolbox has been broken on FCOS for some time; see https://github.com/coreos/fedora-coreos-tracker/issues/693 and https://github.com/coreos/fedora-coreos-tracker/issues/734.

[travier]

Also submitted https://github.com/containers/toolbox/issues/713

[debarshiray]

The following pull requests makes things more compatible with coreos/toolbox:

• https://github.com/containers/toolbox/pull/787: depends on https://github.com/containers/podman/issues/10858
• https://github.com/containers/toolbox/pull/811

[travier]

From https://github.com/containers/toolbox/commit/6c86cabbe5da6e542b50c5c043b4d213c6279bbc, will this still ask for container creation first time?

[travier]

For the configuration file, having the classic pattern of: /usr/lib/toolbox.conf -> /etc/toolbox.conf -> $XDG_CONFIG_DIR/toolbox.conf (all previous 3 in a good format (TOML?)) -> ~/.toolbox.rc (compatibility fallback with support for only a restricted set of options) would enable shipping the /usr config from a package, having installer specified option in /etc, and user ones in $HOME.

[travier]

The only supported options from the legacy toolbox.rc file should be:

REGISTRY=my.special.registry.example.com
IMAGE=debug:latest
TOOLBOX_NAME=special-debug-container"

From https://github.com/coreos/toolbox/blob/main/rhcos-toolbox#L157..L159

[travier]

The new format could have the following options/format:

[root]
interactive = false
default_image = registry.foo/name/image:tag
credentials = /var/lib/foo.conf

[user]
interactive = true
default_image = registry.foo/name/image_user:tag