Toolbox: alignment/comparison with coreos/toolbox

[cgwalters]

Let's discuss overlap with https://github.com/coreos/toolbox/commit/598df78f6b09a9bb19b636bdc983e90f723ae256
here.

One broad concern I have here is the very name of the project; in Fedora we go to a lot of effort to have most of "Fedora" branding inside fedora-release, and there's a generic-release.

This one seems to assume a full desktop login and be run as non-root, where as for the coreos-toolbox we instead assume the user is running as root on a console, and should have full system wide privileges.

But really...the overlap in implementation and scope is big.

My 2c is that we call it "coreos-toolbox" as a project name, but just "toolbox" implicitly when discussing in a Fedora/CoreOS context.

[cgwalters]

My initial impetus to file this issue is https://pagure.io/workstation-ostree-config/pull-request/106 - adding something to the "base image" implies a fairly long term of support.

[yuqi-zhang]

A bit of background on the new coreos-toolbox: it was designed to be a very simple admin/debug container with full privileges, essentially a port from the old one (for CL) but for RHCOS/FCOS systems. In that sense the scope is much smaller.

[debarshiray]

One broad concern I have here is the very name of the project; in Fedora
we go to a lot of effort to have most of "Fedora" branding inside
fedora-release, and there's a generic-release.

(tl;dr: I am happy to switch to a downstream agnostic name.)

Well, this isn't about bits of trademarks, configuration, names and version numbers that define a specific Fedora release, though. It's more like fedora-packager, which has a bunch of fedora prefixed tools, and fedpkg.

Even though a toolbox container is a generic concept, the mix of upstream technologies (ie., Buildah, Podman), and downstream consumers (eg., Fedora Silverblue) make it uniquely "Fedora". It probably already works on, say, Debian, but I haven't tested it, nor is it the primary reason for the tool to exist. At least it's not my priority today.

Hence, the fedora prefix.

That said, if a fedora prefix is considered problematic, then I am happy to go with a name that's completely detached from any downstream.

This one seems to assume a full desktop login and be run as non-root,
where as for the coreos-toolbox we instead assume the user is running
as root on a console, and should have full system wide privileges.

(tl;dr: I am aware of CoreOS' needs and happy to address those.)

Well, it wasn't really a secret that we are working on a toolbox for Silverblue and what our needs are. :)

We have discussed this in #fedora-coreos, #fedora-containers, on fedora-coreos-tracker, etc..

I am aware that Fedora CoreOS' needs are different from Silverblue's. My impression was that we are happy to share as much of the toolbox as we can for the two flavours. However, I haven't heard anything from the CoreOS team after the initial discussion, and I don't know what the current plans or priorities are. Just the other day, I dropped a couple of comments about the current status of fedora-toolbox, but I haven't heard any updates.

Meanwhile, in Silverblue land, the toolbox is pretty crucial. Otherwise we can't use Silverblue as a development environment, and I really want to use my shiny XPS 13 running Silverblue as much as I can. :) So we had to choose a name and move forward.

My 2c is that we call it "coreos-toolbox" as a project name, but just "toolbox"
implicitly when discussing in a Fedora/CoreOS context.

As I mentioned earlier, fedora-toolbox is loosely inspired from coreos/toolbox. Loosely, because the original coreos/toolbox required root and used a different technology stack.

One of the reasons I didn't fork and continue with the original repository, was because I wasn't sure what to do with the governance and process related bits in there. A code of conduct, contribution guidelines, etc. are important things to have, but I also wanted to make it clear that this is happening under the Fedora banner. Not knowing exactly how the Fedora banner looks like, I didn't want to just git rm those bits in a commit, and I badly needed a Git repository to point people at. I had been pointing people to a tarball that I manually kept updated for weeks. :)

Secondly, since the underlying tools were going to be completely different, it meant that the script would have to be rewritten. Hence I was not too worried about preserving the Git history.

[matthiasclasen]

I agree with all that Debarshi said here.

I don't think there's much value in 'genericizing' this. It is about giving access to ready-made fedora tools containers, so calling it fedora-toolbox seems obvious and natural.

[halfline]

fwiw, this is a bit of tangent, but i find the name "toolbox" weird. a toolbox is something you pull tools out of, not something you work inside. I personally wish it was called workbox or something.

[cgwalters]

fwiw, this is a bit of tangent, but i find the name "toolbox" weird.

(Hm, so if we drop fedora and toolbox, the project could just be called - :wink: )

(tl;dr: I am aware of CoreOS' needs and happy to address those.)
Well, it wasn't really a secret that we are working on a toolbox for Silverblue and what our needs are. :)

Yep, I heard there was a discussion, sorry I am only now circling back to this after seeing the Silverblue PR.

Meanwhile, in Silverblue land, the toolbox is pretty crucial.

Agree, though I don't feel blocked myself right now having my own custom scripts for this, I do think we need to have an opinionated out of the box solution too.

Now, I find myself agreeing with this comment:
https://github.com/coreos/fedora-coreos-tracker/issues/38#issuecomment-417714930

It is about giving access to ready-made fedora tools containers, so calling it fedora-toolbox seems obvious and natural.

So, I work on Red Hat CoreOS and...I am not entirely sure that when we ship the tool should default to Fedora userspace. There are a whole lot of implications to that - some good, some bad.

Now particularly when you're talking about the command line, Fedora is packages and particularly given the default to a Fedora userspace, calling it "fedora-toolbox" does make sense. But that said I still have concerns over the branding.

The idea of solving two problems at once by merging the coreos-toolbox and fedora-toolbox and just calling it "toolbox" hence has a lot of appeal.

[matthiasclasen]

Sure, merging the two makes a lot of sense, which is why we reached out earlier to discuss that. If the name is a problem, how about something like "workshop", as the place where you keep your toolboxes and get work done ?

[debarshiray]

Meanwhile, in Silverblue land, the toolbox is pretty crucial.

Agree, though I don't feel blocked myself right now having my own custom
scripts for this, I do think we need to have an opinionated out of the box
solution too.

Well, you are one of those very special people who are well-versed in both containers and modern graphical OSes. Hapless mortals have been known to struggle with little things like leaking in the display server socket, and so on. :)

Now, I find myself agreeing with this comment:
coreos/fedora-coreos-tracker#38 (comment)

Yes, I like Dusty's idea too.

However, I don't know what troubleshooting exactly looks like in this case.

To be fair, my understanding of development is also skewed towards "should be possible to do jhbuild-style development", which is basically "install some tools from the distribution, build and install things into a separate prefix and run them". I'd really appreciate it if people with different workflows and development setups would try it out and give feedback.

So, I work on Red Hat CoreOS and...I am not entirely sure that when we ship
the tool should default to Fedora userspace. There are a whole lot of implications
to that - some good, some bad.

This seems a bit like fedpkg and rhpkg. Although, I haven't studied how those two assert their individual identities.

My assumption was that Fedora CoreOS/Silverblue and Red Hat CoreOS/Silverblue would share the usual relationship between Fedora and RHEL. Admittedly, Red Hat CoreOS is probably a very real thing while Red Hat Silverblue is probably just vapourware. :)

Now particularly when you're talking about the command line, Fedora _is_
packages and particularly given the default to a Fedora userspace, calling it
"fedora-toolbox" does make sense. But that said I still have concerns over the
branding.

Yes, I understand. My operating principal has been to mimic a familiar RPM-based environment with the toolbox. So, I'd expect a Fedora toolbox to have Fedora RPMs, and a RHEL toolbox to have RHEL RPMs. At least by default, because one can always fiddle with the flags and create a Fedora toolbox on RHEL and what not. Just like the toolbox defaults to the host's VERSION_ID.

It's just that I work on Silverblue, and while Fedora Silverblue is a real thing, I have never even heard of Red Hat Silverblue. Hence the bias towards all things Fedora comes from a desire to keep it simple and get the ball rolling. I certainly don't want to enforce Fedora content on RHEL users by default.

The idea of solving two problems at once by merging the coreos-toolbox and
fedora-toolbox and just calling it "toolbox" hence has a lot of appeal.

Yes, just toolbox (or workshop) sounds fine to me. Would we also rename the Git repository, RPM package, etc.? Or shall we just rename the script?

I am apprehensive of claiming a spot in the global namespace. Especially for the Git repository because toolbox is such a generic name.

[halfline]

so the scope of this program isn't just for people working on fedora, right? it's primarily for people who happen to use fedora to get completely unrelated work done. fedpkg is for people packaging programs for fedora. it's for us, not for our users. this program is for our users.

[debarshiray]

so the scope of this program isn't just for people working on
fedora, right? it's primarily for people who happen to use fedora
to get completely unrelated work done. fedpkg is for people
packaging programs for fedora. it's for us, not for our users.
this program is for our users.

Yes, that's right.

However, at least for the time being, we are not focused on making this work on every single distribution out there; and some bits of configuration (eg., the URL of the OCI image registry, the name of the default image, etc.) might change between Fedora and RHEL.

[yuqi-zhang]

From the other issue (regarding coreos v. silverblue):

Yes, we did identify the differences during our chat in #fedora-coreos but my understanding was that we still wanted to share it as much as possible.

Yes, I agree. Apologies for mis-interpreting the initial discussions. Since the coreos use case was much smaller I opted to create a very simple toolbox to build into our existing pipeline. Moving forward I think consolidating the two would definitely make sense.

However, I don't know what troubleshooting exactly looks like in this case.

So I can only speak to what RHCOS originally envisioned to use the toolbox for: essentially run a debug tool inside a failing host in a openshift cluster. In that sense the toolbox command only had to run a privileged container that can mount the host fs and that's all.

So, I'd expect a Fedora toolbox to have Fedora RPMs, and a RHEL toolbox to have RHEL RPMs.

Alternatively, the fedora toolbox can just by default run a fedora image, and the rhel toolbox to default to a rhel image would probably work as well?

Yes, just toolbox (or workshop) sounds fine to me.

The main reasoning from a coreos perspective is that that was the chosen name for CL: https://github.com/coreos/toolbox I have no strong feelings either way regarding naming.

I'll try out @debarshiray's toolbox on coreos systems later this week and see if there's any compatibility issues. Again, apologies for the lack of follow up, and thanks for trying to consolidate!

[halfline]

Hi,

However, at least for the time being, we are not focused on making this work on every single distribution out there; and some bits of configuration (eg., the URL of the OCI image registry, the name of the default image, etc.) might change between Fedora and RHEL.

Okay, my take is that's not a strong enough reason to include fedora in the name. I mean we don't call firefox fedora-firefox because it's default start page is start.fedoraproject.org. Distros are expected to do per distribution customizations of software, I think.

I don't think it's a big issue if it doesn't work on other distros either. I mean that's the responsibility of other distros, if they see the value. of course, if it explicitly should be fedora only that's a different story.

[dustymabe]

I'll try out @debarshiray's toolbox on coreos systems later this week and see if there's any compatibility issues.

thanks @yuqi-zhang !

[yuqi-zhang]

A couple of initial questions:

1. RHCOS systems do not have buildah by default. It may be nicer to have a split between the image creation and the container creation functionality-wise?
2. I'm not very familiar with descriptors, where does 2>&42 pipe the error to? On RHCOS systems this causes an error: 42: Bad file descriptor, or am I using this wrong?
3. The container fails to start with and error that claims it doesn't exist, even though I created it and can see it through podman. This is possibly because some of the flags are causing issues, I will investigate further.
4. Would it be a good idea to allow for dynamic mounts? Or just more/less custom environment vars in genera?

[debarshiray]

1. RHCOS systems do not have buildah by default. It may be nicer to
   have a split between the image creation and the container creation
   functionality-wise?

Yes, sure.

The customized image that's locally created with buildah is there to fill in the pieces that are specific to each user. eg., creating a user to match $USER on the host. If those things are not needed in CoreOS, or if they can be done differently, then the intermediate image isn't necessary.

1. I'm not very familiar with descriptors, where does 2>&42 pipe the
   error to? On RHCOS systems this causes an error: 42: Bad file descriptor,
   or am I using this wrong?

This comes from commit d7219ba512434315a3be0b4bcaa00b16cf7e478c

Usually file descriptor 42 points at /dev/null. When you pass the --verbose flag it points at stderr. I couldn't find a better way to implement --verbose. Ideas welcome.

I am curious why it doesn't work on CoreOS, though.

1. The container fails to start with and error that claims it doesn't exist, even
   though I created it and can see it through podman. This is possibly because
   some of the flags are causing issues, I will investigate further.

Interesting. Was it with local changes in your tree?

I just happened to try the toolbox with --sudo after a long time, and ran into various podman errors. The fedora-toolbox --sudo create worked, but then sudo podman ps --all and fedora-toolbox --sudo enter, etc. are erroring out:

[rishi@bollard ~]$ fedora-toolbox --sudo create
[rishi@bollard ~]$ sudo buildah images
IMAGE ID             IMAGE NAME                                               CREATED AT             SIZE
9ffedc39f7f2         registry.fedoraproject.org/f28/fedora-toolbox:28         Sep 26, 2018 18:07     602 MB
1b2c5602ac02         localhost/fedora-toolbox-rishi:28                        Nov 6, 2018 19:53      602 MB
[rishi@bollard ~]$ sudo podman ps --all
error decoding container status for container dd1c6b885b53c712c177ce14ed23a63572802fd374d8086d1ddc9200e5a725f9: EOF
[rishi@bollard ~]$ fedora-toolbox --sudo -v enter
unable to get container state: error decoding container status for container dd1c6b885b53c712c177ce14ed23a63572802fd374d8086d1ddc9200e5a725f9: EOF
/usr/bin/fedora-toolbox: failed to start container fedora-toolbox-rishi:28
[rishi@bollard ~]$

I have:

[rishi@bollard ~]$ rpm -q podman runc
podman-0.10.1.3-3.gitdb08685.fc28.x86_64
runc-1.0.0-57.dev.git9e5aa74.fc28.x86_64

Looks like a podman bug that needs to be reported.

1. Would it be a good idea to allow for dynamic mounts? Or
   just more/less custom environment vars in general?

Umm... I guess it depends on the specific use-case?

[debarshiray]

So, where do we stand with the naming issue? I am fine with dropping the fedora prefix if you think that toolbox or workbox aren't terribly generic names to put in the global /usr/bin namespace.

Or have we chosen to swallow the fedora-toolbox moniker for the time being?

[yuqi-zhang]

I have no strong feelings either way and defer to more experienced folks regarding naming, but I think it's looking like the generic "toolbox" name is in the majority?

Sorry for the delayed response, I am on a trip until next week. I'll look into the issues above when I'm back.

[dustymabe]

I have no strong feelings either way and defer to more experienced folks regarding naming, but I think it's looking like the generic "toolbox" name is in the majority?

toolbox works for me

[halfline]

i'm surprised you all think toolbox was the majority favorite. i thought consensus had clearly settled on @cgwalters idea of -

[debarshiray]

I have started with the renaming paperwork. With the 0.0.5 release just out and lingering in Fedora's updates-testing repository, this is as good a time as any.

This Git repository itself has now been renamed to toolbox.

Next I will rename the various bits and pieces inside the repository - sources, build system, etc.. Then it's matter of rolling a new tarball and going through the Fedora packaging process.

[debarshiray]

The contents of this Git repository have been renamed in #52.

All that remains is rolling out a new tarball and the downstream packaging. I'd like to wait a bit so that the 0.0.5 release gets a chance to hit the stable updates repositories before rolling the next release.

Closing.