Habitat: Habitat Supervisor should not require running as root

[chetan]

In the recent release of habitat 0.11.0 (via #1325), the handling of root (and sudo) changed quite a bit to become more explicit in certain areas. In particular, the supervisor (hab sup) now requires root to run, as seen here:

    if !am_i_root() {
        try!(ui.warn("Running the Habitat Supervisor requires root or administrator privileges. \
                      Please retry this command as a super user or use a privilege-granting \
                      facility such as sudo."));
        try!(ui.br());
        return Err(sup_error!(Error::RootRequired));
    }

https://github.com/habitat-sh/habitat/blob/master/components/sup/src/command/start.rs#L87-L93

With the following note in the PR:

hab-sup start - the Supervisor itself deeply assumes root permissions for installation, process supervision, etc

Prior to 0.11.0, we were happily running hab sup as non-root under the hab account, as our packages themselves did not require root either. This was a pretty severe change for us as it broke deployments of packages built with 0.11.

Can we get a bit more context around the statement that hab sup "deeply assumes root permissions"? AFAICT, there is no need to run as root unless your application needs (usually it to bind to a privileged port, and even this can be worked around via setcap).

Unless I'm missing something, I would much rather this be a decision made at deploy time, vs requiring it for all packages all the time.

[adamhjk]

If everything is run as hab, you will have some interesting privileges going on. For example, your services will (by definition):

1) be able to write to their own configuration
2) be able to write to packages, if you want to allow any kind of updating
3) you'll never be able to run multiple services with multiple different user contexts

That said, if you're okay with those risks, it would probably work?

[chetan]

So the core issue comes down to being able to write the configs as user A (e.g. root) and run the application as user B (e.g. hab) which makes total sense.

FWIW, in our previous setup we were running containerized applications where:

• /* is owned by root:root
• /hab/svc/<pkg> is owned by hab:hab
• hab-sup runs as hab
• app runs as hab

I see a couple of options to resolve this issue:

• Change the root check in hab-sup to only WARN and not error out
• Initially run as root and write the configs as a different user, i.e. habsup
  
  
  
  • Start app as hab then drop privs to habsup (this option wouldn't allow upgrading or reconfiguring on the fly)
  
  
• Use setcap to grant CAP_SETUID -- this would allow running the supervisor as habsup but still allow it to run apps as hab
  
  
  
  • Needs some investigation - I'm not sure if this would still allow the supervisor to setuid(0)
  
  

[reset]

@chetan Just to be upfront with you, it's unlikely that the team will revisit removing the root requirement from the runtime in the near future. If this is something that you require or just feel very strongly about, I'd suggest pursuing the work on your own. If a solution which provides the same user experience that we currently have is proposed then we'll definitely accept the work.

Let me give you a little context as to how the supervisor will most likely be shaping up over the next few months before you go ahead and make a decision to do any work. The supervisor will be able to run without any packages as a sort of "shell" of a runtime so it can still participate in the gossip ring. You'll then be able to interact with it from a CLI tool (probably just hab) to ask the supervisor to start one or more packages for you. The user experience for people using containers will still be there - you can just ask the supervisor to start and run packages in one command, but this will be how it's working under the hood.

You can see that the runtime becomes a sort of system process and something which could actually run as PID 1 even in non-containerized systems similar to runit/systemd/etc.

[chetan]

Sure, I can see that replacing or supplementing something like systemd is one way to go, but locking into only this strategy removes a lot of the flexibility that made habitat so attractive.

Perhaps the functionality needs to be split up so
there can be different launcher mechanisms depending on your use case.

[reset]

We're not locking into that user story, as I mentioned, the current behavior will be preserved for use in containerized environments 😉

[adamhjk]

What if we did the same thing, but simply made it a warning rather than
fatal? If someone preps the system appropriately, it would run non-root -
it's just not going to work that way out of the box.
On Sat, Oct 29, 2016 at 2:03 PM Jamie Winsor [email protected]
wrote:

We're not locking into that user story, as I mentioned, the current
behavior will be preserved for use in containerized environments 😉

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/habitat-sh/habitat/issues/1408#issuecomment-257115924,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQ0Kue8-4fucuiML3ulU7yUPPwL9XXks5q47SKgaJpZM4KiuLk
.

[adamhjk]

Also, Chetan- can you run docker? Because the daemon is also root.

On Sat, Oct 29, 2016 at 2:05 PM Adam Jacob [email protected] wrote:

What if we did the same thing, but simply made it a warning rather than
fatal? If someone preps the system appropriately, it would run non-root -
it's just not going to work that way out of the box.
On Sat, Oct 29, 2016 at 2:03 PM Jamie Winsor [email protected]
wrote:

We're not locking into that user story, as I mentioned, the current
behavior will be preserved for use in containerized environments 😉

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/habitat-sh/habitat/issues/1408#issuecomment-257115924,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQ0Kue8-4fucuiML3ulU7yUPPwL9XXks5q47SKgaJpZM4KiuLk
.

[bdangit]

FWIW, there are certain container schedulers and resource management
systems like Openshift (which uses Kubernetes underneath) where they don't
allow containers to start their processes as root. If the system does allow
it, you will see a warning in Stdout. However, I'm neutral to this issue.
Security of an app should be thought of as defense in depth.
On Sat, Oct 29, 2016 at 11:12 AM Adam Jacob [email protected]
wrote:

Also, Chetan- can you run docker? Because the daemon is also root.

On Sat, Oct 29, 2016 at 2:05 PM Adam Jacob [email protected] wrote:

What if we did the same thing, but simply made it a warning rather than
fatal? If someone preps the system appropriately, it would run non-root -
it's just not going to work that way out of the box.
On Sat, Oct 29, 2016 at 2:03 PM Jamie Winsor [email protected]
wrote:

We're not locking into that user story, as I mentioned, the current
behavior will be preserved for use in containerized environments 😉

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
https://github.com/habitat-sh/habitat/issues/1408#issuecomment-257115924>,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAAQ0Kue8-4fucuiML3ulU7yUPPwL9XXks5q47SKgaJpZM4KiuLk

.

—

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/habitat-sh/habitat/issues/1408#issuecomment-257116287,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA4-WS4DV40xDDdU-xizUGyAJ5peQCymks5q47a6gaJpZM4KiuLk
.

[chetan]

@reset

The user experience for people using containers will still be there - you can just ask the supervisor to start and run packages in one command, but this will be how it's working under the hood.

and

We're not locking into that user story, as I mentioned, the current behavior will be preserved for use in containerized environments

Preserved but must still run as root?

@adamhjk I think a warning or some sort of flag to ignore+warn would work.

re: docker: we don't currently run docker but assuming that we did, we would need to run the supervisor inside the container as root as well. If you run 10 containers, then every one of those also run supervisors running as root, unless I am mistaken on how things are meant to work.

Stepping back a bit, my preference leans toward tools which are highly modular and composable. Baking the root requirement into the tool itself forces you into a particular deployment model without any way to avoid it -- that is, it makes the choices for you instead of letting me decide how I want to deploy my software.

[reset]

@chetan Yes, the _user experience_ will be preserved - details like permissions aren't an encompassing user story. I was providing you some insight here to help you understand _why_ we made these choices.

With that said, as @adamhjk pointed out, a warning instead of a fatal should be just fine.

[bodymindarts]

I'll have a go at tackling this as part of our CF-exporter task.

[reset]

@bodymindarts you completed this work, yeah?

[jeremymv2]

I'm curious where the discussion on this stands. I have a requirement such that all processes in Docker containers may not be run as user root.

My particular hab service doesn't require root and will happily run as user hab by specifying pkg_svc_user.

However, when I restrict the allowed user process context for the container in my docker-compose.yml to a non-root user, with user: "hab:hab" the supervisor (running as user hab) fails to start properly with:

hab_svc_1    | ∅ Running the Habitat Supervisor with root or superuser privileges is recommended
hab_svc_1    |
hab_svc_1    | hab-sup(SS)[src/manager/service/spec.rs:159:12]: Unable to write or read to a service spec file at /hab/sup/default/specs/hab_svc.spec, Permission denied (os error 13)

[tashimi]

@jeremymv2 so this work didn't end up getting picked up by @bodymindarts but it's still something we plan to do. We were just talking about it as a team today since there are several use cases it's going to be important for (high performance computing clusters, openshift clusters).