Habitat: Revisit behavior around optional bindings

[christophermaier]

From @adamhjk in Slack:

heya - I was having a conversation with some folks today about option/required bindings
and it sounds like the behavior right now is an optional binding is one you can pass or not pass
but if you do pass it, the supervisor blocks on the other side being available
I think that’s a bug - an optional binding should start no matter what, and fill in as its available
rather than a required binding, which will block until receipot
(otherwise, you can’t really build load balancers, etc etc)
does anyone disagree?
if you don’t, we should take a look and see if we can get a fix in

[seth]

We would very much benefit from the behavior described above for managing the applications we are packaging/deploying via Habitat.

[fnichol]

Adding context here from chat:

This was behavior that I made on purpose in #2294 so that once a user explicitly said they were going to use an optional bind, then treat it as a dependency of the service just like required bindings. Once we started talking about the semantics, we realized that there might be a third style of binding: one that isn't pre-declared in a package and would attempt to bind when started (I thought of these as optimistic binds).

The current behavior of optional bindings allows for packages that can operate with a service dependency and by configuration. For example, the Ruby Scaffolding (when detecting a Rails codebase), sets up an optional binding of database which, if specified, waits until the binding is satisfied before the app is booted up (giving you the stronger guarantees of a required binding). However, if you choose to not use the database binding, the package falls back to requiring a few config options, like db_host, db_port, etc (giving you the ability to wire your app to a pre-running database service such as RDS on Amazon). Without the blocking behavior, the app won't wait for the database-bound service group to show up and immediately falls back to using config values which may not be what was intended (i.e. I said to use a service binding, but you gave up right away and did something else??).

I suspect there was a third binding type that was always lurking here that satisfies the above behavior--the only issue (if it is one) is that the meaning of "optional" might be overloaded here. To be clear, changing this behavior and isn't an accidental logic bug. It would be good if we could get some use case examples for this behavior to make sure we haven't missed anything.

Does that make sense?

[fnichol]

The other detail about the current optional bindings is that every optional binding must be declared in your package otherwise the Supervisor will error out (in other words, it guarentees only pre-declared binding names). I have a feeling that this other binding behavior might only work if you allowed for freeform binding names? It's also hard to determine how long do you wait to attempt to satisfy that binding before giving up (immediately try and give up, have a hardcoded retry count and sleep, make it additionally configurable, etc.). That was why the current optional binding behavior is consistent and deterministic--it's the same behavior as required bindings.

[seth]

I can see where the current behavior is useful and solves some cases. Those cases seem to weight towards "apps not designed for habitat/dynamic environment". In the Rails example above, we are assuming naive app logic in terms of how to handle db connectivity.

What's missing is support for apps that want to take more responsibility for handling the availability of their runtime service dependencies. A given service should be able to gracefully handle some of its dependencies being unavailable -- where the unavailability can be at startup as well as at any point at runtime. And where habitat can provide a big lift to help devs writing such apps is in handling the underlying service discovery and membership/availability changes.

In terms of helping apps that are handling their own "how to handle availability of my runtime service dependencies":

• Be able to specify a bind such that hab will start the service regardless of the specified bind's availability providing the guarantee that when/if the bind becomes available, service config is rendered and restart hook run.
• Allow specifying a bind that is not encoded/declared in a services plan. Why? Because we want to make it possible to upgrade a given service in which we've added a new (_really_ optional ) bind in the new version. For us, we have a coordinator service monitoring our deployed set of apps. Allowing to specify "ahead of time" removes coordination burden and lets things achieve desired state over time.

Proposal:

What if we added a new command line flag to setup this optional runtime bind? Perhaps:

hab sup start seth/taco-delivery --runtime-bind fleet:bicycle-hoard.default

[christophermaier]

After extensive discussion, we're going to add a binding behavior that acts as described above; that is, if Service A has a bind on Service B, A can start up just fine with no running instances of Service B around, but as instances of Service B come and go, Service A will have this exposed in templating data, and will be able to dynamically change its configuration, and thus its behavior, as needed.

This is effectively the same behavior that exists now for binds, except that we will not block Service A from starting.

This was apparently the original desired behavior for binds, but the behavior inadvertently evolved away from it for various reasons. By re-introducing the behavior, we'll be able to more fully support the operation of well-behaved distributed systems, which should be robust and gracefully degrade in the absence of their communicating partner services.

The current binding behavior, where we only allow a service to start when its bound services are up, will be retained, as this is important for legacy applications (e.g., applications that simply fail to start without their associated database). However, this behavior will be a "strict" mode, reflecting the Habitat project's view that modern services should be robust by default; the new binding behavior will become the default behavior. Prior to making this the default, we will give advance notice, in order to allow users time to plan either changes in their current deployments (i.e., to prepare to use "strict mode" behavior), or to update their applications to gracefully degrade in the absence of bound services.

[christophermaier]

This was fixed with https://github.com/habitat-sh/habitat/pull/5024