Flux: Move more functionality into fluxd

[paulbellamy]

Notes from discussion with, @paulbellamy, @squaremo, and @awh. Regarding how we can move more functionality into fluxd in order to better support standalone deployments, and better security.

Are there still fluxd and fluxsvc in standalone deployments? or does fluxd serve the http API?
...discussion... You should either tell it to connect to weave cloud or serve http requests. We prefer only having a single "fluxd" image which you run (with memcached) with different flags.

fluxsvc becomes more of a routing proxy to fluxd. So the platform RPC interface will mirror HTTP.

Workflows:

automate      Turn on automatic deployment for a service.
deautomate    Turn off automatic deployment for a service.
lock          Lock a service, so it cannot be deployed.
unlock        Unlock a service, so it can be deployed.
ignore
  Handle with annotation in the service manifest

history       Show the history of a service or all services
  Need push protocol for fluxd to send events to fluxsvc. Long-polling, or
  some other mechanism to push. Consider making history a cloud-exclusive.

list-services List services currently running on the platform.
  Fairly the same, might be some image metadata checking on fluxd

list-images   Show the deployed and available images for a service.
  Fluxd need to fetch images for list-images. Will need a cache (run memcached
  in same pod).

release       Release a new version of a service.
  - Just changes stuff in git. Which is our record.
  - Makes resumable releases much trickier. (maybe a little local database for
    resumable jobs, but now you have volume mounting issues). Third-party
    api server from kubernetes, store a note there, maybe.
  - Slack notifications when finished: we want them in just cloud. Can piggyback
    on the fluxd event pushing.
  - fluxsvc (or standalone fluxd) stores the commit it creates in the repo, and waits
    for an event from fluxd with the same or later commit-hash.
  - Fluxd needs to track "last-commit applied", so we can answer "have I applied
    this commit yet". Adam suggests using a moving git tag, which is more
    persistent. How do we differentiate different instances? Maybe k8s cluster
    ID? To research
  - Rough workflow
    1. Fluxctl asks to do a release of xyz:123
    2. fluxsvc forwards that to fluxd (if in cloud mode)
    3. Fluxd serializes requests and does the git work and returns a commit ID
    4. Fluxd waits for new git commits, and applies them
    5. Fluxd emits an event saying last commit id applied
    6. fluxctl waits for commit id given to have been applied.

check-release Check the status of a release.
  Jobs become simpler, as they can live in fluxd instead of in a DB.
  Most of check-release is checking on job status, so it becomes simpler.

get-config    display configuration values for an instance
set-config    set configuration values for an instance
  Ignore is tricky. Could add a flag to fluxd to load config from file (if running
  independently) or just load from weave cloud

status        display current system status
version       Output the version of fluxctl
  Roughly the same as now

watch
unwatch
  roughly the same

diff
  Easy, can have fluxd send the diff back to the client instead of listing all files.

save
  Unchanged

[paulbellamy]

For migration, having a totally separate api endpoint, e.g. /api/flux2, and run two versions for a while. Seems reasonable. Needs more thought.

[squaremo]

Need push protocol for fluxd to send events to fluxsvc. Long-polling, or some other mechanism to push.

Or .. HTTP POST. Which only just occurred to me.

[squaremo]

Can we do this piecewise? I wonder if we can just move git sync into the daemon, for example, and not expose an HTTP API from it yet. So the command-line client gets targeted at a service, whether running standalone or in weave cloud, as before.

fluxctl list-services

Operates as before; the service asks the daemon over RPC.

fluxctl list-images

Operates as before; the service asks the daemon for the services, then looks up the images.

fluxctl history

We'll have to feed events back to the service, so this needs an endpoint for the daemon to post to, and Slack notifications will have to be driven by that.

fluxctl save

Same as before: the service asks for the config via RPC.

fluxctl release

Trickier. We would have to push this down into the daemon and make it part of the RPC; and, we'll have to implement the "push commits then wait for them to have been synced" mechanism.

Automation

We can keep this running in the service, and simply prod the daemon to do a release when we detect that it's necessary. Since we won't have access to the config files, we'll have to go back to using the machinery for listing services -- perhaps we can add an option for listing the configured rather than the running services into the RPC.

fluxctl [de]automate and fluxctl [un]lock can stay in the service.

[squaremo]

Rough plan for the first bit above (somewhat parallelisable):

1. move the release code to the daemon and make that all work again
   1a. rewrite bits of automation, in particular
2. implement the sync loop in the daemon, change release to update files and commit/push, make that all work again (including waiting for releases to be applied)
3. add hook to service, and follow through to daemon; consider adding fluxctl sync to do a one-off sync.
4. add events endpoint, post release events to it from the daemon, and drive Slack hook from the events

[squaremo]

Can we do this piecewise?
[...]
fluxctl release

Depending on the release specs given, this does some image lookups, so we have to either move the registry code to the daemon, or do lookups in the service before handing exact results on to the daemon.

[squaremo]

On backwards compatibility:

The daemon should end up supporting almost the same API as the service, so that it can be targeted by fluxctl directly. Those calls, when targeted at weave cloud, will be forwarded via RPC. (We also want to remove old methods from the protocol, so that the service does not have a remote control, but it doesn't have to be straight away).

However we still have old daemons connected to the service, who won't understand the new bits of protocol. We can either:

• effectively deprecate old daemons by returning errors for the new methods (e.g., Release), and always forwarding the new methods;

• implement the new methods in terms of the old methods; i.e., more or less what we have already. One problem is that this has to be done in the RPC client, since that's where we get to distinguish between the old daemon and new, which means all requests are sent across NATS and all work happens where the daemon is connected (maybe that's a good thing, since it will have the RPC client right there ...).

• add new API versions of the methods that now get forwarded. One problem here is that using a mix of the new and old API will probably be a disaster; but we can deprecate the old methods in the RPC client at the same time, so that there's no chance of doing this.

[paulbellamy]

all work happens where the daemon is connected (maybe that's a good thing, since it will have the RPC client right there ...).

As noted in discussion this will be the same when we are doing more/all work in fluxd anyway, so this isn't a big loss to me.

[paulbellamy]

Deprecate old daemons and forward new methods seems easiest in terms of implementation and maintenance. Let's move fast while we have a small user base.

[squaremo]

Closing this, as git sync -- which includes moving most things into the daemon -- is merged to master.