Habitat: Streaming log output for Builder and long term storage on job complete

[reset]

When a build is initiated we need to stream the output as close to realtime to a subscribing client on the builder-api gateway. The initial subscriber we should develop for is the builder-web client.

Once a job is complete we will need to store those logs in some sort of long term storage solution.

[christophermaier]

General approach:

Logs will be shipped from the worker to the job server that manages the shard the job resides on. As we accumulate logs, we'll write them to files on disk on the job server. When jobs are finished and logs are all gathered, we move the logs to long-term storage in S3, deleting the file-on-disk after successful upload, and storing the S3 location in the job's record in the database.

Both export from the worker and ingestion on the job server will take place in separate threads communicating over a dedicated ZMQ socket pair. While log retention is important, communication about job state takes priority; combining log and state traffic on the same channel could cause state information to back up. It's fine for logs to lag, though.

Streaming to the API gateway will likely happen with websockets. A client will "subscribe" to a job via WS, indicating how much of the they already have (probably "0" in most cases). The request will be routed down to the appropriate job server and whatever portion of the log the client lacks will be sent back to it. Additionally, as more log chunks come in, they will be forwarded on to any clients that are subscribed.

We'll need new protocol messages for these log interactions, potentially some kind of subscription protocol, and communication of which shards a job server supports to achieve proper routing.

[adamhjk]

We need to design (if not implement) for the long term storage location to be local. We know we won't be able to mandate S3 for everyone. We might mitigate this by doing something like we do in erchef, where we ship a fake S3. :) I'm fine with that. Just worth remembering.

[christophermaier]

@adamhjk Right, air-gapped should be a thing. @reset mentioned the "fake S3" route, possibly using something like https://minio.io/

Long term it might be nice to have some kind of pluggable storage strategy, though at this stage I'm not sure exactly what shape it'd take.

[adamhjk]

Ya - I think saying v1 of the air-gapped private deployment would use a fake s3 makes sense, and I would avoid making it pluggable right this second

[smith]

Streaming to the API gateway will likely happen with websockets.

Have we compared websockets vs. server sent events? SSE seems to be simpler and more in line with this use case. http://stackoverflow.com/questions/5195452/websockets-vs-server-sent-events-eventsource

[christophermaier]

@smith Good point... implementation choices on that portion are still undetermined, but SSE does look appropriate. Thanks for the feedback 👍