Heterogenous silo support
Right now, all silos assume that they all support the same set of grains. We want to remove this limitation.
Why
Removing this assumption could enable these scenarios:
- Controlling grain location: some people want to fine tune grain placement (some grain might require a GPU for computation, some other requires lot of memory, needs to be close to storage, ect.)
- In-place upgrade of silos: support upgrade with different grain type supported (ie: deploy new binary with some grain added, but no modification on already supported grains)
Out of scope (for now)
- Grain versioning
- Stateless grains
- Implicit streaming subscription
- Multi-cluster support
Design
- Add list of excluded grains by assembly loader in config I don't think it is a strong requirement, but it will enable easier testing. It will also allow user to deploy the same package on the cluster even if he want to use this feature.
- All silos need to compute a union of all supported grains across the cluster. Each time a silo join or leave the cluster, this will need to be recomputed
- Gateways need to expose this global list of supported grains to the client
- Update all placement directors to only choose in a list of silos that support the target grain type
Possible optimizations:
- Silos should push the list of supported grain to connected clients when it is updated
Most helpful comment
Merged!