Armeria: Dynamically adding and removing a service

[trustin]

Let's imagine an API gateway with some schema validation where each route has its own schema for its request body. The list of routes and schemas are retrieved in runtime from some centralized registry. The route and schema can be added, removed or updated.

For Armeria to support this scenario, we need the following updates:

• Provide a way to add and remove a Service from a Server without restarting a Server.
  
  
  
  • Service will need additional life cycle callbacks.
  
  
• Provide a way to add and remove a gRPC and Thrift service from GrpcService and THttpService.
• Improve DocService to reconfigure itself on server configuration changes.
  
  
  
  • ServerListener will need additional life cycle callbacks, e.g. routeAdded, routeRemoved.
  
  

Related conversation: https://line-armeria.slack.com/archives/C1NGPBUH2/p1598973931015200

[aio64]

This feature would be incredibly useful for environments in which new virtual hosts need to be added on an ad-hoc basis. This feature, coupled with FileService, TLS support and Armeria's proxy capabilities would allow us to create extremely robust edge routing servers.

Traefik has such facility exposed through a plugin style solution allowing for a whole span of integrations to providers such as Consul etc. Such implementations could always be community contributed.

Notably too, in environments where Armeria is managed from a container - e.g. Guice etc - sophisticated lifecycle and messaging capabilities typically exist to wrap and nudge any internal Armeria Service state machine: the salient point being, that in such environments there would not be a particular need for Armeria to provide this support OTB - simply an API to discover the target Service and an API through which to deliver events etc.

[trustin]

@aio64, it's awesome to find someone who shares the same vision for this feature! It'll be a great addition to Armeria. The icing on the cake is that this can be fulfilled without making a breaking change. (or at least I don't see it coming 😆)

[amitvc]

@trustin I want to work on this issue. I have started reading about the relevant classes that will require some changes for this.

[amitvc]

Current status - Doing some reading on existing classes and trying to understand what needs to be done.

[amitvc]

@trustin I am trying to envision how this feature will work. Lets take a simple scenario -

```
public void init() {
ServerBuilder sb = Server.builder();

   sb.http(8080);

    sb.service("/", (ctx, req) -> HttpResponse.of("Hello, world!"));

    sb.service("/greet/{name}", new AbstractHttpService() {
        @Override
        protected HttpResponse doGet(ServiceRequestContext ctx, HttpRequest req) {
            String name = ctx.pathParam("name");
            return HttpResponse.of("Hello, %s!", name);
        }
    }.decorate(LoggingService.newDecorator())); // Enable logging

    Server server = sb.build();

    server.start().join();

}

// Dynamically add service route
public void someEvent() {
serverBuilder.service("/topic", (ctx, req) -> HttpResponse.of("New topic added"));
}
```

The someEvent method gets called on some event in the system. On occurrence of this event the user wants to register a new service onto the server.

I wonder if there is a simpler solution --
The user builds all routes and services the way they do it now. For some routes they register callbacks. These routes or services only get activated when those callbacks are fired. We will need to build this mechanism of callbacks for route/service activation but seems like this can be done.

In either case we will need to provide the users an ability to register new services without restarting the server. You mention about some sort of registry for schema and service. Can you please elaborate on this little bit ?

[trustin]

How about providing a way to replace the entire routes? e.g.

Server server = ...;

server.reconfigure(serverBuilder -> {
    serverBuilder.http(8081); // Re-bind at 8081 (no need to implement now)
    // Replace the entire routes with the following two services.
    serverBuilder.service(...);
    serverBuilder.service(...);
});

[amitvc]

I like this ^^^ approach better than what I had in mind 😄 . I will start work on this

[amitvc]

@trustin Couple of question on this -

• The Server class has a ServerConfig that List of ServiceConfig that get's built during ServerBuilder::build.
  The ServerBuilder::virtualHostTemplate maintains a list of ServiceConfigBuilder that are then used to construct the List indirectly maintained by Server class. If we do the server.reconfigure and add new routes and Services to do we clear up the existing data maintained by the ServerBuilder::virtualHostTemplate ?

• What happens to the existing services ? What happens if the service is serving a request and we call server.reconfigure ? Do we allow the request to complete ? What if there are more requests in the backlog queue ?

• If I call server.service(..) to add a new service to the server after ServerBuilder::build is called then the new service is not bootstrap so I am guessing there is some more bootstrapping that happens with service on the serverbuilder::build method.

• In the Server::doStart we create the netty's ServerBootstrap and provide the childHandler responsible for handling the incoming requests via HttpServerPipelineConfigurator [Server::doStart] I don't see how we can remove the childHandlers from the ServerBootStrap in netty. Is there a way to do this or we need to destroy the ServerBootStrap and create a new one when we reconfigure the Server with new routes and services ?

[trustin]

Great questions, @amitvc.

If we do the server.reconfigure and add new routes and Services to do we clear up the existing data maintained by the ServerBuilder::virtualHostTemplate ?

Yes.

What happens to the existing services ? What happens if the service is serving a request and we call server.reconfigure ? Do we allow the request to complete ? What if there are more requests in the backlog queue ?

We could treat this like updating a copy-on-write list. The requests being handled with the old configurations could be handled until they are finished. All other requests handled after the reconfiguration could be handled with the new configuration, which means two configurations can be in action during the transition. In the future, we may want to introduce some option that enforces the old configuration to be purged sooner, like cancelling the requests with old configuration.

If I call server.service(..) to add a new service to the server after ServerBuilder::build is called then the new service is not bootstrap so I am guessing there is some more bootstrapping that happens with service on the serverbuilder::build method.

Server's constructor will be given with an immutable configuration which is not modifiable at all currently. We're gonna have to build a new immutable configuration, and then replace the old immutable configuration with the new one, again like we update a copy-on-write list.

In the Server::doStart we create the netty's ServerBootstrap and provide the childHandler responsible for handling the incoming requests via HttpServerPipelineConfigurator [Server::doStart] I don't see how we can remove the childHandlers from the ServerBootStrap in netty. Is there a way to do this or we need to destroy the ServerBootStrap and create a new one when we reconfigure the Server with new routes and services ?

How about limiting what we can change in runtime at the moment? Let's assume we don't want to change the SSL or port in runtime but only routes.

Then, what we need to do is basically reconstruct a new ServerConfig and provide a way to replace it in runtime, like we update a copy-on-write list.

[amitvc]

Thanks for the feedback @trustin. Won't we have to create a new ServerBootStrap with the reconfigured ServerConfig? If we don't do that the existing netty ChannelHandlers will continue to function. See https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/server/Server.java#L473.

[trustin]

Won't we have to create a new ServerBootStrap with the reconfigured ServerConfig?

Yes, but we can refactor things a little bit to swap an old ServerConfig with a new one. I didn't try it by myself, but should be doable as long as we don't change the server port.

[amitvc]

@trustin Thank you. I think I will push an initial PR end of this week.