Vertx-web: RouteImpl.java synchronized

[eun-ice]

I think that RouteImpl.java is synchronizing to much.

Basically the call flow is

router.accept(..) 
  routingContext.next()  
     routingContext.iterateNext() 
        synchronized route.matches(...)
        synchronized route.handleContext(...)
              contextHandler.handle(...)

This means, every contextHandler is called in a synchronized method inside the route it is registered to until the user uses a thread pool (like using a BlockingHandlerDecorator)

If I see correctly, the only reason for the synchronizeds is to protect the access to these collections:

private final Set<HttpMethod> methods = new HashSet<>();
private final Set<String> consumes = new LinkedHashSet<>();
private final Set<String> produces = new LinkedHashSet<>();

These collections could be replaced by copy-on-write variants, since these collections rarely change but are read very often.

[purplefox]

Vert.x code is optimised for the normal case where you're using standard verticles.

In this case your routing context will always be called by the same thread (event loop). Biased locking then kicks in which makes the overhead of synchronized almost zero.

What issues are you finding in practice?

[eun-ice]

I am investigating on whether to port a very large servlet based project to vert.x so for the moment I am just reading code and trying to figure out whether it's worth the invest.

I start the webserver like this:

verty.createHttpServer().requestHandler(router::accept).listen(8080);

If I read the code correctly, the router::accept gets called inside the netty event loop group, which defaults to 2 * Runtime.getRuntime().availableProcessors() threads.

If that is true, you would effectively be blocking the netty loop.

[purplefox]

Vert.x uses the reactor pattern (well.. actually multi-reactor) which means (in the usual/ideal case) all events are dispatched on an event loop.

I don't see anything blocking in router:accept - can you elaborate?

[eun-ice]

router::accept in turn leads to calls to route.matches and route.handleContext which are synchronized. The user handler is called inside handleContext and therefore also synchronized.

[purplefox]

That doesn't mean anything blocks.

Standard Vert.x verticles are always executed by the exact same event loop, so there would be zero contention in the normal case.

For un-contended locks on the JVM synchronized overhead is close to zero due to biased locking. https://blogs.oracle.com/dave/entry/biased_locking_in_hotspot

We use this same locking scheme heavily throughout Vert.x. If you're not convinced it doesn't hurt performance, please take a look at: https://www.techempower.com/benchmarks/#section=data-r12&hw=peak&test=plaintext

As you can see Vert.x provides extremely fast HTTP performance (over 3 million req/resp per sec on a single server) - outperforming almost every other framework on the market.

Every one of those requests will be going through several methods which are synchronized just like the ones you have seen. The key point to take home here is that these are _uncontended_ monitors.

[vietj]

that would be a problem if a method like wait() would be called inside this synchronized method.

however in this case, the synchronized block is only used for ensuring exclusive access and data visibility (which is obvious on an event loop)

On Mar 11, 2016, at 10:13 AM, eun-ice [email protected] wrote:

insider router::accept route.matches and route.handleContext and the user handler is called inside synchronized methods.

—
Reply to this email directly or view it on GitHub https://github.com/vert-x3/vertx-web/issues/328#issuecomment-195279492.

[eun-ice]

I get that un-contended locks are fast.

But if you create a webserver with

vertx.createHttpServer().requestHandler(router::accept).listen(8080);

You are creating a netty event loop with many threads and channeling that through the synchronized methods. These are not un-contended locks.

[vietj]

_netty event loop with many threads_ : can you clarify what you mean because the principle of event loop is to have a single thread

[purplefox]

Creating a web server in Vert.x doesn't create any event loops or threads.

[eun-ice]

@vietj

the correct terminology would be "event loop group", as created by vert.x in VertxImpl:

eventLoopThreadFactory = new VertxThreadFactory("vert.x-eventloop-thread-", checker, false);
eventLoopGroup = new NioEventLoopGroup(options.getEventLoopPoolSize(), eventLoopThreadFactory);

options.getEventLoopPoolSize() defaults to said 2 * cores

The idea behind netty event loop groups is that you have multiple threads each with an event loop. Netty guarantees to some extend that the events of the same connection are handled by the same thread. That's fast since moving data from core to core is slow, even if no synchronization has to happen.

I agree that all events to one connection should be handled in the same thread, however different connections should be distributed across CPU cores.

Having a base component that is called for every connection (like Route) "synchronized" means that you will have contended locks or single thread processing.

[purplefox]

Creating a web server in Vert.x doesn't create any Netty event loop groups either. We maintain a single event loop group per _Vert.x_ instance, not per web server.

You are misunderstanding the Vert.x threading model. Instances of Router and route are never shared between event loops. We guarantee that in the Vert.x threading model. If the standard Vert.x threading model is obeyed the locks you see will never be contended.

[purplefox]

Also, I'm very familiar with the Netty threading model as Netty adopted that model as a direct result of the requirement from us while Norman was working on the Vert.x team :)

[eun-ice]

So there are multiple Routers, one for every thread?

[vietj]

@eun-ice yes there are (now it's clearer)

basically you do in a Verticle:

public void start() {
  Router router = createRouter(httpServer);
  // etc...
}

To scale, this verticle will be instantiated N times where N is the number of instance you specify at deployment time.

That means N HttpServer are started as well, but when this happens, Vert.x only binds one server and the extra servers (provided by the N-1 instances) are registered as handlers along with their event loop.

When the actual server serves a request, it will chose one handler, i.e among the N verticles.

As consequence, each connection is always handled by the same thread among the registered event loops for this http server.

[purplefox]

Let me try and summarise this and hopefully make things clearer.

Vert.x maintains a pool of event loops - by default we choose number of cores * 2 (although this is configurable). Let's say you have 4 event loops.

You write your "webserver" verticle (you can see examples in the examples repo) - inside your verticle code you create your web server, router, whatever.

You then deploy N instances of this verticle. You want to have at least as many instances as there are cores on your machine. You could have 1000 though if you wanted.

When you deploy a verticle instance, Vert.x associates an event loop to that verticle instance, that event loop corresponds to a single thread. Vert.x will then guaranteed that the code in the verticle is _always_ executed by that exact same event loop thread.

Because you know your code is always executed by the exact same thread and never any other threads you don't have to worry about multi-threaded issues in your code (e.g. synchronizing stuff), this is a big win for the developer and one of the major features of Vert.x.

As we know the verticle is always executed by the same thread we also now know that the code in Router, Route etc are also always executed by the same thread, as each verticle instance has its own Router, Routes etc.

That means, for this normal case we don't actually need the synchronized blocks at all in Router, Route etc [*], but since thy are always (in this case) uncontended you don't have to worry about them, as biased locking makes the overhead pretty much zero.

[*] So... you may be asking if we guarantee that the verticle is always executed by the same thread why bother with synchronizing the Router, Route etc at all?

The reason we do this is to support a secondary case - where Vert.x is embedded outside the normal verticle threading model and when using worker verticles (for legacy blocking code). But you shouldn't have to worry about that in the normal use case.

HTH.

[eun-ice]

Ok, now I get it. I did not know that I have to deploy the verticle multiple times.

Thank you guys for explaining it to me!

[purplefox]

This is why we say Vert.x has an "actor like" concurrency model.

To scale you deploy more instances of the "actors" (the verticles), but each verticle is single threaded.

Contrast this with a "traditional Java-style" multi-threading approach to concurrency, where you only have one instance of your code, but have multiple threads executing it concurrently.

Arguably the latter is less effective and more error prone as it's hard to avoid bottlenecks due to contention and MT style concurrency like this is prone to deadlocks, livelock and race conditions.

[eun-ice]

I understand the simplicity of the concept. But I'm not comparing vert.x to traditional java style concurrency but to other modern "reactive" approaches.

This is where my confusion came from. Within the typical reactive observable pattern everything comes inside event loop groups and you just chain in the thread-pools where needed.

observable
.subscribeOn(Schedulers.io())
.observeOn(...)

http://spray.io/msug/#/8

[purplefox]

Could you elaborate? I'm not sure I have followed your point.

[eun-ice]

@purplefox : It's just a slightly different approach. With vert.x I create multiple verticles to have one verticle per thread. With reactive streams I create one observable chain which is then invoked concurrently, and in the chain I can control the level of parallelism and thread pool at every point.

[purplefox]

Well. Vert.x supports both reactive streams and Rx. I don't believe either reactive streams or rx mandates a specific threading model, so I'm not sure I really understand the point still.

[eun-ice]

Reactive streams/rxjava give you the control over the threading model. But vert.x makes sure a verticle is always in one thread, so that somewhat limits you at the start of the pipeline.

[purplefox]

Limit in what way? Perhaps you could provide an example to illustrate the point?

[eun-ice]

It is not a very important debate to have.

Assuming you create an rx stream of http server requests, in vert.x you will have one thread piping requests into your rx stream.

http server -> router verticle -> rx stream -> operation 1 -> operation 2 -> operation 3 -> response

Assume request processing/routing performs best at 8 threads, so you will have to instantiate the verticle 8 times, leading to 8 instances of the rx stream.

Assume operation 2 on the other hand performs best at 2 threads. Then you will need to create a shared thread pool between verticle instances or move the operation on another verticle.

It is still possible, so it's not a big debate to have.

[purplefox]

I don't see the Vert.x threading model precludes pipelines like you describe.

The Vert.x event bus supports Rx so you can send operations off to different numbers of verticles which can send their results back when complete. This can all be composed in the normal Rx way.

[pmlopes]

I'll close this issue since it was more about understanding the internals of vert.x than a real vertx-web issue. It can be re-open later if needed.