Vertx-web: java.lang.IllegalStateException: Request has already been read

[mouse256]

Version

• vert.x core: 3.5.0
• vert.x web: 3.5.0

Context

A handler on a router can be called twice under heavy load.
We've seen in our system when trying to upstep to 3.5.0 that our bodyhandler sometimes got executed twice. It only happened sporadic under heavy load, so it seems like a racecondition.

What I've been able to debug:

• The bodyhandler is called again when calling context.next() from within it's own endHandler(). See attached trace: trace.txt
• It can only be caused if the actualHandlerIndex in io/vertx/ext/web/impl/RouteImpl.java is set to 0, which should be 1 at that time.
• However a reset is done when selecting the route in io/vertx/ext/web/impl/RoutingContextImplBase.java. And I think that's problematic since the route object is global and not linked to the context.

I believe this commit is actually the culprit: https://github.com/vert-x3/vertx-web/commit/6a1da305ff109cdd31d147e7e10352fa07b06db3
The route.resetIndexes(); on line 92 is problematic there.

Do you have a reproducer?

No, could not reproduce it in a test since it seems highly timing related...

[steven-aerts]

I think the problem here is a race condition in the RouteImpl, and has nothing to do with the body handler.

In 6a1da30 two counter were introduced to keep track of the current handler: actualHandlerIndex and actualFailureHandlerIndex. The problem is however that these counters were defined in the Route and a route can handle requests/RoutingContexts in parallel.

This causes a race condition where resetIndexes is called for one request while another request is being handled. This causes the handler for the other request to be called twice.
So handlers which are not IdemPotent, like the BodyHandler will get into troubles.

We can trigger this issue rather easily when putting the system under high load.

To fix this issue, I think the counters need to be moved to the routeCountext.
Or revert all this, and return a copy of the RouteImpl when handler() or failureHandler() is caller, so that one route fill only handle one callback.

[steven-aerts]

Looping in @slinkydeveloper who developed this feature.

[slinkydeveloper]

Working on it!

[ieugen]

We have noticed that pages don't load after moving to vertx 3.5.0 from 3.4.2 . We did some investigation and we believe it has something to do with this.

@vincentDAO wrote this code to reproduce the issue. I only have image, Maybe he can paste the code.

I hope it helps.

p.s. How fast do you think we will have a fix release after a solution is found? This issue currently affects our staging environment. We would not like to roll back.

[vincentDAO]

In our case (at @ieugen comment); the cause of hang page is caused by another library VertxBeans; not relate to Vertx Web;

I reported a bug to them https://github.com/rworsnop/vertx-beans/issues/12

[pmlopes]

Thread safety is now merged.

[razzbee]

for me, I had to enable multipart before the body handler :

           //init server
            val server = vertx.createHttpServer()

            server.requestHandler{
                r->r.setExpectMultipart(true)
            }

                //set routing
                var router = Router.router(vertx)

                //handle request body including file upload
                router.route().handler(BodyHandler.create())