Jetty.project: I/O error while reading input message; nested exception is org.eclipse.jetty.io.EofException: Early EOF

[uedaeita]

I've been using Java8/Springboot and have been bothered by this error, does anyone know what might resolve this error?
There is not much information around on the web, but here's what I have tried:

• updated Springboot from 1.5.19 to 2.1.4. (which also updated org.springframework.boot:spring-boot-starter-jetty)

I was hoping that the latest version would resolve the issue, but it didn't... :/

Error message:

I/O error while reading input message; nested exception is org.eclipse.jetty.io.EofException: Early EOF

Stacktrace:

org.springframework.http.converter.HttpMessageNotReadableException: I/O error while reading input message; nested exception is org.eclipse.jetty.io.EofException: Early EOF
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:216)
    at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:157)
    at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:130)
    at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:126)
    at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:166)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
    at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:117)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:106)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:502)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
    at org.eclipse.jetty.server.HttpChannelOverHttp.earlyEOF(HttpChannelOverHttp.java:238)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1577)
    at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:369)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jetty.io.EofException: Early EOF
    at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1143)
    at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1131)
    at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:318)
    at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:256)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at java.io.PushbackInputStream.read(PushbackInputStream.java:139)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver$EmptyBodyCheckingHttpInputMessage.<init>(AbstractMessageConverterMethodArgumentResolver.java:324)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:193)
    ... 74 more

It would be great if anyone can help out with this, please leave a comment!

[sbordet]

The stack trace shows that Jetty was trying to parse a HTTP request with chunked body, but the client closed the connection before the whole request body was sent.

It appears to me that this is a client issue, so I would analyze the client first.
You can get a hint of what the client does by enabling DEBUG logs on the server.

You can grab a network trace with Wireshark to know for sure what the client sent.

[fs-td-eitaro-ueda]

@sbordet Thank you for the quick reply. As you mentioned, I will work on the client first to see whether this is a problem on the client side or not.

[stale[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[consultantleon]

Convinced this is a regression bug in jetty, MT related.
Initially I thought it's a jersey related issue when using async(), but now have proven that the issue occurs as well doing repeated jersey sync requests.

My test setup:

• jetty server responding to a simple PUT request
• jetty client repeating requests every 30 seconds for 2.5 hours. I got the earlyEOF exception 4 times. Each time the server did not seem to have received anything at all. My current guess is that such 'slow test' causes as many earlyEOF as rapid tests.
• I'll plan to reproduce this in a separate test harness

==> javax.ws.rs.ProcessingException - java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78124bd4::SocketChannelEndPoint@76d5aaa6{localhost/127.0.0.1:7478<->/127.0.0.1:64508,ISHUT,fill=-,flush=-,to=4/0}{io=0/0,kio=0,kro=1}->HttpConnectionOverHTTP@78124bd4(l:/127.0.0.1:64508 <-> r:localhost/127.0.0.1:7478,closed=false)=>HttpChannelOverHTTP@72ea20a4(exchange=HttpExchange@37e8a09d req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@5dce015f(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@69510da{s=START}],recv=HttpReceiverOverHTTP@2559efac(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]
      at: JettyConnectorStreaming.apply(JettyConnectorStreaming.java:323)
  - Cause: java.util.concurrent.ExecutionException - java.io.EOFException: HttpConnectionOverHTTP@78124bd4::SocketChannelEndPoint@76d5aaa6{localhost/127.0.0.1:7478<->/127.0.0.1:64508,ISHUT,fill=-,flush=-,to=4/0}{io=0/0,kio=0,kro=1}->HttpConnectionOverHTTP@78124bd4(l:/127.0.0.1:64508 <-> r:localhost/127.0.0.1:7478,closed=false)=>HttpChannelOverHTTP@72ea20a4(exchange=HttpExchange@37e8a09d req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@5dce015f(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@69510da{s=START}],recv=HttpReceiverOverHTTP@2559efac(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]
      at: InputStreamResponseListener.get(InputStreamResponseListener.java:221)
  - Cause: java.io.EOFException - HttpConnectionOverHTTP@78124bd4::SocketChannelEndPoint@76d5aaa6{localhost/127.0.0.1:7478<->/127.0.0.1:64508,ISHUT,fill=-,flush=-,to=4/0}{io=0/0,kio=0,kro=1}->HttpConnectionOverHTTP@78124bd4(l:/127.0.0.1:64508 <-> r:localhost/127.0.0.1:7478,closed=false)=>HttpChannelOverHTTP@72ea20a4(exchange=HttpExchange@37e8a09d req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@5dce015f(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@69510da{s=START}],recv=HttpReceiverOverHTTP@2559efac(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]
      at: HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:345)

Looking at the code org/eclipse/jetty/client/http/HttpReceiverOverHTTP.java:338

 @Override
    public void earlyEOF()
    {
        HttpExchange exchange = getHttpExchange();
        HttpConnectionOverHTTP connection = getHttpConnection();
        if (exchange == null)
            connection.close();
        else
            failAndClose(new EOFException(String.valueOf(connection)));
    }

It appears that successful runs find exchange==null and do a proper shutdown-close, where in the race condition (?) the code arrives at this method with exchange not being null - that's as far as I currently understand the situation.

On a local host with a jetty server on the other end (separate process) and repeating the test 994 times successfully out of 1000 requests I find it hard to blame this on a badly implemented server (then let's migrate this issue to a jetty server issue and investigate...).

Downgrading to jetty 9.4.12 that we came from does not seem to be an option (I see several CVE's and many fixes since and we need to stay current on those).

So I'd like to see this issue to be dealt with as critical instead of stale!

[joakime]

@consultantleon your issue is happening on Jetty Client. This issue is about Jetty Server.
Please file a new issue for your concern.

[sbordet]

@consultantleon the case you describe is not an issue, just a misconfiguration.
Please open a new issue and we can discuss it there.