Jetty.project: IllegalStateException in HttpOutput with Jersey
Jetty version
9.4.25.v20191220
Java version
OpenJDK 1.8.0_232 (Zulu)
OS type/version
macOS Mojave 10.14.6
Description
Jetty 9.4.25 seems to have a regression (or stricter behavior) in the HttpOutput state machine (likely introduced by #4409) which causes the following exception on write():
java.lang.IllegalStateException: s=OPEN,api=BLOCKED,sc=false,e=null
Unfortunately, I was not able to create a simple reproduction, but it reproduces reliably when running a Presto unit test in the presto-jmx module from this branch: https://github.com/electrum/presto/tree/jetty9425:
mvn clean install -DskipTests -pl presto-jmx -am -T1C
mvn test -pl presto-jmx -Dtest=io.prestosql.plugin.jmx.TestJmxQueries#testQuery
After running the above, look for the following error in the test output after 5-10 seconds. You can kill the test at this point, since it takes a while to eventually time out.
2020-01-07T14:01:47.723-0600 SEVERE An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: com.fasterxml.jackson.databind.JsonMappingException: s=OPEN,api=BLOCKED,sc=false,e=null (through reference chain: io.prestosql.client.QueryResults["data"]->java.util.Collections$UnmodifiableRandomAccessList[12])
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:884)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:916)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:872)
at io.airlift.http.server.AsyncResponseHandler$1.onSuccess(AsyncResponseHandler.java:102)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: s=OPEN,api=BLOCKED,sc=false,e=null (through reference chain: io.prestosql.client.QueryResults["data"]->java.util.Collections$UnmodifiableRandomAccessList[12])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:365)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:338)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at com.fasterxml.jackson.databind.ser.std.IterableSerializer.serializeContents(IterableSerializer.java:107)
at com.fasterxml.jackson.databind.ser.std.IterableSerializer.serialize(IterableSerializer.java:74)
at com.fasterxml.jackson.databind.ser.std.IterableSerializer.serialize(IterableSerializer.java:12)
at com.fasterxml.jackson.databind.ser.std.StdDelegatingSerializer.serialize(StdDelegatingSerializer.java:168)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:721)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:166)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1433)
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:921)
at io.airlift.jaxrs.JsonMapper.writeTo(JsonMapper.java:233)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
... 20 more
Caused by: java.lang.IllegalStateException: s=OPEN,api=BLOCKED,sc=false,e=null
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:767)
at io.airlift.http.server.TimingFilter$TimedServletOutputStream.write(TimingFilter.java:163)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325)
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:224)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:300)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2136)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeCustomStringSegment2(UTF8JsonGenerator.java:1657)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1396)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegments(UTF8JsonGenerator.java:1281)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:502)
at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
... 40 more
electrum
All 5 comments
same here but with other frameworks (spring boot 2.2)
2020-01-08 11:27:17.629 WARN 14955 --- [qtp509559152-14,] org.eclipse.jetty.server.HttpChannel : /vendor-es2015.ab355fce23594e19a032.js
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalStateException: s=OPEN,api=BLOCKED,sc=false,e=null
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:503)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
at XXX.web.filter.CachingHttpHeadersFilter.doFilter(CachingHttpHeadersFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at XXX.web.filter.RequestAndResponseLoggingFilter.doFilterWrapped(RequestAndResponseLoggingFilter.java:65)
at XXX.web.filter.RequestAndResponseLoggingFilter.doFilterInternal(RequestAndResponseLoggingFilter.java:49)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
Caused by: java.lang.IllegalStateException: s=OPEN,api=BLOCKED,sc=false,e=null
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:767)
at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:645)
at org.springframework.util.StreamUtils.copy(StreamUtils.java:143)
at org.springframework.http.converter.ResourceHttpMessageConverter.writeContent(ResourceHttpMessageConverter.java:132)
at org.springframework.http.converter.ResourceHttpMessageConverter.writeInternal(ResourceHttpMessageConverter.java:124)
at org.springframework.http.converter.ResourceHttpMessageConverter.writeInternal(ResourceHttpMessageConverter.java:45)
at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:227)
at org.springframework.web.servlet.resource.ResourceHttpRequestHandler.handleRequest(ResourceHttpRequestHandler.java:489)
at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:53)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:503)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
at XXX.web.filter.CachingHttpHeadersFilter.doFilter(CachingHttpHeadersFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
gbrehmer
on 8 Jan 2020
@electrum thanks for the reproducer, we were able to confirm that's a Jetty bug.
We are working on a fix.
sbordet
on 8 Jan 2020
👍4
A smaller testcase that can replicate this can be found in branch jetty-9.4.x-4461-aggregate-gzip-blocked
joakime
on 8 Jan 2020
Opened PR #4465
joakime
on 8 Jan 2020
Alternate PR #4466 has been opened. (PR #4465 has been closed)
joakime
on 9 Jan 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
joakime
路
8Comments
pedro-abundio-wang
路
3Comments
nathankooij
路
9Comments
julienmoumne
路
3Comments
WalkerWatch
路
4Comments
Most helpful comment
@electrum thanks for the reproducer, we were able to confirm that's a Jetty bug.
We are working on a fix.