Spring-cloud-gateway: DataBufferLimitException: Exceeded limit on max bytes to buffer

Created on 7 Apr 2020 · 13Comments · Source: spring-cloud/spring-cloud-gateway

when i request the body length > 256*1024,tell me the wrong as follow message.
org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
my springboot version is 2.2.6.RELEASE
i set spring.codec.max-in-memory-size not work
how can i do to avoid this problem?

enhancement

Source

jack01234

👍1

Most helpful comment

when i request the body length > 256*1024,tell me the wrong as follow message.
org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
my springboot version is 2.2.6.RELEASE
i set spring.codec.max-in-memory-size not work
how can i do to avoid this problem?

Hey,man i resolve this,When you have config like this

@Configuration
@EnableWebFlux
public class WebFluxWebConfig implements WebFluxConfigurer

spring.codec.max-in-memory-size in you yml or properties will not work,please try this!

@Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
}

Invoker-liu on 16 Apr 2020

👍3

All 13 comments

what version of spring cloud? please post the full stack trace.

spencergibb on 7 Apr 2020

spring cloud version is Hoxton.SR3，when i switch spring cloud to Greenwich.SR5 springboot to 2.1.6.RELEASE is ok，the full stack trace as follow:

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
    at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
    at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:94)
    at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:59)
    at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:132)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1705)
    at reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
    at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
    at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1705)
    at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
    at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252)
    at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
    at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:419)
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:209)
    at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:367)
    at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:363)
    at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:489)
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:90)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:214)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

jack01234 on 8 Apr 2020

Do you have any custom filters?

spencergibb on 8 Apr 2020

yes，i have my filters，is log handle

jack01234 on 8 Apr 2020

Are you reading the body in your custom filter?

spencergibb on 9 Apr 2020

yes i used ModifyRequestBodyGatewayFilterFactory

jack01234 on 9 Apr 2020

when i request the body length > 256*1024,tell me the wrong as follow message.
org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
my springboot version is 2.2.6.RELEASE
i set spring.codec.max-in-memory-size not work
how can i do to avoid this problem?

I have same issue,wish fixed

Invoker-liu on 15 Apr 2020

Are you reading the body in your custom filter?

i think a lot, maybe the GZIP modifyResponseBody whitch added on the Hoxton.SR3

Invoker-liu on 16 Apr 2020

when i request the body length > 256*1024,tell me the wrong as follow message.
org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
my springboot version is 2.2.6.RELEASE
i set spring.codec.max-in-memory-size not work
how can i do to avoid this problem?

Hey,man i resolve this,When you have config like this

@Configuration
@EnableWebFlux
public class WebFluxWebConfig implements WebFluxConfigurer

spring.codec.max-in-memory-size in you yml or properties will not work,please try this!

@Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
}

Invoker-liu on 16 Apr 2020

👍3

it seems still not works in SR5.
My environment is spring boot 2.3.0 + spring cloud Hoxton.SR5

asiaon123 on 1 Jun 2020

Please open a new issue and provide a complete, minimal, verifiable sample that reproduces the problem. It should be available as a GitHub (or similar) project or attached to the new issue as a zip file.

spencergibb on 1 Jun 2020

yes i used ModifyRequestBodyGatewayFilterFactory

Hi jack01234,
I got the same problem, and I solved it like this:

(1) If you use ModifyResponseBodyGatewayFilterFactory
Call ModifyResponseBodyGatewayFilterFactory(ServerCodecConfigurer codecConfigurer)
instead of the default no-param one. As you have to give a correct config to the codec with a larger size.

(2) If you have a child class inheriting ModifyResponseBodyGatewayFilterFactory

@Component
class MyGatewayFilterFactory extends ModifyResponseBodyGatewayFilterFactory {
    /// Give only this constructor here, don't use default no-param one.
    public MyGatewayFilterFactory(ServerCodecConfigurer codecConfigurer) {
        super(codecConfigurer);
    }
    ....
}

Reason:
If you see the source code ModifyResponseBodyGatewayFilterFactory.java, you will find
the default constructor is using a default HandlerStrategies config to give the messageReaders:

/** @deprecated */
@Deprecated
public ModifyResponseBodyGatewayFilterFactory() {
    super(ModifyResponseBodyGatewayFilterFactory.Config.class);
    this.messageReaders = HandlerStrategies.withDefaults().messageReaders();
    this.messageBodyDecoders = Collections.emptyMap();
    this.messageBodyEncoders = Collections.emptyMap();
}

HandlerStrategies.withDefaults().messageReaders() gives default decoder related to AbstractDataBufferDecoder.
the class AbstractDataBufferDecoder has a default nightmare number maxInMemorySize = 262144.

public abstract class AbstractDataBufferDecoder<T> extends AbstractDecoder<T> {
    private int maxInMemorySize = 262144;
    .....
}

Why so many people find it hard to set codec.max-in-memory-size? Because the
RaiseLimitException can be thrown by many classes, in this case, LimitedDataBufferList which is used by AbstractDataBufferDecoder throws RaiseLimitException when finding exceeding. AbstractDataBufferDecoder sets
262144 as max size for RaiseLimitException. So we have to make AbstractDataBufferDecoder has a larger size, so we have to
find the user of AbstractDataBufferDecoder, the user is messageReaders in ModifyResponseBodyGatewayFilterFactory.

I think there may be more ways to solve the problem such as injecting AbstractDataBufferDecoder changing the
maxInMemorySize.

Below is the complete child class I used:

@Slf4j
@Component
public class SignModifyResponseBodyGatewayFilterFactory extends ModifyResponseBodyGatewayFilterFactory {
    @Autowired
    private FilterConfigService filterConfigService;
    @Autowired
    private ClientAuthService clientAuthService;

    public SignModifyResponseBodyGatewayFilterFactory(ServerCodecConfigurer codecConfigurer) {
        super(codecConfigurer);
    }

    @Override
    public GatewayFilter apply(ModifyResponseBodyGatewayFilterFactory.Config config) {
        ModifyResponseBodyGatewayFilterFactory.ModifyResponseGatewayFilter gatewayFilter = new ModifyResponseBodyGatewayFilterFactory.ModifyResponseGatewayFilter(config) {
            @Override
            public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
                config.setRewriteFunction(getRewriteFunction());
                config.setInClass(String.class);
                config.setOutClass(String.class);
                ModifiedServerHttpResponse modifiedServerHttpResponse = new ModifiedServerHttpResponse(exchange, config);
                return chain.filter(exchange.mutate().response(modifiedServerHttpResponse).build());
            }
        };
        gatewayFilter.setFactory(this);
        return gatewayFilter;
    }

    private RewriteFunction<String, String> getRewriteFunction() {
        return (exchange, resp) -> {
            String responseStr = (resp == null ? "" : resp);
            exchange.getResponse().getHeaders().add("Authorization", "1111");
            return Mono.empty();
        };
    }
}

liuyouzhao on 21 Aug 2020

spring.codec.max-in-memory-size acting on the spring internal codec, if you create a new codec like this HandlerStrategies.withDefaults().messageReaders();, that is default size 262144, please use @Autowired CodecConfigurer codecConfigurer codecConfigurer.getReaders() obtain