DataBufferLimitException: 超过最大字节数限制以缓冲 webflow 错误

I suppose this issue is about adding a new spring.codec.max-in-memory-size configuration property in Spring Boot. Add it to the application.yml file like:

spring:
codec:
max-in-memory-size: 10MB

Set the maximum bytes (in megabytes) in your Spring Boot application.properties configuration file like below:

spring.codec.max-in-memory-size=20MB

i was getting this error for a simple RestController (i post a large json string).

here is how i successfully changed the maxInMemorySize

import org.springframework.context.annotation.Configuration;
import org.springframework.http.codec.ServerCodecConfigurer;
import org.springframework.web.reactive.config.ResourceHandlerRegistry;
import org.springframework.web.reactive.config.WebFluxConfigurer;


@Configuration
public class WebfluxConfig implements WebFluxConfigurer {


@Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {


registry.addResourceHandler("/swagger-ui.html**")
.addResourceLocations("classpath:/META-INF/resources/");


registry.addResourceHandler("/webjars/**")
.addResourceLocations("classpath:/META-INF/resources/webjars/");
}


@Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
}
}

this was surprisingly hard to find

This worked for me:

1. Create a @Bean in one of your configuration classes or the main SpringBootApplication class:

   @Bean
   public WebClient webClient() {
   final int size = 16 * 1024 * 1024;
   final ExchangeStrategies strategies = ExchangeStrategies.builder()
   .codecs(codecs -> codecs.defaultCodecs().maxInMemorySize(size))
   .build();
   return WebClient.builder()
   .exchangeStrategies(strategies)
   .build();
   }
   

2. Next, go to your desired class where you want to use the WebClient:

   @Service
   public class TestService {
   
   
   @Autowired
   private WebClient webClient;
   
   
   public void test() {
   String out = webClient
   .get()
   .uri("/my/api/endpoint")
   .retrieve()
   .bodyToMono(String.class)
   .block();
   
   
   System.out.println(out);
   }
   }
   

Instead of retrieving data at once, you can stream:

Mono<String> string = webClient.get()
.uri("end point of an API")
.retrieve()
.bodyToFlux(DataBuffer.class)
.map(buffer -> {
String string = buffer.toString(Charset.forName("UTF-8"));
DataBufferUtils.release(buffer);
return string;
});

Alternatively convert to stream:

    .map(b -> b.asInputStream(true))
.reduce(SequenceInputStream::new)
.map(stream -> {
// consume stream
stream.close();
return string;
});

In most cases you don't want to really aggregate the stream, rather than processing it directly. The need to load huge amount of data in memory is mostly a sign to change the approach to more reactive one. JSON- and XML-Parsers have streaming interfaces.

Another alternative could be creating a custom CodecCustomizer, which is going to be applied to both WebFlux and WebClient at the same time:

@Configuration
class MyAppConfiguration {


companion object {
private const val MAX_MEMORY_SIZE = 50 * 1024 * 1024 // 50 MB
}


@Bean
fun codecCustomizer(): CodecCustomizer {
return CodecCustomizer {
it.defaultCodecs()
.maxInMemorySize(MAX_MEMORY_SIZE)
}
}
}

This worked for me

val exchangeStrategies = ExchangeStrategies.builder()
.codecs { configurer: ClientCodecConfigurer -> configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024) }.build()
return WebClient.builder().exchangeStrategies(exchangeStrategies).build()

worked for me

webTestClient.mutate().codecs(configurer -> configurer
.defaultCodecs()
.maxInMemorySize(16 * 1024 * 1024)).build().get()
.uri("/u/r/l")
.exchange()
.expectStatus()
.isOk()

As of Spring Boot 2.3.0, there is now a dedicated configuration property for the Reactive Elasticsearch REST client.

You can use the following configuration property to set a specific memory limit for the client.

spring.data.elasticsearch.client.reactive.max-in-memory-size= The already existing spring.codec.max-in-memory-size property is separate and only affects other WebClient instances in the application.

For those who had no luck with the myriad of beans, customizers, and properties that could be added to solve this problem, check whether you have defined a bean extending WebFluxConfigurationSupport. If you have, it will disable the autoconfiguration version of the same bean (my personal experience, Boot 2.7.2), somewhere under which Spring loads properties such as the suggested spring.codec.max-in-memory-size. For this solution to work you need to have also configured this property correctly.

To test if this is the cause of your problems, remove your WebFluxConfigurationSupport implementation temporarily. The long term fix that worked for me was to use configuration beans to override attributes for the autoconfigured bean. In my case, WebFluxConfigurer had all of the same methods available and was a drop-in replacement for WebFluxConfigurationSupport. Large JSON messages are now decoding for me as configured.

If you dont want to change the default settings for webClient globally, you can use the following approach to manually merge multiple DataBuffers

webClient
.method(GET)
.uri("<uri>")
.exchangeToMono(response -> {
return response.bodyToFlux(DataBuffer.class)
.switchOnFirst((firstBufferSignal, responseBody$) -> {
assert firstBufferSignal.isOnNext();
return responseBody$
.collect(() -> requireNonNull(firstBufferSignal.get()).factory().allocateBuffer(), (accumulator, curr) -> {
accumulator.ensureCapacity(curr.readableByteCount());
accumulator.write(curr);
DataBufferUtils.release(curr);
})
.map(accumulator -> {
final var responseBodyAsStr = accumulator.toString(UTF_8);
DataBufferUtils.release(accumulator);
return responseBodyAsStr;
});
})
.single();
});

The above code aggregates all the DataBuffers into a single DataBuffer & then converts the final DataBuffer into a string. Please note that this answer wont work as DataBuffers received might not have all the bytes to construct a character (incase of UTF-8 characters, each character can take upto 4 bytes). So we cant convert intermediate DataBuffers into String as the bytes towards the end of buffer might have only part of the bytes required to construct a valid character

Note that this loads all the response DataBuffers into memory but unlike changing global settings for the webClient across the whole application. You can use this option to read complete response only where you want i.e you can narrow down & pick this option only where you expect large responses.

As of Spring boot 2.7.x we should use below property to set the memory size to webclient which is used internally in reactive ElasticSearch

spring.elasticsearch.webclient.max-in-memory-size=512MB

Just add below code in your springboot main class.

@Bean
public WebClient getWebClient() {
return WebClient.builder()
.baseUrl("Your_SERVICE_URL")
.codecs(configurer -> configurer
.defaultCodecs()
.maxInMemorySize(16 * 1024 * 1024))
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.build();
}

This work for me.