logo标签列表

WebFlux DataBufferLimitException: 部分头超过了8192字节的内存使用限制。

javanettyspring-webfluxproject-reactor
10

当在以下端点上传csv文件和JSON对象时：

@PostMapping(value = "dataset/rows/query", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
Mono<List<Integer>> getRowsByQuery(@RequestPart("dataset") Mono<FilePart> file, 
                                   @RequestPart("query") QueryDTO query){
    return Mono.just(new ArrayList<>());
}

我遇到了以下错误：

2020-12-17 12:25:05.142 ERROR 195281 --- [or-http-epoll-3] a.w.r.e.AbstractErrorWebExceptionHandler : [d418565e-17]  500 Server Error for HTTP POST "/dataset/rows/query"

org.springframework.core.io.buffer.DataBufferLimitException: Part headers exceeded the memory usage limit of 8192 bytes
    at org.springframework.http.codec.multipart.MultipartParser$HeadersState.onNext(MultipartParser.java:360) ~[spring-web-5.3.1.jar:5.3.1]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ HTTP POST "/dataset/rows/query" [ExceptionHandlingWebHandler]
Stack trace:
        at org.springframework.http.codec.multipart.MultipartParser$HeadersState.onNext(MultipartParser.java:360) ~[spring-web-5.3.1.jar:5.3.1]
        at org.springframework.http.codec.multipart.MultipartParser.hookOnNext(MultipartParser.java:104) ~[spring-web-5.3.1.jar:5.3.1]
        at org.springframework.http.codec.multipart.MultipartParser.hookOnNext(MultipartParser.java:46) ~[spring-web-5.3.1.jar:5.3.1]
        at reactor.core.publisher.BaseSubscriber.onNext(BaseSubscriber.java:160) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:265) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:371) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:381) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:535) ~[reactor-netty-http-1.0.1.jar:1.0.1]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:229) ~[reactor-netty-http-1.0.1.jar:1.0.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
        at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]

我试图通过defaultCodex().maxinMemorySize()来自定义。

@Component
public class ServerConfiguration implements WebFluxConfigurer {
    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
    }
}

以下是application.yml文件的内容：

server:
  port: ${SERVER_PORT:8080}
  max-http-header-size: 900000000
spring:
  codec:
    max-in-memory-size: 900000000

但它似乎没有任何影响。

此外，奇怪的是，在从 Angular 调用 API 时，服务器端的错误似乎只会出现，而在 postman 中不会出现。

从Angular中，我有以下标题：

POST /dataset/rows/query HTTP/1.1
Host: localhost:4200
Connection: keep-alive
Content-Length: 496570
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
DNT: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarySBh6gJvnTeDzB43Y
Origin: http://localhost:4200
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:4200/app
Accept-Encoding: gzip, deflate, br
Accept-Language: it-IT,it;q=0.9,en-GB;q=0.8,en;q=0.7,ru-RU;q=0.6,ru;q=0.5,en-US;q=0.4

最后，这是该终端点的对应OpenAPI yaml：

openapi: 3.0.1
info:
  title: OpenAPI definition
  version: v0
servers:
- url: http://localhost:8080
  description: Generated server url
paths:
  /dataset/rows/query:
    post:
      tags:
      - dataset-controller
      operationId: getRowsByQuery
      requestBody:
        content:
          multipart/form-data:
            schema:
              type: object
              properties:
                dataset:
                  type: string
                  format: binary
                query:
                  $ref: '#/components/schemas/QueryDTO'
      responses:
        "200":
          description: OK
          content:
            '*/*':
              schema:
                type: array
                items:
                  type: integer
                  format: int32
components:
  schemas:
    PredicateDTO:
      type: object
      properties:
        value:
          type: object
        key:
          type: string
        operator:
          type: string
          enum:
          - EQUAL
          - NOT_EQUAL
          - BELONGING
          - NOT_BELONGING
          - GREATER_THAN
          - GREATER_THAN_EQUAL
          - LESS_THAN
          - LESS_THAN_EQUAL
    QueryDTO:
      type: object
      properties:
        predicates:
          type: object
          additionalProperties:
            $ref: '#/components/schemas/PredicateDTO'

这是我的数据集和json对象。 如何增加Part headers的内存使用限制？

请分享Spring Boot的版本。 - kakabali
1
@1Z10 你找到任何解决方案了吗？我们也遇到了同样的问题。在 Angular 和 Postman 中都可以正常工作，但在我们的情况下，第一次上传失败，之后就可以正常工作了。我们在版本 2.4.1 上发现了这个问题，在 2.4.2 中没有发现任何问题，但现在又出现在 2.4.3 上了。 - Karthik Prasad
1个回答
1

更新： 截至Spring 5.3.13，默认的 DefaultPartHttpMessageReader 类中的 maxHeadersSize 已经从之前的8KiB增加到10KiB。请参见 this Github commit。 尝试更新你正在使用的Spring版本至少到这个版本，希望您的问题会得到解决。

以下是原始答案:

你的想法是正确的。

根据Spring文档，如果要自定义预定义的限制，则必须提供自己的MultipartHttpMessageReader：

对于Multipart解析，maxInMemorySize属性限制非文件部分的大小。对于文件部分，它确定将部分写入磁盘的阈值。对于写入磁盘的文件部分，还有一个maxDiskUsagePerPart属性来限制每个部分的磁盘空间使用量。还有一个maxParts属性来限制多部分请求中的总部件数。为了在WebFlux中配置所有这三个功能，您需要向ServerCodecConfigurer提供预配置的MultipartHttpMessageReader实例。

文档似乎没有提到堆栈跟踪中提到的部分头大小限制。然而，对我来说，那是唯一需要从默认值增加的属性，以解决问题，但您可以轻松自定义DefaultPartHttpMessageReader的任何其他默认限制。

@Configuration
public class CodecConfig implements WebFluxConfigurer {

    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        DefaultPartHttpMessageReader partReader = new DefaultPartHttpMessageReader();
        partReader.setMaxHeadersSize(9216); // 9 KiB, default is 8 KiB
        partReader.setEnableLoggingRequestDetails(true);

        MultipartHttpMessageReader multipartReader = new MultipartHttpMessageReader(partReader);
        multipartReader.setEnableLoggingRequestDetails(true);

        configurer.defaultCodecs().multipartReader(multipartReader);
    }
}
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接