我尝试向客户端流式传输一个实时生成的大型 JSON 文件(可能超过 500 MB)。我试图禁用响应缓冲,原因多种多样,但主要是为了提高内存效率。
我尝试直接写入 HttpContext.Response.BodyWriter
,但在将其写入输出之前,响应似乎被缓冲到内存中。此方法的返回类型为 Task
。
HttpContext.Response.ContentType = "application/json";
HttpContext.Response.ContentLength = null;
await HttpContext.Response.StartAsync(cancellationToken);
var bodyStream = HttpContext.Response.BodyWriter.AsStream(true);
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("["), cancellationToken);
await foreach (var item in cursor.WithCancellation(cancellationToken)
.ConfigureAwait(false))
{
await bodyStream.WriteAsync(JsonSerializer.SerializeToUtf8Bytes(item, DefaultSettings.JsonSerializerOptions), cancellationToken);
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes(","), cancellationToken);
await bodyStream.FlushAsync(cancellationToken);
await Task.Delay(100,cancellationToken);
}
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("]"), cancellationToken);
bodyStream.Close();
await HttpContext.Response.CompleteAsync().ConfigureAwait(false);
注意:我意识到这段代码非常“hacky”,试图让它运行,然后再进行清理
我正在使用Task.Delay
来验证在本地测试时响应是否被缓冲,因为我没有完整的生产数据。我还尝试过IAsyncEnumerable
和yield return
,但是由于响应太大,Kestrel认为可枚举对象是无限的,所以失败了。
我已经尝试过:
- 设置KestrelServerLimits.MaxResponseBufferSize为一个很小的数值,甚至为0;
- 使用
HttpContext.Response.WriteAsync
进行编写; - 使用
HttpContext.Response.BodyWriter.AsStream()
进行编写; - 使用管道编写器模式和
HttpContext.Response.BodyWriter
进行编写; - 删除所有中间件;
- 删除对
IApplicationBuilder.UseResponseCompression
的调用。
更新
- 在设置
ContentType
(即在对响应进行任何写操作之前)之前尝试禁用响应缓冲,但没有效果。
var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
responseBufferingFeature?.DisableBuffering();
样例代码已更新
这个样例代码很简单地重现了该问题。在调用 response.CompleteAsync()
前,客户端将不会接收到任何数据。
[HttpGet]
[Route("stream")]
public async Task<EmptyResult> FileStream(CancellationToken cancellationToken)
{
var response = DisableResponseBuffering(HttpContext);
HttpContext.Response.Headers.Add("Content-Type", "application/gzip");
HttpContext.Response.Headers.Add("Content-Disposition", $"attachment; filename=\"player-data.csv.gz\"");
await response.StartAsync().ConfigureAwait(false);
var memory = response.Writer.GetMemory(1024*1024*10);
response.Writer.Advance(1024*1024*10);
await response.Writer.FlushAsync(cancellationToken).ConfigureAwait(false);
await Task.Delay(5000).ConfigureAwait(false);
var str2 = Encoding.UTF8.GetBytes("Bar!\r\n");
memory = response.Writer.GetMemory(str2.Length);
str2.CopyTo(memory);
response.Writer.Advance(str2.Length);
await response.CompleteAsync().ConfigureAwait(false);
return new EmptyResult();
}
private IHttpResponseBodyFeature DisableResponseBuffering(HttpContext context)
{
var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
responseBufferingFeature?.DisableBuffering();
return responseBufferingFeature;
}