假设我要将几个大小在2MB到5GB之间的文件写入磁盘。
FileStream的合适缓冲区值是什么？
使用几兆字节的缓冲区大小是否合理，还是应该坚持使用千字节缓冲区？
使用几兆字节的缓冲区大小是否合理，还是应该坚持使用千字节缓冲区？
C# FileStream：写入大文件的最佳缓冲区大小是多少？
81
- Pygmy
6
1为什么不让 .Net 处理缓冲区，只需按需编写即可？ - cjk
1试一下吧。使用StopWatch编写一个小型基准测试，并告诉我们结果。 - J. Random Coder
4我怀疑所说的“缓冲区”是指在将数据从一个流复制到另一个流时必须声明的缓冲区（即每次迭代中你要读取和写入多少字节）。 - Pavel Minaev
15为什么不让.Net来处理它？-> 好吧，如果有更适合大量写入的优化方案（我不知道），我想使用它。只是试一下 -> 嗯，秒表只告诉我时间，它不能告诉我是否通过使用极大的缓冲区可能会导致其他问题，如阻塞我的程序或其他程序，或者引起其他未预见的副作用。重点是我不知道缓冲区大小可能会产生什么影响，因此我问你们这群超级聪明的人！:-P - Pygmy
3超级智能的人都不知道你的机器上运行了哪些其他程序，更不用说你客户的机器了。你需要在真实世界条件下进行测试，而且只有你知道你客户的真实世界条件是什么。 - Eric Lippert
1可能是使用流进行文件I/O - 最佳内存缓冲区大小的重复问题。 - StayOnTarget
4个回答
46
默认缓冲区大小为4 KiB。
同时，可以在这里查看：.NET中的顺序文件编程模式和性能
描述了.NET Framework中用于顺序文件访问的编程模式，并对其性能进行了测量。默认行为在单个磁盘上提供了出色的性能 - 50 MBps的读写速度。使用大请求大小并在可能的情况下进行文件预分配具有可量化的好处。当考虑到磁盘阵列时，.NET非缓存IO在16个磁盘阵列上提供800 MBps的速度，而缓存IO仅提供其中的12％的性能。因此，高性能的文件和数据库实用程序仍然被迫使用非缓存IO以获得最大的顺序性能。该报告附带可下载的源代码，演示了这些概念和用于获得这些测量结果的代码。
- Rubens Farias
4
4默认大小为“4096”字节。 - VMAtm
5刚刚反编译了 System.IO.FileStream。内部常量 int DefaultBufferSize = 4096; - Vladimirs
3@VMAtm - 这个参考是虚假的（就回答FileStream类通常在内部使用什么而言）。CopyTo缓冲区大小要大得多，确切地说是80 KiB。我在这里写了一篇关于.NET IO文件性能特征的文章（https://dev59.com/8HI_5IYBdhLWcg3wEOvq#19692949） - John Leidegren
1我想补充一点，在我的电脑上（使用NVMe SSD、Windows 10、.NET Framework 4.8和.NET 5+）进行异步磁盘IO时，最佳的流读写速度发生在缓冲区大小约为1MB的情况下，这与.NET Framework 2.0中相对较小的4096字节默认值相去甚远。当然，你的情况可能会有所不同-始终要进行自己的基准测试！对于异步网络IO，您需要再次选择不同的缓冲区大小。 - Dai
38
根据引用的文档进行的一个小型基准测试表明，在我的系统上，缓冲区大小大于128KB时并没有性能提升。你的情况可能有所不同，可以自由使用下面的内容。
Stopwatch sw = new Stopwatch();
Random rand = new Random(); // seed a random number generator
int numberOfBytes = 2 << 22; //8,192KB File
byte nextByte;
for (int i = 1; i <= 28; i++) //Limited loop to 28 to prevent out of memory
{
sw.Start();
using (FileStream fs = new FileStream(
String.Format(@"C:\TEMP\TEST{0}.DAT", i), // name of file
FileMode.Create, // create or overwrite existing file
FileAccess.Write, // write-only access
FileShare.None, // no sharing
2 << i, // block transfer of i=18 -> size = 256 KB
FileOptions.None))
{
for (int j = 0; j < numberOfBytes; j++)
{
nextByte = (byte)(rand.Next() % 256); // generate a random byte
fs.WriteByte(nextByte); // write it
}
}
sw.Stop();
Console.WriteLine(String.Format("Buffer is 2 << {0} Elapsed: {1}", i, sw.Elapsed));
sw.Reset();
}
- Firestrand
4
3+1 对于基准代码，但是 OP 确实要求写缓冲区大小。不过这种方法是可行的。 - James Dunne
3您的评论存在自相矛盾之处。您表示“大小为256KB”，但同时又声称“i=18”，而实际上“2 << 18”等于512KB。 - Timwi
2 << 18 == 262144 字节，每千字节有1024字节，因此为256 KB。我错过了什么？ - Firestrand
5实际上，“2 << 18”等于512KB。你可能是在使用“1 << 18”（而不是“2 << 18”）来获取256 KB。 - jweyrich
21
简而言之：缓冲区大小大多数情况下并不重要。微调其他东西，只有在处理大文件或设置useAsync=true时，不要使用小于128KiB的缓冲区。
Test Name: FileWritePerformance
Test Outcome: Passed
Result StandardOutput:
File: 1024, Buffer: 1, Elapsed: 00:00:00.0021467
File: 1024, Buffer: 2, Elapsed: 00:00:00.0011164
File: 1024, Buffer: 4, Elapsed: 00:00:00.0012392
File: 1024, Buffer: 8, Elapsed: 00:00:00.0010978
File: 1024, Buffer: 16, Elapsed: 00:00:00.0014263
File: 1024, Buffer: 32, Elapsed: 00:00:00.0010724
File: 1024, Buffer: 64, Elapsed: 00:00:00.0012182
File: 1024, Buffer: 128, Elapsed: 00:00:00.0013605
File: 1024, Buffer: 256, Elapsed: 00:00:00.0010234
File: 1024, Buffer: 512, Elapsed: 00:00:00.0011247
File: 1024, Buffer: 1024, Elapsed: 00:00:00.0013137
File: 1024, Buffer: 2048, Elapsed: 00:00:00.0013527
File: 1024, Buffer: 4096, Elapsed: 00:00:00.0010895
File: 1024, Buffer: 8192, Elapsed: 00:00:00.0017128
File: 1024, Buffer: 16384, Elapsed: 00:00:00.0011272
File: 1024, Buffer: 32768, Elapsed: 00:00:00.0010547
File: 1024, Buffer: 65536, Elapsed: 00:00:00.0011427
File: 1024, Buffer: 131072, Elapsed: 00:00:00.0011571
File: 1024, Buffer: 262144, Elapsed: 00:00:00.0010978
File: 1024, Buffer: 524288, Elapsed: 00:00:00.0010884
File: 1024, Buffer: 1048576, Elapsed: 00:00:00.0011504
File: 1024, Buffer: 2097152, Elapsed: 00:00:00.0011308
File: 1024, Buffer: 4194304, Elapsed: 00:00:00.0026100
File: 1024, Buffer: 8388608, Elapsed: 00:00:00.0015160
File: 1024, Buffer: 16777216, Elapsed: 00:00:00.0011654
File: 1024, Buffer: 33554432, Elapsed: 00:00:00.0013214
File: 1024, Buffer: 67108864, Elapsed: 00:00:00.0012531
File: 1024, Buffer: 134217728, Elapsed: 00:00:00.0031671
File: 1024, Buffer: 268435456, Elapsed: 00:00:00.0032789
File: 1024, Buffer: 536870912, Elapsed: 00:00:00.0049508
File: 1024, Buffer: 1073741824, Elapsed: 00:00:00.0079707
File: 1048576, Buffer: 1, Elapsed: 00:00:00.0175447
File: 1048576, Buffer: 2, Elapsed: 00:00:00.0213337
File: 1048576, Buffer: 4, Elapsed: 00:00:00.0139657
File: 1048576, Buffer: 8, Elapsed: 00:00:00.0135683
File: 1048576, Buffer: 16, Elapsed: 00:00:00.0124353
File: 1048576, Buffer: 32, Elapsed: 00:00:00.0226043
File: 1048576, Buffer: 64, Elapsed: 00:00:00.0123077
File: 1048576, Buffer: 128, Elapsed: 00:00:00.0130151
File: 1048576, Buffer: 256, Elapsed: 00:00:00.0130583
File: 1048576, Buffer: 512, Elapsed: 00:00:00.0130555
File: 1048576, Buffer: 1024, Elapsed: 00:00:00.0130760
File: 1048576, Buffer: 2048, Elapsed: 00:00:00.0111780
File: 1048576, Buffer: 4096, Elapsed: 00:00:00.0097345
File: 1048576, Buffer: 8192, Elapsed: 00:00:00.0096204
File: 1048576, Buffer: 16384, Elapsed: 00:00:00.0085635
File: 1048576, Buffer: 32768, Elapsed: 00:00:00.0081099
File: 1048576, Buffer: 65536, Elapsed: 00:00:00.0076785
File: 1048576, Buffer: 131072, Elapsed: 00:00:00.0069143
File: 1048576, Buffer: 262144, Elapsed: 00:00:00.0078431
File: 1048576, Buffer: 524288, Elapsed: 00:00:00.0065928
File: 1048576, Buffer: 1048576, Elapsed: 00:00:00.0096246
File: 1048576, Buffer: 2097152, Elapsed: 00:00:00.0094253
File: 1048576, Buffer: 4194304, Elapsed: 00:00:00.0089587
File: 1048576, Buffer: 8388608, Elapsed: 00:00:00.0073527
File: 1048576, Buffer: 16777216, Elapsed: 00:00:00.0075993
File: 1048576, Buffer: 33554432, Elapsed: 00:00:00.0069752
File: 1048576, Buffer: 67108864, Elapsed: 00:00:00.0087534
File: 1048576, Buffer: 134217728, Elapsed: 00:00:00.0089197
File: 1048576, Buffer: 268435456, Elapsed: 00:00:00.0072583
File: 1048576, Buffer: 536870912, Elapsed: 00:00:00.0077418
File: 1048576, Buffer: 1073741824, Elapsed: 00:00:00.0217475
File: 1073741824, Buffer: 1, Elapsed: 00:00:08.6674554
File: 1073741824, Buffer: 2, Elapsed: 00:00:08.3731788
File: 1073741824, Buffer: 4, Elapsed: 00:00:08.8015734
File: 1073741824, Buffer: 8, Elapsed: 00:00:08.4734106
File: 1073741824, Buffer: 16, Elapsed: 00:00:08.8242336
File: 1073741824, Buffer: 32, Elapsed: 00:00:08.3961399
File: 1073741824, Buffer: 64, Elapsed: 00:00:08.6481375
File: 1073741824, Buffer: 128, Elapsed: 00:00:08.4404953
File: 1073741824, Buffer: 256, Elapsed: 00:00:08.6811610
File: 1073741824, Buffer: 512, Elapsed: 00:00:08.3297239
File: 1073741824, Buffer: 1024, Elapsed: 00:00:08.9151381
File: 1073741824, Buffer: 2048, Elapsed: 00:00:06.7781371
File: 1073741824, Buffer: 4096, Elapsed: 00:00:05.7471923
File: 1073741824, Buffer: 8192, Elapsed: 00:00:04.6344858
File: 1073741824, Buffer: 16384, Elapsed: 00:00:04.4647610
File: 1073741824, Buffer: 32768, Elapsed: 00:00:04.2576851
File: 1073741824, Buffer: 65536, Elapsed: 00:00:04.3291236
File: 1073741824, Buffer: 131072, Elapsed: 00:00:04.1955776
File: 1073741824, Buffer: 262144, Elapsed: 00:00:04.5730469
File: 1073741824, Buffer: 524288, Elapsed: 00:00:05.1157704
File: 1073741824, Buffer: 1048576, Elapsed: 00:00:05.5262273
File: 1073741824, Buffer: 2097152, Elapsed: 00:00:05.6662534
File: 1073741824, Buffer: 4194304, Elapsed: 00:00:05.6285232
File: 1073741824, Buffer: 8388608, Elapsed: 00:00:05.5857300
File: 1073741824, Buffer: 16777216, Elapsed: 00:00:05.6386955
File: 1073741824, Buffer: 33554432, Elapsed: 00:00:05.6467390
File: 1073741824, Buffer: 67108864, Elapsed: 00:00:05.6969160
File: 1073741824, Buffer: 134217728, Elapsed: 00:00:05.7044284
File: 1073741824, Buffer: 268435456, Elapsed: 00:00:05.8279939
File: 1073741824, Buffer: 536870912, Elapsed: 00:00:05.9805622
File: 1073741824, Buffer: 1073741824, Elapsed: 00:00:06.1534688
useAsync = true：
Test Name: FileWritePerformance
Test Outcome: Passed
Result StandardOutput:
File: 1024, Buffer: 1, Elapsed: 00:00:00.0034314
File: 1024, Buffer: 2, Elapsed: 00:00:00.0026302
File: 1024, Buffer: 4, Elapsed: 00:00:00.0024232
File: 1024, Buffer: 8, Elapsed: 00:00:00.0009797
File: 1024, Buffer: 16, Elapsed: 00:00:00.0047272
File: 1024, Buffer: 32, Elapsed: 00:00:00.0010624
File: 1024, Buffer: 64, Elapsed: 00:00:00.0010657
File: 1024, Buffer: 128, Elapsed: 00:00:00.0039249
File: 1024, Buffer: 256, Elapsed: 00:00:00.0011540
File: 1024, Buffer: 512, Elapsed: 00:00:00.0011479
File: 1024, Buffer: 1024, Elapsed: 00:00:00.0010245
File: 1024, Buffer: 2048, Elapsed: 00:00:00.0011083
File: 1024, Buffer: 4096, Elapsed: 00:00:00.0011263
File: 1024, Buffer: 8192, Elapsed: 00:00:00.0009716
File: 1024, Buffer: 16384, Elapsed: 00:00:00.0011319
File: 1024, Buffer: 32768, Elapsed: 00:00:00.0035012
File: 1024, Buffer: 65536, Elapsed: 00:00:00.0010170
File: 1024, Buffer: 131072, Elapsed: 00:00:00.0010995
File: 1024, Buffer: 262144, Elapsed: 00:00:00.0010381
File: 1024, Buffer: 524288, Elapsed: 00:00:00.0010018
File: 1024, Buffer: 1048576, Elapsed: 00:00:00.0011629
File: 1024, Buffer: 2097152, Elapsed: 00:00:00.0011969
File: 1024, Buffer: 4194304, Elapsed: 00:00:00.0021307
File: 1024, Buffer: 8388608, Elapsed: 00:00:00.0015213
File: 1024, Buffer: 16777216, Elapsed: 00:00:00.0012500
File: 1024, Buffer: 33554432, Elapsed: 00:00:00.0012102
File: 1024, Buffer: 67108864, Elapsed: 00:00:00.0013962
File: 1024, Buffer: 134217728, Elapsed: 00:00:00.0036005
File: 1024, Buffer: 268435456, Elapsed: 00:00:00.0034270
File: 1024, Buffer: 536870912, Elapsed: 00:00:00.0070516
File: 1024, Buffer: 1073741824, Elapsed: 00:00:00.0094566
File: 1048576, Buffer: 1, Elapsed: 00:00:00.1161675
File: 1048576, Buffer: 2, Elapsed: 00:00:00.0867371
File: 1048576, Buffer: 4, Elapsed: 00:00:00.0938809
File: 1048576, Buffer: 8, Elapsed: 00:00:00.0891819
File: 1048576, Buffer: 16, Elapsed: 00:00:00.0856688
File: 1048576, Buffer: 32, Elapsed: 00:00:00.0829121
File: 1048576, Buffer: 64, Elapsed: 00:00:00.1021716
File: 1048576, Buffer: 128, Elapsed: 00:00:00.0819636
File: 1048576, Buffer: 256, Elapsed: 00:00:00.0870802
File: 1048576, Buffer: 512, Elapsed: 00:00:00.0938601
File: 1048576, Buffer: 1024, Elapsed: 00:00:00.0906152
File: 1048576, Buffer: 2048, Elapsed: 00:00:00.0203416
File: 1048576, Buffer: 4096, Elapsed: 00:00:00.0156412
File: 1048576, Buffer: 8192, Elapsed: 00:00:00.0188322
File: 1048576, Buffer: 16384, Elapsed: 00:00:00.0090570
File: 1048576, Buffer: 32768, Elapsed: 00:00:00.0089590
File: 1048576, Buffer: 65536, Elapsed: 00:00:00.0111345
File: 1048576, Buffer: 131072, Elapsed: 00:00:00.0074629
File: 1048576, Buffer: 262144, Elapsed: 00:00:00.0087473
File: 1048576, Buffer: 524288, Elapsed: 00:00:00.0090963
File: 1048576, Buffer: 1048576, Elapsed: 00:00:00.0106754
File: 1048576, Buffer: 2097152, Elapsed: 00:00:00.0072628
File: 1048576, Buffer: 4194304, Elapsed: 00:00:00.0072143
File: 1048576, Buffer: 8388608, Elapsed: 00:00:00.0108595
File: 1048576, Buffer: 16777216, Elapsed: 00:00:00.0078417
File: 1048576, Buffer: 33554432, Elapsed: 00:00:00.0078852
File: 1048576, Buffer: 67108864, Elapsed: 00:00:00.0101042
File: 1048576, Buffer: 134217728, Elapsed: 00:00:00.0107911
File: 1048576, Buffer: 268435456, Elapsed: 00:00:00.0099841
File: 1048576, Buffer: 536870912, Elapsed: 00:00:00.0135941
File: 1048576, Buffer: 1073741824, Elapsed: 00:00:00.0199248
File: 1073741824, Buffer: 1, Elapsed: 00:01:24.9154625
File: 1073741824, Buffer: 2, Elapsed: 00:01:13.0649384
File: 1073741824, Buffer: 4, Elapsed: 00:01:20.7892031
File: 1073741824, Buffer: 8, Elapsed: 00:01:14.3207983
File: 1073741824, Buffer: 16, Elapsed: 00:01:19.1491590
File: 1073741824, Buffer: 32, Elapsed: 00:01:13.1612342
File: 1073741824, Buffer: 64, Elapsed: 00:01:21.2668008
File: 1073741824, Buffer: 128, Elapsed: 00:01:12.1101598
File: 1073741824, Buffer: 256, Elapsed: 00:01:17.3565432
File: 1073741824, Buffer: 512, Elapsed: 00:01:14.2698611
File: 1073741824, Buffer: 1024, Elapsed: 00:01:28.4650820
File: 1073741824, Buffer: 2048, Elapsed: 00:00:20.3608743
File: 1073741824, Buffer: 4096, Elapsed: 00:00:13.4953693
File: 1073741824, Buffer: 8192, Elapsed: 00:00:09.7105073
File: 1073741824, Buffer: 16384, Elapsed: 00:00:07.8587447
File: 1073741824, Buffer: 32768, Elapsed: 00:00:06.9201750
File: 1073741824, Buffer: 65536, Elapsed: 00:00:06.4229520
File: 1073741824, Buffer: 131072, Elapsed: 00:00:05.9591664
File: 1073741824, Buffer: 262144, Elapsed: 00:00:05.7492599
File: 1073741824, Buffer: 524288, Elapsed: 00:00:05.6921116
File: 1073741824, Buffer: 1048576, Elapsed: 00:00:05.6765633
File: 1073741824, Buffer: 2097152, Elapsed: 00:00:05.7068171
File: 1073741824, Buffer: 4194304, Elapsed: 00:00:05.6758360
File: 1073741824, Buffer: 8388608, Elapsed: 00:00:05.5947740
File: 1073741824, Buffer: 16777216, Elapsed: 00:00:05.5885432
File: 1073741824, Buffer: 33554432, Elapsed: 00:00:06.0881748
File: 1073741824, Buffer: 67108864, Elapsed: 00:00:06.2771976
File: 1073741824, Buffer: 134217728, Elapsed: 00:00:06.2455592
File: 1073741824, Buffer: 268435456, Elapsed: 00:00:06.4810064
File: 1073741824, Buffer: 536870912, Elapsed: 00:00:06.4810521
File: 1073741824, Buffer: 1073741824, Elapsed: 00:00:06.5314156
测试用代码：
[TestClass]
public class WritePerfTests
{
[TestMethod]
public async Task FileWritePerformance()
{
var fileSize = 1L;
var useAsync = true;
using (Loggers.Register(ConsoleLogger.CreateDefault()))
using (var logger = Loggers.Register(FileLogger.CreateDefault(typeof(WritePerfTests))))
{
logger.ClearLog();
var bytes = Rng.Pseudo.GetBytes(1024);
for (var i = 0; i < 3; i++)
{
fileSize *= 1024;
for (var p = 0; p < 31; p++)
{
var buffer = (int)Math.Pow(2, p);
var file = new FileInfo($"d:/{fileSize}.{p}.test");
using (Disposable.Action(() => file.Delete()))
using (var stream = new FileStream(file.FullName, FileMode.Create, FileAccess.Write, FileShare.None, buffer, useAsync))
{
var w = Stopwatch.StartNew();
while (stream.Length < fileSize)
await stream.WriteAsync(bytes, 0, bytes.Length);
await stream.FlushAsync();
Loggers.Info($"File: {fileSize}, Buffer: {buffer}, Elapsed: {w.Elapsed}");
}
await Task.Delay(1000); //give file system chance to cleanup so next test wont be skewed
}
}
}
}
}
- Justin
5
1为什么 async:true 总是比同步方式慢？ - Rui Lima
1@RuiLima，我只能猜测useAsync=true之所以较慢，是因为文件系统和应用程序之间来回传输的原因。当缓冲区设置为1字节并写入1GB文件时，这一点变得非常明显。 - Justin
3@Justin您的回答非常好，但是您的TL;DR（总结）是错误的：“缓冲区大小基本上不重要。” 您自己的测试表明这是不正确的。对于大文件，所花费的时间会根据缓冲区大小从简单的变为两倍不等。这是非常重要的。 - Casto
@Casto 谢谢，这就是我说“大多数情况”的原因，但我已经更新了它以反映大文件也应该使用至少128KiB的缓冲区。 - Justin
2我认为应该使用
FileOptions.SequentialScan 重新运行此测试。 - Dai1
简短回答：对于大文件集，请设置缓冲区大小>81920并在写入之前预分配所有文件大小。
长回答：
- 如果缓冲区小于物理扇区大小，则会受到惩罚，不应使用小于4096字节的缓冲区，因为自2011年以来，磁盘正在使用4096字节的扇区（也称为4K本地，4Kn），现在一些SSD的扇区大于4096字节。
- 对于大文件，您必须在写入之前预分配其大小（设置FileStream的长度）。这将加快复制过程，因为文件系统仅会增加文件大小一次。
- 如果查看.NET源代码中的FileStream，默认缓冲区大小为4096。管道使用81920作为默认缓冲区大小，并且对涉及异步操作的复制过程使用2M作为最大缓冲区大小。
- Alexei Shcherbakov
网页内容由stack overflow 提供, 点击上面的可以查看英文原文，
原文链接
原文链接