我正在使用MATLAB中的fwrite命令编写一些较大（~500MB - 3GB）的二进制数据。我希望将数据以表格形式书写，因此使用了skip参数。例如，我有两个uint8值向量
因此，在我的代码中，我会类似于以下操作（我的数据更加复杂）：
a = [1 2 3 4]; b = [5 6 7 8]。我想让二进制文件看起来像这样 1 5 2 6 3 7 4 8。因此，在我的代码中，我会类似于以下操作（我的数据更加复杂）：
fwrite(f,a,'1*uint8',1);
fseek(f,2)
fwrite(f,b,'1*uint8',1);
但写入速度非常慢（ 2MB/s ）。
我运行了下面的代码块，当我设置跳过计数为1时，写入速度大约慢了300倍。
>> f = fopen('testfile.bin', 'w');
>> d = uint8(1:500e6);
>> tic; fwrite(f,d,'1*uint8',1); toc
Elapsed time is 58.759686 seconds.
>> tic; fwrite(f,d,'1*uint8',0); toc
Elapsed time is 0.200684 seconds.
>> 58.759686/0.200684
ans =
292.7971
我能理解skip参数设置为1后需要遍历两倍的字节大小会导致2倍或4倍的减速，但是300倍的减速让我觉得自己做错了什么。
有人遇到过这种情况吗？有没有方法可以加速写入操作？
谢谢！
更新
我编写了以下函数来格式化任意数据集。对于大数据集，写入速度得到了极大的提升（约300MB/s）。
%
% data: A cell array of matrices. Matrices can be composed of any
% non-complex numeric data. Each entry in data is considered
% to be an independent column in the data file. Rows are indexed
% by the last column in the numeric matrix hence the count of elements
% in the last dimension of the matrix must match.
%
% e.g.
% size(data{1}) == [1,5]
% size(data{2}) == [4,5]
% size(data{3}) == [3,2,5]
%
% The data variable has 3 columns and 5 rows. Column 1 is made of scalar values
% Column 2 is made of vectors of length 4. And column 3 is made of 3 x 2
% matrices
%
%
% returns buffer: a N x M matrix of bytes where N is the number of bytes
% of each row of data, and M is the number of rows of data.
function [buffer] = makeTabularDataBuffer(data)
dataTypes = {};
dataTypesLengthBytes = [];
rowElementCounts = []; %the number of elements in each "row"
rowCount = [];
%figure out properties of tabular data
for idx = 1:length(data)
cDat = data{idx};
dimSize = size(cDat);
%ensure each column has the same number of rows.
if isempty(rowCount)
rowCount = dimSize(end);
else
if dimSize(end) ~= rowCount
throw(MException('e:e', sprintf('data column %d does not have the required number of rows (%d)\n',idx,rowCount)));
end
end
dataTypes{idx} = class(data{idx});
dataTypesLengthBytes(idx) = length(typecast(eval([dataTypes{idx},'(1)']),'uint8'));
rowElementCounts(idx) = prod(dimSize(1:end-1));
end
rowLengthBytes = sum(rowElementCounts .* dataTypesLengthBytes);
buffer = zeros(rowLengthBytes, rowCount,'uint8'); %rows of the dataset map to column in the buffer matrix because fwrite writes columnwise
bufferRowStartIdxs = cumsum([1 dataTypesLengthBytes .* rowElementCounts]);
%load data 1 column at a time into the buffer
for idx = 1:length(data)
cDat = data{idx};
columnWidthBytes = dataTypesLengthBytes(idx)*rowElementCounts(idx);
cRowIdxs = bufferRowStartIdxs(idx):(bufferRowStartIdxs(idx+1)-1);
buffer(cRowIdxs,:) = reshape(typecast(cDat(:),'uint8'),columnWidthBytes,[]);
end
end
我对该函数进行了有限的测试，但看起来它已经按照预期工作了。返回的缓冲矩阵可以直接传递给fwrite函数，无需跳过参数，fwrite将以列主序写入缓冲区。
dat = {};
dat{1} = uint16([1 2 3 4]);
dat{2} = uint16([5 6 7 8]);
dat{3} = double([9 10 ; 11 12; 13 14; 15 16])';
buffer = makeTabularDataBuffer(dat)
buffer =
20×4 uint8 matrix
1 2 3 4
0 0 0 0
5 6 7 8
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
34 38 42 46
64 64 64 64
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
36 40 44 48
64 64 64 64