我的MATLAB程序正在读取一个大约有700万行的文件，并且在I/O上浪费了太多时间。我知道每一行都是格式化为两个整数，但我不知道它们占用多少字符。str2num非常慢，我应该使用哪个MATLAB函数代替它？
注意：我必须逐行操作，而不能将整个文件存储在内存中，因此无法使用读取整个矩阵的命令。
fid = fopen('file.txt');
tline = fgetl(fid);
while ischar(tline)
nums = str2num(tline);
%do stuff with nums
tline = fgetl(fid);
end
fclose(fid);
这是一个普遍的难题，没有什么比测试更能回答了。以下是我的假设：
一个格式良好的ASCII文件，包含两列数字。没有标题，没有不一致的行等。
该方法必须适用于读取文件，这些文件太大而无法保存在内存中（尽管我的耐心有限，因此我的测试文件只有500,000行）。
实际操作（OP所谓的“对数字进行处理”）必须逐行执行，不能矢量化。
考虑到这一点，答案和评论似乎鼓励在三个方面提高效率：
我编写了一个快速脚本来测试这些主题的摄入速度（和结果的一致性）。结果如下：
超过原始时间的一半（68-> 27秒）被str2num调用中的低效率消耗掉了，可以通过切换sscanf来消除这种情况。
剩余时间的另外2/3（27-> 8秒）可以通过对文件读取和字符串转换使用更大的批次来减少。
如果我们愿意违反原始帖子中的第三条规则，那么另外7/8的时间可以通过切换到完全数字处理来减少。然而，有些算法不适合这种情况，所以我们将其保留。（请注意，最后一个条目的“检查”值不匹配。）
最后，与我之前在此回复中的编辑直接矛盾的是，通过切换可用的缓存Java单行读取器并不能节省任何时间。事实上，该解决方案比使用本地读取器的相应单行结果要慢2-3倍（63秒对27秒）。
上述所有解决方案的示例代码如下。
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Create a test file
cd(tempdir);
fName = 'demo_file.txt';
fid = fopen(fName,'w');
for ixLoop = 1:5
d = randi(1e6, 1e5,2);
fprintf(fid, '%d, %d \n',d);
end
fclose(fid);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Initial code
CHECK = 0;
tic;
fid = fopen('demo_file.txt');
tline = fgetl(fid);
while ischar(tline)
nums = str2num(tline);
CHECK = round((CHECK + mean(nums) ) /2);
tline = fgetl(fid);
end
fclose(fid);
t = toc;
fprintf(1,'Initial code. %3.2f sec. %d check \n', t, CHECK);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Using sscanf, once per line
CHECK = 0;
tic;
fid = fopen('demo_file.txt');
tline = fgetl(fid);
while ischar(tline)
nums = sscanf(tline,'%d, %d');
CHECK = round((CHECK + mean(nums) ) /2);
tline = fgetl(fid);
end
fclose(fid);
t = toc;
fprintf(1,'Using sscanf, once per line. %3.2f sec. %d check \n', t, CHECK);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Using fscanf in large batches
CHECK = 0;
tic;
bufferSize = 1e4;
fid = fopen('demo_file.txt');
scannedData = reshape(fscanf(fid, '%d, %d', bufferSize),2,[])' ;
while ~isempty(scannedData)
for ix = 1:size(scannedData,1)
nums = scannedData(ix,:);
CHECK = round((CHECK + mean(nums) ) /2);
end
scannedData = reshape(fscanf(fid, '%d, %d', bufferSize),2,[])' ;
end
fclose(fid);
t = toc;
fprintf(1,'Using fscanf in large batches. %3.2f sec. %d check \n', t, CHECK);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Using textscan in large batches
CHECK = 0;
tic;
bufferSize = 1e4;
fid = fopen('demo_file.txt');
scannedData = textscan(fid, '%d, %d \n', bufferSize) ;
while ~isempty(scannedData{1})
for ix = 1:size(scannedData{1},1)
nums = [scannedData{1}(ix) scannedData{2}(ix)];
CHECK = round((CHECK + mean(nums) ) /2);
end
scannedData = textscan(fid, '%d, %d \n', bufferSize) ;
end
fclose(fid);
t = toc;
fprintf(1,'Using textscan in large batches. %3.2f sec. %d check \n', t, CHECK);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Reading in large batches into memory, incrementing to end-of-line, sscanf
CHECK = 0;
tic;
fid = fopen('demo_file.txt');
bufferSize = 1e4;
eol = sprintf('\n');
dataBatch = fread(fid,bufferSize,'uint8=>char')';
dataIncrement = fread(fid,1,'uint8=>char');
while ~isempty(dataIncrement) && (dataIncrement(end) ~= eol) && ~feof(fid)
dataIncrement(end+1) = fread(fid,1,'uint8=>char'); %This can be slightly optimized
end
data = [dataBatch dataIncrement];
while ~isempty(data)
scannedData = reshape(sscanf(data,'%d, %d'),2,[])';
for ix = 1:size(scannedData,1)
nums = scannedData(ix,:);
CHECK = round((CHECK + mean(nums) ) /2);
end
dataBatch = fread(fid,bufferSize,'uint8=>char')';
dataIncrement = fread(fid,1,'uint8=>char');
while ~isempty(dataIncrement) && (dataIncrement(end) ~= eol) && ~feof(fid)
dataIncrement(end+1) = fread(fid,1,'uint8=>char');%This can be slightly optimized
end
data = [dataBatch dataIncrement];
end
fclose(fid);
t = toc;
fprintf(1,'Reading large batches into memory, then sscanf. %3.2f sec. %d check \n', t, CHECK);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Using Java single line readers + sscanf
CHECK = 0;
tic;
bufferSize = 1e4;
reader = java.io.LineNumberReader(java.io.FileReader('demo_file.txt'),bufferSize );
tline = char(reader.readLine());
while ~isempty(tline)
nums = sscanf(tline,'%d, %d');
CHECK = round((CHECK + mean(nums) ) /2);
tline = char(reader.readLine());
end
reader.close();
t = toc;
fprintf(1,'Using java single line file reader and sscanf on single lines. %3.2f sec. %d check \n', t, CHECK);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Using Java scanner for file reading and string conversion
CHECK = 0;
tic;
jFile = java.io.File('demo_file.txt');
scanner = java.util.Scanner(jFile);
scanner.useDelimiter('[\s\,\n\r]+');
while scanner.hasNextInt()
nums = [scanner.nextInt() scanner.nextInt()];
CHECK = round((CHECK + mean(nums) ) /2);
end
scanner.close();
t = toc;
fprintf(1,'Using java single item token scanner. %3.2f sec. %d check \n', t, CHECK);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Reading in large batches into memory, vectorized operations (non-compliant solution)
CHECK = 0;
tic;
fid = fopen('demo_file.txt');
bufferSize = 1e4;
eol = sprintf('\n');
dataBatch = fread(fid,bufferSize,'uint8=>char')';
dataIncrement = fread(fid,1,'uint8=>char');
while ~isempty(dataIncrement) && (dataIncrement(end) ~= eol) && ~feof(fid)
dataIncrement(end+1) = fread(fid,1,'uint8=>char'); %This can be slightly optimized
end
data = [dataBatch dataIncrement];
while ~isempty(data)
scannedData = reshape(sscanf(data,'%d, %d'),2,[])';
CHECK = round((CHECK + mean(scannedData(:)) ) /2);
dataBatch = fread(fid,bufferSize,'uint8=>char')';
dataIncrement = fread(fid,1,'uint8=>char');
while ~isempty(dataIncrement) && (dataIncrement(end) ~= eol) && ~feof(fid)
dataIncrement(end+1) = fread(fid,1,'uint8=>char');%This can be slightly optimized
end
data = [dataBatch dataIncrement];
end
fclose(fid);
t = toc;
fprintf(1,'Fully batched operations. %3.2f sec. %d check \n', t, CHECK);
(译文)
关于Ben所提到的问题，如果您按行读取文件，则瓶颈始终是文件I/O。
我知道有时候您无法将整个文件都放入内存中。我通常会一次性读入大量字符（大约1e5、1e6等，具体取决于您系统的内存）。然后，我将进一步读取单个字符（或返回单个字符）以获取一个整数行数，然后运行字符串解析（例如 sscanf）。
接下来，您可以一次处理生成的大型矩阵的一行，在读取完整个文件之前重复这个过程。
它有点繁琐，但并不难。我通常看到比单行阅读器快90%以上的速度提升。
CHECK值仍然不匹配，这是有意的。但它至少应该表示实际上正在处理所有读取的数据。tic/toc时间没有改变足以引起摘要的更改（实际上略微更快，可能是由于计算机上的其他活动）。 - Pursuitmemmapfile() 取得了良好的结果（速度方面）。这样可以将内存数据复制的数量最小化，并利用内核的IO缓冲。您需要足够的空闲地址空间（但不是实际的空闲内存）来映射整个文件，并且需要足够的自由内存来存储输出变量（很明显！）data 中。fname = 'file.txt';
fstats = dir(fname);
% Map the file as one long character string
m = memmapfile(fname, 'Format', {'uint8' [ 1 fstats.bytes] 'asUint8'});
textdata = char(m.Data(1).asUint8);
% Use textscan() to parse the string and convert to an int32 matrix
data = textscan(textdata, '%d %d', 'CollectOutput', 1);
data = data{:};
% Tidy up!
clear('m')
您可能需要调整textscan()的参数以获得您想要的结果 - 请参阅在线文档。
memmapfile 仅在像这样顺序地读取整个文件时使用时并不占优势。您可以直接对文件进行相同的 textscan() 调用，并使用更少的内存获得相同的结果。Memmapfile 更适用于大型文件的分散（非顺序）访问。 - Andrew Jankechar(m.Data(1).asUint8) 时，你最终不是也只是复制了一遍吗？ - Andrew Janke即使您无法将整个文件放入内存中，也应使用矩阵读取函数读取大批量数据。
可能您甚至可以使用向量操作进行一些数据处理，这将进一步加快速度。
fscanf按块读取数据，这将是原始代码的近似替代品，并且比重复使用num2str或sscanf调用要快得多。 - Andrew Janke
fscanf(fid, '%d %d', 100000)读取一大块内容，然后循环处理该块中的数字，而不是使用sscanf。并使用profile on -timer real确认你花费时间的位置。 - Andrew Janke