我的一个项目存在问题,旨在扫描一个或多个目录以搜索MP3文件,并将其元数据和路径存储到MongoDB中。
运行代码的主计算机是Windows 10 64位,具有8GB RAM,CPU为AMD Ryzen 3.5GHz(4个核心)。 Windows安装在SSD上,而音乐则位于1TB的HDD上。
该nodejs应用程序可以通过命令行或通过NPM手动启动,从这里开始。 我正在使用递归函数来扫描所有目录,大约有20,000个文件左右。
我已经通过graceful-fs解决了EMFILE: too many files open
问题,但现在我遇到了新问题:JavaScript heap out of memory
。
下面是我收到的完整输出:
C:\Users\User\Documents\GitHub\mp3manager>npm run scan
> experiments@1.0.0 scan C:\Users\User\Documents\GitHub\mp3manager
> cross-env NODE_ENV=production NODE_OPTIONS='--max-old-space-size=4096' node scripts/cli/mm scan D:\Musica
Scanning 1 resources in production mode
Trying to connect to mongodb://localhost:27017/music_manager
Connected to mongo...
<--- Last few GCs --->
[16744:0000024DD9FA9F40] 141399 ms: Mark-sweep 63.2 (70.7) -> 63.2 (71.2) MB, 47.8 / 0.1 ms (average mu = 0.165, current mu = 0.225) low memory notification GC in old space requested
[16744:0000024DD9FA9F40] 141438 ms: Mark-sweep 63.2 (71.2) -> 63.2 (71.2) MB, 38.9 / 0.1 ms (average mu = 0.100, current mu = 0.001) low memory notification GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x02aaa229e6e9 <JSObject>
0: builtin exit frame: new ArrayBuffer(aka ArrayBuffer)(this=0x027bb3502801 <the_hole>,0x0202be202569 <Number 8.19095e+06>,0x027bb3502801 <the_hole>)
1: ConstructFrame [pc: 000002AF8F50D385]
2: createUnsafeArrayBuffer(aka createUnsafeArrayBuffer) [00000080419526C9] [buffer.js:~115] [pc=000002AF8F8440B1](this=0x027bb35026f1 <undefined>,size=0x0202be202569 <Number 8.19095e+06>)
3:...
FATAL ERROR: Committing semi space failed. Allocation failed - JavaScript heap out of memory
1: 00007FF6E36FF04A
2: 00007FF6E36DA0C6
3: 00007FF6E36DAA30
4: 00007FF6E39620EE
5: 00007FF6E396201F
6: 00007FF6E3E82BC4
7: 00007FF6E3E79C5C
8: 00007FF6E3E7829C
9: 00007FF6E3E77765
10: 00007FF6E3989A91
11: 00007FF6E35F0E52
12: 00007FF6E3C7500F
13: 00007FF6E3BE55B4
14: 00007FF6E3BE5A5B
15: 00007FF6E3BE587B
16: 000002AF8F55C721
npm ERR! code ELIFECYCLE
npm ERR! errno 134
我尝试使用
NODE_OPTIONS='--max-old-space-size=4096'
,但我甚至不确定Node在Windows上是否考虑了这个选项。 我尝试过p-limit来限制有效运行的promise数量,但是老实说,我现在已经没有新想法了,我开始考虑使用另一种语言来看看它是否能更好地处理这些问题。
任何建议都将不胜感激。
祝你拥有愉快的一天。编辑: 我试图用@Terry发布的
processDir
函数替换该函数,但结果是相同的。更新2019-08-19: 为了避免堆栈问题,我删除了递归,并使用队列添加目录:
const path = require('path');
const mm = require('music-metadata');
const _ = require('underscore');
const fs = require('graceful-fs');
const readline = require('readline');
const audioType = require('audio-type');
// const util = require('util');
const { promisify } = require('util');
const logger = require('../logger');
const { mp3hash } = require('../../../src/libs/utils');
const MusicFile = require('../../../src/models/db/mongo/music_files');
const getStats = promisify(fs.stat);
const readdir = promisify(fs.readdir);
const readFile = promisify(fs.readFile);
// https://github.com/winstonjs/winston#profiling
class MusicScanner {
constructor(options) {
const { paths, keepInMemory } = options;
this.paths = paths;
this.keepInMemory = keepInMemory === true;
this.processResult = {
totFiles: 0,
totBytes: 0,
dirQueue: [],
};
}
async processFile(resource) {
const buf = await readFile(resource);
const fileRes = audioType(buf);
if (fileRes === 'mp3') {
this.processResult.totFiles += 1;
// process the metadata
this.processResult.totBytes += fileSize;
}
}
async processDirectory() {
while(this.processResult.dirQueue.length > 0) {
const dir = this.processResult.dirQueue.shift();
const dirents = await readdir(dir, { withFileTypes: true });
const filesPromises = [];
for (const dirent of dirents) {
const resource = path.resolve(dir, dirent.name);
if (dirent.isDirectory()) {
this.processResult.dirQueue.push(resource);
} else if (dirent.isFile()) {
filesPromises.push(this.processFile(resource));
}
}
await Promise.all(filesPromises);
}
}
async scan() {
const promises = [];
const start = Date.now();
for (const thePath of this.paths) {
this.processResult.dirQueue.push(thePath);
promises.push(this.processDirectory());
}
const paths = await Promise.all(promises);
this.processResult.paths = paths;
return this.processResult;
}
}
module.exports = MusicScanner;
这里的问题是该过程需要54分钟来读取21K个文件,我不确定在这种情况下如何加速该过程。有什么提示吗?