Node.js - 使用createReadStream和createWriteStream时内存使用量高

Question

我正在使用node进行流测试，我设置了一个程序来使用流读取大文件并再次使用流写入。问题是当运行该程序时，Node的内存使用量会上升到1.3 GB，这恰好是要读取的文件的大小。就像它没有使用流，而是将其缓冲并一次性写入OR垃圾回收器没有销毁内存中的“chunk”变量。这是程序：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});
const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});

readStream.on('data', function (chunk) {
    writeStream.write(chunk);
})

readStream.on('end', function () {
    console.log("reading done");
    writeStream.end();
});

writeStream.on('close', function () {
    console.log("Writing done.");
})

奇怪的是，如果我将这些流进行管道处理，它会按照预期工作，并且内存使用率不会超过 20 MB。像这样：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});
const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});

readStream.pipe(writeStream);

什么可能会导致这种行为？

节点版本：v14.15.4

- Miad Abdi

Answer 1

好的，我找到了问题所在。有一个叫做“背压”的条件。在我的情况下，这是因为读取流速度比写入流速度要快很多。而且这是因为readStream将读取的数据缓存在内存中，直到writeMemory写入它。所以解决方案是我们暂停readStream，直到writeStream完成写入，然后再提供更多的数据块。

这是正确的程序:

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});

const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});


readStream.on('data', function (chunk) {
    // according to docs the value of result variable is: 
    // Returns: <boolean> false if the stream wishes for the calling code to wait for the 'drain' event to be emitted before continuing to write additional data; otherwise true.
    const result = writeStream.write(chunk);

    if(!result) {
        console.log("BACKPRESSURE");
        readStream.pause();
    }
});

writeStream.on('drain', () => {
    console.log("DREAINED");
    readStream.resume();
});

readStream.on('end', function () {
    console.log("reading done");
    writeStream.end();
});

writeStream.on('close', function () {
    console.log("Writing done.");
})

drain 事件的文档在这里。

- Miad Abdi

---

非常好，我现在正在阅读官方文档关于流的背压(backpressure)：https://nodejs.org/es/docs/guides/backpressuring-in-streams。你的问题/答案恰好符合我的搜索。 - Rhadamez Gindri Hercilio