我的脚本正在调用 librosa模块来计算短音频片段的梅尔倒谱系数(MFCCs)。在加载音频后,我想尽可能快地计算这些特征(以及其他一些音频特征)- 因此使用了多进程处理。
问题:多进程变体比顺序处理要慢得多。分析显示,我的代码超过90%的时间花费在
据我理解, multiprocessing 模块应该几乎复制所有内容,并且不应该有任何锁定冲突。然而,结果似乎显示了不同的情况。难道是底层的 librosa 模块保持了某种内部锁定吗?
我的简介结果以纯文本形式呈现:https://drive.google.com/open?id=17DHfmwtVOJOZVnwIueeoWClUaWkvhTPc 作为图片:https://drive.google.com/open?id=1KuZyo0CurHd9GjXge5CYQhdWn2Q6OG8Z 复制“问题”的代码:
在我的机器上的结果:
问题:多进程变体比顺序处理要慢得多。分析显示,我的代码超过90%的时间花费在
<method 'acquire' of '_thread.lock' objects>
上。如果是许多小任务,这并不奇怪,但在一个测试案例中,我将音频分成4个块并在单独的进程中处理它们。我认为开销应该很小,但实际上,它几乎和许多小任务一样糟糕。据我理解, multiprocessing 模块应该几乎复制所有内容,并且不应该有任何锁定冲突。然而,结果似乎显示了不同的情况。难道是底层的 librosa 模块保持了某种内部锁定吗?
我的简介结果以纯文本形式呈现:https://drive.google.com/open?id=17DHfmwtVOJOZVnwIueeoWClUaWkvhTPc 作为图片:https://drive.google.com/open?id=1KuZyo0CurHd9GjXge5CYQhdWn2Q6OG8Z 复制“问题”的代码:
import time
import numpy as np
import librosa
from functools import partial
from multiprocessing import Pool
n_proc = 4
y, sr = librosa.load(librosa.util.example_audio_file(), duration=60) # load audio sample
y = np.repeat(y, 10) # repeat signal so that we can get more reliable measurements
sample_len = int(sr * 0.2) # We will compute MFCC for short pieces of audio
def get_mfcc_in_loop(audio, sr, sample_len):
# We split long array into small ones of lenth sample_len
y_windowed = np.array_split(audio, np.arange(sample_len, len(audio), sample_len))
for sample in y_windowed:
mfcc = librosa.feature.mfcc(y=sample, sr=sr)
start = time.time()
get_mfcc_in_loop(y, sr, sample_len)
print('Time single process:', time.time() - start)
# Let's test now feeding these small arrays to pool of 4 workers. Since computing
# MFCCs for these small arrays is fast, I'd expect this to be not that fast
start = time.time()
y_windowed = np.array_split(y, np.arange(sample_len, len(y), sample_len))
with Pool(n_proc) as pool:
func = partial(librosa.feature.mfcc, sr=sr)
result = pool.map(func, y_windowed)
print('Time multiprocessing (many small tasks):', time.time() - start)
# Here we split the audio into 4 chunks and process them separately. This I'd expect
# to be fast and somehow it isn't. What could be the cause? Anything to do about it?
start = time.time()
y_split = np.array_split(y, n_proc)
with Pool(n_proc) as pool:
func = partial(get_mfcc_in_loop, sr=sr, sample_len=sample_len)
result = pool.map(func, y_split)
print('Time multiprocessing (a few large tasks):', time.time() - start)
在我的机器上的结果:
- 单进程时间:8.48秒
- 多进程(许多小任务)时间:44.20秒
- 多进程(几个大任务)时间:41.99秒