这里提供了一种适用于任意数量箱子的向量化方法 -
def vectorized_app(input_data, num_bins):
s0 = input_data.min(0)
s1 = input_data.max(0)
m,n,r = input_data.shape
ids = (num_bins*((input_data - s0)/(s1-s0))).astype(int).clip(max=num_bins-1)
offset = num_bins*(r*np.arange(n)[:,None] + np.arange(r))
ids3D = ids + offset
count3D = np.bincount(ids3D.ravel(), minlength=n*r*num_bins).reshape(n,r,-1)
bins3D = create_ranges_nd(s0, s1, num_bins+1)[...,:-1]
out = np.empty((n,r,num_bins,2))
out[...,0] = bins3D
out[...,1] = count3D
return out
辅助函数 -
def create_ranges_nd(start, stop, N, endpoint=True):
if endpoint==1:
divisor = N-1
else:
divisor = N
steps = (1.0/divisor) * (stop - start)
return start[...,None] + steps[...,None]*np.arange(N)
运行时测试
原始方法 -
def org_app(input_data, num_bins):
q,m,n = input_data.shape
plot_data = np.zeros((m,n,num_bins,2))
for i in range(m):
for j in range(n):
data = input_data[:, i, j]
hist, bins = np.histogram(data, bins=num_bins)
plot_data[i][j] = np.stack((bins[:-1], hist), axis=1)
return plot_data
时间和验证 -
让我们在一个形状为(100, 100, 100)
的大数据数组上进行测试,并将bin的数量设置为10
:
In [967]:
...: num_bins = 10
...: m = 100
...: n = 100
...: q = 100
...: input_data = np.random.rand(q,m,n)
...:
...: out1 = org_app(input_data, num_bins)
...: out2 = vectorized_app(input_data, num_bins)
...: print np.allclose(out1, out2)
...:
True
In [968]: %timeit org_app(input_data, num_bins)
1 loop, best of 3: 748 ms per loop
In [969]: %timeit vectorized_app(input_data, num_bins)
100 loops, best of 3: 12.7 ms per loop
In [970]: 748/12.7
Out[970]: 58.89763779527559
nditer
在这里没有帮助。它只是另一个迭代工具,仍然需要一个 Python 级别的循环。 - hpaulj