Python中Scipy的Spearman矩阵相关性与两个数组相关性以及pandas.Data.Frame.corr()不匹配

Question

Python中Scipy的Spearman矩阵相关性与两个数组相关性以及pandas.Data.Frame.corr()不匹配

5

我正在计算矩阵的斯皮尔曼相关系数。当使用scipy.stats.spearmanr时，我发现矩阵输入和两个数组输入得到的结果不同。这些结果也与pandas.Data.Frame.corr不同。

from scipy.stats import spearmanr # scipy 1.0.1
import pandas as pd # 0.22.0
import numpy as np
#Data 
X = pd.DataFrame({"A":[-0.4,1,12,78,84,26,0,0], "B":[-0.4,3.3,54,87,25,np.nan,0,1.2], "C":[np.nan,56,78,0,np.nan,143,11,np.nan], "D":[0,-9.3,23,72,np.nan,-2,-0.3,-0.4], "E":[78,np.nan,np.nan,0,-1,-11,1,323]})
matrix_rho_scipy = spearmanr(X,nan_policy='omit',axis=0)[0]
matrix_rho_pandas = X.corr('spearman')
print(matrix_rho_scipy == matrix_rho_pandas.values) # All False except diagonal
print(spearmanr(X['A'],X['B'],nan_policy='omit',axis=0)[0]) # 0.8839285714285714 from scipy 1.0.1
print(spearmanr(X['A'],X['B'],nan_policy='omit',axis=0)[0]) # 0.8829187134416477 from scipy 1.1.0
print(matrix_rho_scipy[0,1]) # 0.8263621207201486
print(matrix_rho_pandas.values[0,1]) # 0.8829187134416477

后来我发现Pandas中的rho与R中的rho是相同的。

X = data.frame(A=c(-0.4,1,12,78,84,26,0,0), 
  B=c(-0.4,3.3,54,87,25,NaN,0,1.2), C=c(NaN,56,78,0,NaN, 143,11,NaN), 
  D=c(0,-9.3,23,72,NaN,-2,-0.3,-0.4), E=c(78,NaN,NaN,0,-1,-11,1,323)) 
cor.test(X$A,X$B,method='spearman', exact = FALSE, na.action="na.omit") # 0.8829187

然而，Pandas的相关函数在处理大型表格时会出现问题（例如，这里，我的情况是16,000）。

感谢Warren Weckesser的测试，我发现Scipy 1.1.0版本（但不包括1.0.1版本）得到的两个数组结果与Pandas和R相同。

如果您有任何建议或评论，请告诉我。谢谢。

我使用的是Python: 3.6.2（Anaconda）；Mac OS: 10.10.5。

- Chih-Hsu Jack Lin

请查看 https://github.com/scipy/scipy/issues/3645。 - Warren Weckesser

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Warren Weckesser · Accepted Answer

看起来scipy.stats.spearmanr在输入为数组且给定axis时，无法如预期地处理nan值。以下是一个脚本，比较了几种计算成对Spearman等级相关性的方法：

import numpy as np
import pandas as pd
from scipy.stats import spearmanr


x = np.array([[np.nan,    3.0, 4.0, 5.0, 5.1, 6.0, 9.2],
              [5.0,    np.nan, 4.1, 4.8, 4.9, 5.0, 4.1],
              [0.5,       4.0, 7.1, 3.8, 8.0, 5.1, 7.6]])

r = spearmanr(x, nan_policy='omit', axis=1)[0]
print("spearmanr, array:           %11.7f %11.7f %11.7f" % (r[0, 1], r[0, 2], r[1, 2]))

r01 = spearmanr(x[0], x[1], nan_policy='omit')[0]
r02 = spearmanr(x[0], x[2], nan_policy='omit')[0]
r12 = spearmanr(x[1], x[2], nan_policy='omit')[0]

print("spearmanr, individual:      %11.7f %11.7f %11.7f" % (r01, r02, r12))

df = pd.DataFrame(x.T)
c = df.corr('spearman')

print("Pandas df.corr('spearman'): %11.7f %11.7f %11.7f" % (c[0][1], c[0][2], c[1][2]))
print("R cor.test:                   0.2051957   0.4857143  -0.4707919")
print('  (method="spearman", continuity=FALSE)')

"""
# R code:
> x0 = c(NA, 3, 4, 5, 5.1, 6.0, 9.2)
> x1 = c(5.0, NA, 4.1, 4.8, 4.9, 5.0, 4.1)
> x2 = c(0.5, 4.0, 7.1, 3.8, 8.0, 5.1, 7.6)
> cor.test(x0, x1, method="spearman", continuity=FALSE)
> cor.test(x0, x2, method="spearman", continuity=FALSE)
> cor.test(x1, x2, method="spearman", continuity=FALSE)
"""

输出：

spearmanr, array:            -0.0727393  -0.0714286  -0.4728054
spearmanr, individual:        0.2051957   0.4857143  -0.4707919
Pandas df.corr('spearman'):   0.2051957   0.4857143  -0.4707919
R cor.test:                   0.2051957   0.4857143  -0.4707919
  (method="spearman", continuity=FALSE)

我的建议是不要使用形式为spearmanr(x, nan_policy='omit', axis=<whatever>)的scipy.stats.spearmanr。而是使用Pandas DataFrame的corr()方法，或者使用循环逐一计算值，使用spearmanr(x0, x1, nan_policy='omit')。