我有以下问题:我使用pandas模块打开了一个矩阵,该矩阵的每个单元格都有一个介于-1和1之间的数字。我想找到一行中最大的“可能”值,该值也不是另一行的最大值。
例如,如果2行在同一列具有最大值,则比较两个值并选择较大的一个,然后对于最大值小于其他行的行,我选择第二大的值(并且再次进行相同的分析)。
为了更好地解释自己,请考虑我的代码。
这段代码适用于我使用100x100的矩阵等小型数据集,但如果矩阵大小增加到50,000x50,000,该代码将需要很长时间才能完成。我知道我的代码可能是最低效的处理方式,但我不知道如何解决这个问题。
我已经在阅读有关Python中线程的资料,但如果我放置50,000个线程,这并不能提高计算机的效率。我还尝试使用一些函数,例如
如果有人能够帮助我或给我一些建议来提高代码效率,我将非常感激。
例如,如果2行在同一列具有最大值,则比较两个值并选择较大的一个,然后对于最大值小于其他行的行,我选择第二大的值(并且再次进行相同的分析)。
为了更好地解释自己,请考虑我的代码。
import pandas as pd
matrix = pd.read_csv("matrix.csv")
# this matrix has an id (or name) for each column
# ... and the firt column has the id of each row
results = pd.DataFrame(np.empty((len(matrix),3),dtype=pd.Timestamp),columns=['id1','id2','max_pos'])
l = len(matrix.col[[0]]) # number of columns
while next = 1:
next = 0
for i in range(0, len(matrix)):
max_column = str(0)
for j in range(1, l): # 1 because the first column is an id
if matrix[max_column][i] < matrix[str(j)][i]:
max_column = str(j)
results['id1'][i] = str(i) # I coul put here also matrix['0'][i]
results['id2'][i] = max_column
results['max_pos'][i] = matrix[max_column][i]
for i in range(0, len(results)): #now I will check if two or more rows have the same max column
for ii in range(0, len(results)):
# if two id1 has their max in the same column, I keep it with the biggest
# ... max value and chage the other to "-1" to iterate again
if (results['id2'][i] == results['id2'][ii]) and (results['max_pos'][i] < results['max_pos'][ii]):
matrix[results['id2'][i]][i] = -1
next = 1
举个例子:
#consider
pd.DataFrame({'a':[1, 2, 5, 0], 'b':[4, 5, 1, 0], 'c':[3, 3, 4, 2], 'd':[1, 0, 0, 1]})
a b c d
0 1 4 3 1
1 2 5 3 0
2 5 1 4 0
3 0 0 2 1
#at the first iterarion I will have the following result
0 b 4 # this means that the row 0 has its maximum at column 'b' and its value is 4
1 b 5
2 a 5
3 c 2
#the problem is that column b is the maximum of row 0 and 1, but I know that the maximum of row 1 is bigger than row 0, so I take the second maximum of row 0, then:
0 c 3
1 b 5
2 a 5
3 c 2
#now I solved the problem for row 0 and 1, but I have that the column c is the maximum of row 0 and 3, so I compare them and take the second maximum in row 3
0 c 3
1 b 5
2 a 5
3 d 1
#now I'm done. In the case that two rows have the same column as maximum and also the same number, nothing happens and I keep with that values.
#what if the matrix would be
pd.DataFrame({'a':[1, 2, 5, 0], 'b':[5, 5, 1, 0], 'c':[3, 3, 4, 2], 'd':[1, 0, 0, 1]})
a b c d
0 1 5 3 1
1 2 5 3 0
2 5 1 4 0
3 0 0 2 1
#then, at the first itetarion the result will be:
0 b 5
1 b 5
2 a 5
3 c 2
#then, given that the max value of row 0 and 1 is at the same column, I should compare the maximum values
# ... but in this case the values are the same (both are 5), this would be the end of iterating
# ... because I can't choose between row 0 and 1 and the other rows have their maximum at different columns...
这段代码适用于我使用100x100的矩阵等小型数据集,但如果矩阵大小增加到50,000x50,000,该代码将需要很长时间才能完成。我知道我的代码可能是最低效的处理方式,但我不知道如何解决这个问题。
我已经在阅读有关Python中线程的资料,但如果我放置50,000个线程,这并不能提高计算机的效率。我还尝试使用一些函数,例如
.max()
,但我无法获取列的最大值并将其与其他最大值进行比较...如果有人能够帮助我或给我一些建议来提高代码效率,我将非常感激。