以下是获取最相似单词列表的性能检测结果。在某种程度上,这是一个极端情况,因为模型既没有w.prob也没有w.cluster来缩小搜索空间。我使用了四种方法:上述两种方法、SpaCy的most_similar和Gensim的most_similar。
def spacy_most_similar(word, topn=10):
ms = nlp_ru.vocab.vectors.most_similar(
nlp_ru(word).vector.reshape(1,nlp_ru(word).vector.shape[0]), n=topn)
words = [nlp_ru.vocab.strings[w] for w in ms[0][0]]
distances = ms[2]
return words, distances
def spacy_similarity(word, topn=10):
word = nlp_ru.vocab[str(word)]
queries = [
w for w in word.vocab if w.is_lower == word.is_lower and np.count_nonzero(w.vector)
]
by_similarity = sorted(queries, key=lambda w: w.similarity(word), reverse=True)
return [(w.lower_,w.similarity(word)) for w in by_similarity[:topn+1] if w.lower_ != word.lower_]
def spacy_similarity_numba(word, topn=10):
word = nlp_ru.vocab[str(word)]
queries = [
w for w in word.vocab if w.is_lower == word.is_lower and np.count_nonzero(w.vector)
]
by_similarity = sorted(queries, key=lambda w: cosine_similarity_numba(w.vector, word.vector), reverse=True)
return [(w.lower_,w.similarity(word)) for w in by_similarity[:topn+1] if w.lower_ != word.lower_]
from numba import jit
@jit(nopython=True)
def cosine_similarity_numba(u:np.ndarray, v:np.ndarray):
assert(u.shape[0] == v.shape[0])
uv = 0
uu = 0
vv = 0
for i in range(u.shape[0]):
uv += u[i]*v[i]
uu += u[i]*u[i]
vv += v[i]*v[i]
cos_theta = 1
if uu != 0 and vv != 0:
cos_theta = uv/np.sqrt(uu*vv)
return cos_theta
以下是时间统计结果:
from time import time
import timeit, functools
from timeit import default_timer as timer
print(nlp_ru.vocab.vectors.shape)
arr = ("дерево")
print(f'Gensim most_similar: {timeit.Timer(functools.partial(wv.most_similar, arr)).timeit(1)}')
print(f'Spacy most_similar: {timeit.Timer(functools.partial(spacy_most_similar, arr)).timeit(1)}')
print(f'Spacy cosine_similarity_numba: {timeit.Timer(functools.partial(spacy_similarity_numba, arr)).timeit(1)}')
print(f'Spacy similarity: {timeit.Timer(functools.partial(spacy_similarity, arr)).timeit(1)}')
(1239964, 100)
Gensim most_similar: 0.06437033399993197
Spacy most_similar: 0.4855721250000897
Spacy cosine_similarity_numba: 13.404324778000046
Spacy similarity: 60.58928110700003
所有方法返回的结果都是相同的。如您所见,与其他技术相比,Gensim非常快速。而且您甚至不需要缩小搜索范围。所有测量均在CPU上完成。嵌入取自此处 http://panchenko.me/data/dsl-backup/w2v-ru/all.norm-sz100-w10-cb0-it1-min100.w2v
nlp.vocab
中的所有单词而不是word.vocab
,对吗? - user_1177868dog
的topn=3 most_similar吗? - JJoao[('she', 0.42), ('when', 0.41), ('he', 0.39)]
,这完全荒谬,不幸的是在许多网站上都有类似的例子。不过,感谢提供相关问题链接的点赞。 - Wtower