我想知道在scikit learn的LDA实现中是否有一种方法可以返回主题-词分布,就像genism的show_topics()方法一样。我查看了文档但没有找到任何信息。
我想知道在scikit learn的LDA实现中是否有一种方法可以返回主题-词分布,就像genism的show_topics()方法一样。我查看了文档但没有找到任何信息。
请看sklearn.decomposition.LatentDirichletAllocation.components_
:
components_ : 数组, [主题数, 特征数]
主题单词分布。 components_[i, j] 表示主题 i 中的单词 j。
这是一个最简单的例子:
import numpy as np
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
data = ['blah blah foo bar', 'foo foo foo foo bar', 'bar bar bar bar foo',
'foo bar bar bar baz foo', 'foo foo foo bar baz', 'blah banana',
'cookies candy', 'more text please', 'hey there are more words here',
'bananas', 'i am a real boy', 'boy', 'girl']
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(data)
vocab = vectorizer.get_feature_names()
n_top_words = 5
k = 2
model = LatentDirichletAllocation(n_topics=k, random_state=100)
id_topic = model.fit_transform(X)
topic_words = {}
for topic, comp in enumerate(model.components_):
# for the n-dimensional array "arr":
# argsort() returns a ranked n-dimensional array of arr, call it "ranked_array"
# which contains the indices that would sort arr in a descending fashion
# for the ith element in ranked_array, ranked_array[i] represents the index of the
# element in arr that should be at the ith index in ranked_array
# ex. arr = [3,7,1,0,3,6]
# np.argsort(arr) -> [3, 2, 0, 4, 5, 1]
# word_idx contains the indices in "topic" of the top num_top_words most relevant
# to a given topic ... it is sorted ascending to begin with and then reversed (desc. now)
word_idx = np.argsort(comp)[::-1][:n_top_words]
# store the words most relevant to the topic
topic_words[topic] = [vocab[i] for i in word_idx]
查看结果:
for topic, words in topic_words.items():
print('Topic: %d' % topic)
print(' %s' % ', '.join(words))
Topic: 0
more, blah, here, hey, words
Topic: 1
foo, bar, blah, baz, boy
显然,您应该使用更大的文本体来尝试此代码,但这是获取给定主题的最具信息性的单词的一种方法。
LatentDirichletAllocation
对象本身中,但我可以想到两种(相对简单的)方法来实现这一点:1)如果您正在使用CountVectorizer
,只需按列(即按词汇)逐个求和,并从那里查找;2)使用类似于collections.Counter
的东西,跨整个语料库计算各个令牌的数量(要铺平)。 - blacksite