Scikit Learn TfidfVectorizer：如何获取具有最高tf-idf分数的前n个术语

Question

Scikit Learn TfidfVectorizer：如何获取具有最高tf-idf分数的前n个术语

51

我正在解决关键词提取问题。考虑非常普遍的情况：

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english')

t = """Two Travellers, walking in the noonday sun, sought the shade of a widespreading tree to rest. As they lay looking up among the pleasant leaves, they saw that it was a Plane Tree.

"How useless is the Plane!" said one of them. "It bears no fruit whatever, and only serves to litter the ground with leaves."

"Ungrateful creatures!" said a voice from the Plane Tree. "You lie here in my cooling shade, and yet you say I am useless! Thus ungratefully, O Jupiter, do men receive their blessings!"

Our best blessings are often the least appreciated."""

tfs = tfidf.fit_transform(t.split(" "))
str = 'tree cat travellers fruit jupiter'
response = tfidf.transform([str])
feature_names = tfidf.get_feature_names()

for col in response.nonzero()[1]:
    print(feature_names[col], ' - ', response[0, col])

这让我感到

  (0, 28)   0.443509712811
  (0, 27)   0.517461475101
  (0, 8)    0.517461475101
  (0, 6)    0.517461475101
tree  -  0.443509712811
travellers  -  0.517461475101
jupiter  -  0.517461475101
fruit  -  0.517461475101

很好。对于任何新进来的文档，有没有一种方法可以获取具有最高tfidf分数的前n个术语？

- AbtPst

27

最好不要覆盖 Python 的字符串数据类型 str。 - scottlittle

3个回答

13

使用稀疏矩阵本身的解决方案（不使用.toarray()）！

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english')
corpus = [
    'I would like to check this document',
    'How about one more document',
    'Aim is to capture the key words from the corpus',
    'frequency of words in a document is called term frequency'
]

X = tfidf.fit_transform(corpus)
feature_names = np.array(tfidf.get_feature_names())


new_doc = ['can key words in this new document be identified?',
           'idf is the inverse document frequency caculcated for each of the words']
responses = tfidf.transform(new_doc)


def get_top_tf_idf_words(response, top_n=2):
    sorted_nzs = np.argsort(response.data)[:-(top_n+1):-1]
    return feature_names[response.indices[sorted_nzs]]
  
print([get_top_tf_idf_words(response,2) for response in responses])

#[array(['key', 'words'], dtype='<U9'),
 array(['frequency', 'words'], dtype='<U9')]

- Venkatachalam

2

当我尝试将这些前n个单词作为我的tfidfvectorizer词汇表再次使用时，它也会返回重复的单词，并且抛出值错误，因为词汇表中存在重复单词。我该如何获取前n个唯一的单词？ - Akash Singh

有趣。我正在使用get_feature_names()来获取feature_names，因此get_top_tf_idf_words不应返回任何重复项。你能否发布一个新问题，并附上一个可重现的示例并标记我？ - Venkatachalam

3

这里是一个快速的代码实现：（`documents` 是一个列表）

def get_tfidf_top_features(documents,n_top=10):
  tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2,  stop_words='english')
  tfidf = tfidf_vectorizer.fit_transform(documents)
  importance = np.argsort(np.asarray(tfidf.sum(axis=0)).ravel())[::-1]
  tfidf_feature_names = np.array(tfidf_vectorizer.get_feature_names())
  return tfidf_feature_names[importance[:n_top]]

- Sahar Millis

2

第二行有一个错别字。第一个字符“t”丢失了。 - aoez

no_features 是缺失的变量。 - SriK

感谢您的笔记。已修复。 - Sahar Millis

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- hume · Accepted Answer

你需要稍微花点心思，以将矩阵转化为numpy数组，但这应该能满足你的需求：

feature_array = np.array(tfidf.get_feature_names())
tfidf_sorting = np.argsort(response.toarray()).flatten()[::-1]

n = 3
top_n = feature_array[tfidf_sorting][:n]

这给了我：

array([u'fruit', u'travellers', u'jupiter'], 
  dtype='<U13')

argsort调用真的很有用，这里是它的文档。我们必须做[::-1]，因为argsort只支持从小到大排序。我们调用flatten将维数降至1d，以便使用排序后的索引来索引1d特征数组。请注意，只有在测试一个文档时才包括对flatten的调用才有效。

另外，关于另一件事，你是否想表达类似于tfs = tfidf.fit_transform(t.split("\n\n"))？否则，多行字符串中的每个术语都被视为“文档”。改用\n\n实际上意味着我们正在查看4个文档（每行一个），这在考虑tfidf时更有意义。