我不太理解以下程序中如何得到Tf-Idf值:
我尝试使用网站上给出的概念计算文档2(“And_this_is_the_third_one.”)中a
的值,但是使用上述概念得到的'a'的值为
1/26*log(4/1)
((字符'a'出现次数)/(给定文档中的字符总数)*log( # 所有文档 / # 包含该字符的文档数))
= 0.023156
但是输出结果为0.2203,如输出所示。
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ['This_is_the_first_document.', 'This_document_is_the_second_document.', 'And_this_is_the_third_one.', 'Is_this_the_first_document?', ]
vectorizer = TfidfVectorizer(min_df=0.0, analyzer="char")
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())
print(vectorizer.vocabulary_)
m = X.todense()
print(m)
我预期根据上述概念输出结果为0.023156。
输出结果为:
['.', '?', '_', 'a', 'c', 'd', 'e', 'f', 'h', 'i', 'm', 'n', 'o', 'r', 's', 't', 'u']
{'t': 15, 'h': 8, 'i': 9, 's': 14, '_': 2, 'e': 6, 'f': 7, 'r': 13, 'd': 5, 'o': 12, 'c': 4, 'u': 16, 'm': 10, 'n': 11, '.': 0, 'a': 3, '?': 1}
[[0.14540332 0. 0.47550697 0. 0.14540332 0.11887674
0.23775349 0.17960203 0.23775349 0.35663023 0.14540332 0.11887674
0.11887674 0.14540332 0.35663023 0.47550697 0.14540332]
[0.10814145 0. 0.44206359 0. 0.32442434 0.26523816
0.35365088 0. 0.17682544 0.17682544 0.21628289 0.26523816
0.26523816 0. 0.26523816 0.35365088 0.21628289]
[0.14061506 0. 0.57481012 0.22030066 0. 0.22992405
0.22992405 0. 0.34488607 0.34488607 0. 0.22992405
0.11496202 0.14061506 0.22992405 0.34488607 0. ]
[0. 0.2243785 0.46836004 0. 0.14321789 0.11709001
0.23418002 0.17690259 0.23418002 0.35127003 0.14321789 0.11709001
0.11709001 0.14321789 0.35127003 0.46836004 0.14321789]]