使用kmeans（sklearn）对新文本进行预测？

Question

我有一个非常小的短字符串列表，我想要(1)对它们进行聚类，(2)使用该模型预测新字符串属于哪个聚类。

第一部分可以正常运行，但是对新字符串进行预测的过程中出现问题。

第一部分

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

# List of 
documents_lst = ['a small, narrow river',
                'a continuous flow of liquid, air, or gas',
                'a continuous flow of data or instructions, typically one having a constant or predictable rate.',
                'a group in which schoolchildren of the same age and ability are taught',
                '(of liquid, air, gas, etc.) run or flow in a continuous current in a specified direction',
                'transmit or receive (data, especially video and audio material) over the Internet as a steady, continuous flow.',
                'put (schoolchildren) in groups of the same age and ability to be taught together',
                'a natural body of running water flowing on or under the earth']


# 1. Vectorize the text
tfidf_vectorizer  = TfidfVectorizer(stop_words='english')
tfidf_matrix = tfidf_vectorizer.fit_transform(documents_lst)
print('tfidf_matrix.shape: ', tfidf_matrix.shape)

# 2. Get the number of clusters to make .. (find a better way than random)
num_clusters = 3

# 3. Cluster the defintions
km = KMeans(n_clusters=num_clusters, init='k-means++').fit(tfidf_matrix)

clusters = km.labels_.tolist()

print(clusters)

这会返回:

tfidf_matrix.shape:  (8, 39)
[0, 1, 0, 2, 1, 0, 2, 0]

第二部分

出现问题的部分：

predict_doc = ['A stream is a body of water with a current, confined within a bed and banks.']

tfidf_vectorizer  = TfidfVectorizer(stop_words='english')
tfidf_matrix = tfidf_vectorizer.fit_transform(predict_doc)
print('tfidf_matrix.shape: ', tfidf_matrix.shape)

km.predict(tfidf_matrix)

错误：

ValueError: Incorrect number of features. Got 7 features, expected 39

顺便说一句：我有点理解，在向量化后，训练和预测具有不同数量的特征...

我可以接受任何解决方案，包括从kmeans更改到更适合短文本聚类的算法。

提前感谢。

- Itay Livni

---

你正在为预测任务训练一个新的tfidfvectorizer。因此，它会生成不同的特征集。你应该使用原始的tfidf_vectorizer。 - Vivek Kumar

@VivekKumar 当然可以 :). 呃，那很合理。[https://dev59.com/6IXca4cB1Zd3GeqPECMd#26943563] - Itay Livni

Answer 1

为了完整性，我将回答自己的问题，并提供来自这里的答案，该答案并没有回答那个问题。但是回答了我的问题

from sklearn.cluster import KMeans

list1 = ["My name is xyz", "My name is pqr", "I work in abc"]
list2 = ["My name is xyz", "I work in abc"]

vectorizer = TfidfVectorizer(min_df = 0, max_df=0.5, stop_words = "english", charset_error = "ignore", ngram_range = (1,3))
vec = vectorizer.fit(list1)   # train vec using list1
vectorized = vec.transform(list1)   # transform list1 using vec

km = KMeans(n_clusters=2, init='k-means++', n_init=10, max_iter=1000, tol=0.0001, precompute_distances=True, verbose=0, random_state=None, n_jobs=1)

km.fit(vectorized)
list2Vec = vec.transform(list2)  # transform list2 using vec
km.predict(list2Vec)

荣誉归功于@IrshadBhat。

- Itay Livni