CountVectorizer
支持部分拟合吗?
我想使用不同的数据批次训练CountVectorizer
。
CountVectorizer
支持部分拟合吗?
我想使用不同的数据批次训练CountVectorizer
。
def partial_fit(self , data):
if(hasattr(vectorizer , 'vocabulary_')):
vocab = self.vocabulary_
else:
vocab = {}
self.fit(data)
vocab = list(set(vocab.keys()).union(set(self.vocabulary_ )))
self.vocabulary_ = {vocab[i] : i for i in range(len(vocab))}
from sklearn.feature_extraction.text import CountVectorizer
CountVectorizer.partial_fit = partial_fit
vectorizer = CountVectorizer(stop_words=l)
vectorizer.fit(df[15].values[0:100])
vectorizer.partial_fit(df[15].values[100:200])
sajiad 的实现是正确的,我很感激他们分享他们的解决方案。通过修改对 hasattr()
的调用以引用 self
而不是 vectorizer
,可以使其更加灵活。
我使用了一个简短的可重现示例来说明 partial_fit()
与 fit()
的作用:
def partial_fit(self , data):
if(hasattr(self , 'vocabulary_')):
vocab = self.vocabulary_
else:
vocab = {}
self.fit(data)
vocab = list(set(vocab.keys()).union(set(self.vocabulary_ )))
self.vocabulary_ = {vocab[i] : i for i in range(len(vocab))}
from sklearn.feature_extraction.text import CountVectorizer
CountVectorizer.partial_fit = partial_fit
vectorizer = CountVectorizer()
corpus = ['The quick brown fox',
'jumps over the lazy dog']
# Without partial fit
for i in corpus:
vectorizer.fit([i])
print(vectorizer.get_feature_names())
# With partial fit
for i in corpus:
vectorizer.partial_fit([i])
print(vectorizer.get_feature_names())