我使用一个自定义分词方法的CountVectorizer。当我序列化它,然后再反序列化它时,我会得到以下错误信息:
AttributeError: module '__main__' has no attribute 'tokenize'
我该如何“序列化”
tokenize
方法?以下是一个小例子:
import nltk
from nltk.stem.snowball import FrenchStemmer
stemmer = FrenchStemmer()
def stem_tokens(tokens, stemmer):
stemmed = []
for item in tokens:
stemmed.append(stemmer.stem(item))
return stemmed
def tokenize(text):
tokens = nltk.word_tokenize(text)
stems = stem_tokens(tokens, stemmer)
return stems
tfidf_vec = TfidfVectorizer(tokenizer=tokenize, stop_words=stopwords.words('french'), ngram_range=(1,1))
clf = MLPClassifier(solver='lbfgs', alpha=0.02, hidden_layer_sizes=(400, 50))
pipeline = Pipeline([("tfidf", tfidf_vec),
("MLP", clf)])
joblib.dump(pipeline,"../models/classifier.pkl")