如何从Pipeline中提取词汇表

Question

我可以通过以下方式从CountVectorizerModel中提取词汇：

fl = StopWordsRemover(inputCol="words", outputCol="filtered")
df = fl.transform(df)
cv = CountVectorizer(inputCol="filtered", outputCol="rawFeatures")
model = cv.fit(df)

print(model.vocabulary)

以上代码将打印出带有索引id的词汇列表。

现在，我已经创建了上述代码的管道如下：

rm_stop_words = StopWordsRemover(inputCol="words", outputCol="filtered")
count_freq = CountVectorizer(inputCol=rm_stop_words.getOutputCol(), outputCol="rawFeatures")

pipeline = Pipeline(stages=[rm_stop_words, count_freq])
model = pipeline.fit(dfm)
df = model.transform(dfm)

print(model.vocabulary) # This won't work as it's not CountVectorizerModel

这将会抛出以下错误

print(len(model.vocabulary))

AttributeError: 'PipelineModel' object has no attribute 'vocabulary'

那么如何从管道中提取Model属性呢？

- user2377528

Answer 1

同样地，与任何其他阶段属性一样，提取stages：

stages = model.stages

找到你感兴趣的那个（们）：

from pyspark.ml.feature import CountVectorizerModel

vectorizers = [s for s in stages if isinstance(s, CountVectorizerModel)]

获取所需字段：

[v.vocabulary for v in vectorizers]

- zero323