我是一个PySpark的新手。我在Windows 10上安装了Spark 2.3.0。我想使用带有交叉验证的线性SVM分类器进行训练,但数据集有3个类别。因此,我正在尝试从Spark ML中应用One vs Rest策略。但是,我的代码似乎有问题,因为我收到了一个错误,显示LinearSVC仅用于二元分类。
以下是在调试时执行“crossval.fit”行时出现的错误:
pyspark.sql.utils.IllegalArgumentException: u'requirement failed: LinearSVC only supports binary classification. 1 classes detected in LinearSVC_43a48b0b70d59a8cbdb1__labelCol'
以下是我的代码: (我只在一个非常小的数据集上尝试了10个实例)
from pyspark import SparkContext
sc = SparkContext('local', 'my app')
from pyspark.ml.linalg import Vectors
from pyspark import SQLContext
sqlContext = SQLContext(sc)
import numpy as np
x_train=np.array([[1,2,3],[5,6,7],[9,10,11],[2,4,5],[2,7,9],[3,7,6],[8,3,6],[5,8,2],[44,11,55],[77,33,22]])
y_train=[1,0,2,1,0,2,1,0,2,1]
#converting numpy array to dataframe
df_list = []
i = 0
for element in x_train: # row
tup = (y_train[i], Vectors.dense(element))
i = i + 1
df_list.append(tup)
Train_sparkframe = sqlContext.createDataFrame(df_list, schema=['label', 'features'])
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.ml.classification import OneVsRest
from pyspark.ml.classification import LinearSVC
LSVC = LinearSVC()
ovr = OneVsRest(classifier=LSVC)
paramGrid = ParamGridBuilder().addGrid(LSVC.maxIter, [10, 100]).addGrid(LSVC.regParam,
[0.001, 0.01, 1.0,10.0]).build()
crossval = CrossValidator(estimator=ovr,
estimatorParamMaps=paramGrid,
evaluator=MulticlassClassificationEvaluator(metricName="f1"),
numFolds=2)
cvModel = crossval.fit(Train_sparkframe)
bestModel = cvModel.bestModel