我正在使用sklearn pipeline构建一个机器学习流程。在预处理步骤中,我尝试对两个不同的字符串变量进行两种不同的处理:1)对BusinessType进行独热编码 2)对AreaCode进行均值编码,如下所示:
preprocesses_pipeline = make_pipeline (
FeatureUnion (transformer_list = [
("text_features1", make_pipeline(
FunctionTransformer(getBusinessTypeCol, validate=False), CustomOHE()
)),
("text_features2", make_pipeline(
FunctionTransformer(getAreaCodeCol, validate=False)
))
])
)
preprocesses_pipeline.fit_transform(trainDF[X_cols])
以下是定义为TransformerMixin类的内容:
class MeanEncoding(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
tmp = X['AreaCode1'].map(X.groupby('AreaCode1')['isFail'].mean())
return tmp.values
class CustomOHE(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
tmp = pd.get_dummies(X)
return tmp.values
并且 FunctionTransformer 函数返回所需的字段
def getBusinessTypeCol(df):
return df['BusinessType']
def getAreaCodeCol(df):
return df[['AreaCode1','isFail']]
现在当我运行上述管道时,它会生成以下错误
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-146-7f3a31a39c81> in <module>()
15 )
16
---> 17 preprocesses_pipeline.fit_transform(trainDF[X_cols])
~\Anaconda3\lib\site-packages\sklearn\pipeline.py in fit_transform(self, X, y, **fit_params)
281 Xt, fit_params = self._fit(X, y, **fit_params)
282 if hasattr(last_step, 'fit_transform'):
--> 283 return last_step.fit_transform(Xt, y, **fit_params)
284 elif last_step is None:
285 return Xt
~\Anaconda3\lib\site-packages\sklearn\pipeline.py in fit_transform(self, X, y, **fit_params)
747 Xs = sparse.hstack(Xs).tocsr()
748 else:
--> 749 Xs = np.hstack(Xs)
750 return Xs
751
~\Anaconda3\lib\site-packages\numpy\core\shape_base.py in hstack(tup)
286 return _nx.concatenate(arrs, 0)
287 else:
--> 288 return _nx.concatenate(arrs, 1)
289
290
ValueError: all the input arrays must have same number of dimensions
看起来在管道中有"MeanEncoding"的那一行出现了错误,因为将其删除后管道可以正常工作。不确定具体出了什么问题,需要帮助。