我正在使用jupyter笔记本尝试spark。
一旦在我的笔记本中,我尝试了Kmean:
from pyspark.ml.clustering import KMeans
from sklearn import datasets
import pandas as pd
spark = SparkSession\
.builder\
.appName("PythonKMeansExample")\
.getOrCreate()
iris = datasets.load_iris()
pd_df = pd.DataFrame(iris['data'])
spark_df = spark.createDataFrame(pd_df, ["features"])
estimator = KMeans(k=3, seed=1)
一切都很顺利,然后我拟合模型:
estimator.fit(spark_df)
我遇到了一个错误:
16/08/16 22:39:58 ERROR Executor: Exception in task 0.2 in stage 0.0 (TID 24)
java.io.IOException: Cannot run program "jupyter": error=2, No such file or directory
Caused by: java.io.IOException: error=2, No such file or directory
Spark在哪里寻找Jupyter?如果我可以使用Jupyter Notebook,为什么它找不到它?应该怎么办?
export PYSPARK_PYTHON
无效,有什么解决办法吗? - rosefun