我按照此链接进行操作,以创建用于IPython的PySpark配置文件。
00-pyspark-setup.py
# Configure the necessary Spark environment
import os
import sys
spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, spark_home + "\python")
# Add the py4j to the path.
# You may need to change the version number to match your install
sys.path.insert(0, os.path.join(spark_home, '\python\lib\py4j-0.8.2.1-src.zip'))
# Initialize PySpark to predefine the SparkContext variable 'sc'
execfile(os.path.join(spark_home, '\python\pyspark\shell.py'))
当我在ipython-notebook中键入
sc
时,遇到了问题,输出为空字符串''
,实际上应该输出类似于<pyspark.context.SparkContext at 0x1097e8e90>
的结果。你有什么解决办法吗?