在使用Phoenix 4.5在CDH 5.4上运行Spark作业时，出现了PhoenixOutputFormat未找到的问题。

Question

我通过重新编译源代码成功地配置了Phoenix 4.5在Cloudera CDH 5.4上。 sqlline.py 运行良好，但与spark存在问题。

spark-submit --class my.JobRunner \
--master yarn --deploy-mode client \
--jars `ls -dm /myapp/lib/* | tr -d ' \r\n'` \
/myapp/mainjar.jar

/myapp/lib 文件夹包含 Phoenix 核心库，其中包含 org.apache.phoenix.mapreduce.PhoenixOutputFormat 类。但似乎驱动程序/执行器无法看到它。

Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:971)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:903)
    at org.apache.phoenix.spark.ProductRDDFunctions.saveToPhoenix(ProductRDDFunctions.scala:51)
    at com.mypackage.save(DAOImpl.scala:41)
    at com.mypackage.ProtoStreamingJob.execute(ProtoStreamingJob.scala:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.mypackage.SparkApplication.sparkRun(SparkApplication.scala:95)
    at com.mypackage.SparkApplication$delayedInit$body.apply(SparkApplication.scala:112)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:71)
    at scala.App$$anonfun$main$1.apply(App.scala:71)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
    at scala.App$class.main(App.scala:71)
    at com.mypackage.SparkApplication.main(SparkApplication.scala:15)
    at com.mypackage.ProtoStreamingJobRunner.main(ProtoStreamingJob.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
    ... 30 more

我该怎么做才能克服这个异常？

- Nicola Ferraro

---

你是如何准确地构建mainjar.jar的？ - Gillespie

你比 Stack Overflow 上的任何人都更有资格回答这个问题。检查你的参数：最终会出现什么在 --jar 中？检查 JAR 文件：它们是否包含这个类？使用 rdd.foreach 在执行器上打印类路径。它是否包含该 JAR 文件？这个问题需要调试，赏金也无法帮助你。 - Daniel Darabos

Answer 1

将phoenix-core添加到classpath.txt中即可解决该问题。该文件通常位于/ etc / spark / conf文件夹下。

- Hakan İlter