Sparkのインストールにハマる[3]
無理やりインストールしてあるバージョンのCDHとは違うバージョンを指定してみました.
$ time SPARK_HADOOP_VERSION=2.3.0-cdh5.0.0 SPARK_YARN=true sbt/sbt assembly Launching sbt from sbt/sbt-launch-0.12.4.jar [info] Loading project definition from /usr/local/spark-0.9.1-bin-hadoop2/project/project [info] Loading project definition from /usr/local/spark-0.9.1-bin-hadoop2/project (中略) [error] /usr/local/spark-0.9.1-bin-hadoop2/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:669: type mismatch; [error] found : org.apache.spark.streaming.dstream.DStream[(K, (com.google.common.base.Optional[V], W))] [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V], W)] [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type [error] joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v), w)} [error] ^ [error] 44 errors found [warn] two warnings found [error] (streaming/compile:compile) Compilation failed [error] Total time: 147 s, completed May 11, 2014 8:47:23 PM
もっと酷いことになりました...