Hadoop Nativeにハマる
まず,ライブラリを入れているはずなのに動かない理由が1つはすぐ分かり.
$ file /usr/local/hadoop/lib/native/Linux-amd64-64/lib/libhadoop.so.1.0.0 /usr/local/hadoop/lib/native/Linux-amd64-64/lib/libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
そこでHadoopのソースを取ってきて
$ mvn compile -Pnative -DskipTests -e (snip) [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.137s [INFO] Finished at: Fri Aug 02 13:29:50 JST 2013 [INFO] Final Memory: 26M/340M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.0.5-alpha:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: proto c failure -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.0.5-alpha:protoc (compile-protoc) on project hadoop-common: org.a pache.maven.plugin.MojoExecutionException: protoc failure
プロトコルバッファを入れ忘れていたようです.
$ wget http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz $ tar xvf protobuf-2.5.0.tar.gz $ cd protobuf-2.5.0 $ ./configure $ make $ make install $ cd java $ mvn compile $ mvn install $ mvn install:install-file -Dpackaging=jar -DgeneratePom=true -DgroupId=com.google.protobuf -DartifactId=protobuf-java -Dfile=protobuf-java-2.5.0.jar -Dversion=2.5.0
しかしエラーが出続ける...そこでmvn -Xでデバッグログを見ると,protocol bufferの2.4.0aというのをわざわざダウンロードしていました.
そこで,hadoop-2.0.5-alpha-src/hadoop-common-project/hadoop-common/pom.xmlに
<dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> <version>2.5.0</version> <scope>compile</scope> </dependency>
バージョンを指定してやると,
$ mvn compile -Pnative -DskipTests -e (snip) [exec] Linking C shared library target/usr/local/lib/libhadoop.so [exec] [ 47%] Built target hadoop [exec] [ 94%] Built target hadoop_static [exec] [100%] Built target test_bulk_crc32 [INFO] Executed tasks [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ $ ls -l target/native/target/usr/local/lib/libhadoop.* -rw-r--r-- 1 users 653130 Aug 2 14:54 target/native/target/usr/local/lib/libhadoop.a lrwxrwxrwx 1 users 18 Aug 2 14:54 target/native/target/usr/local/lib/libhadoop.so -> libhadoop.so.1.0.0 -rwxr-xr-x 1 users 359917 Aug 2 14:54 target/native/target/usr/local/lib/libhadoop.so.1.0.0 $ cp -fpvl target/native/target/usr/local/lib/libhadoop.* /usr/local/hadoop/lib/native/Linux-amd64-64/lib/
良い感じにできました.