Hadoop Nativeにハマる

まず，ライブラリを入れているはずなのに動かない理由が1つはすぐ分かり．

$ file /usr/local/hadoop/lib/native/Linux-amd64-64/lib/libhadoop.so.1.0.0 
/usr/local/hadoop/lib/native/Linux-amd64-64/lib/libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped

そこでHadoopのソースを取ってきて

$ mvn compile -Pnative -DskipTests -e

(snip)

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.137s
[INFO] Finished at: Fri Aug 02 13:29:50 JST 2013
[INFO] Final Memory: 26M/340M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.0.5-alpha:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: proto
c failure -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.0.5-alpha:protoc (compile-protoc) on project hadoop-common: org.a
pache.maven.plugin.MojoExecutionException: protoc failure

プロトコルバッファを入れ忘れていたようです．

$ wget http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
$ tar xvf protobuf-2.5.0.tar.gz
$ cd protobuf-2.5.0
$ ./configure
$ make
$ make install
$ cd java
$ mvn compile
$ mvn install
$ mvn install:install-file -Dpackaging=jar -DgeneratePom=true -DgroupId=com.google.protobuf -DartifactId=protobuf-java -Dfile=protobuf-java-2.5.0.jar -Dversion=2.5.0

しかしエラーが出続ける．．．そこでmvn -Xでデバッグログを見ると，protocol bufferの2.4.0aというのをわざわざダウンロードしていました．
そこで，hadoop-2.0.5-alpha-src/hadoop-common-project/hadoop-common/pom.xmlに

    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-java</artifactId>
      <version>2.5.0</version>
      <scope>compile</scope>
    </dependency>

バージョンを指定してやると，

$ mvn compile -Pnative -DskipTests -e

(snip)

     [exec] Linking C shared library target/usr/local/lib/libhadoop.so
     [exec] [ 47%] Built target hadoop
     [exec] [ 94%] Built target hadoop_static
     [exec] [100%] Built target test_bulk_crc32
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
$ ls -l target/native/target/usr/local/lib/libhadoop.*
-rw-r--r-- 1 users 653130 Aug  2 14:54 target/native/target/usr/local/lib/libhadoop.a
lrwxrwxrwx 1 users     18 Aug  2 14:54 target/native/target/usr/local/lib/libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x 1 users 359917 Aug  2 14:54 target/native/target/usr/local/lib/libhadoop.so.1.0.0
$ cp -fpvl target/native/target/usr/local/lib/libhadoop.* /usr/local/hadoop/lib/native/Linux-amd64-64/lib/

良い感じにできました．