Druid: LzoCodec not found in index_hadoop task

[erankor]

Hi all,

I'm getting the following exception when submitting an index_hadoop task:

Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) ~[?:?]
    at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) ~[?:?]
    at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180) ~[?:?]
    at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:59) ~[?:?]
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:399) ~[?:?]
    at org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:115) ~[?:?]
    at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) ~[?:?]
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) ~[?:?]
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) ~[?:?]
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_131]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_131]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) ~[?:?]
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]
    at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:203) ~[druid-indexing-hadoop-0.10.0.jar:0.10.0]
    at io.druid.indexer.JobHelper.runJobs(JobHelper.java:349) ~[druid-indexing-hadoop-0.10.0.jar:0.10.0]
    at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:95) ~[druid-indexing-hadoop-0.10.0.jar:0.10.0]
    at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:276) ~[druid-indexing-service-0.10.0.jar:0.10.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
    at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:208) ~[druid-indexing-service-0.10.0.jar:0.10.0]

The task json is:

{
    "type": "index_hadoop",
    "spec": {
        "dataSchema": {
            ...
        },
        "ioConfig": {
            "type": "hadoop",
            "inputSpec": {
                "type": "static",
                "paths": "s3n://path-to-data-file.gz"
            }
        },
        "tuningConfig": {
            "type": "hadoop"
        }
    },
    "hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.7.3",
    "org.apache.hadoop:hadoop-aws:2.7.3"]
}

We're using Amazon EMR's Hadoop 2.7.3, and from what I see in core-site.xml & mapred-site.xml, it's configured with LZO:

  <property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
  </property>

  <property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
  </property>

Hadoop merge tasks (inputSpec->type = dataSource) are working properly in our setup.
Is there anything we need to install on the Middle Managers/Overlord to make it work, or is this error coming from the Hadoop cluster?

Thank you!

Eran

[erankor]

Ah, btw, at first I tried with "mapreduce.job.user.classpath.first": "true" in the tuningConfig, that didn't work as well

[b-slim]

was this working before or it is the first time you run into it ? seems like the jar is not in the class path of the container, am suspecting the class loader isolation is knocking it off the class path.

[erankor]

Thanks @b-slim, no, it wasn't working before, I'm just setting it up..
Any idea what to do about this?

[erankor]

Just to update this thread that I managed to solve it, pasting the commands that I used, in case it will be helpful to someone...

1. In Druid middlemgrs/overlord, ran:

## build libgplcompression
apt-get install gcc-multilib maven

# lzo
cd /opt
wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
tar -zxvf lzo-2.10.tar.gz
cd lzo-2.10/
./configure --enable-shared --prefix /usr/local/lzo-2.06
make
make install

# hadoop lzo
cd /opt
git clone https://github.com/twitter/hadoop-lzo/
cd hadoop-lzo/
C_INCLUDE_PATH=/usr/local/lzo-2.06/include LIBRARY_PATH=/usr/local/lzo-2.06/lib mvn clean package -Dmaven.test.skip=true

## pull hadoop dependencies
java -cp "/opt/druid-0.10.0/lib/*" -Ddruid.extensions.directory="/opt/druid-0.10.0/extensions" -Ddruid.extensions.hadoopDependenciesDir="/opt/druid-0.10.0/hadoop-dependencies" io.druid.cli.Main tools pull-deps --no-default-hadoop -r "http://maven.twttr.com/" -h "org.apache.hadoop:hadoop-client:2.7.3" -h "org.apache.hadoop:hadoop-aws:2.7.3" -h "com.hadoop.gplcompression:hadoop-lzo:0.4.20"

1. When running Druid as middlemgr / overlord, add LD_LIBRARY_PATH=/opt/hadoop-lzo/target/native/Linux-amd64-64/lib/ environment var
2. Set "hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.7.3", "org.apache.hadoop:hadoop-aws:2.7.3", "com.hadoop.gplcompression:hadoop-lzo:0.4.20"] in the task JSON
3. Set mapreduce.job.classloader=true in mapred-site.xml

[kalyanjanaki]

Hello erankor,

You said you are using EMR and telling "apt-get" Amazon linux uses YUM right ?

Regards
Kalyan

[erankor]

@kalyanjanaki, the commands above were not executed on EMR servers, they were executed on Druid servers, and we're installing Druid on Ubuntu

[kalyanjanaki]

Hello Erankor,

Thanks for your response. My questions might seem silly please bear with me.

So you have installed Druid on separate nodes than Hadoop nodes. In that case does Druid have Hadoop client installed with in its installation ? Do we have to setup password less sssh from Middlemanager servers to Our EMR cluster ? because for hadoop client application to work there should be password less ssh from client node to all hadoop nodes. This was not part of druid Documentation that was confusing to me.

Regards
Kalyan

[erankor]

No, Druid nodes don't need to ssh into Hadoop nodes, I think they communicate over port 8020

[kalyanjanaki]

Hello Erankor,

Thanks for your writeup. You made my day. I was able to fix issue with your steps. Many thanks.

Regards
Kalyan

[erankor]

Great! glad to hear :)