Deeplearning4j: ND4J: CUDA + Spark dependency conflict

[AlexDBlack]

CUDA on Spark can do this:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang3/ThreadUtils
    at org.nd4j.jita.concurrency.CudaAffinityManager.getNextDevice(CudaAffinityManager.java:184)

Full stack trace:
https://gist.github.com/AlexDBlack/716a76014e3248e8142475f9b19a687b

ThreadUtils was added in commons lang 3 version 3.5.

Looking at dependency tree:
Spark 1.6.3 pulls in Hadoop 2.2.0 which pulls in commons-lang3 3.3.2
When we don't control the classpath (i.e., spark submit) we can get that exception.

Spark 2.4.2 uses commons-lang3 3.5, but that doesn't really matter as long as we still support Spark 1...

[saudet]

@jentfoo Any recommendations?

[saudet]

Looks like we only use findThreadById() from that though, doesn't look like a big issue...

[raver119]

It's pure cosmetic use, and in most of cases it'll be Thread.currentThread(). Fix merged.

[jentfoo]

Fix by @raver119 looks good to me

Do feel free to keep pinging me if there is anything you feel I can help with on the project :)

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.