Incubator-mxnet: clojure: cross-platform unique library dependency

[leontalbot]

I am building a clojure library. I'd like to use MxNet in it.

I found these different dependencies I could use:
[org.apache.mxnet.contrib.clojure/clojure-mxnet-osx-cpu "1.3.1"]
[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-cpu "1.3.1"]
[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-gpu "1.3.1"]

Say I want to build a Clojure library that needs to use Clojure MXNet
I cannot tell if someone who will be using my library will be on Windows, Mac or Linux.

How should I handle this?
I'd like to avoid packaging the native files up on one jar myself and having it choose the right one
https://github.com/swannodette/native-deps

This idea was added here: https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs

Would it be a good idea to have a unique cross-platform jar as dependency?

[gigasquid]

Thanks for reporting the issue @leontalbot 😸

@nswamy @lanking520 @andrewfayres @frankfliu

I'm wondering if it would be possible to have another "single" artifact for scala that was packaged for both Linux and OSX (and eventually windows). It would help cross platform portability and adoption and enable others to build on top of the mxnet and include it as a dependency for all systems.

One problem is, I don't know how the gpu/cpu would work though....

Here is a possible maven plugin https://github.com/maven-nar/nar-maven-plugin.

Thoughts?

[lanking520]

@gigasquid There are three issues inside:

• GPU and CPU are not compiled as the same types, GPU operators are compiled to .cu for CUDA acceleration. We cannot have a universal types in here
• OSX likes the target to dylib instead of so file. We cannot merge these too. Same with Windows likes dll.
• If we just put all files together, it will be large (Around 500 (GPU) + 60 (CPU) + 60 (OSX) = 620).

[leontalbot]

Thank you for taking a look @gigasquid @lanking520 ! Was wondering, globally, would a single artefact take much more disk space then platform specific ones? Is that an issue?

[piyushghai]

Theoretically, a single global artefact would in fact take a lot more disk space as well, which might or might not be an issue given the standards of modern day computing.

But, @leontalbot just @lanking520 mentioned, the issue is not about the space for the artefact, rather the platform specific dependencies that we need to bundle into the jars. There are technical limitations to doing so.

In Scala, when we are not sure whether our end users will be using Ubuntu CPU/GPU or MacOS, we add these profiles in pom.xml and let Maven pick it up for us.

eg snippet taken from here : https://github.com/apache/incubator-mxnet/blob/master/scala-package/mxnet-demo/java-demo/pom.xml

    <profile>
          <id>osx-x86_64</id>
          <activation>
            <os>
              <family>mac</family>
            </os>
          </activation>
          <properties>
            <mxnet.profile>osx-x86_64-cpu</mxnet.profile>
          </properties>
        </profile>
        <profile>
          <id>linux-x86_64</id>
          <activation>
          <os>
            <family>unix</family>
            <name>Linux</name>
          </os>
          </activation>
          <properties>
            <mxnet.profile>linux-x86_64-cpu</mxnet.profile>
          </properties>
        </profile>

Hope that answers your question :)
@lanking520 I suggest this issue to be closed.

@leontalbot Please feel free to re-open if closed in error.

[gigasquid]

Thanks @piyushghai and @lanking520 - I think we could do something similar with Leiningen https://stackoverflow.com/questions/4688336/what-is-an-elegant-way-to-set-up-a-leiningen-project-that-requires-different-dep

[piyushghai]

Thanks @piyushghai and @lanking520 - I think we could do something similar with Leiningen https://stackoverflow.com/questions/4688336/what-is-an-elegant-way-to-set-up-a-leiningen-project-that-requires-different-dep

This seems a good approach to me, as we'll be abstracting away the complexity of choosing which jar/artifact from the user.

[piyushghai]

I created a JIRA ticket to track this request in our backlog :
https://issues.apache.org/jira/browse/MXNET-1319

[lanking520]

@piyushghai Thanks for filing this as an item in backlog. Close this issue for now. @leontalbot Please feel free to reopen it if you believe your problems is unsolved.

[leontalbot]

Really nice, thanks everyone!