Incubator-mxnet: Scala/Java Library for mxnet

[terrytangyuan]

Anyone interested? I'll jump in to help whenever I have time.

[piiswrong]

Hi @terrytangyuan
We don't have anyone working on that yet. You are well come to open a repo!

[terrytangyuan]

Could @tqchen or anyone else create a repo for mxnet.scala for me? It says I have insufficient permission. We'll go from there.

[piiswrong]

They are probably off line at the moment. You can create your own repo and transfer it's ownership later.

[tqchen]

@yanqingmen seems has already have some progress on JNI, maybe you can sync up on this.

Let us consider directly put things in main repo when it is ready, as xgboost did. The reason for mxnet.js and mxnet.jl was mainly the for the convention of packaging and requirement on hosting additional big files that do not suitable for the main code repo.

[terrytangyuan]

Ok sounds good. Please share some updates on that.

[tqchen]

@yanqingmen if you have time maybe comment on this issue?

[yzhliu]

Hi all,

I've done some beginning work here https://github.com/javelinjs/mxnet-scala

Any suggestion or help is highly appreciated.

[terrytangyuan]

@javelinjs Cool. Could you grant me write access so I can jump in to help? Also, are we able to move your lib under the main mxnet repo as @tqchen mentioned? Are you and @yanqingmen working on different libraries and whice one should we stick with?

[yzhliu]

@terrytangyuan Thank you for your interest. Since it is still a fledging project and the code base may change rapidly, I suggest to follow the fork & pull-request process at this moment. I didn't know @yanqingmen was also working on the JNI package when I started. @yanqingmen Would you mind sync your progress here?

[yzhliu]

Added @terrytangyuan to collaborators :)

[ieee8023]

+1 for Java

[yanqingmen]

hi, very glad to see this issue, I‘ve done some beginning work for jni package, and would be glad to help.since I don't have too much jni program experience, I've done it in a more naive way than @javelinjs. Maybe we could have a discuss group~

[yanqingmen]

https://github.com/yanqingmen/mxnet/tree/java_wrapper this is my fork:)

[yzhliu]

Hi @yanqingmen Great to see your work! I quickly browsed your code base and found you've already finished the NDArray and IO part, and you really did a nice work. From my side, I just finished the NDArray class (with a few test cases). And the automatic JNI compiling settings in maven seems to work well. Are you interested in Scala? It writes more naturally, (e.g. we can write (ndarray + 1.0) or (1.0 + ndarray) directly in scala code), and can also provide java-friendly interfaces. @terrytangyuan has created a gitter here https://gitter.im/javelinjs/mxnet-scala , would you like to join to have a discussion?

[yanqingmen]

great, I'll join you

[terrytangyuan]

@yanqingmen @javelinjs The repo has been moved to https://github.com/terrytangyuan/mxnet
cc: @tqchen

We can discuss issues from the Gitter in mxnet-scala folder.

[yzhliu]

@tqchen As you mentioned here https://github.com/javelinjs/mxnet-scala/issues/2 , Did you mean to keep mxnet-scala as an independent project, or make it as a folder in mxnet as @terrytangyuan suggested?

[tqchen]

both works for me, I think making it officially part as mxnet can make the integration more up to date easily

[terrytangyuan]

Yeah it would be easier to with with Travis for example. We have both just agreed to work on https://github.com/terrytangyuan/mxnet and later merge to dmlc/mxnet.

[yzhliu]

@tqchen Thank you, we moves to https://github.com/terrytangyuan/mxnet/scala-package
@yanqingmen I sent an email to you, please check your mail box, thank you :)

[yzhliu]

Correct url https://github.com/terrytangyuan/mxnet/tree/master/scala-package

[saudet]

Hi, just FYI, it shouldn't be too hard to create full bindings using JavaCPP. Check the ones for Caffe and TensorFlow, for example:
https://github.com/bytedeco/javacpp-presets/tree/master/caffe
https://github.com/bytedeco/javacpp-presets/tree/master/tensorflow
Let me know if interested!

[yzhliu]

Hi @saudet Your project is awesome. We'll check it later to see if we can learn something from javacpp-presets. I think it'll be a great tool if we need full bindings some day.

[piiswrong]

@tqchen I vote for independent repo for java binding. The integrated R package is already making travis test very slow. If we put more language binding and tests into the main repo integration tests is going to take forever.

[yzhliu]

@piiswrong I think what tianqi considered is to keep the contribution history. I wonder If we transfer it's ownership later to dmlc, will all these commit history also be transferred?

[terrytangyuan]

Maybe we should try to speed up Travis tests instead. For example:

1. only run Scala tests when there are files changed in Scala directory.
2. run non-vital tests only once a while on a regular basis, e.g. lint tests
3. for major developers, I recommend to activate their Travis for the forked repo.

[tqchen]

Let us keep things in one repo, so contribution to the scala package also seen as major contribution to mxnet. We have done this for matlab, python and R and it works well so far.

@piiswrong The major problem on R tests is the cost to install R env, instead of running the test. For java and others, it is likely that we can do it in the docker images and things should be faster

[saudet]

@tqchen Just for my information, I can't seem to find where the the CI is done for Windows. Do you have another build server for that?

[tqchen]

We don't yet have a CI for windows setup.

[saudet]

@tqchen I see, that's one of the major problems I'm having with JavaCPP. People who can't do builds by themselves are mostly the Windows users...

[terrytangyuan]

@piiswrong @tqchen There are only a few R tests currently so it shouldn't take long for the unit tests to run. It only took several minutes to install those R dependencies and tools when I was setting it up long time ago but it looks like it's in a very different shape now. I will take a look to try to speed it up once we are done with the Scala package.

[javadba]

@saudet How are the javacpp-presets for mxnet going? I have joined the sparknet project github.com/amplab/SparkNet and we want to add mxnet bindings . Presently they have only caffe.

[yzhliu]

Hi @javadba , the scala package for mxnet will soon be released. And we also have plan to integrate it with spark. If I didn't misunderstand, Caffe/Tensorflow in SparkNet follows the distributed weight calculation - average workflow, i.e. the way described in SparkNet paper, instead of a parameter server strategy, right?

[javadba]

I am new to the SparkNet project but believe what you were saying to be correct.

[javadba]

So what does that mean wrt javacpp-presets for mxnet?

[yzhliu]

You're free to use javacpp-presets in SparkNet if it contains mxnet bindings. But I doubt whether the synchronized training strategy is a right way for deep networks. We'll discuss it later.
And you are so welcomed to help us test the scala package https://github.com/javelinjs/mxnet/tree/scala-package-l, if you would like to :)

[javadba]

@javelinjs My team is investigating approaches to distributed gpu computations especially for CNN. We will be using Spark for the resource scheduling. Beyond that we are open to considering different approaches . The parameter server approach has been considered as well. Your input would be taken into account.

[saudet]

@javadba The C API is very simple, so I could provide presets for that right away if that's enough. The C++ API would take a bit more time, but not that much...

[javadba]

OK. Re: Spark with mxnet. My takeaway is your preferred approach is to use a Parameter Server vs the synchronized training (the SparkNet approach).

There might be two solutions here:
(a) Write your own Spark interface
(b) Collaborate with SparkNet to enhance their platform to support a Parameter Server

Have you also considered the (b) approach? If so why was it not chosen?

Thanks!

[javadba]

Re: the C API Presets: they would be helpful yes! thanks. I am new to this and would not be able to answer if C++ were required though I would venture a guess that no.

[yzhliu]

@javadba Since SparkNet, according to its published paper, is a synchronized approach, it seems that ps cannot take advantage of any exist interface in SparkNet. But I'm still open to any possible choice. Also correct me if I'm wrong.

[javadba]

Hi Yizhi, thanks for the response. I am new to both projects and am
learning myself what are their characteristics. My team is looking for a
distributed gpu support for deep learning - and both mxnet and SparkNet
seem to have pieces of the answer that may be useful. We will look more
into/understand better the Parameter Server vs Synchronous approaches to
be more helpful in these disucssions.
Thanks.

2016-02-17 8:45 GMT-08:00 Yizhi Liu [email protected]:

@javadba https://github.com/javadba Since SparkNet, according to its
published paper, is a synchronized approach, it seems that ps cannot take
advantage of any exist interface in SparkNet. But I'm still open to any
possible approach. Also correct me if I'm wrong.

—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/842#issuecomment-185292076.

[tqchen]

I think we are open for the options in terms of integrations. MXNet already support YARN and I think spark integration is one the plan.

One key difference between the project you mentioned here is. While caffe style model follows a config and run strategy, which is fine for pre-defined CNN. MXNet focuses on maximum customization via hybrid declarative and imperative API.

Here are some examples I believe which is easier in mxnet:

• Use scala to define customized optimization algorithm that runs on GPU
• Fully take advantage of the scala NDArray API, so you can do other strange stuffs besides data parallel training of a model.

This being said, the goal of mxnet is being portable, this should include being able to easily ported to other projects, as well as being able to port to major platforms. We have done this for MPI, YARN and I suppose spark or even flink is another natural option

[javadba]

Thanks Tianqui for that insight. MxNet sounds like a great project! We
are interested in the dynamic / runtime / flexible algorithmic declaration
approach that you mention. So we will be interested in these developments
going forward.

2016-02-17 9:46 GMT-08:00 Tianqi Chen [email protected]:

I think we are open for the options in terms of integrations. MXNet
already support YARN and I think spark integration is one the plan.

One key difference between the project you mentioned here is. While caffe
style model follows a config and run strategy, which is fine for
pre-defined CNN. MXNet focuses on maximum customization via hybrid
declarative and imperative API.

Here are some examples I believe which is easier in mxnet:

• Use scala to define customized optimization algorithm that runs on
  GPU
• Fully take advantage of the scala NDArray API, so you can do other
  strange stuffs besides data parallel training of a model.

This being said, the goal of mxnet is being portable, this should include
being able to easily ported to other projects, as well as being able to
port to major platforms. We have done this for MPI, YARN and I suppose
spark or even flink is another natural option

—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/842#issuecomment-185321990.

[yzhliu]

Totally agreed with @tqchen . I hope scala package can make MXNet more portable to other JVM platforms.

[Yunrui]

Is there any plan to integrate scale-package to mxnet project? Can I ask when will it happen?

[yzhliu]

@Yunrui Yes, I will make a PR within two weeks. We are doing more tests for models on multiple devices. Thank you for paying close attention to this project.

[Yunrui]

@javelinjs thanks, looking forward to integrating this feature to my pipeline

[saudet]

@javadba I've finally been able to take a few hours to work on a wrapper for the C API along with some sample code: https://github.com/bytedeco/javacpp-presets/tree/master/mxnet

[javadba]

Thanks! We will be looking at it probably in mid March- had a bit of
rearrangement of tasks.

2016-02-28 7:28 GMT-08:00 Samuel Audet [email protected]:

@javadba https://github.com/javadba I've finally been able to take a
few hours to work on a wrapper for the C API along with some sample code:
https://github.com/bytedeco/javacpp-presets/tree/master/mxnet

—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/842#issuecomment-189891997.

[liuagen]

Hi, due to sync difference bewteen parameter server and Spark, how to integrate scala package with Spark? Thanks!

[yzhliu]

@kelly-lxg Good question. It needs some hack efforts to make mxnet run in one spark stage. If you have any idea, feel free to let us know. @CodingCat and @tqchen are doing similar things with xgboost4j, they have experiences on this.

[liuagen]

We may don't agree the spark interface plan which sparknet take, because it may allocate memory twice both in c++ and jvm. Do you have some feasible way? thanks!

[CodingCat]

@javelinjs @kelly-lxg ,

for making mxnet run in one stage, you have to call mxnet within a spark task and block the thread until the training done....e.g. trainingData.mapPartition( partition => mxnet.train ...).foreachPartition(() => _) (each partition maps to a task )

for memory cost, I think we have to accept the overhead in off-heap memory part, unless you rely on 3rd-party solutions to share the memory space between JVM and c++ process (e.g. Apache Arrow

[liuagen]

I've got your idea and I will try,Thanks.