Incubator-mxnet: Running the mxnet scala jar in the interpreter throws an error

[aaronmarkham]

Description

If you try following the docs for Scala --> NDArray, it uses the interpreter. The latest jar doesn't work in the interpreter.

You can recreate this by running:

scala -classpath ../scala-package/assembly/linux-x86_64-cpu/target/mxnet-full_2.11-linux-x86_64-cpu-1.4.0-SNAPSHOT.jar 

Then paste in one of the examples:

import org.apache.mxnet._
// all-zero array of dimension 100x50
val a = NDArray.zeros(100, 50)
// all-one array of dimension 256x32x128x1
val b = NDArray.ones(256, 32, 128, 1)
// initialize array with contents, you can specify dimensions of array using Shape parameter while creating array.
val c = NDArray.array(Array(1, 2, 3, 4, 5, 6), shape = Shape(2, 3))

Errors

First one you see:

scala> import org.apache.mxnet._
import org.apache.mxnet._

scala> val x = NDArray.zeros(100, 200)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
java.lang.NoSuchMethodError: scala.Predef$.byteArrayOps([B)Lscala/collection/mutable/ArrayOps;
  at org.apache.mxnet.NDArrayInternal$$anonfun$12.apply(NDArray.scala:1189)

Then you also get:

scala> val a = NDArray.zeros(100, 50)
java.lang.NoSuchMethodError: scala.Predef$.byteArrayOps([B)Lscala/collection/mutable/ArrayOps;

[aaronmarkham]

@mxnet-label-bot add [scala]

[ddavydenko]

@mxnet-label-bot add [Bug]

[larroy]

Is this a bug? I think you should use -cp instead of -classpath with scala, some of the errors you mention is that scala can't find the scala standard library. The error about the logging is fine. Could you try again with -cp and see if it works?

[aaronmarkham]

Same errors...
Also this one:

scala> val c = NDArray.array(Array(1, 2, 3, 4, 5, 6), shape = Shape(2, 3))
java.lang.NoSuchMethodError: scala.Predef$.byteArrayOps([B)Lscala/collection/mutable/ArrayOps;

[larroy]

I see, I think they are equivalent. That error means that it can't find the scala standard library. It could be something wrong with your scala instalation is not related to mxnet from what I see.

[aaronmarkham]

This probably needs to be reported as another issue, but I tried the scala setup steps to see if something was messed up with my install, and met with further errors.
First, the sample setup commands don't appear to work. I had to remove the parenthesis. I ran:

export SCALA_VERSION_PROFILE=2.11 SCALA_VERSION=2.11.8 MXNET_VERSION=1.3.0
export SCALA_PKG_PROFILE=
mvn package -Dmxnet.profile=$SCALA_PKG_PROFILE \
        -Dmxnet.scalaprofile=$SCALA_VERSION_PROFILE \
        -Dmxnet.version=$MXNET_VERSION \
        -Dscala.version=$SCALA_VERSION

Then it fails with:

[ERROR] Failed to execute goal on project mxnet-scala-demo_2.11: 
Could not resolve dependencies for project Demo:mxnet-scala-demo_2.11:pom:1.0-SNAPSHOT: 
Could not find artifact org.apache.mxnet:mxnet-full_2.11-:jar:1.3.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]

I tried this on my Mac and on Ubuntu. Same thing happens. I messed with the version too... I thought maybe 1.3.1 would work, but it does not.

[larroy]

Im trying to follow the instructions on setting up scala, they need some love... :/

[larroy]

Why are we packing the standard library in the scala Jar?

  1495 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/TreeCheckers$TreeChecker$precheck$$anonfun$mk$default$3$1$1.class
  1284 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/TypeStrings$$anonfun$7.class
  1435 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/TypeStrings$$anonfun$isAnonClass$1.class
  1955 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/Typers$Typer$$anonfun$46.class
  6087 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/Typers$Typer$$anonfun$checkEphemeral$1.class
  2966 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/Typers$Typer$$anonfun$checkMethodStructuralCompatible$1.class
  2440 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/Typers$Typer$$anonfun$scala$tools$nsc$typechecker$Typers$Typer$$addLocals$1$2.class
  2884 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/typechecker/Typers$Typer$$anonfun$typedDefDef$5$$anonfun$apply$24.class
   984 Fri Mar 04 14:16:12 CET 2016 scala/tools/nsc/util/Exceptional$.class
  1576 Fri Mar 04 14:16:12 CET 2016 scala/tools/reflect/FormatInterpolator$$typecreator1$1.class
  2977 Fri Mar 04 14:16:12 CET 2016 scala/tools/reflect/ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$compile$1.class
 10381 Thu Apr 30 13:51:56 CEST 2015 scala/xml/NamespaceBinding.class
  1063 Thu Apr 30 13:51:56 CEST 2015 scala/xml/Utility$$anonfun$getName$1.class
   949 Thu Apr 30 13:51:56 CEST 2015 scala/xml/parsing/ConsoleErrorHandler.class
  6033 Thu Apr 30 13:51:58 CEST 2015 scala/xml/pull/EvElemStart.class
pllarroy@186590d670bd:0:~/Downloads/kk$
pllarroy@186590d670bd:1:~/Downloads/kk$ jar tvf ../mxnet-full_2.11-linux-x86_64-cpu-1.3.1.jar |less

[larroy]

pllarroy@186590d670bd:0:~/Downloads$ scala -cp mxnet-full_2.11-linux-x86_64-cpu-1.3.1.jar
Welcome to Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.mxnet._
import org.apache.mxnet._

scala> val x = NDArray.zeros(100, 200)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
java.lang.UnsatisfiedLinkError: Couldn't find the resource libmxnet-scala.jnilib
  at org.apache.mxnet.util.NativeLibraryLoader$.loadLibrary(NativeLibraryLoader.scala:90)
  at org.apache.mxnet.Base$.<init>(Base.scala:70)
  at org.apache.mxnet.Base$.<clinit>(Base.scala)
  at org.apache.mxnet.NDArray$.initNDArrayModule(NDArray.scala:159)
  at org.apache.mxnet.NDArray$.<init>(NDArray.scala:39)
  at org.apache.mxnet.NDArray$.<clinit>(NDArray.scala)
  ... 33 elided

[andrewfayres]

I did some testing and it looks to me like this might be a problem that comes up depending upon your scala version. When I use version 2.11.8 the commands run correctly. When using 2.12.6 I get the errors that @aaronmarkham reported.

I'm not sure of the reason. Naively, I'd expect that the 2.12.6 scala binary would be backwards compatible with 2.11.x.

[larroy]

Works for me with 2.11.7 compiled from master. Still we are packing the standard library, this doesn't seem right:

$jar tvf ./scala-package/assembly/osx-x86_64-cpu/target/mxnet-full_2.11-osx-x86_64-cpu-1.4.0-SNAPSHOT.jar | less
[...]
  2249 Fri Mar 04 14:12:52 CET 2016 scala/collection/mutable/ArrayOps$ofBoolean$.class
  2241 Fri Mar 04 14:12:52 CET 2016 scala/collection/mutable/ArrayOps$ofDouble$.class
 45151 Fri Mar 04 14:12:52 CET 2016 scala/collection/mutable/ArrayOps$ofDouble.class
  2624 Fri Mar 04 14:12:52 CET 2016 scala/collection/mutable/ArrayOps$ofUnit$.class
 46362 Fri Mar 04 14:12:52 CET 2016 scala/collection/mutable/ArrayOps$ofUnit.class

 scala -cp mxnet-full_2.11-osx-x86_64-cpu-1.4.0-SNAPSHOT.jar
Welcome to Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181).
Type in expressions to have them evaluated.
Type :help for more information.

scala> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".pllarroy@186590d670bd:127:/tmp$ scala -cp mxnet-full_2.11-osx-x86_64-cpu-1.4.0-SNAPSHOT.jar
Welcome to Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.mxnet._
import org.apache.mxnet._

scala> val c = NDArray.array(Array(1, 2, 3, 4, 5, 6), shape = Shape(2, 3))
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
c: org.apache.mxnet.NDArray = org.apache.mxnet.NDArray@ac08bfda

scala> :quit

[larroy]

@nswamy @lanking520

[lanking520]

Could you please try again, are you still facing the same issue?

[frankfliu]

Scala doesn't support binary compatibility between major versions. It only promise source code level compatibility. Means if you want to run scala 2.12, you have to recompile our scala project.

[zachgk]

@aaronmarkham I'm not able to find the instructions involving the interpreter, otherwise I would try to clarify them. Since it seems to just be a problem with your installed scala version, do you think this issue is resolved?

As for the problem with the scala demo https://github.com/apache/incubator-mxnet/issues/13468#issuecomment-443889321, we improved upon that by using the nightly snapshot repository so it should be much simpler now.

[larroy]

@frankfliu doesn't many projects release in maven with different scala versions? https://stackoverflow.com/questions/13872226/how-to-support-multiple-scala-versions-in-a-library/13880582#13880582

[zachgk]

@larroy What I see the most is that they will release different artifacts for each supported Scala version and append _scalaVersion to the artifact name (see https://mvnrepository.com/artifact/org.scalacheck/scalacheck or https://search.maven.org/search?q=g:org.scalatest for an example).

We do need to take a look at our Scala versions. We currently publish 2.11.8 when we really should be producing both 2.11.12 and 2.12.8. We also publish an artifact that does not indicate the scala version: org.apache.mxnet:mxnet-full_2.11-linux-x86_64-gpu (it indicates the os instead and we have linux-cpu, linux-gpu, and osx).

[zachgk]

I added interpreter instructions in #14169 to avoid this problem in the future. In addition, the scala standard library was removed from the final jar as part of #13626 and fixing the released versions was added to the JIRA backlog as [MXNET-1230]

[larroy]

Thanks @zachgk @lanking520 and others. The assembly & core jar artifacts look much better now. This is certainly an improvement.

I think linking with OpenCV statically would help, in Ubuntu 18.04 for example there's opencv 3.2 but not OpenCV 2.4 with results in being unable to load the library:

piotr@storage:130:~/devel/mxnet/scala-package (feature_discovery)+$ scala -cp ./assembly/target/mxnet-full_2.11-INTERNAL.jar
Welcome to Scala 2.11.12 (OpenJDK 64-Bit Server VM, Java 10.0.2).
Type in expressions for evaluation. Or try :help.

scala> import org.apache.mxnet._
import org.apache.mxnet._

scala> val c = NDArray.array(Array(1.0, 2, 3, 4, 5, 6), shape = Shape(2, 3))
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
java.lang.UnsatisfiedLinkError: /tmp/mxnet10078689559699346137/mxnet-scala: libopencv_core.so.2.4: cannot open shared object file: No such file or directory
  at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
  at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2424)
  at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2481)
  at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2678)
  at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2611)
  at java.base/java.lang.Runtime.load0(Runtime.java:814)
  at java.base/java.lang.System.load(System.java:1838)
  at org.apache.mxnet.util.NativeLibraryLoader$.loadLibraryFromFile(NativeLibraryLoader.scala:131)
  at org.apache.mxnet.util.NativeLibraryLoader$.loadLibrary(NativeLibraryLoader.scala:99)
  at org.apache.mxnet.Base$.<init>(Base.scala:77)
  at org.apache.mxnet.Base$.<clinit>(Base.scala)
  at org.apache.mxnet.NDArray$.initNDArrayModule(NDArray.scala:165)
  at org.apache.mxnet.NDArray$.<init>(NDArray.scala:42)
  at org.apache.mxnet.NDArray$.<clinit>(NDArray.scala)
  ... 32 elided

scala>

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

[zachgk]

@larroy We do link opencv statically (and everything else statically linked by the pip package) following the publish instructions (https://github.com/apache/incubator-mxnet/tree/master/ci/publish) that we use to publish the releases and nightly snapshots. The normal build instructions for building from source do not.

[lanking520]

Seemed to be good to close now 👍