Spark: Error while running sample program for Dotnet for spark

[dkjha811]

Problem encountered on https://dotnet.microsoft.com/learn/data/spark-tutorial/run
Operating System: windows

I have gone through this tutotials and intall every package listed down and was able to get the desired results till the last command to get the word count .
Here is the command which gives the error. I am not able to understand where is the gap .
%SPARK_HOME%bin\spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.0\microsoft-spark-2.4.x-0.6.0.jar dotnet bin\Debug\netcoreapp3.0\mySparkApp.dll

C:\Users\mc73892\mySparkApp>%SPARK_HOME%bin\spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.0\microsoft-spark-2.4.x-0.6.0.jar dotnet bin\Debug\netcoreapp3.0\mySparkApp.dll
20/01/24 13:18:53 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/01/24 13:18:53 WARN DependencyUtils: Local jar C:\Users\mc73892\mySparkAppbin\Debug\netcoreapp3.0\microsoft-spark-2.4.x-0.6.0.jar does not exist, skipping.
20/01/24 13:18:53 WARN SparkSubmit$$anon$2: Failed to load org.apache.spark.deploy.dotnet.DotnetRunner.
java.lang.ClassNotFoundException: org.apache.spark.deploy.dotnet.DotnetRunner
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:810)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/01/24 13:18:53 INFO ShutdownHookManager: Shutdown hook called
20/01/24 13:18:53 INFO ShutdownHookManager: Deleting directory C:\Users\mc73892\AppData\Local\Temp\spark-fd869262-8ec5-47aa-8392-dad3b6b02f04

C:\Users\mc73892\mySparkApp>

[elvaliuliuliu]

Can you try run the following command?

%SPARK_HOME%\bin\spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.1\microsoft-spark-2.4.x-0.6.0.jar dotnet bin\Debug\netcoreapp3.1\mySparkApp.dll

[imback82]

20/01/24 13:18:53 WARN DependencyUtils: Local jar C:\Users\mc73892\mySparkAppbin\Debug\netcoreapp3.0\microsoft-spark-2.4.x-0.6.0.jar does not exist,

Can you try @elvaliuliuliu's suggestion?

[sharadagarwal]

I was getting a similar error, and @elvaliuliuliu 's suggestion fixed it. Thanks @elvaliuliuliu !

[dkjha811]

Yes
Thank you .it's fixed now with the updated command.

On Sun, Jan 26, 2020, 9:04 PM Sharad Agarwal [e-mail]
wrote:

I was getting a similar error, and @elvaliuliuliu
https://github.com/elvaliuliuliu 's suggestion fixed it. Thanks
@elvaliuliuliu https://github.com/elvaliuliuliu !

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/spark/issues/401?email_source=notifications&email_token=AFTMYJSWLET34AIZFTYFU2DQ7ZFKVA5CNFSM4KLK4RUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ6GE3Y#issuecomment-578577007,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTMYJSMNUTXET7ELJG6DS3Q7ZFKVANCNFSM4KLK4RUA
.

[elvaliuliuliu]

This PR should address this issue.