spark-submit command does not work after installation

[anand035]

Problem encountered on https://dotnet.microsoft.com/learn/data/spark-tutorial/install-spark
Operating System: windows

Provide details about the problem you are experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.

[imback82]

Can you provide more details? (error messages, commands you ran, etc.)

[anand035]

I installed all prerequisites and extracted Apache Spark artifacts in c:bin, set the environment variables. After that when I run spark-submit (from C:binspark-2.4.1-bin-hadoop2.7bin) I get error
The system cannot find the path specified.

[imback82]

did you install Java? Can you run java on the command line?

[anand035]

yes,
C:\bin\spark-2.4.1-bin-hadoop2.7\bin>echo %JAVA_HOME%
C:\Program Files\Java\jdk1.8.0_231\bin

[elvaliuliuliu]

Can you see spark-submit.cmd in your folder C:\bin\spark-2.4.1-bin-hadoop2.7\bin?

[anand035]

yes,
Volume in drive C is OSDisk
Volume Serial Number is XYZ

Directory of C:\bin\spark-2.4.1-bin-hadoop2.7\bin

10/23/2019 04:39 PM <DIR> .
10/23/2019 04:39 PM <DIR> ..
03/26/2019 05:13 PM 1,089 beeline
03/26/2019 05:13 PM 1,064 beeline.cmd
03/26/2019 05:13 PM 5,427 docker-image-tool.sh
03/26/2019 05:13 PM 1,933 find-spark-home
03/26/2019 05:13 PM 2,681 find-spark-home.cmd
03/26/2019 05:13 PM 1,892 load-spark-env.cmd
03/26/2019 05:13 PM 2,025 load-spark-env.sh
03/26/2019 05:13 PM 2,987 pyspark
03/26/2019 05:13 PM 1,170 pyspark.cmd
03/26/2019 05:13 PM 1,540 pyspark2.cmd
03/26/2019 05:13 PM 1,030 run-example
03/26/2019 05:13 PM 1,223 run-example.cmd
10/23/2019 04:51 PM 3,205 spark-class
03/26/2019 05:13 PM 1,180 spark-class.cmd
03/26/2019 05:13 PM 2,545 spark-class2.cmd
03/26/2019 05:13 PM 3,122 spark-shell
03/26/2019 05:13 PM 1,178 spark-shell.cmd
03/26/2019 05:13 PM 1,818 spark-shell2.cmd
03/26/2019 05:13 PM 1,065 spark-sql
03/26/2019 05:13 PM 1,173 spark-sql.cmd
03/26/2019 05:13 PM 1,118 spark-sql2.cmd
03/26/2019 05:13 PM 1,040 spark-submit
03/26/2019 05:13 PM 1,180 spark-submit.cmd
03/26/2019 05:13 PM 1,155 spark-submit2.cmd
03/26/2019 05:13 PM 1,039 sparkR
03/26/2019 05:13 PM 1,168 sparkR.cmd
03/26/2019 05:13 PM 1,097 sparkR2.cmd
10/23/2019 04:39 PM 109,568 winutils.exe
28 File(s) 156,712 bytes
2 Dir(s) 853,160,673,280 bytes free

[imback82]

can you run java -version? not from JAVA_HOME, but anywhere else.

[anand035]

yes
D:\temp>java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

[elvaliuliuliu]

So my echo %JAVA_HOME% returns C:\Program Files\Java\jdk1.8.0_221 without bin. Not sure if this is some factor

[anand035]

I got rid of bin, and now its working 👍

[geyungjen]

I have the same problem, with jdk1.8.0_231, resolved after I have installed jdk1.8.0_191, strange indeed.

My environment:
Windows 10 professional.

[jolurf]

Hey guys. I got the same issue.
My %JAVA_HOME% is C:Program FilesJavajdk-15.0.1
MY %SPARK_HOME% C:Sparkspark-3.0.1-bin-hadoop2.7
and when i try to run %SPARK_HOME%/bin/spark-shell I get
"System couldn't find specified path'

[zelgawis2]

Hey,
I got the same issue today. Reading this topic and suggested solutions I wanted to install a new Java version to get rid of the problem.
but... before I did this I checked my %JAVA_HOME% environment variabale in cmd (echo %JAVA_HOME%) and it returned some version number.
Then I checked java version in cmd (java -version) and it returned some other version number.
So I just simply updated %JAVA_HOME% to the newest installed Java version and it worked!

When u have Java Updater prompt saying that the new Java version is ready to install, make sure to update the %JAVA_HOME% variable after, coz it turns out it's not being updated automatically.