Spark: [BUG]: Path does not exist error (despite correct path)

[bamurtaugh]

Describe the bug
I believe I may have encountered a bug with the .Text() method (for loading from a text file into a DataFrame).

I am following the getting started tutorial: https://dotnet.microsoft.com/learn/data/spark-tutorial/run. I have all the installations completed and paths set correctly (I can successfully run other Spark apps locally).

However, for mySparkApp specifically, when I run spark-submit, I get the following error:

[Error] [JvmBridge] org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:/Users/brmurtau/mySparkApp/input.txt.

To Reproduce

I followed all of the exact steps on the .NET tutorial site, including creating a new mySparkApp, adding input.txt, and modifying Program.cs.

dotnet build runs correctly. I also set the path to my app's dll in the assembly search environment variable (as discussed in a previous issue).

Expected behavior
Logging and word count table, as shown on the tutorial at the run stage: https://dotnet.microsoft.com/learn/data/spark-tutorial/run.

Screenshots
I have input.txt at the exact path that was listed in the error message:

I even tried changing the path to be full rather than relative in Program.cs but still got the same error.

Desktop:

• OS: Windows 10
• Version: Spark.NET 0.4, .NET Core 2.2

Additional context
I believe this could be some sort of bug with the .Text() method not being able to find my file. I am able to successfully submit/run other Spark apps that read in CSVs. I have tried deleting and re-creating mySparkApp and input.txt, but to no avail.

[GoEddie]

This is odd, all Text does is to pass on the path to spark - i.e. the spark-dotnet code doesn't look at the path at all.

If you start spark-shell and run:

spark.read.text("file:/C:/Users/brmurtau/mySparkApp/input.txt").show

Does that work or do you get the error there?

[bamurtaugh]

I just tried that and I got the same error: org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:/Users/brmurtau/mySparkApp/input.txt;

It's so strange because when I copy the path into my file directory, input.txt opens right up, so it's definitely there? Not sure what type of bug/issue this is? Is it some sort of issue because it appends file:/ to the beginning when it should just start with C:/?

[bamurtaugh]

I realize now my file name had ".txt" at the end of it, so the file was getting confused and had two ".txt" endings. Mistake I couldn't catch at the end of a long day :-) Issue solved!

[GoEddie]

Great news :)

[abhaysibal]

@bamurtaugh I ran into the same issue just now, and was stuck for long. Thanks for posting this!