Spark: [BUG]: Broadcast throws exception on Databricks on AWS.

[jonathantonic]

Describe the bug
Broadcast throws exception on Databricks on AWS.

Error Logs

20/10/26 16:43:55 WARN DotnetBackendHandler: cannot find matching method class org.apache.spark.api.python.PythonRDD.setupBroadcast. Candidates are:
20/10/26 16:43:55 WARN DotnetBackendHandler: setupBroadcast(class org.apache.spark.api.java.JavaSparkContext,class java.lang.String)
20/10/26 16:43:55 ERROR DotnetBackendHandler: Failed to execute 'setupBroadcast' on 'NullObject' with args=([Type=java.lang.String, Value: /local_disk0/spark-0b4511c2-19cd-4849-9fe6-583b1ea94fac/sparkdotnet/5w3ohffw.2m0])

HelloSpark Program.cs Code

using Microsoft.Spark;
using Microsoft.Spark.Sql;
using Microsoft.Spark.Sql.Types;

namespace HelloSpark
{
    class Program
    {
        static void Main(string[] args)
        {
            var spark = SparkSession.Builder().GetOrCreate();
            var broadcastTest = spark.SparkContext.Broadcast("test");
        }
    }
}

To Reproduce

1. Create a simple HelloSpark program that just gets the context and does a broadcast.
2. Setup a jar job using the HelloSpark program in databricks as per the deployment guide:
   https://github.com/dotnet/spark/tree/master/deployment#using-set-jar
3. Run the job.
4. There should be an error in the Standard Output logs complaining about setupBroadcast only existing with two parameters (apache.spark.api.java.JavaSparkContext, java.lang.String) but being called with one (java.lang.String).

Expected behavior
Broadcast works without any errors.

Environment Info

• AWS
• Databricks Runtime: 6.6 (Scala 2.11, Spark 2.4.5)
• using jar: microsoft_spark_2_4_2_11_1_0_0.jar
• Dotnet Spark v1.0.0

Additional Info
For whatever reason it seems like in the Databricks environment PythonRDD setupBroadcast takes the javaSparkContext and the path.
https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark/Broadcast.cs#L177

Updating the Broadcast.cs to also pass in the javaSparkContext resolved the issue when running on Databricks but of course breaks it in other enviornments like AWS EMR.

This example uses 2.4, but I have seen the same issues on spark 3.

This may also happen on Azure, but can only confirm on AWS.

[Niharikadutta]

Hi @jonathantonic , thanks for reporting this and sorry for the error you are facing. Could you please attach the full stacktrace of the error logs?

[jonathantonic]

Hi @jonathantonic , thanks for reporting this and sorry for the error you are facing. Could you please attach the full stacktrace of the error logs?

full error logs:
log4j-active.txt
stderr.txt
stdout.txt

[Niharikadutta]

Thanks for providing the complete logs @jonathantonic . Just to clarify, are you able to run your application in AWS EMR without any of the changes you had to make for AWS Databricks? (Such as passing javaSparkContext to setupBroadcast ?)

[jonathantonic]

Thanks for providing the complete logs @jonathantonic . Just to clarify, are you able to run your application in AWS EMR without any of the changes you had to make for AWS Databricks? (Such as passing javaSparkContext to setupBroadcast ?)

Yes that is correct.

[Niharikadutta]

Thanks @jonathantonic I'm going to repro this bug and get back to you with our suggestions.

[jonathantonic]

Hey @Niharikadutta, any luck reproducing?

[Niharikadutta]

Hey @jonathantonic I am sorry for the delay have been a little occupied, I am working on it and will get back to you soon. Thanks for your patience!

[Niharikadutta]

Hi @jonathantonic I was able to reproduce this on AWS as well as Azure Databricks for Spark versions 2.4.5 and 3.0.1. We are working on coming up with a solution/workaround and will update you as soon as we have a plan. Thanks!

[Niharikadutta]

HI @jonathantonic , the fix for this bug is in PR - https://github.com/dotnet/spark/pull/766 and will be available in v1.0.1, which is the next release.