Spark: Investigate Signer mismatch issue for microsoft-spark-*.jar

[halfbro]

Is your feature request related to a problem? Please describe.
My company uses Spark on Nomad to handle deployment and running of our Spark programs. I was recently tasked with evaluating whether or not it would be possible to use .NET Spark in conjunction with Spark on Nomad. After making sure that I am able to run the basic Scala examples, I tried to switch from the example .jar to microsoft-spark-2.4.x-0.3.0.jar, but got errors from the driver program.

Our specific setup for Spark on Nomad is to use Docker images based off of hashicorp/spark-nomad running in cluster mode. My idea for using .NET Spark is to replace the --class org.apache.spark.examples.SparkPi with --class org.apache.spark.deploy.DotnetRunner, and local:/opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar with local:/app/microsoft-spark-2.4.x-0.3.0.jar (we have patched versions of the docker image with Spark updated to 2.4.0). When I tried this, I immediately get this exception:

java.lang.SecurityException: class "org.apache.spark.deploy.DotnetRunner"'s signer information does not match signer information of other classes in the same package

This leads me to believe that it _should_ be possible to get this setup running, but I don't have enough knowledge of the JVM/Spark side of things to know where to begin. I also noticed that there is a separate worker used in the other deployment methods to translate the .NET code for the executors, I'm not sure how that should be done.

Describe the solution you'd like
Ideally all I would have to do is replace --class org.apache.spark.examples.SparkPi with --class org.apache.spark.deploy.DotnetRunner, and local:/opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar with local:/app/microsoft-spark-2.4.x-0.3.0.jar

Describe alternatives you've considered
We do not have the option of using a different orchestrator, so we would have to use normal Spark with Scala instead of this library

[imback82]

We heard the similar feedbacks from other users about the signer issue. Our build pipeline is signing the jar, which is conflicting with yours. We plan to resolve it in the next release. Meanwhile, can you build the jar from the source and use it instead of the jar shipped with the Nuget?

Here is the instruction: https://github.com/dotnet/spark/blob/master/docs/building/windows-instructions.md#building-spark-net-scala-extensions-layer

[ylibrach]

@imback82 is there any ETA on next release?

[imback82]

We are aiming for the second week of July.

[halfbro]

I built the jar from source, and I can confirm that this solved our issues.

Should I change the PR to a bug report and change the title to something more fitting (ie [BUG]: java.lang.SecurityException during startup)? Or just leave it as is?

[imback82]

Cool. I will handle changing the title.

[imback82]

@eerhardt, @safern

As far as I understand, the following https://github.com/dotnet/spark/blob/eb9cffb8c92879887972a934d3009ec56bf91572/eng/Signing.props#L3
will sign all the files inside a nuget package. Is there a way to skip signing files inside the nuget package? I am trying to skip signing jar files.

I looked at https://github.com/dotnet/arcade/blob/master/Documentation/CorePackages/Signing.md#usage-examples but didn't find this scenario.

[eerhardt]

Is not signing the .jar files really the right thing to do here? The error message says:

class "org.apache.spark.deploy.DotnetRunner"'s signer information does not match signer information of other classes in the same package

Is the more correct thing to do here to make our classes in a unique "dotnet" package? That way we still sign all the code that Microsoft is shipping, but since it is all contained in its own unique package, we don't get the error above that multiple classes in the same package are signed differently.

See https://stackoverflow.com/questions/2877262/java-securityexception-signer-information-does-not-match

This happens when classes belonging to the same package are loaded from different JAR files, and those JAR files have signatures signed with different certificates - or, perhaps more often, at least one is signed and one or more others are not (which includes classes loaded from directories since those AFAIK cannot be signed).
So either make sure all JARs (or at least those which contain classes from the same packages) are signed using the same certificate, or remove the signatures from the manifest of JAR files with overlapping packages.

[eerhardt]

If you want to skip signing the .jar files, modify the following

https://github.com/dotnet/spark/blob/eb9cffb8c92879887972a934d3009ec56bf91572/eng/Signing.props#L11-L14

to be:

  <ItemGroup>
    <!-- extend arcade's FileExtensionSignInfo with snupkg information -->
    <FileExtensionSignInfo Include=".snupkg" CertificateName="NuGet" />

    <!-- skip signing jar files -->
    <FileExtensionSignInfo Remove=".jar" />
  </ItemGroup>

[imback82]

Thanks @eerhardt for the suggestion! We are aware of the root cause and looking into different solutions. One is to move all classes into our own package names as you suggested (but this will break all published documentations). The other is to skip signing just this class (if Arcade supports this). As you suggested, skipping signing the whole jar may not be a good idea.

In the long run, moving all classes into our own package names may be the best solution (we are also working to see if we can put our Scala code into apache/spark). Please let me know if you have any question/suggestion.

[imback82]

Closed by #186.