Spark: [BUG]: Spark 3.0 and latest azure event hub library from microsoft requires encrypting connection strings

[viskumar-microsoft]

With Spark3.0 and latest Azure event Hub library (2.3.15 and beyond) we need to encrypt the event hub connection strings. Take a look here
https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/PySpark/structured-streaming-pyspark.md#event-hubs-configuration

When creating connections to event hub from spark dotnet we dont have a direct way to call this encryption method from .Net

Without this support we wont be able to connect event hubs from spark dotnet.

If you think the bug depends on external factors (e.g., dataset), please provide us with a minimal reproducible example that consists of the following items:

• a minimal dataset, necessary to reproduce the error
• the minimal runnable code necessary to reproduce the error, which can be run on the given dataset
• the necessary information on any used packages, .NET runtime version, and system it is run on
• in the case of random processes, a seed for reproducibility

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[suhsteve]

@viskumar-microsoft some possible options:

• Use the IgnoresAccessChecksToGenerator nuget to access the internal JVMBridge class
  
  
  
  • In our conversation you mentioned that you must use nugets available in a special feed. Can you publish this nuget there? If not, why not ?
  
  
• Use reflection to access internal classes/methods.
• Convert the encrypt method implementation to C#

[viskumar-microsoft]

Agreed we can use reflection or use IgnoresAccessChecksToGenerator to call the JVMBridge class but I believe this is a general problem that everyone using spark dotnet and event hubs will face. So it will be great if there is an option provided in spark dotnet library so that we dont have to go round about way to make it work.

[suhsteve]

The request here seems to be an ask to add access of the JVMBridge as public APIs.

cc: @imback82, @rapoth

[suhsteve]

Another option is to add an azure-event-hubs-spark .Net for Apache Spark Extension

[ksiomelo]

Here is how you can use EventHubUtils to encrypt the connection string in dotnet:

add a reference to IgnoresAccessChecksToGenerator along with the following properties:
<PackageReference Include="IgnoresAccessChecksToGenerator" Version="0.4.0" />

  <PropertyGroup>
    <InternalsAssemblyNames>Microsoft.Spark</InternalsAssemblyNames>
  </PropertyGroup>

  <PropertyGroup>
    <InternalsAssemblyUseEmptyMethodBodies>false</InternalsAssemblyUseEmptyMethodBodies>
  </PropertyGroup>

Make the following changes in your class:

using Microsoft.Spark.Interop;
using Microsoft.Spark.Interop.Ipc;

public class EventHubExample
{
    private static IJvmBridge Jvm { get; } = SparkEnvironment.JvmBridge;

    private Dictionary<string, string> GetEventHubConfiguration()
        {
            var eventHubConnectionStringEncrypted = (string)Jvm.CallStaticJavaMethod(
                "org.apache.spark.eventhubs.EventHubsUtils", "encrypt", "<eventhub-connection-string>");

            var eventhubConf = new Dictionary<string, string>
            {
                { "eventhubs.connectionString", eventHubConnectionStringEncrypted },
                { "eventhubs.useExclusiveReceiver", "false" },
            };

            return eventhubConf;
        }
}

[suhsteve]

Workaround to access EventHubUtils to encrypt the connection string in a notebook session

using System.Reflection;
var jvmBridge = Assembly
    .Load("Microsoft.Spark")
    .GetType("Microsoft.Spark.Interop.SparkEnvironment")
    .GetProperty("JvmBridge", BindingFlags.Static | BindingFlags.NonPublic)
    .GetValue(null);
var callStaticJavaMethod = jvmBridge
    .GetType()
    .GetMethod(
        "CallStaticJavaMethod",
        BindingFlags.Public | BindingFlags.Instance,
        null,
        new Type[] { typeof(string), typeof(string), typeof(object) },
        null);
var eventHubConnectionStringEncrypted = (string)callStaticJavaMethod
    .Invoke(
        jvmBridge,
        new object[]
        {
            "org.apache.spark.eventhubs.EventHubsUtils",
            "encrypt",
            "<eventhub-connection-string>"
        });

[suhsteve]

@viskumar-microsoft , @ksiomelo we have added some public APIs to access the JVM and these should support your use case. I will close this issue, please feel free to reopen if the APIs we've made available do not fulfill your needs.