Spark: [Question]: .NET for Apache Spark to stream from Azure EventHub

Created on 1 Nov 2019  路  22Comments  路  Source: dotnet/spark

Question
I'm new to .net for spark and wondering if there is any way we could stream the EventHub events?

To Reproduce

Actually, I tried to give the format as "eventHubs" when set up the spark ReadStream, but it throws an error when try to run locally as below:

image

Also, i could not find the equivalence of 'azure-eventhubs-spark_2.11' in nuget store.

Desktop (please complete the following information):

  • Windows 10
  • Spark version 2.3.*
question
Source
picture rkbalu

Most helpful comment

Example works! Thanks @joperezr !
I have to reference package not jars

spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.13 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.7.0.jar TestSparkStreaming.exe

picture MaxKhlupnov on 24 Feb 2020
👍2

All 22 comments

@rkbalu if you are trying to use azure eventhubs then you may need to add the --packages option to your spark-submit command as documented here

suhsteve picture suhsteve on 1 Nov 2019

thanks @suhsteve . But look like the example given is to write it in python/scala. How can we use that in .net spark application? Are there any example link that you can point to?

rkbalu picture rkbalu on 1 Nov 2019

Cc: @joperezr

rapoth picture rapoth on 8 Nov 2019

Hello @rkbalu. I recently wanted to write a structured Streaming app reading data streaming from Azure eventHubs as well, and this is how I was able to do it:

The library that I used in order to get this working was this one: https://github.com/Azure/azure-event-hubs-spark which is probably the one you found examples for and the reason why you passed in eventhubs expecting for it to work.

1.- Copy the following code into your Console App so that it has a basic object model that knows how to read things from EventHub, and so that it gets your connection string details. I added comments to the places where you have to add the info specific to your own EventHub: https://gist.github.com/joperezr/de0c963213c69b870f2e65b92c5406a4
2.- After getting that code ready and compiled, all you need to do is to make sure that you pass in the --jars method to the spark-submit command and include the jar from azure-event-hubs-spark repo. By adding this jar, it will add the support of understanding the format "eventhubs" when you pass it into the StreamReader.

After that, you should be able to succesfully read data from EventHub using .NET for Apache Spark.

Let me know if you hit any issues.

joperezr picture joperezr on 11 Nov 2019

Also, if the above code and instructions work for people, I'll go ahead and submit a PR into this repo to add some docs in the repo on how to do this.

joperezr picture joperezr on 11 Nov 2019
👍2

thanks @joperezr for giving this info. I'll give it a go and get back if i face any issues.

rkbalu picture rkbalu on 13 Nov 2019

thanks @joperezr!

imback82 picture imback82 on 13 Nov 2019

Thanks for the code @joperezr !
The only change I have to made to be able compile this code was events.Select expression:
Functions.FromJson(Functions.Col("body").Cast("string"), "BlobPath String").Alias("Raw"),
Functions.Col("properties"),
Functions.Col("enqueuedTime")

MaxKhlupnov picture MaxKhlupnov on 8 Jan 2020

I tried run example but get an error:
java.lang.NoSuchMethodError: com.microsoft.azure.eventhubs.EventHubClient.createSync(Ljava/lang/String;Ljava/util/concurrent/ScheduledExecutorService;)Lcom/microsoft/azure/eventhubs/EventHubClient;

SparkVersion: 2.4.1

spark-submit --jars azure-eventhubs-spark_2.11-2.3.13.jar,azure-eventhubs-3.0.2.jar --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.7.0.jar dotnet TestSparkStreaming.dll

Error message:
Exception in thread "stream execution thread for [id = 8d72a79d-2cff-46cd-a5d3-50a52edb77f5, runId = f41f2314-852c-4fbd-8634-4ac32f9b6b80]" java.lang.NoSuchMethodError: com.microsoft.azure.eventhubs.EventHubClient.createSync(Ljava/lang/String;Ljava/util/concurrent/ScheduledExecutorService;)Lcom/microsoft/azure/eventhubs/EventHubClient; 374-4ce4-4e74-8a6e-fb3a4a85b937

MaxKhlupnov picture MaxKhlupnov on 8 Jan 2020

Example works! Thanks @joperezr !
I have to reference package not jars

spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.13 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.7.0.jar TestSparkStreaming.exe

MaxKhlupnov picture MaxKhlupnov on 24 Feb 2020
👍2

Hi @joperezr - tried following command, but it failed.

spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.11.0.jar IotHubConsumer.exe

Here is the full message:

E:\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.11.0.jar IotHubConsumer.exe
Ivy Default Cache set to: C:\Users\Sanket\.ivy2\cache
The jars for the packages stored in: C:\Users\Sanket\.ivy2\jars
:: loading settings :: url = jar:file:/C:/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.microsoft.azure#azure-eventhubs-spark_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-71591438-593e-4204-a1e6-105cdfab201b;1.0
        confs: [default]
        found com.microsoft.azure#azure-eventhubs-spark_2.11;2.3.15 in central
        found com.microsoft.azure#azure-eventhubs;3.0.2 in central
        found org.apache.qpid#proton-j;0.31.0 in central
        found com.microsoft.azure#qpid-proton-j-extensions;1.2.0 in central
        found org.slf4j#slf4j-api;1.7.6 in central
        found com.microsoft.azure#azure-client-authentication;1.6.12 in central
        found com.microsoft.azure#azure-client-runtime;1.6.12 in central
        found com.microsoft.rest#client-runtime;1.6.12 in central
        found com.google.guava#guava;24.1.1-jre in central
        found com.google.code.findbugs#jsr305;1.3.9 in local-m2-cache
        found org.checkerframework#checker-compat-qual;2.0.0 in local-m2-cache
        found com.google.errorprone#error_prone_annotations;2.1.3 in local-m2-cache
        found com.google.j2objc#j2objc-annotations;1.1 in local-m2-cache
        found org.codehaus.mojo#animal-sniffer-annotations;1.14 in local-m2-cache
        found com.squareup.retrofit2#retrofit;2.5.0 in central
        found com.squareup.okhttp3#okhttp;3.11.0 in central
        found com.squareup.okio#okio;1.14.0 in central
        found com.squareup.okhttp3#logging-interceptor;3.12.2 in central
        found com.squareup.okhttp3#okhttp-urlconnection;3.12.2 in central
        found com.squareup.retrofit2#converter-jackson;2.5.0 in central
        found com.fasterxml.jackson.core#jackson-databind;2.9.9 in central
        found com.fasterxml.jackson.core#jackson-annotations;2.9.9 in central
        found com.fasterxml.jackson.core#jackson-core;2.9.9 in central
        found com.fasterxml.jackson.datatype#jackson-datatype-joda;2.9.8 in central
        found joda-time#joda-time;2.7 in central
        found org.apache.commons#commons-lang3;3.8.1 in central
        found io.reactivex#rxjava;1.2.4 in central
        found com.squareup.retrofit2#adapter-rxjava;2.4.0 in central
        found com.microsoft.azure#azure-annotations;1.7.0 in central
        found commons-codec#commons-codec;1.10 in local-m2-cache
        found com.microsoft.azure#adal4j;1.6.3 in central
        found com.nimbusds#oauth2-oidc-sdk;6.13 in central
        found com.sun.mail#javax.mail;1.6.1 in central
        found javax.activation#activation;1.1 in local-m2-cache
        found com.github.stephenc.jcip#jcip-annotations;1.0-1 in central
        found net.minidev#json-smart;2.3 in central
        [2.3] net.minidev#json-smart;[1.3.1,2.3]
        found net.minidev#accessors-smart;1.2 in central
        found org.ow2.asm#asm;5.0.4 in central
        found com.nimbusds#lang-tag;1.5 in central
        [1.5] com.nimbusds#lang-tag;[1.4.3,)
        found com.google.code.gson#gson;2.8.5 in central
        found com.nimbusds#nimbus-jose-jwt;6.0.1 in central
        found org.scala-lang.modules#scala-java8-compat_2.11;0.9.0 in central
:: resolution report :: resolve 6741ms :: artifacts dl 79ms
        :: modules in use:
        com.fasterxml.jackson.core#jackson-annotations;2.9.9 from central in [default]
        com.fasterxml.jackson.core#jackson-core;2.9.9 from central in [default]
        com.fasterxml.jackson.core#jackson-databind;2.9.9 from central in [default]
        com.fasterxml.jackson.datatype#jackson-datatype-joda;2.9.8 from central in [default]
        com.github.stephenc.jcip#jcip-annotations;1.0-1 from central in [default]
        com.google.code.findbugs#jsr305;1.3.9 from local-m2-cache in [default]
        com.google.code.gson#gson;2.8.5 from central in [default]
        com.google.errorprone#error_prone_annotations;2.1.3 from local-m2-cache in [default]
        com.google.guava#guava;24.1.1-jre from central in [default]
        com.google.j2objc#j2objc-annotations;1.1 from local-m2-cache in [default]
        com.microsoft.azure#adal4j;1.6.3 from central in [default]
        com.microsoft.azure#azure-annotations;1.7.0 from central in [default]
        com.microsoft.azure#azure-client-authentication;1.6.12 from central in [default]
        com.microsoft.azure#azure-client-runtime;1.6.12 from central in [default]
        com.microsoft.azure#azure-eventhubs;3.0.2 from central in [default]
        com.microsoft.azure#azure-eventhubs-spark_2.11;2.3.15 from central in [default]
        com.microsoft.azure#qpid-proton-j-extensions;1.2.0 from central in [default]
        com.microsoft.rest#client-runtime;1.6.12 from central in [default]
        com.nimbusds#lang-tag;1.5 from central in [default]
        com.nimbusds#nimbus-jose-jwt;6.0.1 from central in [default]
        com.nimbusds#oauth2-oidc-sdk;6.13 from central in [default]
        com.squareup.okhttp3#logging-interceptor;3.12.2 from central in [default]
        com.squareup.okhttp3#okhttp;3.11.0 from central in [default]
        com.squareup.okhttp3#okhttp-urlconnection;3.12.2 from central in [default]
        com.squareup.okio#okio;1.14.0 from central in [default]
        com.squareup.retrofit2#adapter-rxjava;2.4.0 from central in [default]
        com.squareup.retrofit2#converter-jackson;2.5.0 from central in [default]
        com.squareup.retrofit2#retrofit;2.5.0 from central in [default]
        com.sun.mail#javax.mail;1.6.1 from central in [default]
        commons-codec#commons-codec;1.10 from local-m2-cache in [default]
        io.reactivex#rxjava;1.2.4 from central in [default]
        javax.activation#activation;1.1 from local-m2-cache in [default]
        joda-time#joda-time;2.7 from central in [default]
        net.minidev#accessors-smart;1.2 from central in [default]
        net.minidev#json-smart;2.3 from central in [default]
        org.apache.commons#commons-lang3;3.8.1 from central in [default]
        org.apache.qpid#proton-j;0.31.0 from central in [default]
        org.checkerframework#checker-compat-qual;2.0.0 from local-m2-cache in [default]
        org.codehaus.mojo#animal-sniffer-annotations;1.14 from local-m2-cache in [default]
        org.ow2.asm#asm;5.0.4 from central in [default]
        org.scala-lang.modules#scala-java8-compat_2.11;0.9.0 from central in [default]
        org.slf4j#slf4j-api;1.7.6 from central in [default]
        :: evicted modules:
        com.nimbusds#nimbus-jose-jwt;[6.0.1,) by [com.nimbusds#nimbus-jose-jwt;6.0.1] in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   43  |   2   |   0   |   1   ||   42  |   0   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
                [NOT FOUND  ] org.checkerframework#checker-compat-qual;2.0.0!checker-compat-qual.jar (1ms)

        ==== local-m2-cache: tried

          file:/C:/Users/Sanket/.m2/repository/org/checkerframework/checker-compat-qual/2.0.0/checker-compat-qual-2.0.0.jar

                [NOT FOUND  ] com.google.errorprone#error_prone_annotations;2.1.3!error_prone_annotations.jar (0ms)

        ==== local-m2-cache: tried

          file:/C:/Users/Sanket/.m2/repository/com/google/errorprone/error_prone_annotations/2.1.3/error_prone_annotations-2.1.3.jar

                [NOT FOUND  ] com.google.j2objc#j2objc-annotations;1.1!j2objc-annotations.jar (0ms)

        ==== local-m2-cache: tried

          file:/C:/Users/Sanket/.m2/repository/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar

                [NOT FOUND  ] org.codehaus.mojo#animal-sniffer-annotations;1.14!animal-sniffer-annotations.jar (1ms)

        ==== local-m2-cache: tried

          file:/C:/Users/Sanket/.m2/repository/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::              FAILED DOWNLOADS            ::

                :: ^ see resolution messages for details  ^ ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: org.checkerframework#checker-compat-qual;2.0.0!checker-compat-qual.jar

                :: com.google.errorprone#error_prone_annotations;2.1.3!error_prone_annotations.jar

                :: com.google.j2objc#j2objc-annotations;1.1!j2objc-annotations.jar

                :: org.codehaus.mojo#animal-sniffer-annotations;1.14!animal-sniffer-annotations.jar

                ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: org.checkerframework#checker-compat-qual;2.0.0!checker-compat-qual.jar, download failed: com.google.errorprone#error_prone_annotations;2.1.3!error_prone_annotations.jar, download failed: com.google.j2objc#j2objc-annotations;1.1!j2objc-annotations.jar, download failed: org.codehaus.mojo#animal-sniffer-annotations;1.14!animal-sniffer-annotations.jar]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1302)
        at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:304)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>

My Spark version is: 2.4.5.

Can you please help with recommended solution?

sanket-t-shah picture sanket-t-shah on 12 May 2020

mm unfortunately I'm not an expert on package dependencies so I'm not sure if I would be able to help here, @imback82 do you have any idea of what might be going on in @sanket-t-shah case?

joperezr picture joperezr on 12 May 2020

Hi @imback82 - can you please help on error that am getting? I'll be highly obliged if you can provide some guidance over the same.

sanket-t-shah picture sanket-t-shah on 12 May 2020

I cannot repro your problem by running ./spark-shell --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15. Maybe can you try to delete C:/Users/Sanket/.m2 and try again?

imback82 picture imback82 on 12 May 2020

@imback82 - Thanks, it worked in a way, but started giving me different problem now.

I've used the code sample as given by @joperezr. Below is my command:

spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.11.0.jar IotHubConsumer.exe

Here is the output am getting.

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.15 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.11.0.jar IotHubConsumer.exe
Ivy Default Cache set to: C:\Users\Sanket\.ivy2\cache
The jars for the packages stored in: C:\Users\Sanket\.ivy2\jars
:: loading settings :: url = jar:file:/C:/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.microsoft.azure#azure-eventhubs-spark_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-1fcc0bc0-f777-4f79-a9cf-90191511ef12;1.0
        confs: [default]
        found com.microsoft.azure#azure-eventhubs-spark_2.11;2.3.15 in central
        found com.microsoft.azure#azure-eventhubs;3.0.2 in central
        found org.apache.qpid#proton-j;0.31.0 in central
        found com.microsoft.azure#qpid-proton-j-extensions;1.2.0 in central
        found org.slf4j#slf4j-api;1.7.6 in central
        found com.microsoft.azure#azure-client-authentication;1.6.12 in central
        found com.microsoft.azure#azure-client-runtime;1.6.12 in central
        found com.microsoft.rest#client-runtime;1.6.12 in central
        found com.google.guava#guava;24.1.1-jre in central
        found com.google.code.findbugs#jsr305;1.3.9 in local-m2-cache
        found org.checkerframework#checker-compat-qual;2.0.0 in local-m2-cache
        found com.google.errorprone#error_prone_annotations;2.1.3 in local-m2-cache
        found com.google.j2objc#j2objc-annotations;1.1 in local-m2-cache
        found org.codehaus.mojo#animal-sniffer-annotations;1.14 in local-m2-cache
        found com.squareup.retrofit2#retrofit;2.5.0 in central
        found com.squareup.okhttp3#okhttp;3.11.0 in central
        found com.squareup.okio#okio;1.14.0 in central
        found com.squareup.okhttp3#logging-interceptor;3.12.2 in central
        found com.squareup.okhttp3#okhttp-urlconnection;3.12.2 in central
        found com.squareup.retrofit2#converter-jackson;2.5.0 in central
        found com.fasterxml.jackson.core#jackson-databind;2.9.9 in central
        found com.fasterxml.jackson.core#jackson-annotations;2.9.9 in central
        found com.fasterxml.jackson.core#jackson-core;2.9.9 in central
        found com.fasterxml.jackson.datatype#jackson-datatype-joda;2.9.8 in central
        found joda-time#joda-time;2.7 in central
        found org.apache.commons#commons-lang3;3.8.1 in central
        found io.reactivex#rxjava;1.2.4 in central
        found com.squareup.retrofit2#adapter-rxjava;2.4.0 in central
        found com.microsoft.azure#azure-annotations;1.7.0 in central
        found commons-codec#commons-codec;1.10 in local-m2-cache
        found com.microsoft.azure#adal4j;1.6.3 in central
        found com.nimbusds#oauth2-oidc-sdk;6.13 in central
        found com.sun.mail#javax.mail;1.6.1 in central
        found javax.activation#activation;1.1 in local-m2-cache
        found com.github.stephenc.jcip#jcip-annotations;1.0-1 in central
        found net.minidev#json-smart;2.3 in central
        [2.3] net.minidev#json-smart;[1.3.1,2.3]
        found net.minidev#accessors-smart;1.2 in central
        found org.ow2.asm#asm;5.0.4 in central
        found com.nimbusds#lang-tag;1.5 in central
        [1.5] com.nimbusds#lang-tag;[1.4.3,)
        found com.google.code.gson#gson;2.8.5 in central
        found com.nimbusds#nimbus-jose-jwt;6.0.1 in central
        found org.scala-lang.modules#scala-java8-compat_2.11;0.9.0 in central
:: resolution report :: resolve 7031ms :: artifacts dl 65ms
        :: modules in use:
        com.fasterxml.jackson.core#jackson-annotations;2.9.9 from central in [default]
        com.fasterxml.jackson.core#jackson-core;2.9.9 from central in [default]
        com.fasterxml.jackson.core#jackson-databind;2.9.9 from central in [default]
        com.fasterxml.jackson.datatype#jackson-datatype-joda;2.9.8 from central in [default]
        com.github.stephenc.jcip#jcip-annotations;1.0-1 from central in [default]
        com.google.code.findbugs#jsr305;1.3.9 from local-m2-cache in [default]
        com.google.code.gson#gson;2.8.5 from central in [default]
        com.google.errorprone#error_prone_annotations;2.1.3 from local-m2-cache in [default]
        com.google.guava#guava;24.1.1-jre from central in [default]
        com.google.j2objc#j2objc-annotations;1.1 from local-m2-cache in [default]
        com.microsoft.azure#adal4j;1.6.3 from central in [default]
        com.microsoft.azure#azure-annotations;1.7.0 from central in [default]
        com.microsoft.azure#azure-client-authentication;1.6.12 from central in [default]
        com.microsoft.azure#azure-client-runtime;1.6.12 from central in [default]
        com.microsoft.azure#azure-eventhubs;3.0.2 from central in [default]
        com.microsoft.azure#azure-eventhubs-spark_2.11;2.3.15 from central in [default]
        com.microsoft.azure#qpid-proton-j-extensions;1.2.0 from central in [default]
        com.microsoft.rest#client-runtime;1.6.12 from central in [default]
        com.nimbusds#lang-tag;1.5 from central in [default]
        com.nimbusds#nimbus-jose-jwt;6.0.1 from central in [default]
        com.nimbusds#oauth2-oidc-sdk;6.13 from central in [default]
        com.squareup.okhttp3#logging-interceptor;3.12.2 from central in [default]
        com.squareup.okhttp3#okhttp;3.11.0 from central in [default]
        com.squareup.okhttp3#okhttp-urlconnection;3.12.2 from central in [default]
        com.squareup.okio#okio;1.14.0 from central in [default]
        com.squareup.retrofit2#adapter-rxjava;2.4.0 from central in [default]
        com.squareup.retrofit2#converter-jackson;2.5.0 from central in [default]
        com.squareup.retrofit2#retrofit;2.5.0 from central in [default]
        com.sun.mail#javax.mail;1.6.1 from central in [default]
        commons-codec#commons-codec;1.10 from local-m2-cache in [default]
        io.reactivex#rxjava;1.2.4 from central in [default]
        javax.activation#activation;1.1 from local-m2-cache in [default]
        joda-time#joda-time;2.7 from central in [default]
        net.minidev#accessors-smart;1.2 from central in [default]
        net.minidev#json-smart;2.3 from central in [default]
        org.apache.commons#commons-lang3;3.8.1 from central in [default]
        org.apache.qpid#proton-j;0.31.0 from central in [default]
        org.checkerframework#checker-compat-qual;2.0.0 from local-m2-cache in [default]
        org.codehaus.mojo#animal-sniffer-annotations;1.14 from local-m2-cache in [default]
        org.ow2.asm#asm;5.0.4 from central in [default]
        org.scala-lang.modules#scala-java8-compat_2.11;0.9.0 from central in [default]
        org.slf4j#slf4j-api;1.7.6 from central in [default]
        :: evicted modules:
        com.nimbusds#nimbus-jose-jwt;[6.0.1,) by [com.nimbusds#nimbus-jose-jwt;6.0.1] in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   43  |   2   |   0   |   1   ||   42  |   0   |
        ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-1fcc0bc0-f777-4f79-a9cf-90191511ef12
        confs: [default]
        0 artifacts copied, 42 already retrieved (0kB/43ms)
20/05/13 02:27:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[2020-05-12T20:57:45.3490685Z] [SANKETSHAH-HP] [Info] [ConfigurationService] Using port 1510 for connection.
[2020-05-12T20:57:45.3596496Z] [SANKETSHAH-HP] [Info] [JvmBridge] JvMBridge port is 1510
20/05/13 02:27:50 ERROR MicroBatchExecution: Query [id = c7d1aa16-3a3e-4941-943b-58ad562ec5ed, runId = c3cf99f6-6a24-4b74-a96d-dd0e4af440cd] terminated with error
java.lang.NoClassDefFoundError: com/microsoft/azure/eventhubs/impl/EventHubClientImpl
        at org.apache.spark.eventhubs.client.EventHubsClient$.userAgent_$eq(EventHubsClient.scala:336)
        at org.apache.spark.sql.eventhubs.EventHubsSourceProvider.createSource(EventHubsSourceProvider.scala:79)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:255)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:88)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:85)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:79)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:85)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:257)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:247)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:65)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:269)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.lang.ClassNotFoundException: com.microsoft.azure.eventhubs.impl.EventHubClientImpl
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 41 more
Exception in thread "stream execution thread for [id = c7d1aa16-3a3e-4941-943b-58ad562ec5ed, runId = c3cf99f6-6a24-4b74-a96d-dd0e4af440cd]" java.lang.NoClassDefFoundError: com/microsoft/azure/eventhubs/impl/EventHubClientImpl
        at org.apache.spark.eventhubs.client.EventHubsClient$.userAgent_$eq(EventHubsClient.scala:336)
        at org.apache.spark.sql.eventhubs.EventHubsSourceProvider.createSource(EventHubsSourceProvider.scala:79)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:255)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:88)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:85)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:79)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:85)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:257)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:247)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:65)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:269)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.lang.ClassNotFoundException: com.microsoft.azure.eventhubs.impl.EventHubClientImpl
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 41 more
20/05/13 02:27:50 ERROR DotnetBackendHandler: Failed to execute 'awaitTermination' on 'org.apache.spark.sql.execution.streaming.StreamingQueryWrapper' with args=()
[2020-05-12T20:57:51.2557891Z] [SANKETSHAH-HP] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'awaitTermination' failed for class '15' when called with no arguments
[2020-05-12T20:57:51.2560947Z] [SANKETSHAH-HP] [Error] [JvmBridge] org.apache.spark.sql.streaming.StreamingQueryException: com/microsoft/azure/eventhubs/impl/EventHubClientImpl
=== Streaming Query ===
Identifier: [id = c7d1aa16-3a3e-4941-943b-58ad562ec5ed, runId = c3cf99f6-6a24-4b74-a96d-dd0e4af440cd]
Current Committed Offsets: {}
Current Available Offsets: {}

Current State: INITIALIZING
Thread State: RUNNABLE
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:297)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.lang.NoClassDefFoundError: com/microsoft/azure/eventhubs/impl/EventHubClientImpl
        at org.apache.spark.eventhubs.client.EventHubsClient$.userAgent_$eq(EventHubsClient.scala:336)
        at org.apache.spark.sql.eventhubs.EventHubsSourceProvider.createSource(EventHubsSourceProvider.scala:79)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:255)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:88)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:85)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:79)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:85)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:257)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:247)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:65)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:269)
        ... 1 more
Caused by: java.lang.ClassNotFoundException: com.microsoft.azure.eventhubs.impl.EventHubClientImpl
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 41 more

[2020-05-12T20:57:51.3268926Z] [SANKETSHAH-HP] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'awaitTermination' failed for class '15' when called with no arguments
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
Unhandled exception. System.Exception: JVM method execution failed: Nonstatic method 'awaitTermination' failed for class '15' when called with no arguments
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmObjectReference.Invoke(String methodName, Object[] args)
   at Microsoft.Spark.Sql.Streaming.StreamingQuery.AwaitTermination()
   at IotHubConsumer.Program.Main(String[] args) in E:\self-training\spark\IotHubConsumer\Program.cs:line 91

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>
sanket-t-shah picture sanket-t-shah on 12 May 2020

@imback82 @joperezr - Here is a new hack that kind of moved me to different error.

I copied following 3 files to spark --> jars folder.
azure-eventhubs-3.1.1.jar
azure-eventhubs-spark_2.11-2.3.15.jar
microsoft-spark-2.4.x-0.11.0.jar

Seems that above 3 JARs are not getting passed via my earlier command. Is there any way to put these files via Spark Submit job as I intend to publish them as a Job on Azure Databricks?

Post copying of files, I modified the command to look like below:
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local[*] microsoft-spark-2.4.x-0.11.0.jar IotHubConsumer.exe

Now, the error is: 

java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit

Can you please help here?

My versions: 

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>spark-shell --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.5
      /_/

Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_202
Branch HEAD
Compiled by user centos on 2020-02-02T19:38:06Z
Revision cee4ecbb16917fa85f02c635925e2687400aa56b
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>java -version
java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>dotnet --version
3.1.202

Error Stack Trace: 

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local[*] microsoft-spark-2.4.x-0.11.0.jar IotHubConsumer.exe
20/05/13 16:37:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[2020-05-13T11:07:17.2027957Z] [SANKETSHAH-HP] [Info] [ConfigurationService] Using port 34401 for connection.
[2020-05-13T11:07:17.2157641Z] [SANKETSHAH-HP] [Info] [JvmBridge] JvMBridge port is 34401
20/05/13 16:37:24 ERROR MicroBatchExecution: Query [id = 73ea0a6c-6cb5-41e9-921f-d6fdb177eaca, runId = d9694079-294a-42e3-a531-5b30bbe1cf3b] terminated with error
java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit
        at java.util.Base64$Decoder.decode0(Unknown Source)
        at java.util.Base64$Decoder.decode(Unknown Source)
        at java.util.Base64$Decoder.decode(Unknown Source)
        at org.apache.spark.eventhubs.EventHubsUtils$.decrypt(EventHubsUtils.scala:158)
        at org.apache.spark.eventhubs.EventHubsConf$.toConf(EventHubsConf.scala:553)
        at org.apache.spark.sql.eventhubs.EventHubsSource.<init>(EventHubsSource.scala:77)
        at org.apache.spark.sql.eventhubs.EventHubsSourceProvider.createSource(EventHubsSourceProvider.scala:82)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:255)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:88)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:85)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:79)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:85)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:257)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:247)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:65)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:269)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
20/05/13 16:37:24 ERROR DotnetBackendHandler: Failed to execute 'awaitTermination' on 'org.apache.spark.sql.execution.streaming.StreamingQueryWrapper' with args=()
[2020-05-13T11:07:24.4102208Z] [SANKETSHAH-HP] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'awaitTermination' failed for class '15' when called with no arguments
[2020-05-13T11:07:24.4103844Z] [SANKETSHAH-HP] [Error] [JvmBridge] org.apache.spark.sql.streaming.StreamingQueryException: Input byte array has wrong 4-byte ending unit
=== Streaming Query ===
Identifier: [id = 73ea0a6c-6cb5-41e9-921f-d6fdb177eaca, runId = d9694079-294a-42e3-a531-5b30bbe1cf3b]
Current Committed Offsets: {}
Current Available Offsets: {}

Current State: INITIALIZING
Thread State: RUNNABLE
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:297)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit
        at java.util.Base64$Decoder.decode0(Unknown Source)
        at java.util.Base64$Decoder.decode(Unknown Source)
        at java.util.Base64$Decoder.decode(Unknown Source)
        at org.apache.spark.eventhubs.EventHubsUtils$.decrypt(EventHubsUtils.scala:158)
        at org.apache.spark.eventhubs.EventHubsConf$.toConf(EventHubsConf.scala:553)
        at org.apache.spark.sql.eventhubs.EventHubsSource.<init>(EventHubsSource.scala:77)
        at org.apache.spark.sql.eventhubs.EventHubsSourceProvider.createSource(EventHubsSourceProvider.scala:82)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:255)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:88)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:85)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:79)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:85)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:258)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:257)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:328)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:247)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:83)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:65)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:269)
        ... 1 more

[2020-05-13T11:07:24.4458731Z] [SANKETSHAH-HP] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'awaitTermination' failed for class '15' when called with no arguments
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
Unhandled exception. System.Exception: JVM method execution failed: Nonstatic method 'awaitTermination' failed for class '15' when called with no arguments
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmObjectReference.Invoke(String methodName, Object[] args)
   at Microsoft.Spark.Sql.Streaming.StreamingQuery.AwaitTermination()
   at IotHubConsumer.Program.Main(String[] args) in E:\self-training\spark\IotHubConsumer\Program.cs:line 88

E:\self-training\spark\IotHubConsumer\bin\Debug\netcoreapp3.1>
sanket-t-shah picture sanket-t-shah on 13 May 2020

Hi @imback82 / @joperezr - just wanted to check with you about any solution you have for the problem am facing or if you can connect me with someone. Would be highly obliged.

sanket-t-shah picture sanket-t-shah on 13 May 2020

The following is happening from https://github.com/Azure/azure-event-hubs-spark. You may want to ask in that repo?

Caused by: java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit
        at java.util.Base64$Decoder.decode0(Unknown Source)
        at java.util.Base64$Decoder.decode(Unknown Source)
        at java.util.Base64$Decoder.decode(Unknown Source)
        at org.apache.spark.eventhubs.EventHubsUtils$.decrypt(EventHubsUtils.scala:158)
        at org.apache.spark.eventhubs.EventHubsConf$.toConf(EventHubsConf.scala:553)
        at org.apache.spark.sql.eventhubs.EventHubsSource.<init>(EventHubsSource.scala:77)
        at org.apache.spark.sql.eventhubs.EventHubsSourceProvider.createSource(EventHubsSourceProvider.scala:82)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:255)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:88)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:85)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:79)
imback82 picture imback82 on 14 May 2020

@joperezr, @imback82 - I recognized the problem. Event Hub Connection string needs to be encrypted with latest version. I don't have a sample for the same. Would it be kindly possible for you to give one?

Also, I had to move version back, and had to copy .jar files to spark\jars folder. Would there be any possibility to have dependencies installed / downloaded automatically? Pardon me if this is basic question, as I belong primarily to .NET world.

Here is the list of .jar files I placed to get code working.

scala-java8-compat_2.11-0.9.0.jar
proton-j-0.31.0.jar
qpid-proton-j-extensions-1.2.0.jar
azure-eventhubs-2.3.2.jar
azure-eventhubs-spark_2.11-2.3.13.jar
microsoft-spark-2.4.x-0.11.0.jar

Here is the command I used for running application. @MaxKhlupnov - did you also copied .jar files or were they downloaded automatically?

spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.13 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.11.0.jar IotHubConsumer.exe

sanket-t-shah picture sanket-t-shah on 15 May 2020

Here is the list of
Here is the command I used for running application. @MaxKhlupnov - did you also copied .jar files or were they downloaded automatically?

Hello!
Jars was uploaded by spark-submit. I don't copy jars into spark folder.

MaxKhlupnov picture MaxKhlupnov on 16 May 2020

Thanks, @MaxKhlupnov for a super quick response. Closing this for now as obtained solution by lot of experiments.

sanket-t-shah picture sanket-t-shah on 17 May 2020

FYI, documentation for connecting to Azure Event hubs now available here. Closing this issue as it is believed it has been resolved. Please feel free to re-open it if the issue persists and/or there are any new updates. Thank you!

Niharikadutta picture Niharikadutta on 14 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

WhitWaldo picture WhitWaldo  路  6Comments
jonathantonic picture jonathantonic  路  9Comments
voltcode picture voltcode  路  6Comments
viskumar-microsoft picture viskumar-microsoft  路  7Comments
bamurtaugh picture bamurtaugh  路  5Comments