Azure-sdk-for-java: [BUG] Cant use azure-storage-file-datalake in Spring Framework

[nandorrb]

Describe the bug
I am trying to implement azure-storage-file-datalake into a Spring Framework however DataLakeServiceClientBuilder crashes with the following error:

***Exception or Stack Trace***
***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:101)

The following method did not exist:

    reactor.netty.http.client.HttpClient.doAfterResponseSuccess(Ljava/util/function/BiConsumer;)Lreactor/netty/http/client/HttpClient;

The method's class, reactor.netty.http.client.HttpClient, is available from the following locations:

    jar:file:/Users/fernandorubio/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/0.8.6.RELEASE/4afa198f187a2eef5361a28cbaf3b63d181cb00c/reactor-netty-0.8.6.RELEASE.jar!/reactor/netty/http/client/HttpClient.class

It was loaded from the following location:

    file:/Users/fernandorubio/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/0.8.6.RELEASE/4afa198f187a2eef5361a28cbaf3b63d181cb00c/reactor-netty-0.8.6.RELEASE.jar


Action:

Correct the classpath of your application so that it contains a single, compatible version of reactor.netty.http.client.HttpClient

To Reproduce
Try to implement azure-storage-file-datalake in Spring Framework

Code Snippet

import com.fasterxml.jackson.databind.ObjectMapper
import org.slf4j.LoggerFactory
import org.springframework.boot.context.properties.ConfigurationProperties
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.context.annotation.Primary
import org.springframework.context.annotation.Profile
import org.springframework.core.env.Environment
import java.net.InetSocketAddress
import com.azure.core.http.HttpClient
import com.azure.core.http.ProxyOptions
import com.azure.core.http.netty.NettyAsyncHttpClientBuilder
import com.azure.identity.ClientSecretCredential
import com.azure.identity.ClientSecretCredentialBuilder
import com.azure.storage.file.datalake.DataLakeServiceClientBuilder

@Configuration
@ConfigurationProperties(prefix = "azureconfiguration")
class AzureStorageConfig(
    var endpoint: String = "",
    var containerorfilesystem: String = "",
    var tenantid: String = "",
    var clientid: String = "",
    var clientsecret: String = ""

) {
    companion object {
        private val logger = LoggerFactory.getLogger(AzureStorageConfig::class.java)
    }

    @Bean
    fun azureStorageControllerLocal(env: Environment, objectMapper: ObjectMapper): AzureStorageController {
        logger.info("setting up Azure Local config")

        return configureAzureStorage(objectMapper, getLocalHttpClient())
    }

    @Profile("cloud")
    @Primary
    @Bean
    fun azureStorageControllerCloud(env: Environment, objectMapper: ObjectMapper): AzureStorageController {
        logger.info("setting up Azure Cloud config")

        return configureAzureStorage(objectMapper, getCloudHttpClient())
    }

    private fun configureAzureStorage(objectMapper: ObjectMapper, httpClient: HttpClient?): AzureStorageController {

        if (configurationIsEmpty()) {
            return AzureStorageFakeController()
        }

        val clientSecretCredential: ClientSecretCredential = ClientSecretCredentialBuilder()
            .clientId(clientid)
            .clientSecret(clientsecret)
            .tenantId(tenantid)
            .build()

        val builder = DataLakeServiceClientBuilder()

        try {
            val dataLakeClient = if (httpClient == null) {

                builder.credential(clientSecretCredential)
                    .endpoint(endpoint)
                    .buildClient()
            } else {
                builder.credential(clientSecretCredential)
                    .endpoint(endpoint)
                    .httpClient(httpClient)
                    .buildClient()
            }
            val filesystemClient = dataLakeClient.getFileSystemClient(containerorfilesystem)

            logger.info("Returned freshly created AzureStorageContainerController")
            return AzureStorageContainerController(
                objectMapper,
                filesystemClient
            )
        } catch (e: Exception) {
            logger.error("AzureStorageContainerController ERROR!")
            logger.error(e.localizedMessage)
        }
        return AzureStorageFakeController()
    }

    private fun getLocalHttpClient(): HttpClient? { val authUser = System.getProperty("https.proxyUser", "")
        val authPassword = System.getProperty("https.proxyPassword", "")
        val host = System.getProperty("https.proxyHost", "")
        val port = System.getProperty("https.proxyPort", "80").toInt()

        val proxyOptions = ProxyOptions(
            ProxyOptions.Type.HTTP,
            InetSocketAddress(host, port)
        )
        proxyOptions.setCredentials(
            authUser,
            authPassword
        )

        return NettyAsyncHttpClientBuilder()
          //  .proxy(proxyOptions)
            .build()
    }

    private fun getCloudHttpClient(): HttpClient? {
        val authUser = System.getProperty("https.proxyUser", "")
        val authPassword = System.getProperty("https.proxyPassword", "")
        val host = System.getProperty("https.proxyHost", "")
        val port = System.getProperty("https.proxyPort", "80").toInt()

        val proxyOptions = ProxyOptions(
            ProxyOptions.Type.HTTP,
            InetSocketAddress(host, port)
        )
        proxyOptions.setCredentials(
            authUser,
            authPassword
        )

        return NettyAsyncHttpClientBuilder()
            .proxy(proxyOptions)
            .build()
    }

    private fun configurationIsEmpty(): Boolean {
        if (endpoint.isEmpty() ||
            containerorfilesystem.isEmpty() ||
            tenantid.isEmpty() ||
            clientid.isEmpty() ||
            clientsecret.isEmpty()
        ) {
            logger.info("Missing credentials in azure_reporting_data_credentials service instance")
            return true
        }
        return false
    }
}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: Mac OS big sur 11.0.1
• IDE : Intellij IDEA
• Version of the Library used
  -Spring: :: Spring Boot :: (v2.1.4.RELEASE)
  Additional context
  Gradle Configuration Snippet

 azureCoreHttpNettyVersion = "1.7.0"
        azureIdentityVersion = "1.2.0"
        azureStorageFileDataLakeVersion = "12.3.0"
     implementation("com.azure:azure-storage-file-datalake:${azureStorageFileDataLakeVersion}")
    implementation("com.azure:azure-identity:${azureIdentityVersion}")

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [x] Bug Description Added
• [x] Repro Steps Added
• [x] Setup information Added

[alzimmermsft]

Hi @nandorrb, unfortunately Spring Boot 2.1.x versions resolve to using Reactor 3.2.x and Reactor Netty 0.8.x versions while Azure Storage Datalake, and other SDKs underneath the com.azure group, use Reactor 3.3.x and Reactor Netty 0.9.x versions. Between these versions of Reactor and Reactor Netty there are incompatibilities leading to this issue you see. Upgrading to Spring Boot 2.2.x or 2.3.x should resolve the Reactor and Reactor Netty incompatibilities.

cc: @gapra-msft @rickle-msft @chenrujun @saragluna

[nandorrb]

Hello Mr @alzimmermsft thanks for your help. I updated Spring to version 2.4.1 and now I get this Issue

java.lang.NoClassDefFoundError: com/nimbusds/oauth2/sdk/http/CommonContentTypes
    at com.microsoft.aad.msal4j.TokenRequestExecutor.createOauthHttpRequest(TokenRequestExecutor.java:46) ~[msal4j-1.8.0.jar:1.8.0]
    at com.microsoft.aad.msal4j.TokenRequestExecutor.executeTokenRequest(TokenRequestExecutor.java:36) ~[msal4j-1.8.0.jar:1.8.0]
    at com.microsoft.aad.msal4j.AbstractClientApplicationBase.acquireTokenCommon(AbstractClientApplicationBase.java:119) ~[msal4j-1.8.0.jar:1.8.0]
    at com.microsoft.aad.msal4j.AcquireTokenByAuthorizationGrantSupplier.execute(AcquireTokenByAuthorizationGrantSupplier.java:63) ~[msal4j-1.8.0.jar:1.8.0]
    at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:59) ~[msal4j-1.8.0.jar:1.8.0]
    at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:17) ~[msal4j-1.8.0.jar:1.8.0]
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1700) ~[na:na]
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java) ~[na:na]
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) ~[na:na]
    at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:290) ~[na:na]
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java) ~[na:na]
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[na:na]
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[na:na]
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[na:na]
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) ~[na:na]
Caused by: java.lang.ClassNotFoundException: com.nimbusds.oauth2.sdk.http.CommonContentTypes
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[na:na]
    ... 15 common frames omitted

[alzimmermsft]

Hi @nandorrb, do you mind trying with Spring version 2.2.x or 2.3.x, your preference, as Spring version 2.4.x took on new dependencies of Reactor and Reactor Netty which aren't compatible with the versions the Azure SDKs use. Spring is using Reactor 3.4.x and Reactor Netty 1.0.x which has backwards compatibility breaking changes with Reactor 3.3.x and Reactor Netty 0.9.x that we are using in the SDKs. We are actively working on an update to bring support for Spring 2.4.x.

[nandorrb]

Thanks for your reply @alzimmermsft ,
do you know when will you update the library in order to be compatible to Spring 2.4.x?
There are some breaking changes between Spring 2.3.x and 2.4.x related to hateoas library so I would like to be sure when to make the update.

[chenrujun]

Hi, @nandorrb , we will release azure-spring-boot-starter-xxx GA version before 2021.01.31

[saragluna]

FYI, all the Azure Spring starters compatible with Spring Boot 2.4 have been released.