Elasticsearch: Use Azure Storage SDK 12 in Azure Repository plugin

[russcam]

Describe the feature:

repository-azure plugin uses the azure-storage SDK 8 to interact with Azure Storage:

https://github.com/elastic/elasticsearch/blob/349d2ce153d4be78bf0d8ae2472bfc88e18530c1/plugins/repository-azure/build.gradle#L26

SDK 8 is legacy and superseded by Azure Storage SDK 11, which offers async IO for all operations. From conversations with folks at Microsoft, SDK 11 yields marked throughput improvements, which would be beneficial for snapshotting in Azure.

SDK 11 is a complete overhaul of the library however, with significant changes from SDK 8. This may require changes to the public API surface of repository-azure plugin.

/cc @elastic/cloud-azure

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

Just a note on this:
I already looked into this a bit. Unfortunately, as it stands right now it's pretty hard to make the SDK v11 work in our security environment. It sets up its own Netty+Jackson infrastructure and doesn't have the APIs to customize it enough to make it run with our security manager. We will probably need some changes made to the Azure SDK to become compatible.

We have a TODO for this in the codebase though and in general it would be great to have this. Currently, Azure requires a lot more threads (and hence resources in general) than our other cloud repository implementations since it doesn't have bulk deletes and upgrading here would nicely improve that situation.

[tlrx]

@original-brownbear Could you create an issue in the Azure SDK repository to explain what would be needed?

[original-brownbear]

@tlrx sure will do once I have all the details of this worked out. It's on my list of TODOs :)

[original-brownbear]

I raised https://github.com/Azure/azure-sdk-for-java/issues/4009 and https://github.com/Azure/azure-sdk-for-java/issues/3999 in the Azure SDK repo now. Those two issues are definitive blockers here as far as I can see. There could be more that are hidden underneath, but it's not that likely judging by my experiments so far (turning off the security manager and hacking around the serialization issue allowed me to execute all the basic operations we run on the repository).

[fcofdez]

Finally I've been able to run some basic upload benchmarks using the latest Azure SDK:

I've run the benchmarks using an azure instance with type Standard D4s v3. According to the documentation, that instance type have an expected network bandwidth of 2000 Mbps.

The latest SDK provides a blocking and a non-blocking client, I've run the benchmarks using both clients to verify if it improves throughput.

The benchmarks try to simulate how snapshots are uploaded to the repository, it splits a file into chunks and upload those chunks independently submitting the jobs into a thread pool (for the non-blocking client we just issue the requests and wait for all of them to finish in a single thread).

Results:

Blocking client
-----------
4 iterations
6 files (composed of 7 chunks of 128 MB)
12 threads
-----------
BenchmarkReport{ avg MB/s=221.69 runs=[BenchmarkRun{MB/s=215.04}, BenchmarkRun{MB/s=224.0}, BenchmarkRun{MB/s=224.0}, BenchmarkRun{MB/s=224.0}]}

Blocking client
-----------
4 iterations
6 files (composed of 7 chunks of 128 MB)
6 threads
-----------
BenchmarkReport{avg MB/s=228.76 runs=[BenchmarkRun{MB/s=215.04}, BenchmarkRun{MB/s=233.73}, BenchmarkRun{MB/s=233.73}, BenchmarkRun{MB/s=233.73}]}

Blocking client
-----------
4 iterations
6 files (composed of 7 chunks of 128 MB)
4 threads
-----------
BenchmarkReport{avg MB/s=224.0 runs=[BenchmarkRun{MB/s=224.0}, BenchmarkRun{MB/s=224.0}, BenchmarkRun{MB/s=224.0}, BenchmarkRun{MB/s=224.0}]}

Non blocking client
-----------
4 iterations
6 files (composed of 7 chunks of 128 MB)
1 thread
-----------
BenchmarkReport{avg MB/s=228.76 runs=[BenchmarkRun{mb/s=215.04}, BenchmarkRun{MB/s=233.73}, BenchmarkRun{MB/s=233.73}, BenchmarkRun{MB/s=233.73}]}

As we can observe, either using the blocking or non-blocking client the results are close to the advertised maximum network bandwidth.

As a next step I'll check if the problems with the Security manager persist in the latest versions.

[fcofdez]

Download benchmarks

Non blocking client (Reading the entire file in the same connection and writing it sequentially to disk)
-----------
4 iterations
1 files (composed of 1 chunk of 2048 MB)
1 thread
-----------
BenchmarkReport {avg Mb/s=101.977 runs=[BenchmarkRun{MB/s=128.212}, BenchmarkRun{MB/s=91.405}, BenchmarkRun{MB/s=91.002}, BenchmarkRun{MB/s=105.301}]}

Non blocking client (Reading the entire file in the same connection and writing it sequentially to disk)
-----------
4 iterations
4 files (composed of 1 chunk of 2048 MB)
1 thread
-----------
BenchmarkReport{avg MB/s=290.462 runs=[BenchmarkRun{MB/s=296.410 }, BenchmarkRun{MB/s=266.919}, BenchmarkRun{MB/s=316.662}, BenchmarkRun{MB/s=286.281}]}

Non blocking client, SDK built-in parallel download in chunks
-----------
4 iterations
4 files (composed of 1 chunk of 2048 MB)
1 thread
-----------
BenchmarkReport {avg Mb/s=73.969 runs=[BenchmarkRun{MB/s=77.564}, BenchmarkRun{MB/s=72.501}, BenchmarkRun{MB/s=70.228}, BenchmarkRun{MB/s=76.036}]}

As we can observe there is quite a bit of overhead while using the SDK high level APIs to download a file to disk, AFAIK it splits the file into 4 MB chunks and tries to download them in parallel, but there's a high cost associated with managing so many tasks and connections that don't improve performance at all.

Additionally while downloading a file using streaming, we can observe that we get a throughput around 100 MB/s per connection, that's in line with the behavior observed in other providers.