Elasticsearch: Add support for Azure Blob Storage

[casperOne]

With the emphasis of allowing ElasticSearch to store data "in the cloud", it would be great if ElasticSearch could store in more than just two clouds (currently, S3 and RackSpace).

Originally, it seems that there was a general cloud module which used jCloud to provide access to various clouds, but that support has been pulled.

However, Microsoft now has a Java-native Azure API (http://www.windowsazure4j.org/) which can be used to access Azure Blob Storage.

Additionally, to help progress this issue along, I have an Azure account that I would be _more_ than happy to contribute to the cause (with a 10GB limit).

Note that this is _not_ a request for discovery between Azure instances, but rather, just to use Azure storage. I'll file discovery separately when the need arises. =)

[simnova]

I'm interested in this as well.

[flippyhead]

As am I.

[flippyhead]

Also, I'm curious, could S3 storage be used from instancing running on Azure? Any downside to doing this?

[peteot]

Any update? What about using Windows Azure Drive (XDrive)?

The direct use of the blob API sounds cool, but there should also be some way to use Azure blobs via the XDrive which lets you mount the blob as a drive. However, I have only seen examples accessing it with .NET code.

Interested to hear if anyone has ElasticSearch running on Windows Azure....

[casperOne]

@flippyhead You _could_ do it, but given that the machines would not be co-located, the performance would probably suffer.

[jakenuts]

I'd definitely be interested as well, perhaps we can pool some budget to encourage someone familiar with Azure to implement a good solution?

[dadoonet]

Heya!

We released elasticsearch-cloud-azure plugin.
We can now close this issue I guess.

Note that snapshot and restore on Azure Storage will come pretty soonish.

Thanks @spinscale for pointing me to this issue :-)

[netgh0st]

The elasticsearch-cloud-azure is a plugin to discover nodes using Azure environment information. The request was for a completely different feature - supporting native Azure blob storage instead of having to use blob mapped as a VHD (which is yet another level of abstraction on top of IO, with additional redundant triplication of data, because ES triplicates shards while Azure blob storage also triplicates data on the backend).