Elasticsearch: Azure snapshot request returns 500 error: "null_pointer_exception"

[tdoman]

NEST 6.0.0-beta1
ES 6.1.1
repository-azure-6.1.1

I am able to successfully create the azure repository but when I attempt to take the snapshot like so:

PUT <server>/_snapshot/es-2018-01/snapshot-2018-01-17?pretty=true&wait_for_completion=false
{
  "include_global_state": true
}

I get the following response:

Response: {
  "error" : {
    "root_cause" : [
      {
        "type" : "null_pointer_exception",
        "reason" : null
      }
    ],
    "type" : "null_pointer_exception",
    "reason" : null
  },
  "status" : 500
}

Here's the log:

[2018-01-17T18:23:58,846][WARN ][r.suppressed             ] path: /_snapshot/es-2018-01/snapshot-2018-01-17, params: {pretty=true, repository=es-2018-01, wait_for_completion=false, snapshot=snapshot-2018-01-17}
java.lang.NullPointerException: null
    at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.generateOperationContext(AzureStorageServiceImpl.java:182) ~[?:?]
    at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.lambda$listBlobsByPrefix$10(AzureStorageServiceImpl.java:312) ~[?:?]
    at org.elasticsearch.cloud.azure.blobstore.util.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:55) ~[?:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
    at org.elasticsearch.cloud.azure.blobstore.util.SocketAccess.doPrivilegedVoidException(SocketAccess.java:54) ~[?:?]
    at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.listBlobsByPrefix(AzureStorageServiceImpl.java:309) ~[?:?]
    at org.elasticsearch.cloud.azure.blobstore.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:118) ~[?:?]
    at org.elasticsearch.cloud.azure.blobstore.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:123) ~[?:?]
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:769) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:747) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:599) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.snapshots.SnapshotsService.createSnapshot(SnapshotsService.java:236) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.masterOperation(TransportCreateSnapshotAction.java:82) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.masterOperation(TransportCreateSnapshotAction.java:41) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167) ~[elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) [elasticsearch-6.1.1.jar:6.1.1]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.1.jar:6.1.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

[dadoonet]

@tdoman Thanks for reporting. Could you share what you have in elasticsearch.yml (please replace any secret by XXXXX)?
Also is that a fresh new installation? How did you create the repository?

[dadoonet]

Looking at the code, I see where this NPE is coming from but it seems to indicate that the Azure client can not be found.

[tdoman]

@dadoonet Ok, that'd be a good message in lieu of the NPE but I am able to successfully use the keys, etc. in other tools to access the blob storage. Also, I had been successfully doing this w/ ES 6.0 so I'm not sure what changed. Here's my repository creation request:

PUT _snapshot/es-2018-01?pretty=true
{
  "settings": {
    "container": "elasticsearch-backups",
    "base_path": "es-2018-01",
    "compress": true,
    "chunk_size": "64mb"
  },
  "type": "azure"
}

Here's the elasticsearch.yml config:

cloud:
    azure:
        storage:
            degreed_account:
                account: <my_account>
                key: <my_key>

Oh, and yes it's a fresh installation. I've simply been doing reindex from remote from my ES 1.7.5 cluster but it's not an upgrade of any kind.

[dadoonet]

I believe that you are seeing some deprecation notice may be? Anyway, to use the plugin in 6.1, you have to follow the guide here: https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure-usage.html

Define the key and secret:

bin/elasticsearch-keystore add azure.client.default.account
bin/elasticsearch-keystore add azure.client.default.key

Remove credentials from elasticsearch.yml.

Then you need to restart the node and you should be good.

Otherwise, you could may be do something like:

_snapshot/es-2018-01?pretty=true
{
  "settings": {
    "client": "degreed_account",
    "container": "elasticsearch-backups",
    "base_path": "es-2018-01",
    "compress": true,
    "chunk_size": "64mb"
  },
  "type": "azure"
}

That "might work" but I'm unsure though.

[tdoman]

@dadoonet Ah, ok I see, the configuration methodology changed between 6.0 and 6.1. Ok, I'll try 6.1 approach out today and report back. I did try your "might work" suggestion and it didn't. :) Thank you very much for the prompt response, I really appreciate it.

[dadoonet]

Yeah it changed but it is supposed to be backward compatible. That's why I marked this as a bug and also because a NPE is always bad. 😉

[tdoman]

@dadoonet Yep, purely configurational, thanks again for the support David.

[tdoman]

Sorry, I re-opened it so the NPE can be addressed.

[dadoonet]

Thanks for confirming. I'm planning to address the issues indeed (backward comp and NPE). Let's keep that open.

[ywelsch]

Closed by #28769