Elasticsearch: Deleting S3 snapshot returns 404

[Hexta]

Elasticsearch version: 5.2.2

Plugins installed: [repository-s3, discovery-file, x-pack]

JVM version: openjdk 8u111

OS version: Ubuntu 16.04

Description of the problem including expected versus actual behavior:
Deleting S3 snapshot taken about a minute and completed with error 404. But actually, the snapshot has been removed from S3.

Steps to reproduce:

1. Create snapshot 01/2017-03-27-01
2. Create snapshot 01/2017-03-27-02
3. Delete snapshot 01/2017-03-27-01

Provide logs (if relevant):

{
  "error": {
    "root_cause": [
      {
        "type": "snapshot_missing_exception",
        "reason": "[01:2017-03-27-01] is missing"
      }
    ],
    "type": "snapshot_missing_exception",
    "reason": "[01:2017-03-27-01] is missing"
  },
  "status": 404
}

[abeyad]

@Hexta is there a stack trace in the log files that you can share?

[Hexta]

@abeyad I don't see any stack in the log file.

[abeyad]

@Hexta can you consistently reproduce this? are you sure the snapshot wasn't deleted first and then the command was re-run again?

if the SnapshotMissingException was thrown after confirming the snapshot exists before proceeding with the delete, the exception should show up in the logs

[Hexta]

@abeyad I performed queries to ES from Kibana Console. Kibana accesses ES through HAProxy with defined inactivity timeout (1m).
Delete snapshot request took more than 1m and HAProxy closed connection which caused retrying request by Kibana Console.

[Hexta]

Is it OK that delete snapshot request doesn't run asynchronously?

[abeyad]

My guess is that the request succeeded the first time, but for some reason there was a problem getting the response back from the proxy to kibana. You could turn on debug logging for org.elasticsearch.snapshots and org.elasticsearch.repositories and we could see if there are any particular problems in ES with regards to deleting the snapshot.

Is it OK that delete snapshot request doesn't run asynchronously?

The snapshot deletion executes asynchronously, but from an API perspective, the HTTP response waits until the snapshot deletion operation completes (either successfully or unsuccessfully or times out) before returning a response. Not sure if that is your question?

[Hexta]

My guess is that the request succeeded the first time, but for some reason there was a problem getting the response back from the proxy to kibana.

I inspected requests to ES HTTP service (using tcpflow). I saw that DELETE request performed twice - first immediately after starting the query in Kibana Console, and then after HAProxy timeout.

After setting HAProxy timeout to 5 minutes, 404 errors have gone away. But such configuration is vulnerable to slow DoS attack.

The snapshot deletion executes asynchronously, but from an API perspective, the HTTP response waits until the snapshot deletion operation completes (either successfully or unsuccessfully or times out) before returning a response.

For ES client delete snapshot query acts like synchronous - DELETE request completes only after deleting all snapshot files. This requests can take too much time, and it would be great to run it in the same manner as creating snapshot request (return acknowledgment and provide /_status for checking operation progress).

[clintongormley]

We could add the wait_for_completion parameter to delete snapshot

[Hexta]

@clintongormley I think that would be great.

[tlrx]

We talked about this issue today with @ywelsch. While we understand the need to track snapshot deletions and/or wait for them to complete, we didn't feel like it is a high requested feature. Considering that implementing the feature of tracking snapshot deletion progress is not easy, we'd like to close this issue to indicate that we don't have immediate plan to work on it. Please, feel free to add any comment and +1.

[regnarock]

+1

[jmleoni]

+1

[gmhabchi]

+1