Helm-charts: Elasticsearch cluster backup/restore

[rimusz]

Is there Elasticsearch cluster backup/restore supported?

[Crazybus]

Yes, although there is no "helm" specific configurations to enable snapshotting so you just enable it as you normally would following the snapshot docs. Instead you need to install whichever snapshot plugin you need into the docker image and add the configuration and keystore values yourself. You can see how to do that in the FAQ.

For actually running the snapshots you can use a tool like curator to automate the snapshot and restore process.

There are plans to manage the automation of snapshots inside Elasticsearch with Snapshot Lifecycle Management which would mean not needing to also deploy curator to manage them.

To make this whole process easier I can imagine the helm chart having a way to automate installing cluster settings and a way to automatically add secrets into the keystore would make this process a lot easier. But for now this is the way to do it (and also what we do in my team).

[rimusz]

@Crazybus thanks for your answers.
Can the backup process examples doc be added to the helm chart, so anybody else can read it?

[Crazybus]

Great idea. I just opened https://github.com/elastic/helm-charts/pull/146 to add it.

[rimusz]

ok, I tried to add gcs plugin, got weird warning messages:
$ docker build -t elasticsearch-gcs:7.1.1 --build-arg elasticsearch_version=7.1.1 .:
Dockerfile:

ARG elasticsearch_version
FROM docker.elastic.co/elasticsearch/elasticsearch:${elasticsearch_version}

RUN bin/elasticsearch-plugin install --batch repository-gcs

Output:

Sending build context to Docker daemon  2.048kB
Step 1/3 : ARG elasticsearch_version
Step 2/3 : FROM docker.elastic.co/elasticsearch/elasticsearch:${elasticsearch_version}
 ---> b0e9f9f047e6
Step 3/3 : RUN bin/elasticsearch-plugin install --batch repository-gcs
 ---> Running in ceab4119ef0a
-> Downloading repository-gcs from elastic
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/usr/share/elasticsearch/lib/tools/plugin-cli/bcprov-jdk15on-1.61.jar) to constructor sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission setFactory
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-gcs
Removing intermediate container ceab4119ef0a
 ---> 62f332bbcb24
Successfully built 62f332bbcb24
Successfully tagged elasticsearch-gcs:7.1.1

[rimusz]

Would make more sense that ES docker images comes with s3, gcs and other plugins preinstalled?

[Crazybus]

ok, I tried to add gcs plugin, got weird warning messages:

Here is the Elasticsearch issue for it: https://github.com/elastic/elasticsearch/issues/41478 and it appears that it has been fixed in this PR https://github.com/elastic/elasticsearch/pull/41620 which has been tagged to released in 7.2.0. So it looks like it is an actual warning, but fine as long as you accept it (which --batch mode does automatically).

Would make more sense that ES docker images comes with s3, gcs and other plugins preinstalled?

I'd like it very much too! I know it has been discussed in the past but I can't seem to find any of the issues. The dockerfile itself recently moved from https://github.com/elastic/elasticsearch-docker/ into https://github.com/elastic/elasticsearch. I'm going to ping some of the Elasticsearch folks and see what the latest status is on doing that. If there isn't anything existing I'll open an issue in the Elasticsearch repo to get the discussion going again.

A bit more detailed instruction on how to do the whole setup would be very welcome, specially for new Elasticsearch users :)

Were you able to get snapshots working in the end? If so letting me know which parts were missing or you had trouble with would be very helpful!

[rimusz]

thanks the updates on gcs plugin weird warning and docker images comes with s3, gcs ... 👍

I was able to get snapshots working, but as for new ES user it wasn't easy as docs are all over the place.
So what I did:
1) made custom docker image as per https://github.com/elastic/helm-charts/tree/master/elasticsearch#how-to-install-plugins
2) Create a keystore file

docker run -v "$PWD:/usr/share/elasticsearch/tmp" docker.elastic.co/elasticsearch/elasticsearch:7.1.1 \
  bash -c "bin/elasticsearch-keystore create && bin/elasticsearch-keystore add-file gcs.client.es_backup.credentials_file tmp/gcloud_es-backup_sa_key.json && cp config/elasticsearch.keystore tmp/elasticsearch.keystore"
#

then https://github.com/elastic/helm-charts/tree/master/elasticsearch#how-to-use-the-keystore
3) Registered snapshot repo (running a pod in the k8s cluster):

        curl -X PUT "$ELASTIC_URL/_snapshot/es_backup" -H 'Content-Type: application/json' -d '{
          "type": "gcs",
          "settings": {
            "bucket": "'${GCS_BUCKET}'",
            "client": "es_backup"
          }
        }'

4) and have k8s cronjob (simple helm chart) running to trigger snapshots with:

curl -X PUT '{{ .Values.elasticsearch.url }}/_snapshot/es_backup/snapshot_{{ randAlpha 6 | lower }}?wait_for_completion=true'

I have no idea how to use curator so went that way ^^^.

I provided more detailed steps so maybe it can be used as some doc for easier ES backup story.

@Crazybus I see you have PR #154 for PoC of Keystore integration, if that gets working and we can have ES docker images with s3, gcs and other plugins preinstalled, then would be much easier to setup snapshotting.

[Crazybus]

I have no idea how to use curator so went that way ^^^.

So the one downside with the approach you have taken is that you don't have an easy way to rotate the snapshots. To give you an idea of how this looks with curator this is what my configuration looks like:

actions:
  1:
    action: snapshot
    description: 'Take a snapshot'
    options:
      repository: 'default'
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
      name: 
    filters:
    - filtertype: none
  2:
    action: delete_snapshots
    description: 'Clean up snapshots'
    options:
      repository: 'default'
      retry_interval: 120
      retry_count: 3
      ignore_empty_list: True
    filters:
    - filtertype: age
      source: name
      direction: older
      prefix: 
      timestring: '%Y%m%d%H%M%S'
      unit: days
      unit_count: 10
      exclude: False

This is setup to run hourly and keeps 10 days of snapshots.

I was able to get snapshots working, but as for new ES user it wasn't easy as docs are all over the place.

I 100% agree which is why I hope https://github.com/elastic/elasticsearch/issues/38461 is going to make this a lot lot easier. When combined with the keystore integration I'm working on in #154 the process is going to be a lot easier.

if that gets working and we can have ES docker images with s3, gcs and other plugins preinstalled

I'm going to follow up on this again. I too would love this since the GCS snapshot plugin is the only thing we install for our own Elasticsearch clusters.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This issue has been automatically closed because it has not had recent activity since being marked as stale.

[harsh4870]

i have tried adding GCS service account key in keystone but getting error

created kubernetes secret and added file into it and added secret into keystore

but pod are crashing also my i am using custom docker image with plugin installed but getting error like :

csearch", "node.name": "elasticsearch-master-0",  "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [gcs_backup_key.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.2.jar:7.3.2]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.2.jar:7.3.2]",
"Caused by: java.lang.IllegalArgumentException: unknown secure setting [gcs_backup_key.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:531) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:357) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.2.jar:7.3.2]",
"... 6 more"] }

created secret using

kubectl create secret generic gcs-backup-key --from-file=gcs_backup_key.json=gcs_backup_key.json

here

keystore:
  - secretName: gcs-backup-key

Update

Image running and ready 1/1.

secretMounts:
  - name: gcs-backup-key
    secretName: gcs-backup-key
    path: /usr/share/elasticsearch/config/gcs_backup_key.json
    subPath: gcs_backup_key.json

please correct me if missing anything or wrong way

getting

{"error":{"root_cause":[{"type":"blob_store_exception","reason":"Unable to check if bucket [elasticsearch-backup-manvar-co] exists"}],"type":"repository_exception","reason":"[backup] cannot create blob store","caused_by":{"type":"blob_store_exception","reason":"Unable to check if bucket [elasticsearch-backup-manvar-co] exists","caused_by":{"type":"security_exception","reason":"access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")"}}},"status":500}

i want to set cronjob for setting regular backup let me know if missing anything thanks in advance for help.

[denismatyu]

@harsh4870
-XPOST 127.0.0.1:9200/_nodes/reload_secure_settings
Try this one
How did you fix the unknown secure setting?

[christidis]

Same issue with @harsh4870 when I try enabling snapshoting in gcs.
I have tried using the json file directly I have also tried creating the keystore file offline in a docker container like @rimusz did in his example.
But I always end up in a java.lang.IllegalArgumentException: unknown secure setting [elasticsearch.keystore] fatal error.

Snapshotting instructions in the README are not very helpful especially obscure step 2:

Add any required secrets or credentials into an Elasticsearch keystore following the how to use the keystore guide.

What are the "required secrets or credentials" for GCS plugin to work? Does it need a JSON SA file or a pre-generated keystore file or something else?

[harsh4870]

@christidis you can please follow this...https://medium.com/@harsh.manvar111/backup-and-restore-elasticsearch-using-gcs-fa70003b9b5e

if you are still facing issues please let me know i will share my files and configuration and check-in more depth. Sorry again little busy right now so.

[christidis]

@harsh4870 thanks! this small detail from your post in Medium helped me make some progress

kubectl create secret generic gcp-backup-key --from-file=gcs.client.default.credentials_file=./key.json

as I was creating the secret using --from-file=key.json and I was trying to map the keystore with gcs.client.default.credentials_file like this example.