Helm-charts: How to resize ES storage volume but keeping old data

[milijander88]

Hi all,
I'll need a help with resizing a storage for elasticsearch. We have deployed elasticserach with helm but we have an issue with resizing the storage. This are the resources that we deployed with helm:

> pod/sandbox-elasticsearch-test2-0                             1/1     Running   0          63m
> pod/sandbox-elasticsearch-test2-1                             1/1     Running   0          63m
> 
> service/sandbox-elasticsearch-test2                    ClusterIP   172.20.15.222    <none>        9200/TCP,9300/TCP            63m
> service/sandbox-elasticsearch-test2-headless           ClusterIP   None             <none>        9200/TCP,9300/TCP            63m
> statefulset.apps/sandbox-elasticsearch-test2                     2/2     63m

We tried to resize the storage by changing nodeGroup and masterService and we ended with new nodes but old one are deleted, only what is left is old storage disks :

sandbox-elasticsearch-test-sandbox-elasticsearch-test-0                                    Bound    pvc-f45fe905-7f09-43aa-b4a5-5ef8124ad69e   120Gi      RWO            gp2-csi        18h
sandbox-elasticsearch-test-sandbox-elasticsearch-test-1                                    Bound    pvc-5ed567f6-0aa4-4ebd-8162-42f7fa27ee0d   120Gi      RWO            gp2-csi        18h
sandbox-elasticsearch-test2-sandbox-elasticsearch-test2-0                                  Bound    pvc-46a87003-3c90-453c-a3dc-de65877d60b7   130Gi      RWO            gp2-csi        64m
sandbox-elasticsearch-test2-sandbox-elasticsearch-test2-1                                  Bound    pvc-334cfc32-a376-469a-ae9d-e7e0e2cf3c7a   130Gi      RWO 

I think data is still on old disks but how to migrate old data to new nodes ? After the attempt to resize the storage we ended up with a brand new cluster and old data are lost .

[ebuildy]

Hello,

You changed cluster name, so PVC name changed , thats why there are new PVs.

This is not really related to this Helm chart, but did you read https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/?

If you cant resize, you can consider to scale out, by creating more replicas, something like that:

1. Remove existing StatefulSet without removing pods (kubectl delete sts XXX --cascade=false)
2. Set bigger size in Helm values, then deploy values (this will not change existing PVC)
3. Increase replicas (this will create new pod with the right PVC size)
4. Move all data to new nodes (using https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cluster.html#cluster-shard-allocation-filtering)
5. Remove old pod / PVC manually (see https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/#statefulset-considerations)

I am not sure about "4", but this is the idea.

[jmlrt]

Hi @milijander88,
The good news is that PVC are never deleted by this chart so changing back nodeGroup and masterService to previous values should be enough to re-use the disks with datas.

As for the procedure to resize disk, changing volumeClaimTemplate.resources.requests.storage value would fail with Updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden error because this field is immutable at Kubernetes level. That's why even @ebuildy procedure would fail at step 1.

The recommended workaround for Elasticsearch chart is using a new chart release, with different storage size to replace the existing one.:

1. Install a new deployment of the helm chart, with nodeGroup set to a unique value, such as data-1 and the appropriate PVC claim size.
2. Ensure that these new ES nodes have joined the cluster.
3. Trigger Elasticsearch to move all data off of the data nodes. This can be achieved by using Cluster-level shard allocation filtering, using the IP's of the 3 original data nodes.
4. Once all data has been migrated off of the original data nodes, then delete that helm deployment.

(source: https://github.com/elastic/helm-charts/issues/647#issuecomment-636778450)

This is related to #772, #893 and #647.

[Xat59]

@jmlr I am now able to resize the pvc using a second release in parallel, wait for the data sync, then removing the original.

But, doing that, it also renames the elasticsearch service regarding $clusterName-$nodeGroup . And this make the service unavailable because it has been renamed.

Any idea how to do it properly ?

EDIT : actually I've resolve it by creating the multi chart release, at least one for the master and one for the data, as described here : https://github.com/elastic/helm-charts/tree/master/elasticsearch/examples/multi

[ebuildy]

Hi @milijander88,
The good news is that PVC are never deleted by this chart so changing back nodeGroup and masterService to previous values should be enough to re-use the disks with datas.

As for the procedure to resize disk, changing volumeClaimTemplate.resources.requests.storage value would fail with Updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden error because this field is immutable at Kubernetes level. That's why even @ebuildy procedure would fail at step 1.

The recommended workaround for Elasticsearch chart is using a new chart release, with different storage size to replace the existing one.:

1. Install a new deployment of the helm chart, with nodeGroup set to a unique value, such as data-1 and the appropriate PVC claim size.
2. Ensure that these new ES nodes have joined the cluster.
3. Trigger Elasticsearch to move all data off of the data nodes. This can be achieved by using Cluster-level shard allocation filtering, using the IP's of the 3 original data nodes.
4. Once all data has been migrated off of the original data nodes, then delete that helm deployment.

(source: #647 (comment))

This is related to #772, #893 and #647.

I forgot to mention: "remove StatefulSet before" (this will not remove associated pods), I used it many-times without data loss

[dhinakar2707]

@ebuildy Please can you once again share me the detailed steps to re-size the elastic search disk online.

Even as part of new project I have deployed elasticsearch with latest helm charts. Now I have to take it to production, but i'm afraid of re-sizing disk without downtime or data loss.
Your help would be much appriciated, so that i can test the upgradation before i move into production.