Charts: Mongodb replicaset chart should support pre-existing PVC

[pswenson]

FEATURE REQUEST?

Mongodb replicaset chart should support pre-existing PVC

Version of Helm and Kubernetes:
1.6.4k8s, 2.7.0 helm

Which chart:
mongo replicaset

What happened:
I see no indication that the chart supports pre-existing PVC via claim name.
The mongodb chart seems to support pvc via a claim name

[naveensrinivasan]

This would be a great feature.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[sjentzsch]

/remove-lifecycle rotten

[scottcrespo]

@pswenson
@unguiculus

I think this is a very critical feature to have, as far as database management best practices is concerned. I believe the ability to utilize existing PVCs is important for 3 different scenarios: (1) disaster recovery, (2) rapid horizontal scaling, and (3) major volume changes.

Perhaps specifying existing PVCs can occur on helm install and it would be really nice if it could also performed with helm upgrade --set existingPVC=foo-pvc-claim

Disaster Recovery

If the data in a MongoDB cluster is corrupted due to hacking or disk failure, operators may need to detach the tainted volumes and mount new volumes that were created from disk snapshots. In fact, I came upon this issue while researching disaster recovery options because I'm looking for this exact capability for disaster recovery.

I think if the mongodb-replicaset has the ability to use an existing PVC, the operator could manually create the new PVCs, and then perform (or something similar to) helm upgrade <release name> stable/mongodb-replicaset --set existingPVC=foo-pvc-claim

Rapid Horizontal Scaling

Sometimes, production databases get slammed with unexpected traffic volumes, and you need to rapidly increase the replica count. If you have large data volumes, it can take hours (or even days) for all of the data to replicate into a new instance if launched with an empty volume.

Thus, to speed up replication, the operator can launch the new replica with a snapshotted volume. The instance will come online faster because the instance will only need to replicate a smaller data delta since the time of last snapshot.

Major Volume Changes

If you want to change the type of volume being utilized, such as migrating from gp2 to io1 on AWS, you will need to create a new volume and attach it to the mongodb instance. Once again, I suspect updating the release with an existing PVC might help solve this problem.

Other considerations

Graceful Stepdown and Shutdown

If you need to perform some sort of detach-reattach volume procedure, it's important the replica is gracefully taken out of service. This requires connection draining, and "stepdown and re-election" if the particular instance is serving as the primary. I believe this is already taking place automatically wth this chart, but it is worth noting.

Automatic Volume Snapshots

I should also note that I'm currently using k8s-snapshots to automatically create volume snapshots on AWS EBS.

Interestingly, it seems that an official k8s incubator project is in the works for a full-featured external storage api, called external storage which includes automated snapshot capabilities.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[nicroto]

If this remains unsupported, what is the proper way to upgrade your chart to a newer version, while keeping your data?
// cc: @ssalaues

[ssalaues]

Since I've been tagged I'll reply to the original requests for future onlookers:
The ability to specify an existingClaim you see common in charts, including the mongodb chart in standalone mode, is only relevant when using deployments. An entire deployment will attach to a single specified claim thus sharing the storage across pods. However the mongodb-replicaset chart (and even the mongodb when ReplicaSet is enabled) do not have any option to specify an existingClaim because they are Stateful Sets which means each individual pod is expected to use its own claim to request individual/unique storage.

That being said if you wanted to scale or do disaster recovery you should be able to manually create the PVCs for individual pods based off how the claims are naturally created by the StatefulSet and the new pods would, in theory, attach to their corresponding claim automatically.

Now to answer @nicroto : Most charts typically enable upgrading via the simple helm upgrade command. If you're having issues with upgrades it is likely for the best that you open a new issue with a reference to this issue explaining the details of your specific problems.

[nicroto]

Thank you for the reply and I apologize for taking of your time.