Postgres-operator: Switch to having Kubernetes do (EBS) volume management / resizing

[frittentheke]

Right now the trusty postgres-operator does use the AWS API directly to resize EBS volumes and requires a call of of resize2fs inside the running pod after the volume has been grown - (https://github.com/zalando-incubator/postgres-operator/blob/master/docs/user.md#increase-volume-size).

Starting with 1.11 Kubernetes got more capable of managing persistent volumes and also does resizing of them now, see: https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/

Snapshotting volumes also comes along (currently in alpha) - https://kubernetes.io/blog/2019/01/17/update-on-volume-snapshot-alpha-for-kubernetes/ if that would ever be sensible to use (instead of using pg_basebackup to clone an instance).

My point here is to consider switching to only using Kubernetes for volume management / resizing as this will make the postgres-operator even more agnostic to the cloud provider / volume types and frees the operator from having to do yet another job.

[hjacobs]

:+1:, I know that @Jan-M was already commenting on one or more Kubernetes issues (can't find them now) regarding resize/snapshot.

[erthalion]

Starting with 1.11 Kubernetes got more capable of managing persistent volumes and also does resizing of them now, see: https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/

Looks cool, thank you. We indeed need to implement something like that, but probably only as an alternative for now, because as I can see from the docs, it requires a pod to be restarted, and to resize online we need to have ExpandInUsePersistentVolumes enabled (which is in alpha so far).

[Jan-M]

We want to rely as much as possible on Kubernetes to solve our problems, thus we will look into supporting this. Due to the lack of online resize we will continue to support it the way we do right now for AWS to avoid the pod restart.

[Jan-M]

As for EBS snapshots, I had some uncertainty in the past on how well this works for Postgres, but @CyberDem0n thinks this should be fine to use as a method so maybe a use case can be specced out.

We will spend a bit of work on logical backup support, this can be more useful due to smaller size on s3 or similar.

[frittentheke]

@Jan-M online resizing is availailable with Kubernetes 1.11, https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/.

Check out the last paragrah.

[sdudoladov]

the implementation of this issue must also properly handle pvc updates, currently increasing volume size is no reflected in persistent volume claims

[yujunz]

Encountered an error when trying to resize the volume. Do I need to grant additional roles to the operator or it is just not supported yet?

time="2019-07-09T08:29:49Z" level=warning msg="error while syncing cluster state: could not sync persistent volumes: could not sync volumes: could not resize EBS volume \"vol-02e9bfe3e20df9693\": could not get information about the volume: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 48a09385-58ab-4fc2-8412-75afcd61df9c" cluster-name=postgres-operator/hl-dev pkg=cluster worker=2
time="2019-07-09T08:29:49Z" level=error msg="could not sync cluster: could not sync persistent volumes: could not sync volumes: could not resize EBS volume \"vol-02e9bfe3e20df9693\": could not get information about the volume: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 48a09385-58ab-4fc2-8412-75afcd61df9c" cluster-name=postgres-operator/hl-dev pkg=controller worker=2

[hex2a]

@yujunz
(for future documentation) this probably occurs because the IAM role for the operator pod/kubernetes node is missing the ec2:ModifyVolume action.

Is there interest in a PR to use PVC expansion (only for the blockdevice, without FS resizing) instead of the current implementation that calls the AWS API directly? (that solves the PVC update (@sdudoladov)/IAM permission problems and should add support for other providers), it is enabled by default since v1.11.

[Jan-M]

@hex2a Thanks for the offer, but my current understanding is that this resizing using K8s built ins is not "restart" / "rolling upgrade" free. So not an ideal solution from our point of view.

If this has changed, any PR removing code and custom code paths is always welcome.

[frittentheke]

@Jan-M the container storage interface (CSI) as well as the EBS CSI driver does have online-resizing capability:

• https://kubernetes-csi.github.io/docs/volume-expansion.html#volume-expansion
• https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/resizing/README.md#volume-resizing

Certainly one requires K8S 1.15 or 1.16 ... but things regarding the EBS driver are moving along quite nicely in the recent releases. Until now doing things "online" does require a feature gate set for Kubelet still though: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/374#issuecomment-535811284

[frittentheke]

CSI 1.2 is coming along now as well ....

https://github.com/container-storage-interface/spec/pull/381

[dave08]

Is there currently any workaround for using k8s volume resizing (we're not on AWS...) until this feature is developed? We're considering this operator, but we need to make sure that volume expansion is at least possible.

Thanks!

[fischerman]

@dave08 have you tried simply resizing the PVCs by hand? The operator might complain about the out of band changes, though.

[dave08]

I haven't tried, I'm really considering our needs before actually adopting the system and was wondering if the operator would try to fight manual changes...

I'd suppose from your answer that the operator doesn't tamper with such changes... but the state of the postresql yaml will just be wrong and GitOps wouldn't be possible here (since the manifests are created by the operator...).

Also, if in the near future this would be implemented, maybe it should check that somebody didn't do this (manually patching the PVCs) before "trying to resize" the volume according to the CRD...

[frittentheke]

@Jan-M I suppose with 1.6 released and https://github.com/zalando/postgres-operator/pull/958 included this feature is implemented and the issue should be closed. Thanks a bunch!

^^ @dave08 FYI

[dave08]

Thanks, that's great!

[Jan-M]

Yes. With 1.6 we merged this. Thanks for your patience. Looking forward to see more happy users of this.