Terraform-provider-kubernetes: PVC increase size causes destroy

Created on 28 Aug 2019  ·  7Comments  ·  Source: hashicorp/terraform-provider-kubernetes

Hi,
TF K8s provider flags storage increase as a destroy/create (-/+) op as opposed to update (~). If using kubectl edit pvc, size increases without destroying the PVC.

Terraform Version

0.11.10
terraform-provider-kubernetes_1.8.0

Affected Resource(s)

  • kubernetes_persistent_volume_claim

Terraform Configuration Files

resource "kubernetes_storage_class" "standard" {
  metadata {
    name = "${var.environment}-standard"
  }

  storage_provisioner = "kubernetes.io/gce-pd"
  reclaim_policy      = "Retain"

  parameters = {
    type = "pd-standard"
  }

  allow_volume_expansion = true
}

resource "kubernetes_persistent_volume_claim" "vol1" {
  metadata {
    generate_name = "${var.environment}-vol-1-"
  }

  spec {
    access_modes = ["ReadWriteOnce"]

    resources {
      requests {
        storage = "${var.kubernetes_persistent_volume_disk_size}"
      }
    }

    storage_class_name = "${kubernetes_storage_class.standard.metadata.0.name}"
  }

  wait_until_bound = true

  lifecycle {
    create_before_destroy = true
  }
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

Using kubectl edit pvc <pvc-name> and increasing storage did not destroy the PVC but rather increased the size as per https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes

Actual Behavior

TF forces new resource

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

Important Factoids

Used in GKE. Haven't tested in EKS or other K8s PaaS.

References

None

acknowledged bug needs investigation themlifecycle
Source
picture dcloud9
👍8 👀1

Most helpful comment

I can confirm this using terraform v0.12.9 on AWS EBS backed pvcs. This behavior happens even with storage class set to allow volume expansion

allowVolumeExpansion: true

picture haggishunk on 1 Nov 2019
👍4

All 7 comments

I can confirm this using terraform v0.12.9 on AWS EBS backed pvcs. This behavior happens even with storage class set to allow volume expansion

allowVolumeExpansion: true

haggishunk picture haggishunk on 1 Nov 2019
👍4

Still an issue with Terraform v0.12.21 + provider.kubernetes v1.10.0.

TeroPihlaja picture TeroPihlaja on 28 Feb 2020
👍1

The PVC objects are immutable unless the associated storage class has allow_volume_expansion set on it.

We don't really have a way to selectively allow updates to the PVC based on attributes of it's associated SC. As such, I think we should just implement the update to the PVC and make sure we accurately convey the errors returned by the API when such an operation doesn't apply correctly.

alexsomesan picture alexsomesan on 29 Apr 2020
👍1

Example of kubectl being unable to edit the PVC without an appropriate storage class:

~/test-pvc-storageclass » kubectl edit -v=7 persistentvolumeclaim test-pvc                                                                                                                    alex@alex-macbook
I0429 18:35:51.941579   97530 loader.go:375] Config loaded from file:  /Users/alex/.kube/config
I0429 18:35:51.945056   97530 cert_rotation.go:137] Starting client certificate rotation controller
I0429 18:35:51.954085   97530 round_trippers.go:420] GET https://192.168.64.46:8443/api/v1/namespaces/default/persistentvolumeclaims/test-pvc
I0429 18:35:51.954098   97530 round_trippers.go:427] Request Headers:
I0429 18:35:51.954102   97530 round_trippers.go:431]     Accept: application/json
I0429 18:35:51.954105   97530 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (darwin/amd64) kubernetes/9e99141
I0429 18:35:51.968326   97530 round_trippers.go:446] Response Status: 200 OK in 14 milliseconds
I0429 18:35:51.971891   97530 editor.go:128] Opening file with editor [vi /var/folders/zf/33sr_43n343f_wls_994vkkm0000gn/T/kubectl-edit-6azl5.yaml]
I0429 18:36:13.417689   97530 editoptions.go:288] User edited:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"ebe46ebf-89f8-11ea-b62c-0a7b8fad38ba","leaseDurationSeconds":15,"acquireTime":"2020-04-29T16:34:15Z","renewTime":"2020-04-29T16:34:45Z","leaderTransitions":0}'
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: "2020-04-29T16:34:15Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: test-pvc
  namespace: default
  resourceVersion: "454024"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc
  uid: 6db17edd-c049-4fa2-b7a7-b37b01ac7c85
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: pvc-6db17edd-c049-4fa2-b7a7-b37b01ac7c85
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound
I0429 18:36:13.420635   97530 round_trippers.go:420] GET https://192.168.64.46:8443/openapi/v2?timeout=32s
I0429 18:36:13.420675   97530 round_trippers.go:427] Request Headers:
I0429 18:36:13.420684   97530 round_trippers.go:431]     Accept: application/[email protected]+protobuf
I0429 18:36:13.420691   97530 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (darwin/amd64) kubernetes/9e99141
I0429 18:36:13.432163   97530 round_trippers.go:446] Response Status: 200 OK in 11 milliseconds
I0429 18:36:13.524628   97530 round_trippers.go:420] PATCH https://192.168.64.46:8443/api/v1/namespaces/default/persistentvolumeclaims/test-pvc
I0429 18:36:13.524652   97530 round_trippers.go:427] Request Headers:
I0429 18:36:13.524659   97530 round_trippers.go:431]     Content-Type: application/strategic-merge-patch+json
I0429 18:36:13.524665   97530 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (darwin/amd64) kubernetes/9e99141
I0429 18:36:13.524673   97530 round_trippers.go:431]     Accept: application/json
I0429 18:36:13.533842   97530 round_trippers.go:446] Response Status: 403 Forbidden in 9 milliseconds
error: persistentvolumeclaims "test-pvc" could not be patched: persistentvolumeclaims "test-pvc" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
You can run `kubectl replace -f /var/folders/zf/33sr_43n343f_wls_994vkkm0000gn/T/kubectl-edit-6azl5.yaml` to try this update again.
F0429 18:36:13.534905   97530 helpers.go:115]
------------------------------------------------------------
alexsomesan picture alexsomesan on 29 Apr 2020

And here is how kubectl resolves an edit by patching the PVC when its storage class allows for it.

~/test-pvc-storageclass » kubectl edit -v=7 persistentvolumeclaim test-pvc                                                                                                                    alex@alex-macbook
I0429 18:59:30.152162    5987 loader.go:375] Config loaded from file:  /Users/alex/.kube/config
I0429 18:59:30.153810    5987 cert_rotation.go:137] Starting client certificate rotation controller
I0429 18:59:30.160933    5987 round_trippers.go:420] GET https://192.168.64.46:8443/api/v1/namespaces/default/persistentvolumeclaims/test-pvc
I0429 18:59:30.160946    5987 round_trippers.go:427] Request Headers:
I0429 18:59:30.160950    5987 round_trippers.go:431]     Accept: application/json
I0429 18:59:30.160953    5987 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (darwin/amd64) kubernetes/9e99141
I0429 18:59:30.177144    5987 round_trippers.go:446] Response Status: 200 OK in 16 milliseconds
I0429 18:59:30.179127    5987 editor.go:128] Opening file with editor [vi /var/folders/zf/33sr_43n343f_wls_994vkkm0000gn/T/kubectl-edit-z8d49.yaml]
I0429 18:59:53.004107    5987 editoptions.go:288] User edited:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"ebe46ebf-89f8-11ea-b62c-0a7b8fad38ba","leaseDurationSeconds":15,"acquireTime":"2020-04-29T16:59:24Z","renewTime":"2020-04-29T16:59:28Z","leaderTransitions":0}'
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: "2020-04-29T16:59:24Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: test-pvc
  namespace: default
  resourceVersion: "457285"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc
  uid: 33bd1de9-95ff-4521-9f02-24df1bea9f21
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: test-standard
  volumeMode: Filesystem
  volumeName: pvc-33bd1de9-95ff-4521-9f02-24df1bea9f21
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound
I0429 18:59:53.006382    5987 round_trippers.go:420] GET https://192.168.64.46:8443/openapi/v2?timeout=32s
I0429 18:59:53.006409    5987 round_trippers.go:427] Request Headers:
I0429 18:59:53.006419    5987 round_trippers.go:431]     Accept: application/[email protected]+protobuf
I0429 18:59:53.006450    5987 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (darwin/amd64) kubernetes/9e99141
I0429 18:59:53.021173    5987 round_trippers.go:446] Response Status: 200 OK in 14 milliseconds
I0429 18:59:53.127034    5987 round_trippers.go:420] PATCH https://192.168.64.46:8443/api/v1/namespaces/default/persistentvolumeclaims/test-pvc
I0429 18:59:53.127058    5987 round_trippers.go:427] Request Headers:
I0429 18:59:53.127067    5987 round_trippers.go:431]     Accept: application/json
I0429 18:59:53.127074    5987 round_trippers.go:431]     Content-Type: application/strategic-merge-patch+json
I0429 18:59:53.127081    5987 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (darwin/amd64) kubernetes/9e99141
I0429 18:59:53.133581    5987 round_trippers.go:446] Response Status: 200 OK in 6 milliseconds
persistentvolumeclaim/test-pvc edited
------------------------------------------------------------
alexsomesan picture alexsomesan on 29 Apr 2020
❤1

I'm able to reproduce this in minikube too. This is the config I'm using:

resource "kubernetes_storage_class" "example" {
  metadata {
    name = "example"
  }
  allow_volume_expansion = true
  storage_provisioner = "k8s.io/minikube-hostpath"
}

resource "kubernetes_persistent_volume_claim" "example" {
  wait_until_bound = true
  metadata {
    name = "example"
  }
  spec {
    access_modes = ["ReadWriteOnce"]
    storage_class_name = kubernetes_storage_class.example.metadata.0.name
    resources {
      requests = {
        storage = "1G"
      }
    }
  }
}

I'm going to start working on modifying our Update function to allow in-place updates of the PVC attributes, as Alex suggested.

dak1n1 picture dak1n1 on 29 Jul 2020
🚀1 ❤1

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

hashibot[bot] picture hashibot[bot] on 10 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

synhershko picture synhershko  ·  3Comments
landorg picture landorg  ·  3Comments
hashibot[bot] picture hashibot[bot]  ·  4Comments
joe-a-t picture joe-a-t  ·  3Comments
bugb picture bugb  ·  4Comments