Service-catalog: Deprovisioned service instance is never deleted

[jwforres]

If I delete an Instance, the deprovision operation against the broker succeeds and the conditions on the Instance are updated correctly. But then the Instance object is never deleted. This will just leak instance objects over time.

Example deprovisioned instance:

    {
      "metadata": {
        "name": "user-provided-service-6cfjw",
        "generateName": "user-provided-service-",
        "namespace": "test2",
        "selfLink": "/apis/servicecatalog.k8s.io/v1alpha1/namespaces/test2/instances/user-provided-service-6cfjw",
        "uid": "b2b9a898-5129-11e7-a167-0242ac110004",
        "resourceVersion": "492",
        "creationTimestamp": "2017-06-14T17:48:39Z",
        "deletionTimestamp": "2017-06-14T19:51:01Z",
        "deletionGracePeriodSeconds": 0,
        "finalizers": [
          "foregroundDeletion"
        ]
      },
      "spec": {
        "serviceClassName": "user-provided-service",
        "planName": "default",
        "parameters": {},
        "externalID": "d878bf7a-982b-47c8-b9d1-1e56e1b0a308"
      },
      "status": {
        "conditions": [
          {
            "type": "Ready",
            "status": "False",
            "lastTransitionTime": "2017-06-14T19:51:01Z",
            "reason": "DeprovisionedSuccessfully",
            "message": "The instance was deprovisioned successfully"
          }
        ],
        "asyncOpInProgress": false,
        "checksum": "6ce4c601559f165600303782c244aeb7b983a260eb5d7168efed7895faf0a3e4"
      }
    }

[jwforres]

This is most likely an issue with async deprovision.

[jwforres]

We are seeing a very similar issue with deleted bindings. Not sure if you want a separate issue to track that, some details here https://github.com/openshift/origin-web-console/issues/1711

[MHBauer]

Is this reliably triggered? Sounds like we are missing a test case.

[jwforres]

Yes, every time from our console when you delete an instance or a binding. By default we send all delete requests with the delete options set for Foreground deletion. That could be the difference between the tests and what is happening here?

So the DELETE request body looks like:

{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}

[MHBauer]

Definitely not familiar with foreground deletion policy.

[liggitt]

there are two issues:

1. when not running the service catalog server behind the aggregator, the garbage collector is unable to discover the service catalog types, in order to watch/finalize them.
2. even when the service catalog server is running behind the aggregator, the garbage collector does not (as of kube 1.7) watch TPR/CRD/aggregated APIs (https://github.com/kubernetes/kubernetes/issues/44507)

until the garbage collector supports non-core APIs, aggregated apiservers should not enable garbage collection (see https://github.com/kubernetes/kubernetes/pull/47432/files for a similar change to CRD objects in 1.7)

[pmorie]

@liggitt @derekwaynecarr I do not think the garbage collector is actually in play here at all.

[jwforres]

@pmorie the finalizers for GC were being added, and never being resolved. At least for Binding it definitely resolved the issue when I stopped sending propagationPolicy during the Delete request. But we don't want to special case deletion of catalog items unless necessary.

I can double check the same is true for deleting Instances if that would help.

[derekwaynecarr]

@jwforres @pmorie - to capture what i think was discussed for future readers, the UI will need to special case this at the moment otherwise no other finalizer can be used on service catalog.

[jwforres]

But for the record, the CLI will still be broken if someone decides to do
kubectl delete with --cascade=false

On Jun 19, 2017 5:12 PM, "Derek Carr" [e-mail] wrote:

@jwforres https://github.com/jwforres @pmorie https://github.com/pmorie

• to capture what i think was discussed for future readers, the UI will
  need to special case this at the moment otherwise no other finalizer can be
  used on service catalog.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/service-catalog/issues/942#issuecomment-309575214,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABZk7WxHFcGzudSLSc-Q8Aqnt0H7e87iks5sFuRFgaJpZM4N6XPI
.

[andutt]

Seeing this issue in Openshift 3.7. Is there anyway to delete a Service Binding in limbo state? The service and project är deleted but the Service Binding are still there and hammering the logs:

1 controller_binding.go:880] ServiceBinding "space/mysql-persistent-bcgfg-jxdpf": Setting condition "Ready" to False
Dec 13 12:37:55 sys1 dockerd-current[37557]: I1213 11:37:55.877258 1 controller_binding.go:959] ServiceBinding "space/mysql-persistent-bcgfg-jxdpf": Updating Ready condition to False (Reason: "ErrorGettingAuthCredentials", Message: "Error getting auth credentials. Error getting broker auth credentials for broker \"template-service-broker\": secrets \"templateservicebroker-client\" not found")

[MHBauer]

Is this related to https://github.com/kubernetes-incubator/service-catalog/issues/1562 or different?

[andutt]

Seems like similar, but not sure if its the same case. The above is triggered everytime you create a lamp (httpd + mysql with servicebinding) within a project, and then try to delete the project. The same result both from cli and by console (Delete project). Everything is removed, but the servicebinding remains in limbo state.

[propyless]

@andutt Hey there, so a workaround seems to be to remove the finalizer token in the metadata section of the serviceinstance and servicebinding objects.
oc edit servicebinding <name>
oc edit serviceinstance <name>

[andutt]

Nope, that didnt work in this case, it was not possible to remove the finaliser token. I had to remove it in etcd.

NOTE! this is a unsupported and not popular fix! and can in some cases brick things.

But solved the problem in my clusters. I made sure that no other objects to the project remained, and then manually removed it in etcd.

etcdctl3 --cacert=/etc/etcd/ca.crt --cert=/etc/etcd/peer.crt --key=/etc/etcd/peer.key --endpoints=https://hostname:2379 get --keys-only --prefix / | grep MYPROJECT

etcdctl3 --cacert=/etc/etcd/ca.crt --cert=/etc/etcd/peer.crt --key=/etc/etcd/peer.key --endpoints=https://hostname:2379 del --prefix /registry/servicecatalog.k8s.io/servicebindings/MYPROJECT

[propyless]

Ah ok, interesting. then we probably had different issues all together. :)

[MHBauer]

/assign @jboyd01
to follow up

[jboyd01]

I discussed with pmorie and given the age of this issue and progress made with catalog in terms of bug fixes and reliability and the lack of recent issues with current versions of the software (Catalog was in Alpha back in OpenShift 3.7) I'm going to close this issue. We've certainly seen issues where if a broker is unable to delete a resource (or gives catalog an error on a delete request) the resource may become wedged in K8s and get stuck in a deletion state, not really alive, not really dead. In instances like this generally deleting the finalizer enables the resource to be deleted. I've seen one other issue where the finalizer couldn't be deleted because when you tried to update the object removing the finalizer, some other field was being called out as invalid. In that specific case we've had to resort to deleting from etcd storage. If anyone is hitting deletion, issues with current versions of Catalog we should definitely open a new issue with supporting detail and steps to reproduce.