Keda: Azure Event Hubs scaler checkpointing should be language agnostic

[tomkerkhove]

Azure Event Hubs scaler check-pointing should be language agnostic.

We need to find a way to decouple the frameworks and know how to lookup the checkpoints.

This is an aggregate to track issues for languages/SDKs we don't support properly:

• Java #767
• Spring Cloud #762
• Python #741

[programaka]

This issue will also influence Dapr projects that use the Event Hubs input binding because it uses Go SDK.

Tried it today and got the same issue as in #767

E0512 16:12:36.417570       1 provider.go:94] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="unable to get checkpoint from storage: unable to download file from blob storage: -\u003e github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=BlobNotFound) =====\nDescription=The specified blob does not exist.\nRequestId:13867e13-701e-0069-4c78-2855a6000000\nTime:2020-05-12T16:12:36.4050651Z, Details: \n   Code: BlobNotFound\n   GET https://dapreventhubbinding.blob.core.windows.net/dapr-event-hub-container/dapr/0?timeout=61\n   Authorization: REDACTED\n   User-Agent: [Azure-Storage/0.7 (go1.13.3; linux)]\n   X-Ms-Client-Request-Id: [11f13cb3-8a84-440e-4e43-f7732534dabf]\n   X-Ms-Date: [Tue, 12 May 2020 16:12:36 GMT]\n   X-Ms-Version: [2018-11-09]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 404 The specified blob does not exist.\n   Content-Length: [215]\n   Content-Type: [application/xml]\n   Date: [Tue, 12 May 2020 16:12:35 GMT]\n   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [BlobNotFound]\n   X-Ms-Request-Id: [13867e13-701e-0069-4c78-2855a6000000]\n   X-Ms-Version: [2018-11-09]\n\n\n"  "ScaledObject.Name"="azure-eventhub-scaledobject" "ScaledObject.Namespace"="default" "Scaler"={}

[tomkerkhove]

@jeffhollan @zroubalik I think we should plan for this in 2.0; wdyt?

[christle]

if no one already work on this issue, i think i can start with it, the next couple of days

[tomkerkhove]

That would be great, thanks @christle !

[christle]

Update: i've tested a few frameworks in different languages ​​and searched for differences. I think I can make a PR proposal soon.

[destumme]

Would very much appreciate if node was to be added to the list of languages to support.

The node library stores the checkpoints at the same path as detailed in the Java issue: https://github.com/kedacore/keda/issues/767#issuecomment-616784583

The main difference it looks like with what is implemented (beyond that path) is the node library stores the properties sequencenumber and offset on the Checkpoint as opposed to what is defined in the Checkpoint struct.