Aws-sdk-go: Feature request for Kinesis API - decode protobuf

[rayjlinden]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

v1.16.18

Version of Go (go version)?

1.11

What issue did you see?

Trying to read kinesis stream that was written by Java KPL library. Messages are encoded with protobuf to aggregate multiple records into one kinesis record. Shouldn't there be a way to decode those messages with this API?

Steps to reproduce

This can be worked around by turning off the Aggregation feature in the properties file for the KPL. However, the feature is nice and allows for higher throughput. It would be nice if the Go library could decode it.

Please add a way to decode the protobuf compressions and aggregation that is the default behavior of the Java KPL.

[jasdel]

Thanks for the request @rayjlinden. There currently is not an Amazon KPL for Go library. A feature to decode the protobuffer would best be in a KPL for Go instead in the base SDK. I think this is related to #594.

[cb-xav]

While I understand the reasoning is because there is no KPL for Go, I am sure there are still cases where folks are writing stream consumers using the Go AWS SDK and running into the issue that some records are in that KPL Aggregate Format.

For our use case, a different team populates the kinesis stream using the Java KPL, and we are on the other end consuming it. I know the KCL normally handles that, but unless there is a native Go KCL very soon on the horizon, I think this would be a nice feature to have. We currently have a little module in our code to do that KCL deaggregation.

We started putting together a PR and then saw in the PR opening text that anything autogenerated should be opened as an issue instead of a PR, so not sure how we could integrate our stuff (with service/kinesis/api.go) if that code is autogenerated, but would love to share and contribute what we have done.

[rayjlinden]

Either AWS should add support to decode compressed streams in aws-go-adk or
they should release a go version of the KCL. The bull shit answers they
give now are pretty lame.

Also - I'n my use case which was a lambda that reads from a kinesis event -
the entire KCL library is unneeded and overkill. Lambda create a new
lambda instance for each shard and restarts the lambda on every event. So
most of the KCL functionality should not even be used in such a use case.
So frankly I think think the support should be added to the aws-go-sdk even
if a Go version of KCL is ever created.

I did end up implementing it myself for our code base. (Stole parts of is
from other stuff I found on the net.) In the end it is about a page and
1/2 of code. Which makes it seem even more ridiculous that it isn't
supported out of the box.

(I did hear some aws developer whine that they had some other part of the
code using proro3 instead of proto2 and they are incompatible or something
like that anyway. However, there is not a lot of value in regenerating the
proto generated Go code every time. Just do it once and check it in. It
will compile fine on all platforms. I just do not get this excuse.

Ray

On Tue, Jun 11, 2019 at 12:25 PM Xavier Thierry [e-mail]
wrote:

While I understand the reasoning is because there is no KPL for Go, I am
sure there are still cases where folks are writing stream consumers using
the Go AWS SDK and running into the issue that some records are in that KPL
Aggregate Format.

For our use case, a different team populates the kinesis stream using the
Java KPL, and we are on the other end consuming it. I know the KCL normally
handles that, but unless there is a native Go KCL very soon on the horizon,
I think this would be a nice feature to have. We currently have a little
module in our code to do that KCL deaggregation.

We started putting together a PR and then saw in the PR opening text that
anything autogenerated should be opened as an issue instead of a PR, so not
sure how we could integrate our stuff (with service/kinesis/api.go) if that
code is autogenerated, but would love to share and contribute what we have
done.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-go/issues/2408?email_source=notifications&email_token=AKE5LWXOFVMSYVSO6CYMFPTPZ7327A5CNFSM4GQFJCE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXOHLQY#issuecomment-500987331,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKE5LWUAI6W5P5PUCVTA7GLPZ7327ANCNFSM4GQFJCEQ
.

[cdemonchy-pro]

Working implementation available here https://github.com/awslabs/kinesis-aggregation/blob/master/go/deaggregator/deaggregator_test.go

[github-actions[bot]]

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.