Chalice: [proposal] Add support for Kinesis and DynamoDB stream events

[kgutwin]

Similar to the S3, SNS and SQS event sources implemented earlier, it would be awesome (especially for my current use case) to also support DynamoDB streams - and while we're at it, supporting Kinesis streams should also be easy.

Fortunately, the implementation in #886 (proposed in #884) should be relatively easy to adapt to support these other sources, since the Lambda event source mapping function is the same.

Proposal

Public API

The DynamoDB streams and Kinesis streams would be implemented nearly identically to the existing event sources - a parameterized decorator.

@app.on_dynamodb_stream_event(table='mytable', batch_size=30, starting_position='TRIM_HORIZON')
def handler(event):
    for record in event:
        print(record.body)

@app.on_kinesis_stream_event(stream='mystream', batch_size=100, starting_position='LATEST')
def handler(event):
    for record in event:
        print(record.body)

Backend

While we could maintain the existing SQS event source as-is, because a significant amount of functionality would be shared between the three stream sources (SQS, DynamoDB and Kinesis), it may make more sense to rename the backend "sqs_event_source" methods as "stream_event_source" methods, and then map each of the three stream decorators to the same core event source with different configs. Then you could build up the event source arn using the source as another parameter.

[kyleknap]

@kgutwin Thanks for the proposal! We will get to looking at it and give you feedback on it.

[Sytten]

I think this would be very useful if chalice is to become the standard for AWS lambda in python

[benlvovsky]

I really need this for my project. I architected with that in mind but Chalice cannot do it at the moment. And I found it out just now. Any workaround currently available?

[henkesde92]

I'm also looking forward to see this soon. Do you already have any news regarding this?

I want to implement a fairly easy REST API with AWS chalice, but this is something that I'm missing right now. So Chalice might not be the right choice when this is not on the timeline yet.

[benlvovsky]

I have setup manually via AWS console trigger my lambda (easy in Chalice
'@app.lambda_function(name='MyFunction')) from DynamoDB stream and it
works. I think the ease of this approach is the main reason for Chalice
team is so slow on this to implement and not too many people complain about
lack of this functionality. I'd prefer to have it in code of course. When
you deploy to another environment you need to remember to do it again or
better script it. Which in turn requires learning curve with no real need
for this.

On Fri, 3 May 2019 at 00:12, henkesde92 [e-mail] wrote:

I'm also looking forward to see this soon. Do you already have any news
regarding this?

I want to implement a fairly easy REST API with AWS chalice, but this is
something that I'm missing right now. So Chalice might not be the right
choice when this is not on the timeline yet.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/chalice/issues/987#issuecomment-488689380, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAN225TA73MHO4NN7JERTVLPTLZF3ANCNFSM4GE2EV3A
.

[lacatus]

+1 on this

[dperconti]

Curious if this was still on the plan for the next release?

[TraMZzz]

Any progress?

[MichaelBoselowitz]

This would be a great feature to have! :+1:

[lucasmpaim]

+1

[BrettFeldman]

This would be an awesome addition to Chalice!

[me2resh]

+1

[haduart]

+1

[techytoes]

+1

[sw33tr0ll]

+1 here. Going to try to workaround by using chalice to make a pure lambda function and triggering it separately through Dynamo Triggers.

[KapitalEarth]

Good thinking. Slow and steady with the Dynamo Triggers as lots of colleagues here for one.

Range O through carrier B ch uffed with planning OS-ÐAV architecture.

R.

Sent from my iPhone

On 19 Aug 2020, at 12:27 PM, sweetroll [e-mail] wrote:


+1 here. Going to try to workaround by using chalice to make a pure lambda function and triggering it separately through Dynamo Triggers.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[jamesls]

I've got a PR for the kinesis stream, and I'm working on the dynamodb stream PR now. The only hiccup here is that I don't think we'd be able to support the table name when configuring a stream. The full event source ARN has a timestamp in it, e.g. arn:aws:dynamodb:us-west-2:12345:table/MyTable/stream/2020-09-28T16:49:14.209. For the chalice deploy codepath, we can look up this value by making a list-stream API call, however to support SAM/terraform templates via chalice package, this approach isn't going to work, as we don't know the timestamp portion of the ARN.

I think for the time being we'll have to require a stream_arn instead of a table param. Thoughts?

[kyleknap]

@jamesls yeah arn makes sense to me given that constraint

[kgutwin]

Would it make sense to create the DynamoDB stream as part of this request? That way you could specify the table and it would be easier for the user to use it, rather than having to hunt down the stream ARN. Thoughts?

[jamesls]

Would it make sense to create the DynamoDB stream as part of this request?

Ideally yes, but we'd still have a problem with supporting cloudformation/terraform. The streamspecification is part of the table resource, which means we'd have to create the DynamoDB table as well, which now means we need an API for how to specify additional resources.

There's a few proposals on generic resource support (@kyleknap has one that's been updated), and it is something we'd like to support. Once that happens then we'd also be able to support creating any resource associated with an event handler. We might also be able to leverage the CDK as well.

[skwashd]

I know this doesn't solve the terraform part, but for Cloudformation there is the template merge as a work around.

[jamesls]

This is now part of Chalice v1.21.0, closing issue.

[lucaghera]

@jamesls Thanks for implementing this feature. Could you let us know if this is supposed to work locally? I tried it, but I could not get the lambda to trigger. I computed the ARN used in the decorator as follows:

dynamodb_resource = boto3.resource('dynamodb', endpoint="http://dynamodb-local:8000")
table = dynamodb_resource.Table('MyTable')
stream_arn = table.latest_stream_arn

Thanks again!