Faas: Research: Integrate with AWS SNS

[alexellis]

Since some of our users are deploying OpenFaaS to production with Kops (on AWS) - we should look at the work required to integrate events from a AWS SNS as triggers. This may be through the use of a side-car for the gateway or a new "queue" type similar to NATS Streaming's "queue worker".

SQS vs SNS

SNS faq

Expected Behaviour

Events in AWS such as DynamoDB triggers, S3 updates etc can trigger functions in OpenFaaS.

Current Behaviour

Can be built with custom solution/side-car.

Possible Solution

Create a PoC with either/or:

• A function exposed on the gateway (I believe the SNS handshake involves GET + POST)
• A sidecar for the Gateway (new micro-service)
• A queue-worker type similar to NATS streaming

Steps to Reproduce (for bugs)

1. Configure SNS for AWS assets
2. Point SNS to relevant route/service on OpenFaaS
3. Trigger update
4. Profit

Context

Easier integration with AWS eco-system

Your Environment

• Docker version docker version (e.g. Docker 17.0.05 ):

• Are you using Docker Swarm or Kubernetes (FaaS-netes)?

Should cover Swarm + K8s.

[VitorFalcao]

@alexellis I would like to work on this. Do you have any further instructions? Thank you.

[alexellis]

Yes I'd start with the documentation and do a minimal example that can handle the handshake and receive a message. After that it should be a case of digging on the detail and finding out the best way to integrate. You're also welcome to join our slack community - email [email protected].

[VitorFalcao]

Great. Thank you for the instructions, I will start working on it and join the slack community.

[alexellis]

@VitorFalcao did you make any progress with this?

[austinfrey]

cc @viveksyngh you might be able to collaborate with @VitorFalcao here

[alexellis]

I think we'll need to work on this through the contributors group instead of wider community. Austin please could you look into the analysis / high level details around handshakes and configuring SNS to emit events? Maybe see if Robby can help since he has worked with EventGrid which is concentually the same thing.

[viveksyngh]

@austinfrey I would like to work on this one. I am part of contributors group of open-faas slack community. Please guide me on how to get started for this.

[alexellis]

Start at the beginning with analysis about how to register, and handshake for receiving from SNS.

[viveksyngh]

Thanks. I will start working on it.

[viveksyngh]

@alexellis @austinfrey I have written a sample HTTP subscriber for SNS in Go, which makes subscription request, confirms the subscription, receives and process notification. Please have look at this repo.

https://github.com/viveksyngh/aws-sns-subscriber

[austinfrey]

@alexellis I don't believe it supports all the flags yet

[alexellis]

There should be examples for configuring SNS events to be published within AWS and all the steps required. Also need to have the analysis / documentation on the hand-shake so we can understand how to integrate.

[alexellis]

@austinfrey what doesn't support which flags?

[austinfrey]

@alexellis apologies posted in the wrong issue thread :/

[viveksyngh]

Please help me in understanding. What kind of integration will we do with open faas and SNS? Will open faas functions push an event to AWS SNS topic (Publisher) ? or Open faas function will be used as an HTTP subscriber to receive an event from SNS topic?

What I have tried to do till now is write an HTTP subscriber in go which will programmatically subscribe to an SNS topic, confirm subscription and receive events from SNS topic.

[mitrilmad]

@viveksyngh I think that aws events (like a S3 operation, triggered cloudwatch metric) can trigger functions on openfaas like a lambda.

[alexellis]

So if we look at the Kakfa implementation and the work @kinghuang did for University of Calgary the concept is that we should somehow bind a topic to a function. If we were to put that in YAML at a guess it may look like this:

bindings:
  - topic: arn:s3:images:put
    - function: func_resizer
  - topic: arn:s3:thumbnails:put
    - function: func_update_catalog
    - function: func_rebuild_homepage

Whatever we think is appropriate should be driven from actual use-cases with AWS - rather than pure conjecture or high-level design. We also need to consider whether we can create an abstract way of doing this such that we could use similar config with a RabbitMQ, NATS Streaming or Kakfa log.

[alexellis]

Derek close: fixed via kafka-connector and annotations field in YAML