Faas: Proposal: Generic trigger event source plugin system

[Templum]

Description

I want to introduce support for trigger/event sources.

Currently, a user of Open Faas has to call the function he likes. And in a system where he likes to react to events from a special source he is required to create a connector which stores not only the mapping from the event -> Lambda, but further also invoke the lambda with the payload.

Possible Solution

Introduce a defined interface which allows the creation of (official) connectors for different sources which will invoke lambda function. With a standardized (rest) interface user can create a variety of trigger for different event sources, language agnostic.

The trigger has to register them with the gateway, this will be used later on the verify that deployed function really can use the defined trigger. The faas-cli or more precisely the yaml definition file needs to be modified to also introduce a trigger section, where multiple triggers can be added, maybe a function might be a trigger. Which would allow a chaining.

The mapping between trigger -> function can be either stored directly in the connector or be accessible from the gateway through an API endpoint. And the connector would cache it.

[alexellis]

@Templum thanks for your suggestion. Just curious - is this an idea or something you personally have a business use-case for? What limitations are you finding with HTTP for triggering? What is the missing producer that you need?

Btw are you on Slack? You may want to attend some of the contributor meetings too.

[Templum]

@alexellis Just to highlight I propose a standardized way of connecting event trigger with the openfaas framework and this should happen using the HTTP API, but with additional endpoints and additional CLI support. This would enable developers to create connectors for event sources they need like internal systems or more general approaches like the observation of databases. Further, it would allow a more declarative definition of trigger-lambda relation.This would be kind of similar to the events in the serverless framework. As far as I know, something like that is currently not directly supported by openfaas. If required I can write a more detailed proposal including some graphics and additional material to highlight benefits etc.

Haven't joined the slack yet. For an invitation, I have to send you an email correct?

For our business use-case, we have a couple of event sources which will later trigger functions. It would really simplify the adaption process if we could directly define the trigger in the template and have the connector query all "registered" function at the gateway.

[reiz]

@alexellis I think the proposal from @Templum makes sense. The Apache OpenWhisk project has a concept for Triggers and Rules. Having a concept for triggers makes sense if you want to react to an event from a WebSocket for example.

Let's say there is a WebSocket API. You are connecting to it by opening a WebSocket connection. Then you are waiting util the server is pushing actively a message to you. If that happens you want to trigger a function on openFaaS. Right now we have to implement that glue code between WebSocket an openFaaS by our own and we have to host it somewhere outside of openFaaS. In OpenWhisk I can implement that glue code as a Trigger and host it inside of OepnWhisk. And I can even have multiple rules which connect a trigger with multiple functions.

Would be great if openFaaS would have a concept for triggers as well. But I'm not sure if it's the best solution to implement it directly into openFaaS. Right now the beauty of openFaaS is the simplicity. By adding more concepts, it becomes more complex, too. Maybe the best solution is a side project which builds on top of openFaaS. A side project which extends openFaaS with Triggers & Rules. That way it would be decoupled from each other.
What do you think?

[austinfrey]

Derek add label: proposal

[alexellis]

I think I have to say what @reiz is saying aligns best with the values of the project. Simplicity for the developer and operator is key to the value. Being an open-platform is also very important so that means something like this can be built and run-alongside the existing solution to add the functionality.

One of the examples I really like is this from @kinghuang which takes around 90 lines of Python code to integrate the API gateway with triggers on Kakfa topics. https://github.com/ucalgary/ftrigger

In terms of the use-cases I would love to hear more. One specific integration I need help with is the AWS queue/pub/sub integration - #263 - I could see this bringing a lot of value to the project.

[Templum]

Okay, I will try to come up with a possible solution for a generic approach and show it on some use cases. By the way, haven't heard back from you since my email with my little intro.

[alexellis]

I'll send you an invite today. I'd like to start thinking about the concrete use-cases.

[Templum]

Use cases:

1. Using a Lambda function to react on database modification

In a typical microservice environment services tend to have their own data and also manage their data explicitly. Other services, in this case, need to request data directly from the service. With a trigger system and openfaas, the services could store that information in there own data storage, of cause only for reading access. Using triggers modification to data can be propagated to services affected.

Of cause this is a specific use case. However, theire might be a couple of use cases where you want to react to changes to the database and based on the occurred event trigger functions/logic

1. Using a Lambda function to react to file uploads

Imagine you have a system that should execute a couple of functionality each time a file is uploaded. For example, scan it for security vulnerabilities, compress it and notify the user. This could be a cloud service like amazon s3 or an internal system.

1. Reacting to event in Kafka / SQS

Reacting to messages published to a queue and executing business logic.

1. Reacting to received events (via WebSockets)

A more concrete use case. If you want to offer your customers the ability to modify and extend your platform using lambda functions, similar to auth-0 rules, a good way to do this would be openfaas and lambda functions. Those function then can be executed to enforce rules provided by the customer and alter the existing platform based on the customers need. In such a scenario incoming events from the WebSocket connection would be encapsulated by a trigger source and then forwarded to the gateway which will invoke lambdas.

Concept:

Openfaas already provides a functionality through the API-Gateway that allows a mapping from a function to a concrete URL. I would propose a solution that enhances the current implementation by a mapping from a trigger to function. The trigger sources would be able to call a specified URL with a specified JSON payload. The gateway will do the mapping based on the event type and execute the registered functions.

Modifications:

A potential trigger-source could be implemented pretty freely the only restriction being the gateway URL and the JSON object that must be provided during the call.

The cli must be extended to also allow the specification of triggers under an own attribute. Labels in the deployment can be used to store information about a function reacting to a provided trigger. Or those mappings get stored in an own datastore.

{
  "trigger": "com.trigger.event",
  "type": "kafka",
  "token": "some-jwt",
  "payload": {
    "application": "specific"
  }
}

Legend:

• _trigger_ is the event type that gets specified by the user. A package format here would allow for safety in greater environments
• _type_ specifies the connector-type. For example Kafka or database-name. Based on the type a payload schema validation can be done.
• _token_ allows verifying that the message came from a trusted source.
• _payload_ trigger dependent payload

[radu-matei]

Hi, everyone!
Has there been any progress here?

I am interested in building something that aggregates events from multiple sources (cloud storages, database updates, pub/sub services).

Here's how an S3 blob created event looks like (taken from AWS docs):

{  
   "Records":[  
      {  
         "eventVersion":"2.0",
         "eventSource":"aws:s3",
         "awsRegion":"us-east-1",
         "eventTime":"1970-01-01T00:00:00.000Z",
         "eventName":"ObjectCreated:Put",
         "userIdentity":{  
            "principalId":"AIDAJDPLRKLG7UEXAMPLE"
         },
         "requestParameters":{  
            "sourceIPAddress":"127.0.0.1"
         },
         "responseElements":{  
            "x-amz-request-id":"C3D13FE58DE4C810",
            "x-amz-id-2":"FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
         },
         "s3":{  
            "s3SchemaVersion":"1.0",
            "configurationId":"testConfigRule",
            "bucket":{  
               "name":"mybucket",
               "ownerIdentity":{  
                  "principalId":"A3NL1KOZZKExample"
               },
               "arn":"arn:aws:s3:::mybucket"
            },
            "object":{  
               "key":"HappyFace.jpg",
               "size":1024,
               "eTag":"d41d8cd98f00b204e9800998ecf8427e",
               "versionId":"096fKKXTRTtl3on89fVO.nfljtsv6qko",
               "sequencer":"0055AED6DCD90281E5"
            }
         }
      }
   ]
}

It is very obvious that the event should not contain the blob object itself, but the bucket ID (and this is the case for Azure Storage, as well as other event types) - so there should be another component that takes the event and gathers all objects required for the execution of the function, as well as put things back to the event source (like other blobs, database entries or pub/sub messages) - this means abstracting the implementation details from the user.

This would significantly close the gap between the FaaS offerings from various cloud providers and OpenFaaS / other self-hosted serverless frameworks.

I would like to start working on the generic event aggregator, but have OpenFaaS in mind for first-class integration - for this I would like to get some insights on where to plug-in with the events, and how to pass the event data back to the function runtime.

Thanks!

[Templum]

Hi @radu-matei ,

I have waited for a response to my previous post. In the meanwhile, we started a custom implementation for our company use case. I would say you will require a connector solution which receives the amazon/azure events and then goes ahead and transforms them into a common format. Also, they can handle transforming / forwarding the received response.

I updated the presented schema based on experiences I made.

{
    "send-timestamp":"1234123123", 
    "event-source": "IP / Name", 
    "event-type":"Datastore.Amazon.Created",
    "invocation-type":"Sync/Async/Fire&Forget", 
    "response-address":"http://send-response.here/path (Used for asynchronous operations)",
    "payload":{}
}

• The event-source field would allow a better tracking of invocations. You might have a configuration which also states which event source will be processed on the configured event source gateway.

• The event-type field follows a defined structure like Domain.Provider.Event based on that you could store json schema for the payload. Which allows validating the received payload. So you could have a list of officially supported events but are still able to add on events + schemas.

• The invocation-type field is for configuring the behaviour. If its a sync invocation the gateway will hold the connection return the function response in a synchronous matter. If its an async invocation it will make use of the provided response-address. If it's a fire&forget scenario the gateway just verify that the request was received

• The payload will vary based on the event, a json schema could be used to verify that the payload matches the schema of the event.

• Authentification could be done via a header or by providing a token

What do you think of that idea?

[alexellis]

I haven't got notifications on this issue recently. I'm interested in defining a pattern for people to follow to provide event-connectors to the OpenFaaS API gateway.

The initial event sources I have in mind are:

• Kakfa
• AWS SNS

The pattern would allow people to implement Azure Blob storage or similar on their own and then contribute that back if they wanted to.

I like the response from @reiz - we need to make sure we honour the core value of a developer first experience (i.e. 1 click setup, simplicity) and also operational simplicity. Tightly coupling OpenFaaS to an event source like Kafka is going to make it hard to maintain these values - but doing it in a way that makes them pluggable / extensible I think makes more sense.

I'd like to make some progress with this. Maybe we can join together for a Zoom call in a week or two?

[nicholasjackson]

Personally I like the fact that OpenFaaS has single responsibility,and I am not sure event sources should be built into the core. The main problem with OpenWhisk is that because it is tightly integrated with a bunch of components it complicates the deployment regardless of what features you need. The modularity of OpenFaaS makes it super easy to deploy requiring and get up and running.

My personal approach is to start to put together a standalone component which integrates into openfaas. I am currently working expanding light weight server which would accept a variety of sources as inputs and take a number of output. This includes, message queues, openfaas functions, web hooks. Messages are translatable using go templates and also chainable.

https://github.com/nicholasjackson/faas-nats

My internal proposal to try and figure out the configuration syntax and which introduces the new types can be found here.

https://github.com/nicholasjackson/faas-nats/issues/3

[alexellis]

Thanks for sharing the NATS.io example.

It would be useful to fully understand the problem before coming up with a solution but there are elements of what Nic has put together which could be re-used. It may be that the events are so different that making a generic handler for all is too hard, but defining a pattern may be easier.

We've already defined some of the inputs and the actions that need to take place for instance:

At a simplistic view:

A message is published on a Kafka topic. That should invoke function 1, 2 and 3.

More complex concerns

• should retry be available?
• is this bi-directional? where is the response published? Another Kakfa topic? Or is it the duty of the function to report the result somewhere?
• are transformations of input messages necessary or required for a minimum viable product?
• how do you store the list of topics / subscriptions and their mappings to functions? Should that be stateful or delcarative? If so - where?

[robertkeizer]

I'm in favour of not having OpenFAAS core handle multiple sources, although I am very much in favour of having the number of sources extended.

Reasons for not supporting many triggering systems in core

• It isn't too hard to imagine >20 different triggering systems as people start to adopt.
  
  
  
  • A single robust triggering system can be maintained much easier than many.
  
  
• Features such as retry, bi-directional, and any transformations are highly specific to different triggering systems. If OpenFAAS starts supporting live streaming debugging logs as an example, exposing that back across all interfaces becomes much more labour intensive.
• Any end user of OpenFAAS who wants to customize the particular triggering system could be left forking the entire project, which isn't ideal.

Proposed solution

• Expose a single triggering system that is as simple, generic, and featureful, as possible.
• Support containers that provide translation between triggering systems and the triggering system.
• Basically exactly what the faas-nats repository does, with individual repositories for each kind of trigger.
• Custom or proprietary triggering mechanisms could be created and not be bound by the same license agreement if committed back to core project.

My suggestion boils down to a system similar to PouchDBs Plugins, with the expectation that the core offering should be robust and well documented, providing examples of how to create a custom triggering system.

As an example of a developer requirement: I would like to see AMQP triggers, hence how I got on this issue in the first place. I'm going to be creating faas-amqp, very similar to faas-nats.

[alexellis]

@robertkeizer how would you see using AMQP in your specific use-cases? Can you give us some idea of how that would look?

[MarsWarrior]

As no one is mentioning MQTT - the standard protocol for IoT - as a source for triggering events, I thought I throw this in :smiley:

The use cases are numerous:

• A device that updates a topic, triggers a function call to initiate a function / alarm / email etc.
• A device that updates a value within a topic, triggers a function depending on the value of the topic
• A function can also be a microservice. That is not part of OpenFaaS of course, but in our opinion we can mix functions and microservices.

All functions (or microservices) publish the result back on a MQTT Topic:

• Devices can subscribe to this result
• Other functions can be triggered based on this result topic: a simple chain of functions can be created by simply publishing data.

Not a single device knows that it (he/she) is firing these events/functions of that it is listening to the result of such an event.

Currently this kind of functionality is build as NodeJS microservices packaged in Docker containers.

[alexellis]

Closing due to work around connector pattern and SDK:

https://github.com/openfaas-incubator/connector-sdk

https://docs.openfaas.com/reference/triggers/