Graphql-engine: Could we have PRE event-trigger for customize data validation?

[hengsovandara]

I wonder whether I could have a pre-event-trigger to customize data validation before saving record. I known that Hasura has remote-schema which could do this, however, I think I don't need to create server to run the remote-schema. I just want to create Lambda function to check validation before saving records.

[hengsovandara]

Here is the example of the process that I need, where it is acting after client submit, but before writing stage into database.

[tirumaraiselvan]

I concur it would be great to have a pre-commit webhook, but currently it is not easy to implement this in the present event triggers design. This is because the events are generated atomically with a mutation using native Postgres triggers. To accomplish pre-commit webhook, we will need to use something other than Postgres triggers so it will be a different sub-system.

@coco98 @0x777 thoughts on this feature?

For the time being, there are 2 solutions:

1. Like you mentioned, you can implement your own graphql API on Lambda and stitch it with Hasura.
2. You can perform a mutation to a temp table, which will trigger your validation lambda and in that validation lambda you can verify and perform an insert on the actual table. You can clean up the temp table afterwards.

[hengsovandara]

@tirumaraiselvan , I am happy to get answer from you. And if I choose the second option, how could I get the error message to the client side, while it stores on event logs, not return into the client.

[hengsovandara]

And How about the authentication system Hasura_Qraphql_Auth_Hook, is it Pre-Hook?

[tirumaraiselvan]

@hengsovandara You will have to handle it asynchronously. You will have to emulate a sync like process for e.g. after performing the mutation to the temp table, you should wait/block the UI and poll (or use subscriptions) for the validation status.

Since validation is more naturally a sync process, hence approach 1) is recommended :)

Regarding auth system, yes it is pre-hook and that is exactly the kind of system which would be needed to achieve pre-hook event triggers. Let's see if there is enough interest in this.

[tirumaraiselvan]

@hengsovandara A 3rd way is to use CHECK constraints or postgres stored proc for validation although this is best suited for validations on data based on regexes and simple constraints.

[hengsovandara]

I have been doing on that too, but what the problem is we don't have
postgres developer and also don't want to waste time on learning while
doing a project.

On Thu, Jan 3, 2019, 6:34 PM Tirumarai Selvan <[email protected]
wrote:

@hengsovandara https://github.com/hengsovandara A 3rd way is to use
postgres stored proc for validation although this is best suited for
validations on data based on regexes and constraints.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hasura/graphql-engine/issues/1291#issuecomment-451119193,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aadv6RBV7shqBJLBKk1v8QaAfat2tsK1ks5u_ercgaJpZM4ZnpWI
.

[tirumaraiselvan]

@hengsovandara Understand :)

Why don't you give Remote Schemas a try? With the boilerplates available at https://github.com/hasura/graphql-serverless, I think it should be a very easy process. It would be same as your Lambda code and the wrapper code is already present. If there is any trouble, we will be happy to solve it for you.

[hengsovandara]

I just seen the example lately. I will have a try. However, I think if
Hausra could give a option like I raised above, it would make life easier.

Thank you every much for supporting all the time.

On Thu, Jan 3, 2019, 6:55 PM Tirumarai Selvan <[email protected]
wrote:

@hengsovandara https://github.com/hengsovandara Understand :)

Why don't you give Remote Schemas a try? With the boilerplates available
at https://github.com/hasura/graphql-serverless, I think it should be a
very easy process. It would be same as your Lambda code and the wrapper
code is already present. If there is any trouble, we will be happy to solve
it for you.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hasura/graphql-engine/issues/1291#issuecomment-451123071,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aadv6evNZ3PYUvd0JcFPQt33hzMdO0cIks5u_e-YgaJpZM4ZnpWI
.

[hmexx]

@hengsovandara

When I was using firebase, I bumped into similar issues. Reading state from a database directly, with an ACL for access control, is amazing, and saves time. You also get "real-time" updates for free on both platforms. But modifying/creating state, almost always ends up needing some custom code.

Therefore the convention I came up with was simple. All state mutations go through cloud functions, and (almost all) read-only queries go straight to the DB.

I suspect a simliar convention would work well with Hasura so as to not get stuck down the line; i.e. use a remote schema for EVERYTHING that modifies state in the database. Use Hasura's awesome autogenerated schema for everything else.

[brodie-hodges]

If Hasura had this feature then you wouldn't need a separate remote schema server for 95% of cases. Most other business logic could be handled with events and subscriptions. Couldn't Hasura leverage the remote schema functionality internally to achieve this? Would also be great to have common field validations easily configurable: email, credit card / mod-10, US state / zip, sql injection prevention for text fields, URL format, etc.

[tirumaraiselvan]

@brodie-hodges For common field types like email, credit card, zip, etc , you can perhaps use Postgres CHECK constraints : https://www.postgresql.org/docs/9.5/ddl-constraints.html#DDL-CONSTRAINTS-CHECK-CONSTRAINTS

For more complex use-cases, we have a solution in the works called "actions" which will be out pretty soon. This will eliminate remote schemas use for tasks like these.

[mnlbox]

@tirumaraiselvan good news brow :wink:

[brodie-hodges]

@tirumaraiselvan Thanks for the update. Is this on a roadmap somewhere?

[icewind]

Thanks for the update! Are there any details about upcoming "actions"? 😊

[iamjrock]

Thanks @tirumaraiselvan! Support for custom pre-insert/update/delete business logic validation via the upcoming "Actions" functionality will be incredibly useful.

Where can we follow the roadmap for this please?

I checked and the release candidate roadmap does not seem to mention Actions.

[sarendsen]

Subscribe

[cantaspinar]

It would be great to have pre triggers for custom logic. Are there any updates on that?

[ec-raphael]

For more complex use-cases, we have a solution in the works called "actions" which will be out pretty soon. This will eliminate remote schemas use for tasks like these.

Any word on the "actions" solution @tirumaraiselvan ?

[0x777]

Hi, we've been actively working on this. We'll have a preview build by the end of this week.

[cfharyadi]

Looking forward for any updates regarding this feature
Many thanks for your awesome work.

[PedramMarandi]

Guys, it's an amazing feature which let us rely on Hasura even more. I've had to create custom endpoints to be able to just handle my data validation.

Thank you for this incredible effort.

[plmercereau]

Hello! Any update on this? I'm curious about what you meant by "actions", if it's related to backend validations or broader synchronous hooks prior to the postgres operation...

[victorferreira]

I know it's hard to share a timeline, but it'd be awesome if you guys could share the rational behind this "actions". How do you plan to implement this feature and we might be able to help with feedback. I'm also looking forwarding to this feature.

[Ant59]

Any updates around when this will be available? Lack of validation for inserts is the only thing stopping my team from adopting Hasura into production.

[piroszhog]

Definitely, it's one of the most important thing for the custom business logic implementation!

[MikeBosw]

This issue is my blocker. It's a make-or-break issue for me.

The reason I almost went with Hasura for my company was the beautiful simplicity of using it relative to other backend architectures. Models, migrations, permissions, and a super-flexible frontend API, with support for realtime pushes... hurray! But I do need business validation (beyond authorization). If I had that, ermahgerd. And yet without that, the value proposition is unfortunately undone. But I'll keep an eye on this thread and I very, very much look forward to your upcoming feature.

[beepsoft]

@MikeBosw as far as I understand the "Actions" feature will make this possible/easier. But even without this you _can do_ validation using Postgresql trigger functions today:

https://blog.hasura.io/postgres-triggers-on-graphql-mutations-682bf48db023/

https://docs.hasura.io/1.0/graphql/manual/guides/auditing-tables.html

[lukaspili]

A high level overview of actions has been made available here: https://deploy-preview-3042--hasura-docs.netlify.com/graphql/manual/actions/index.html (found on PR https://github.com/hasura/graphql-engine/pull/3042)

[MikeBosw]

@MikeBosw as far as I understand the "Actions" feature will make this possible/easier. But even without this you _can do_ validation using Postgresql trigger functions today:

https://blog.hasura.io/postgres-triggers-on-graphql-mutations-682bf48db023/

https://docs.hasura.io/1.0/graphql/manual/guides/auditing-tables.html

Thanks. As you say. Mentioned also in an earlier comment. The trade-off from an architectural standpoint is the conflation of your database with your application, but yes, if I had to pick between the one approach (remote schema) and the other (Postgres constraints) I'd definitely pick the latter.

[MikeBosw]

A high level overview of actions has been made available here: https://deploy-preview-3042--hasura-docs.netlify.com/graphql/manual/actions/index.html (found on PR #3042)

Exciting :)

[jorgecolonconsulting]

@lukaspili

I’m a little thrown off by the Python example. Isn’t it supposed to be in Node?

On Oct 11, 2019, at 3:12 PM, Michael Bosworth [e-mail] wrote:

A high level overview of actions has been made available here: https://deploy-preview-3042--hasura-docs.netlify.com/graphql/manual/actions/index.html (found on PR #3042)

Exciting :)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[lukaspili]

@2upmedia I just found the PR and read the doc, I don't have more info.
From my understanding, actions trigger a webhook so this logic happens outside of Hasura. But yeah, the most common architecture will be a serverless function running on node.

[jorgecolonconsulting]

@lukaspili thanks. I'll reach out to the person that created the PR.

@0x777 any ideas why there's a Python example in the documentation?

[MikeBosw]

@lukaspili thanks. I'll reach out to the person that created the PR.

@0x777 any ideas why there's a Python example in the documentation?

@2upmedia, if my understanding is correct, the HTTP endpoint is something you yourself create, which means that you can use any language, be it Node, Python, or Racket. In what sense is it supposed to be Node?

[valstu]

Python syntax is almost like writing some sort of pseudo code so I guess it makes sense. But yeah, your backend could be written in any language so it really doesn’t matter like @MikeBosw said.

[coco98]

Also, we're planning to add support for the action handler to be a postgres function (say, plv8) or an HTTP endpoint. So you can move from one to the other quite easily. For example, start off with a postgres function if its simple data validation type code and move to a HTTP endpoint later without affecting anything.

[jorgecolonconsulting]

https://github.com/hasura/graphql-engine/issues/1291#issuecomment-541206594

@MikeBosw

This makes sense now. I wasn’t aware that the action could be an HTTP endpoint.

[mklueh]

Stumbled across this amazing project and handling pre-processing / validation logic was my only concern when adopting it. I'm very glad to hear you are working on a solution

[yasinarik]

@tirumaraiselvan , @coco98 , @0x777 , I've been waiting for a long time for actions feature to be released.

How long will it take? It will change our DB structure dramatically. Here is an example of a use case: #3536

Thanks for building the amazing Hasura!

[KirillSuhodolov]

Hi.
What is current status?
Lack of this feature is a blocker for me start using hasura.io for new projects

[rkawajan]

I'm so excited for this feature! Are there any updates on that?

[jmonday]

Maybe these are two different things, but I'd love to be able to specify when the event should be triggered. Right now, it looks like it's being triggered after the update/mutation is made. I'd like to be able to configure the event to run before an update/mutation. This way, my webhook can determine if the update is allowed and/or modify the response for additional processing (thinking ETL cases). If my response is, say 403, the mutation is not executed/blocked.

Thoughts?

[jmonday]

I was reading the documentation further and it seems that something similar is already in place but I'm not seeing it work as described (or I'm misunderstanding the documentation).

Webhook response structure

A 2xx response status code is deemed to be a successful invocation of the webhook. Any other response status will be deemed as an unsuccessful invocation which will cause retries as per the retry configuration.

I see that if I return a non 2xx status code, the retries are fired as expected but the database update has already been executed and not rolled back. When first reading about this feature, I thought of it almost like a database transaction where the transaction would not be committed if the status code was non 2xx.

[tirumaraiselvan]

@jmonday The docs you are referring to are for Event-Triggers which are very different from Actions -- the topic of this thread.

Anyway, in event-triggers, you have transactionality between the database mutation and the creation of the event. But, you don't have transactionality with the invocation of the webhook (as it is an external system).

[tirumaraiselvan]

Hey everyone

Happy to announce that we have released Actions in v1.2 (currently in beta) to solve this problem. Please try out the beta from here https://github.com/hasura/graphql-engine/releases. Expecting to release stable by week-end.

Why did we build Actions to solve this problem?

• Data validations, in general, are NOT simply static in nature (if they are static or even declarative, consider using CHECK constraints or permission rules). This means that most validations will involve checking things against the db: i) see if inventory stock > 0, ii) see if shopping cart size < 10 and so on.

• In view of above, the validations and the operation cannot be separate. Otherwise, you will have a stricter programming environment, no transactionality, brittle business logic, etc

• Complete control: You may need greater functionality in your operation. For e.g. suppose you want to validate in article to be "clean" in content during an insert_article, if the article is not "clean" you might want to perform red flagging the author by performing a different mutation instead (update_author).

How does Actions work?

• You type your desired mutation and click on Derive action on graphiql (you can also define it explicitly): https://hasura.io/docs/1.0/graphql/manual/actions/derive.html. You can customize the derived action, if you want.

• You also get a boilerplate which you can fill with your custom logic and you can host this anywhere to get a HTTP endpoint.

• Provide this endpoint in your action metadata and that's it. You can also start creating relationships on an Action with your existing tables: https://hasura.io/docs/1.0/graphql/manual/actions/action-connect.html

All this is available via the CLI as well to fit nicely with your existing dev environment.

Closing this issue now, but feel free to ask any questions here or on Discord !

[dariocravero]

Nice @tirumaraiselvan! Amazing work on actions.

For anyone else seeing this, we've been using them for a couple of weeks in production and they are super useful. Right off the bat, it saves you from having to write the graphql stuff surrounding your logic which means that a simple function like the one you were using in events pretty much works. 👏

One question I had was whether you'll be opening actions to use them as queries apart from mutations. I remember that when we discussed this @coco98 said it should be almost as easy as letting the keyword query be used instead of mutation. What do you think?

Another thing that comes to mind is being able to use the existing types within the app. enums would be particularly handy but also uuid, timestamptz etc. I know that I can write my own and we've been doing it so far but being able to access the system's ones would help keep things consistent. Is that something you thought about?

Thanks!

[tirumaraiselvan]

@dariocravero Yes, query support is almost merged: https://github.com/hasura/graphql-engine/issues/4032 and so is re-using postgres scalars: https://github.com/hasura/graphql-engine/issues/4125 . We will keep improving other things as we go along.

[dariocravero]

That's class! :)

On Tue 14 Apr 2020, 12:15 Tirumarai Selvan, [e-mail]
wrote:

@dariocravero https://github.com/dariocravero Yes, query support is
almost merged: #4032
https://github.com/hasura/graphql-engine/issues/4032 and so is re-using
postgres scalars: #4125
https://github.com/hasura/graphql-engine/issues/4125 . We will keep
improving other things as we go along.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hasura/graphql-engine/issues/1291#issuecomment-613352189,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC33TEQPS4YLBE4IWOVRZ3RMQZS5ANCNFSM4GM6SWEA
.

[RodolfoSilva]

@tirumaraiselvan has any plan to add support for the Postgres functions?