Prisma1: Add support for more PubSub systems

[mavilein]

Is your feature request related to a problem? Please describe.
In the issue on horizontal scaling it became apparent that our community would like to see support for other PubSub systems than RabbitMQ.

Describe the solution you'd like
We should add support for more PubSub systems. This should probably be configurable through the Prisma server config.

Describe alternatives you've considered
People do not really like the additional dependency for a PubSub. We need it for those reasons:

1. to power subscriptions over Websockets
2. to store ServerSideSubscriptions/Webhooks in a Queue. This allows us to eventually deliver Webhooks even if the Server goes down due a crash or reboot.
3. to propagate information about schema changes to all servers. We need this as each server holds a cache of the schema of service. Not having a cache would mean adding significant latency to query execution.

Many people are probably fine with not having support for point 1 and 2. So we could come up with an alternative solution for 3 that could involve polling or a database level pubsub.

[goulderb]

My team uses Kafka as an integral part of our stack already, and I would vote for having Kafka as a pubsub option so that we could directly subscribe to messages from Prisma rather than our current solution (which is basically using the WebSocket subscriptions to then push things into Kafka for the rest of our system to consume).

[mavilein]

@goulderb : Sounds good to me. We are looking into Kafka for other features anyway. As a temporary workaround i would suggest using our Server Side Subscriptions aka Webhooks that support the same subscription queries. That is probably easier to use than the Websockets.

[goulderb]

@mavilein Thanks for the suggestion. I will definitely look into using server side subscriptions.

[mcmar]

Any of the systems that Apache ActiveMQ supports.
JMS, NMS, AMQP (1.0, not 0.9), STOMP, MQTT

[develomark]

I'd like to cast my vote for Google PubSub also as we currently have a cloud function consuming Prisma subscriptions over HTTP and passing them into GCP PubSub. This would reduce the latency/complexity.

[mcmar]

It looks like there are 2 separate issues being discussed in this thread:

• An alternative/replacement for the RabbitMQ dependency.
  Currently Prisma uses the RabbitMQ client library (AMQP 0.9) for synchronizing the primary/secondary servers and propagating various events internally to all the prisma servers in a cluster.
• An alternative/replacement for the Server-Side Subscriptions in HTTP.
  Currently Prisma only accepts webhooks for server-side subscriptions.
  The Prisma team has said before that they may also support direct Lambda invocation in the future.
  Kafka/Google PubSub are both great for this. They're both data pipelines made for consuming events in your application.

I see no reason that we can't use different technologies for solving these two problems.
ActiveMQ is a replacement for RabbitMQ and supports more messaging formats. As a client, Prisma only needs to support 1 of the many messaging formats.
Kafka and Google PubSub would both be good additions to the Server-Side Subscription formats. I don't think anyone is asking for Kafka or Google PubSub to be implemented for horizontal scaling purposes. I think they're just asking for it as an alternative format for SSS.

[develomark]

Google PubSub would be useful also for horizontal scaling if your Prisma servers are in GCP. GCP PubSub is infinitely globally scalable and far simpler to implement and support over spinning up an additional PubSub server to support messaging.

[frandiox]

graphql-subscriptions supports a bunch of PubSub implementations. It would be nice to see some of them in Prisma in order to use the same infrastructure we already have for the app servers: https://github.com/apollographql/graphql-subscriptions#pubsub-implementations
Perhaps if the PubSub API part of Prisma server is documented and refactored as necessary, these PubSub implementations could be made by the community just like in graphql-subscriptions.

That said, I also vote for Google PubSub 👍

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

[j4jefferson]

I am currently trying to deploy graphql-yoga/prisma backend to GCP Kubernetes cluster and hooked up to CloudSQL Postgres and finding it very hard to figure out. Having got the app node and prisma node successfully deployed with a custom prisma image packing cloud_sql_proxy I then tried to connect to prisma cloud and failed. That's when I discovered this issue of the management api and pubsub. Getting prisma to production ain't easy folks! I definitely want to cast a vote for GCP pubsub for primary/secondary sync as well as subscriptions.