Dgraph: Build subscription logic

[manishrjain]

Build the equivalent of subscribe in GraphQL. So, users can subscribe to a query, and every time something changes, they'll automatically get an updated version of that response. This would be like streaming the results of a query.

[manishrjain]

A user should be able to specify a query, which would stream results back every time the corresponding data changes. The idea I have is this:

• Any server in the cluster gets this query.
• It identifies the predicates required to run this query.
• Identifies the servers serving these preds.
• Opens up GetUpdateStream request to these servers, with the list of preds.
• Every time it gets an updated ping from any of these servers, it sends off the query to be executed.
• It then compares the new result's hash with the previous result's hash.
• If they differ, it would stream the result back to the user.

This is a simple design, and it should achieve what we want here.

[jasonkuhrt]

I think what you're looking for is less subscribe and more live? https://dev-blog.apollodata.com/new-features-in-graphql-batch-defer-stream-live-and-subscribe-7585d0c28b07

[jasonkuhrt]

Would the following use-case be in-scope:

• Fetch a segment of users based on some property conditions
• may return millions of IDs on first result
• have any changes to this segment be streamed, just the delta

[manishrjain]

@jasonkuhrt : Dgraph responses can be nested, and not flat like SQL / NoSQL. So, it's trickier to do a diff and send the diff. Thought, it might be possible.

For now, the task scope is just to return the full results, if there's a change.

[jasonkuhrt]

For now, the task scope is just to return the full results, if there's a change.

Understood. So for a use-case where the result set is very large, streaming change probably doesn't make much sense.

[smkhalsa]

@manishrjain how far out is this feature from being implemented?

It's an important feature for my use case (real-time collaborative problem solving).

[manishrjain]

We had a feature freeze to launch v1.0. We'll start working on features in the new year. Expect this in mid-Q1 2018.

[insanitybit]

Curious to hear if there's been any work on this feature. I currently roll my own eventing for dgraph, and this feature would simplify things.

[good-idea]

@insanitybit Any chance that what you have built is public? I'd be curious to see (or hear more about) how you're handling this.

[insanitybit]

Sure, it's public.

https://github.com/insanitybit/grapl

I pity anyone who reads that code though - I wrote it before I really understood much of dgraph and it's horribly inefficient and verbose. It's the graph-service graph-merger binary.

It's not nearly as efficient as a dgraph-provided system. Basically, when merges occur, I emit events to SNS, and subsequent systems perform queries using the metadata (timestamps).

If I had a native system that would just re-execute the queries and emit the results directly it would cut out quite a lot of complexity.

[insanitybit]

"@insanitybit insanitybit unassigned janardhan1993 an hour ago"

No idea what that's referring to btw

[peter-clemenko]

Any update on this? This would be very useful

[danielmai]

Subscriptions is on the 2019 roadmap.

[seanlaff]

Are there any public design docs for this feature? Any mutation could theoretically match any subscription and it seems like you could hit a performance wall pretty fast with lots of subscriptions. Im guessing there's a way to bound the problem, and Im interested to see what the team is thinking :)

Also on the topic of subscriptions- Will dgraph support piping subscribed queries to a message bus as opposed to requiring a client to have an open connection/subscription? ( which can result in missed data if either go down). Cockroach recently added this and I think it's a killer feature https://www.cockroachlabs.com/docs/stable/change-data-capture.html

[campoy]

This should be part of our next release v1.2, on which we will start working as soon as v1.1 is released by the end of July 2019.

[liveforeverx]

@manishrjain Question, can this be used than from native GraphQL+- interface or only from GraphQL interface? Even if this will support only limited features of GraphQL+-, it would be really great to access from GraphQL+- directly.

Thank you guys for a great database with a great plans 👍

[campoy]

This issue is about adding subscriptions to GraphQL+-.

GraphQL has its own subscription system, which we will indeed implement as part as our native GraphQL layer support.

[nickname32]

So... Is it implemented yet?

[shekarm]

@asminozhka We are working on this and other features. This issue has been accepted and in our backlog. Please watch for new releases and corresponding logs for more information.

[stevefan1999-personal]

@asminozhka don't be begging too much, it's open source and it is to the generosity of the amazing contributors to support this project. if you can you could try and make it happen

[MichaelJCompton]

We've added subscriptions to Dgraph's GraphQL support. That should be released in 20.07.0.

There's no current roadmap for addressing this for GraphQL+-.

[liveforeverx]

@MichaelJCompton Means, the plan has changed, that most probably for next year there willn't be a support for native GraphQL+-, right?

[github-actions[bot]]

Use Discuss Issues for reporting issues about this repository.