Dgraph: Product Roadmap 2018

[manishrjain]

Work In Progress

• Support Open Cypher
• Support Gremlin
• [ ] Support GraphQL spec
• [ ] Subscriptions
• [ ] Stronger Type System
• [ ] Backup
• [x] Python Driver
• [x] Simplify TLS setup
• [ ] Authentication (full control)
• [x] Jepsen test integration (http://jepsen.io/analyses/dgraph-1-0-2)

[WolfDan]

The GraphQL spec also includes specs like Union and enum types @manishrjain ?

[manishrjain]

Yeah, we'll try and support it as close to the spec as possible. I think GraphQL compatibility is needed by a lot of users.

[manishrjain]

Would anyone need support for JDBC driver?

[smkhalsa]

With support for Cypher, Gremlin, and GraphQL as the first 3 items on the Roadmap (not sure if the roadmap is in order of priority), is the intention to simply appeal to a broader base of users who may already be familiar with one of them? Or is there a valid reason why a user would want to use all three (e.g. tradeoffs in performance, simplicity for certain types of queries, etc?)

[smkhalsa]

@manishrjain What's your vision for Subscriptions? Do you intend on implementing the GraphQL Subscription spec or provide full reactivity that could enable things like @live queries?

Any ideas on what the API will look like?

[WolfDan]

Yeah, we'll try and support it as close to the spec as possible. I think GraphQL compatibility is needed by a lot of users.

That could be amazing! In my case I'm creating a Graphql api with Elixir, have the same schema in the app and the database can simplify the work a lot, I hope this get done soon :D

[akuckartz]

Is SPARQL already supported?

[sebastianmacias]

Great news! I'm really looking forward to Subscriptions. It would be a huge plus if dgraph could borrow some of the offline-first ideas from gun.js, extremely useful for PWAs.

[MichelDiz]

"Support GraphQL spec" - So, Will Dgraph have an optional "virtual" Schema type just for GraphQL? @manishrjain

[thetuxedo]

how about key cardinality? such as "Tom Hanks" has a predicate of type to multiple values "actor,director"

[rogerwaldvogel]

@manishrjain Can you tell when approximately Open Cypher support is implemented?

[MichelDiz]

Graphcool Graphql IDE turns out to be opensource in november 2017.
i believe that we could attach dgraph-js as a wrapper to graphcool IDE.
Or Dgraph Team observe them as an example to create Dgraph support for GraphQL IDE like.
https://youtu.be/WhYjSFKNhBA

See:
https://github.com/graphcool
https://github.com/graphcool/prisma
https://github.com/graphcool/graphql-playground

Check it out!

[sorenbs]

@MichelDiz - thank you for mentioning Graphcool :-)

GraphQL Playground is an open source replacement for GraphiQL and in no way specific to Graphcool or Prisma.

We'd be happy to jump on a call if anybody from Dgraph want an introduction or help integrating it. In the future we'll be adding a plugin system as well as support for custom styling that should make it easier to tie into Dgraph.

[MichelDiz]

@sorenbs Hey, I'm interested in the connector for Prisma. But I have no concept of programming with SCALA.

I think Prisma uses the prisma-binding, correct? If not, what are the principles for assembling a connector for the Prisma?

I think this would be enough to make Dgraph align with the graphcool tools.

[Textras]

Any update on how the top 2 items are going (Cypher and Gremlin) now that it is March?

[manishrjain]

We're a bit bogged down right now -- but expect to start the work on Cypher in early Q2.

[jimanvlad]

@manishrjain

Would anyone need support for JDBC driver?

We will need to extract tabular data out of dgraph for machine learning models. I assume JDBC would be the best way?

[chengyu-liu-cs]

It would be really nice to have scala driver in addition to others.

Do you have any plan to support subquery, i.e., something like

[manishrjain]

Dgraph supports sub-queries like these. We have the concept of query blocks. One query block's results can be used by another.

https://docs.dgraph.io/query-language/#multiple-query-blocks

[chengyu-liu-cs]

@manishrjain I knew query blocks. Probably I was using wrong term, maybe "subgraph query". Or I was totally wrong.
I was asking in slack how to query "Find all users who have type "user" and who have "visited" places "from" more than N distinct cities* , for example, more than 2 cities.."

Pawan suggested a solution like this and commented that I should do one more step in my own application. It seems the filtering step is not supported by dgraph currently. If I can do one more filter ontop of this result, then I do not need to have my application do that.
{
query(func: eq(type, "user")){
uid
type
visited @groupby(from) {
count(uid)
}
}
}

Another non-supported function is that it is impossible to return or count other scalar values.
The reason is that in my case, name is used as external ID (unique) that identifies my application item. So in the current situation, I have to make another query to get the "name" and combine the two results.

[manishrjain]

Dgraph supports filtering among blocks, which is what you're referring to as subquery filtering. The issue with this specific query is that Dgraph currently doesn't have a distinct count feature. That's what is needed to run this query.

Also, I think we need a feature where we can expand out two or more edges at the same level.

{
var(func: eq(type, "user")){
  u as uid
  type
  d as distinct(visited>from)
}
// Then use d to filter u further in another block.
}

[smkhalsa]

@manishrjain Can we get an update on the roadmap and on which features are currently prioritized? There hasn't been an update in a couple months.

[experimenti]

If you support graphQL, you don't need Prisma. It's a semi-auto-magic ORM for databases needing graphQL. (There may be some other functions, but that's the gist of it).

[manishrjain]

@smkhalsa: We're focused entirely on fixing up the various bugs in the distributed cluster setting. So, features have taken a back seat. We'll start working on them over the next couple of months.

[pmualaba]

Hello, could you please consider adding the following feature https://github.com/dgraph-io/dgraph/issues/2000 to this roadmap. It is probable low hanging fruit on the implementation side, but enables so much power for our GraphQL+- queries.

[doofin]

Support for query languages like Gremlin is absolutely necessary,for I think GraphQL has almost no connection with graph database (unable to represent graph theory related things intuitively) .. How about the progress?

[ian19861]

No release from two months. Please share the timeline for features

[manishrjain]

Preparing a release with many bug fixes. Should have something out this week. Features are not the top priority at the moment.

[zhicwu]

@manishrjain

Would anyone need support for JDBC driver?

If GraphQL compatibility is still on the list, it would be great that the JDBC driver can cover both DGraph and GraphQL(without transaction support). The driver can be implemented as an adapter on top of Apache Calcite, or standalone for least dependencies.

Once SQL to GraphQL implemented, maybe you guys can also consider ODBC driver to benefit tools like Power BI :) It is easier to analyze tabular data set while it usually requires more work (expand/flatten/split/merge/transform) for nested JSON.

[billylindeman]

@manishrjain Is cypher support still in the works for this year?

[manishrjain]

Cypher support won't happen this year. We're working on Backups and then on ACLs. GraphQL official spec takes a higher priority than Cypher.

[manishrjain]

Update: We've punted on Cypher and Gremlin support for the roadmap. The focus is on GraphQL and other features mentioned here. Two of them are already being worked upon, i.e. binary backups and access control lists.

[jccq]

I guess punted means "you're not going to do it". If that's the case i would suggest it to be a bad choice manish. Remember when I was commenting that "proprietary modified Apache licence wasnt going to work" and you guys pushed back saying "nooo its ok" but then i was right, it was not ok and now you're back to Apache.

:) ok that was just an example:

now my prediction is that if you want to be relevat in the graph db world the sooner you get cyber or gremlin support the better.

Graphql can be great for very simple stuff but for arbitrary ad hoc queries the syntax i am afraid is way too far from what anyone would care to learn. Just trying to help here really!

[manishrjain]

I guess punted means "you're not going to do it". If that's the case i would suggest it to be a bad choice manish. Remember when I was commenting that "proprietary modified Apache licence wasnt going to work" and you guys pushed back saying "nooo its ok" but then i was right, it was not ok and now you're back to Apache.

I guess punted means "you're not going to do it" bad choica manish. Remember when i told you that a "proprietary modified Apache licence wasnt going to work" and you told me "nooo it will why not bla bla" .. and you went back to that.

I'm not sure what your point is, but your language (both the original version and the update) feels unnecessarily aggravating and derogatory. Dgraph was the first company to adopt Commons Clause, which was then used by Redis Labs and is now being used by Confluent. A similar license, called SSPL, is being used by MongoDB and being pushed to OSI. We actually had an uptick in usage when we switched from AGPL to Apache + Commons Clause. So, if anything, you were completely wrong about it. Nothing to be proud of.

Now, coming to your "prediction" -- Dgraph is already relevant in the graph DB world, without having ever touched Gremlin or Cypher. When I updated this thread, I didn't mean to say that we would "never" support Gremlin. But, it takes a lower priority for us than the other items on the roadmap.

Graphql can be great for very simple stuff but for arbitrary ad hoc queries the syntax i am afraid is way too far from what anyone would care to learn

You can make a less offensive and more logical argument here by talking about what queries you are unable to solve with our current QL, that can only be solved by Cypher -- and how important are these to your use case. Then we can think about how best to provide you with a solution.

[manigandham]

Since it's almost 2019, is there a new roadmap? Perhaps it's better to have a consolidated page since the issue here shows Cypher/Gremlin crossed out rather than delayed.

From our perspective, interfacing with the database through server-side and backend systems is much easier with standard graph languages like Gremlin or Cypher which already have many libraries and frameworks available.

[jccq]

@manishrjain back when you switched to common clause I (and now I read many others) immediately raised that that was going to wipe the apache benfits away compeltely. One year after that conversation your own words "While the promise of Commons Clause is a great one, i.e. restricting resale while allowing proprietary usage, its subjective implementation generates a lot of friction and confusion for the open source community."

So i am not sure how I would be completely wrong :) but anyway really i am trying to help here. I am no competitor to you guys just a believer that you do have great technology and that commercial/positioning choices are very important.

All i am saying here is that going "solo" with your own, arguably unclear, query language puts the rests of the great technology greatly at risk.

Do you want another example? remember when you spent time arguing that a dgraph was not a transactional system and that people didnt really need transactionality all the time just to realize they did and implementing a great transactional system?

What i am arguing here is that your energies might be better spent to implement a great gremlin or cypher support rather than fighting it . Its all similar to the Nosql vs SQL (As everyone is going back to SQL now). Just trying to help, take it easy. (Also apologies if it sounded derogarory, it wasnt meant to )

[MichelDiz]

IMHO GraphQL is way superior than gremlin or cypher for a Graph DB.

And remembering that GraphQL+- is an almost direct inspiration from GraphQL.

IMHO again, Put Both SQL like languages as main importance here ​​is a logical throwback to Graph DBs. Because you are forcing a SQL-like syntax to comfort SQL mind users. Not that they are not important, they are, but a GraphDB has its own language/ways by its very essence. For me to focus resources on giving this support is like asking a mighty eagle not to fly.

Let's see numbers - Math never lies :P
First, Google Trends: https://trends.google.com.br/trends/explore?q=graphql,Cypher,Gremlin

On Github:

GraphQL - 32,259 repository results (not saying that repos are important)
10,513 stars - 76 contributors

Gremlin - 677 repository results
1,755 stars - 14 contributors

OpenCypher - 1,628 repository results
312 stars - 24 contributors

Dgraph - 458 repository results
7,190 stars - 90 contributors

The numbers tell me that GraphQL will be even more significant in 2019. This community is growing every day. They support many languages. Maybe more than Gremlin and Cypher combined. And they have more than 10 times the number of users for sure.

And personally, I do not see fun at Gremlin or Cypher. But why not? Also PR are always welcome.

In my view the cost benefit and ease of implementation is in favor of GraphQL is in light years.

Cheers.

[smkhalsa]

@jccq if your goal is to advocate for cypher / gremlin, you may want to try a different approach. Simply pointing out what you believe to be Dgraph's missteps, and suggesting that since you knew they were mistakes all along, the Dgraph team should just do as you say is misguided at best, bordering on trolling.

[smkhalsa]

@manishrjain I'd like to offer my use case.

TLDR: I'd be happy with just GraphQL (i.e. no Cypher) so long as you can make it play nice with GraphQL spec compliant tooling while retaining support for arbitrarily complex graph queries.

I'm building a GraphQL Server as the backend for a web / mobile app. I'm leaning towards using neo4j, coupled with the neo4j-graphql-js library because it allows me to:

1. Easy delegate GraphQL queries from my GraphQL API Server to the graph database

Since most of the time the fields in my GraphQL API schema map directly to the structure of the data in my graph, the ability to defer the query and automatically fetch the requested properties / relations makes building my GraphQL API a breeze (i.e. I don't have to write any resolvers!). If my client needs to request fields that don't map to the structure of my graph, I can still handle this easily with cypher directives (see below). Authorization is managed as a separate layer using middleware.

2. Use a single database query to resolve the entire GraphQL request

By deferring the incoming GraphQL query to neo4j-graphql-js, which generates the appropriate cypher query, I'm able to map each GraphQL API query from my client to a single database query. Eliminating multiple calls between by my GraphQL API server and database is a big win in network latency.

3. Retain ability to make arbitrarily complex graph queries using the cypher directive

GraphQL is somewhat limited in its ability to express complex queries / operations (e.g. aggregations), which I believe is why you created Graphql+-. neo4j-graphql-js provides an escape hatch with the cypher directive to allow for arbitrarily complex cypher queries. It also allow me to map fields if my GraphQL API schema doesn't exactly match the structure of the data in my graph.

Conclusion

My issue with GraphQL+- is NOT any limitation in its ability to query / operate on my data. Rather, it's that it doesn't provide an easy way to interact with the exiting tooling for building GraphQL APIs.

[jccq]

@MichelDiz your analysis would work if GraphQL was a query language, which is not.

I dont claim to be an expert but it seems well known that GraphQL may be a much better alternative than Rest for many APIs, might resemble querying a graph for certain applications but it was not designed as the interface to a graph DB to ask arbitrary ad hoc questions (possibly Touring complete a la SQL or SQL extentions commonly used in commercial DBs) .

Now, i don't know graphQL+ and it might be it does all really.

It might be you guys proven its complexity, its expressivity compared to others, the difficulty in optimizing as they have done for other QLs (e.g. https://users.dcc.uchile.cl/~jperez/papers/reasoning-web09.pdf )

I am not going to debate (but please consider it internally)

What i am doing here is sharing a first impression

Assuming i could get over my expressivity concerns, as IT manager I would still be quite puzzled in choosing a syntax for asking ad hoc/analytical questions that is different from what everyone in my team already knows.

I would say that being proud of being "NoSQL" is something that lost its appeal years ago, and the trend has reverted (for clear commercial/utility reason) to exposing SQL similarity as a strong positive (for all e.g. Elasticsearch new SQL) but also for graph e.g. see TigerGraph, ArangoDB, i believe Orient also and others

anyway to wrap up, while for certain use case (e.g. the graphQL crowd) being a "graphQL native" db is great and i would by all mean keep it, I would suggest you guys consider adopting a syntax that is closer to SQL to make adoption easier by:

• being easier to understand and get started
• being perceived as "more sound" language, age tested, theoretically sound, known to work and all.

(.. and i guess this rules out gremlin :P :) ) . Cheers.

[MichelDiz]

Now, i don't know graphQL+ and it might be it does all really.
What i am doing here is sharing a first impression

If you do not know GraphQL+- you have never used Dgraph, did you?
Try it first and share your experience later.

You're funny, tell me who decides what is a Query Language or not?
Just because it is not a language created for a DB or SQL Database.
It does not mean that it is not a query language. This statement is very arbitrary.

As Manish said, shifting focus does not mean giving up support other Queries langs. Period.

If you know the history of GraphQL you know that its creation was not only to give an improvement in API's context, but to overcome old fails of SQL DBs. And GraphQL was created to make life easier for the front-end. Better indirect communication between devs and more versatility and suppleness.

Unlike SQL, GraphQL was not created with "tables and Joins" in mind. It was created to match the reality of the objects inserted in the DB's in an intelligible way. Having as philosophy "the format of the Query corresponds to the format of the result of the query". That's like GraphQL is for butter on bread, just as SQL is for stack up stones.

And it happen that the GraphQL (Graph Query Language) it's like gloves for a Graph DB.

I am not going to debate (but please consider it internally)

I think this discussion is going on for almost a year internally. GraphQL won.

I think the next one could indeed be the gremlin. Because this has open source
interpreters for Cypher. Virtually Dgraph would be accessible for both languages.
But not for now.

Cheers.

[jccq]

The approach "use it first and then find out 6 months later that the design is inherently limited" won't work for me as well as for any sensible decision makers in the IT world.

I of course didn't say GraphQL is not a query language which can ask questions to a graph. It can but what class of questions can i ask, and what is the efficiency?

These are questions which are crucial in enterprise projects and people are spending considerable resources (example for Gremlin, example for RDF, example, example for GSQL, example covering Cypher ) to prove the expressivity and properties of their graph languages exactly for this reason.

If you're going a different way (adding some proprietary extensions to something that was originally intended only for relatively simple data structure retrieval), well... it might work, but as DB maker it is up to you to build that credibility.

In fact, the reason i was here originally is exactly because i found some worrying messages in this sense on your own forum indicating that things as simple as "sort results by nested object property" might not be possible your response (you do pointing alternatives, but sorry i cant understand the examples to evaluate if its a valid one or not in that case)

Maybe i am wrong of course and it can all be done in graphQL+-. But on my side, being far from the others, I would feel much more confident if there was a bit of "formal" review of its design and expressivity available.

Maybe you could get someone to look at it like you did with Jepsen? I found that very useful and worth of praise on your side.

[darshanDevrai]

@manishrjain Any plans for Hosted solution in 2019?

[manishrjain]

Thank you all for your input. We've now created a new roadmap for 2019. Closing this issue. But, I still want to hear what Dgraph users want built in 2019. So, go on and comment here: #2894 .

[kanihal]

Is SPARQL already supported?

SPARQL support would be nice.

[rdinnik]

I need to support of jdbc/odbc for several reasons:

• to connect to dgraph from etl-tools
• to connect to dgraph from BI
• to have dgraph as a backed for existing metadata repository solution which can be represented as any datasource supporting jdbc/odbc connection. In other words I need a bridge between regulare sql queries including ddl and dgraph. The bridge should be represented as a jdbc/odbc driver.
  One of important question and challedge would be: how to map/interprete "create table..." statement to dgraph structures/api.

[kant111]

GraphQL is for web. Cypher is for graphdb. Apache Spark (one of the top ETL frameworks) supports open cypher so users can run cypher queries on streaming and batch data so if you want to leverage the big data community I would recommend cypher.