Keystone-classic: Consider BackboneORM for database

[kmalakoff]

I just listened to NodeUp and noticed you were not aware of BackboneORM for cross-database, client/server databases.

We support browser over HTTP, SQL variants (via knex), mongo, and in memory.

[morenoh149]

so this is like mongoose?

[JedWatson]

@kmalakoff thanks for pointing that out, it looks great.

Keystone currently relies on several mongoose features (e.g. virtual properties, schema methods, validation, etc.) that I haven't seen in any other ORM including, by the looks of it, BackboneORM.

i.e. taking the password field as an example, we add a compare method to the model that abstracts the complexity of dealing with bcrypt away, e.g.

var user = new User.model();
user.password = "hello";
// the save method fires middleware that automatically hashes the new password
user.save(function(err) {
  // the compare method is a virtual method on the schema
  user._.password.compare("hello", function(err, isMatch) {
    console.log(isMatch); // true
  });
});

The biggest blocker we have running Keystone against any other ORM (or database platform) is the ability to include functionality like this.

Is this something that BackboneORM could (or may in the future) provide? or do you know of any other system that _does_?

[kmalakoff]

@morenoh149 as far as I understand, Mongoose is Mongo-only. BackboneORM's main value proposition is the ability to interface with databases (Mongo, SQL-variant) and servers (via HTTP) with relationships and a powerful mongo-inspired query syntax.

@JedWatson I remember when I looked at Mongoose, it had a lot of special terminology, seemed to reinvent the wheel when it comes to classes, and had a few problems with relationship dependency cycles.

1) validations - Backbone.Models have bare-bones validations hooks and you can look to the Backbone community for more advanced solutions.

2) virtual properties - class and instance functions/variables are supported. For example:

class User extends Backbone.Model
  schema: {} # this can include relationships and use a function to break dependency cycles
  url: 'mongodb://same_database/users'
  sync: (require 'backbone-mongo').sync(User)

  @classFunction: ->  
  instanceFunction: ->  

User.classFunction()

user = new User()
user.instanceFunction()

3) schema methods - not supported, but I'm not sure how often these are used. We just use class and instance methods, and you can create mixins or a base class if you have reused functionality.

If you want to compare a password, you can define an instance method like:

class User extends Backbone.Model
  setPassword: (password, callback) -> # do encryption
  checkPassword: (password, callback) -> # do encryption and compare

user = new User()
user.setPassword "Hello", ->
  user.checkPassword "hello", (err, isMatch) ->

One of the main advantages is providing keystone users with choice in database.

[grabbou]

Well, can somebody explain pros of changing Mongoose to another ORM? Apart from having ability to use other databases, I can't see any advantages. Generally, Keystone uses the database engine of its choice and I think that's fine as long as all the current features are working and we are not being blocked with further improvements by Mongoose or MongoDB design choices.

[sebmck]

@grabbou That's the main/only advantage/incentive right now and it's a massive one. The ability to allow different databases while retaining all keystone features would be awesome.

[JedWatson]

I think it's worth offering people the ability to choose a different database _as long as it doesn't limit KeystoneJS_, and _as long as we don't end up devoting a large amount of effort to cross-database support_.

I've never (personally) seen the promise of "database agnostic" work out well at a project level - you limit yourself from being able to take advantage of the strengths of any one platform, in order to support them all. Cross-platform ORMs I've worked with suffer for this (much more so at the level of abstracting the differences between document stores like MongoDB and relational databases like MySQL).

However, Keystone really doesn't _need_ a lot of database features; projects that are built on Keystone do. So as long as we can get what we need at the core level from another database and underlying ORM / package, great.

So while I wouldn't advocate "database agnostic" as a goal for a project, if we could implement it for Keystone, then at the outset you could choose the database platform that meets your requirements best and then build on that.

I have been thinking about whether we could move _some_ of the database interaction that we currently rely on Mongoose to provide into Keystone APIs (e.g. so you could List.query instead of List.model.query) as a fairly light-weight wrapper. That would let us build a very light-weight abstraction for ourselves (to handle basic tasks like querying / pagination) which would then let us plug in alternative underlying adapters.

We have enough roadmap features I think are really high value that I don't want to get bogged down developing or supporting a database abstraction layer. However, if we can find a good solution, or someone is willing to take on (and commit to supporting) a project that achieve this, then I'm on board.

[jamlen]

Just to throw another one in the ideas pot.... I've been looking at https://github.com/balderdashy/waterline recently.

[morenoh149]

I think waterline is a great option. Though I'm biased - I looked at sails a lot before getting into keystone.
And I've grown used to Mongoose haven't run into anything I'd need postgres support for. Though I'd love the flexibility to do so. Which both BackboneORM and Waterline offer.

[grabbou]

@jamlen still under 1.0 which kinda discards it as a main ORM for Keystone for now (as long as they follow semver)

[jamlen]

@grabbou seriously??? Backbone ORM is also only at 0.7.x, node is only on 0.12....

A quick look through the packages that are "in" keystone show plenty of key packages that are not a 1.0, async, react, marked to point out a few.

[grabbou]

That's why I said "as long as they follow semver", the whole point was to think if Backbone ORM is stable enough or not ;)

[mxstbr]

I'll close this due to inactivity, we don't have any plans to integrate this currently. Moved to ProductPains: https://productpains.com/post/keystonejs/custom-orms

[kmalakoff]

Sounds like a good thing to do. It is hard to find a great cross-platform library in Node.js for abstracting various databases. Hopefully, a great solution will emerge soon.

[morenoh149]

I think nodal's ORM layer is very well thoughtout http://www.nodaljs.com/static/docs/index.html may have some RDBMS-isms due to primarily targeting postgres. But the author expressed support for other databases should be straightforward by writing an adapter https://github.com/keithwhor/nodal/tree/920dfbced62339b738d583dda7930392019e81d2/core/required/db/adapters

[bellsaladin]

I've started a project with the goal to support Mysql. The project has now a good foundation & implementation strategy.
https://github.com/bellsaladin/Keystone-Mysql

I have been using Keystone for quite a while now as light admin tool for some small projects and I aquired good knowledge on it's core functionning, at some point i felt the need for Mysql databases support, so i started a project with that goal.

I work on this project on free time only so I need help for this project to progress. Any help is welcome.