Horizon: Composition of queries

[deontologician]

From #93

1. The client should be able to compose separate queries into a single object that describes the model the app needs. fusion.model({foo: fusion('foo').find({id: myFoo}), bar: fusion('bar').find({id: myBar})})
2. The server should be able to coalesce a composed query like this into a single union changefeed. This would allow efficient updates and the possibility of doing incremental updates of a client-side model when any constituent part of the model changes, without opening lots of separate changefeeds.

The idea is that right now, the fusion client has no idea what kind of model you're trying to build. Each query/subscription is a separate entity and the server needs to manage multiple changefeeds to the database because it doesn't know that they'll be created or destroyed together. In practice, this flexibility is needed less with React style apps (I'm including Cycle.js and Elm in this as well). In these apps, one big model is created all at once to represent the app state.

[mlucy]

Just for the record, a.union(b).changes() is rewritten internally to a.changes().union(b.changes()), and the only difference between that query and two separate changefeeds is a small amount of overhead associated with the open connection to the client. So probably the efficiency gains of trying to cleverly union changefeeds together won't be very large.

[deontologician]

I guess the server side of this is the same point being made about not needing to dedupe changefeeds. If they're cheap enough on the db, fusion doesn't need to do anything fancy. I think there still may be value in trying to combine queries on the client, though maybe observables get us that for free...

[deontologician]

Ok, as a concrete proposal, I think we should add a top-level function called aggregate, that would work like this:

const horizon = Horizon()
horizon.aggregate({
  user: horizon('users').find(userId).fetch(),
  pets: horizon.aggregate({
    cats: horizon('cats').above({ age: 30 }).watch(),
    dogs: horizon('dogs').below({ age: 15 }).watch(),
  })
})

This would make use of combineLatest to automatically emit the entire data structure any time any of the constituent queries change. This is similar to watching the individual queries and combining them yourself, only a bit nicer syntax, and I think it's a common enough use-case to warrant its own function

[mlucy]

Why do you need the second horizon.aggregate inside the object?

[deontologician]

In this case, it makes all of the values the same type, an observable. So you can just line up keys with values without having to descend. An alternate syntax would be like:

const horizon = Horizon()
horizon.aggregate({
  user: horizon('users').find(userId).fetch(),
  pets: {
    cats: horizon('cats').above({ age: 30 }).watch(),
    dogs: horizon('dogs').below({ age: 15 }).watch(),
  }
})

Where if we hit an object we can just wrap it in a horizon.aggregate call for them. This might be a leaky abstraction though. It might lead people to add arrays, constants etc. I guess we could do an instanceof check... it just gets a little hairier, but probably not insurmountable.

[marshall007]

@deontologician it would be nice if these "aggregations" could be parameterized for reuse, more like a GraphQL query. Also, I don't think you should call .fetch() or .watch() on each of the nested elements. Instead, I think it would make sense to do that on the entire aggregation. Continuing with your example, I'm thinking something like this:

const horizon = Horizon()
const UserPetsModel = horizon.aggregate(({ userId }) => {
  return {
    user: horizon('users').find(userId),
    pets: {
      cats: horizon('cats').findAll({ userId }),
      dogs: horizon('dogs').findAll({ userId }),
    }
  }
})

let model = UserPetsModel({ userId: '12345' }).watch() // or .fetch()

[deontologician]

Wow, that is pretty nice

[mglukhovsky]

That looks awesome. :100:

[marshall007]

Yea, I'm pretty stoked about this feature. Thinking about it a bit more, building parameterized models should probably be exposed as something else like .model(...). I think .aggregate(...) as originally described by @deontologician would still be quite useful as a separate utility method.

It should be fairly trivial to support nesting and composition of models too since they're really just collections of TermBase.

const PetsModel = horizon.model(({ userId, type }) => {
  return horizon(type).findAll({ userId })
})

const UserPetsModel = horizon.model(({ userId }) => {
  return {
    user: horizon('users').find(userId),
    pets: {
      cats: PetsModel({ userId, type: 'cats' }),
      dogs: PetsModel({ userId, type: 'dogs' })
    }
  }
})

// note how `.aggregate()` is still nice for one-offs
let model = horizon.aggregate({
  user: UserPetsModel({ userId: '12345' }),
  friends: horizon('users').findAll({ friends: '12345' })
}).watch()

This thread feels like the future and it's beautiful.

[deontologician]

This thread feels like the future and it's beautiful.

:100:

[coffeemug]

I have to think about it a little bit, but my initial reaction is that this should be a v2 feature (I.e. I wouldn't work on this until we nail down the basics like security).

[ghost]

This is actually exactly how we handle things in our Elm app when passing data from the back-end to the front-end for display. 👍

[deontologician]

Implemented in #329