Apollo-client: Ideas for additional subscription APIs

[stubailo]

Yesterday, we published all of the pieces of the initial Apollo GraphQL subscriptions implementation. Read about it here: https://medium.com/apollo-stack/graphql-subscriptions-in-apollo-client-9a2457f015fb#.qd8rnff9d

Right now, the API looks like this:

client.subscribe({ query, variables }) => Observable

It's the most basic thing we could come up with, and makes sense for an initial release. This API is good if what you need is a stream of events, and it's an API we can build on top of to enable other features. But it's definitely not the final thing we want, because it doesn't help developers display those subscription results in your UI.

Here are some ideas for additional APIs we can build on top of the base:

fetchMore-style subscription

In the GitHunt UI, we use a subscription to update the result of the original comments query. We think this is going to be a common pattern.

In usage, this is very similar to fetchMore, where you do a second query to update the results of the first. But in this case, the second query will be a subscription, and it will get multiple results. You can think of it as reactive pagination.

So one proposed API addition is to add a method to ObservableQuery, called something like subscribeToMore, that has an updateQuery reducer just like the fetchMore query does.

mutation-style subscription

You can think of subscriptions as "someone else's mutation result". In that context, it makes sense to have all of the same features as mutations, and perhaps even reuse the same code. So we should have all of the ways you can use mutation results:

1. Updating objects by matching up IDs
2. updateQueries, so that you can reuse your mutation reducers in your subscription results.
3. refetchQueries, so that subscriptions can trigger page refetches

I'm not sure what the API for this would look like, but open to suggestions.

And if anyone else has ideas, let's talk!

[jbaxleyiii]

@stubailo first off, congrats to the team for client.subscribe! I'll be adding support for this in react-apollo 👍

Both of the other types of subscriptions you mentioned make a ton of sense to me. I personally can't think of another use case currently but I'll keep thinking on it!

[stubailo]

Oh, use case 3 is right here:

subscription as a replacement for a query

Your subscription represents a constantly changing value, and you want to put that in your UI directly. In that case, you probably want to just have a container that pipes the most recent value from the subscription into your props.

[jbaxleyiii]

Wouldn't that be the default?

[stubailo]

The default is, you just get a callback from the subscription. I guess that would be the default for react-apollo though.

[jbaxleyiii]

@stubailo awesome, I'll try to get the current api reflected in react-apollo this week!

[rricard]

Once again, great work! Let me add my 2cents here 😉 …

An another thing to explore that would flow naturally into apollo-react afterwards would be having a kind of centralized system attached to the whole client that would allow attaching a subscription to the client alongside an updateQueries system that would change in-store all queries with a matching name. That way this would trigger a new observed value on every subscribed entities (react components or basically anything else!).

It would look like this:

const updateQueriesWithCommentsSubscription = {
  feedQuery(currentQueryState, subscriptionPayload) {
    return /* a new queryState with the comment inside */;
  },
  notifications(currentQueryState, subscriptionPayload) {
    return /* a new queryState with the comment added */;
  },
};


client.addSubscriptionOverQueries(gql`
  subscription comments($repoName: String!) {
    newComments(repoName: $repoName) {
      content
      postedBy {
        username
      }
      postedAt
    }
  }
`, updateQueriesWithCommentsSubscription);

I'll start playing with this concept tonight, if that works, expect a PR!

[stubailo]

That seems identical to the option above for "mutation-style subscription"

But I agree moving updateQueries to a more centralized approach is a good idea soon. We should decouple that from subscriptions, though.

[rricard]

@stubailo yes that's exactly the same I was just trying to formalize an actual api so I can get to work on something! I'm kind of missing stuff today, sorry

[stubailo]

Yeah, I think the API should be identical to mutations, to start with.

[rricard]

We should decouple that from subscriptions, though.

Seems to be actually a very powerful idea... So what you are implying if I understand is an API where I can feed an observable on one side and an updateQueries on the other right ?

[stubailo]

Yes, but let's do that as a next step, after implementing some of these subscriptions APIs.

[rricard]

I would see something like this:

const updateQueriesWithCommentsSubscription = {
  feedQuery(currentQueryState, subscriptionPayload) {
    return /* a new queryState with the comment inside */;
  },
  notifications(currentQueryState, subscriptionPayload) {
    return /* a new queryState with the comment added */;
  },
};

client.updateQueriesForEach(this.props.client.subscribe({
  query: gql`
    subscription comments($repoName: String!) {
      newComments(repoName: $repoName) {
        content
        postedBy {
          username
        }
        postedAt
      }
    }
  `,
}), updateQueriesWithCommentsSubscription);

I find it quite simple and I would totally use that!

[rricard]

@stubailo I actually tried something and based the update system on a generic "observable query updater" in #649 to expose a "subscription query updater". It works pretty well and is quite simple in my opinion. It may need some more thoughts but I think that's some code we can start with...

[rricard]

I could see if we can do a similar system with a fetch-more style API. It would simply be based on updateQuery with an updateQueryFromObservable and finally an updateQueryFromSubscription.

[tmeasday]

@stubailo I think the idea that you expressed to me, where we treat mutation/subscription results analogously to redux actions is the way to go. To follow the analogy, a query's result is a store and we should define a (set of) reducers on that store to keep it up to date.

Concretely, this might looks something like:

client.watchQuery({
  query: gql`{ entry(...) { comments { content } } }`,
  reducer(prev, { type, name, result } ) {
     if (type === 'mutation' && name === 'submit') {
       return update(prev, {
           entry: {
             comments: {
               $unshift: [result.data.submitComment],
             },
           },
        });
     } else if (type === 'subscription' && name = 'repoComments') {
        return // something similar
     } else {
        return prev;
     }
  }
});

(I guess we might call it updateQuery rather than reducer though.)

[tmeasday]

If we like that, I'd propose "bundling" the query and the reducer somehow (maybe just by convention), when using this in practice:

// the "data" part
const repoCommentsQueryAndReducer = {
    query: gql`{ entry(...) { comments { content } } }`,
    reducer(prev, { type, name, result } ) {
      ...
    }
}

// possibly in a different file, certainly separated a little
@graphql(repoCommentsQueryAndReducer)
class CommentsList extends Component {
}

[stubailo]

Yeah I'm on board with Tom's approach. It seems to me that it makes a lot more sense to put the reducer next to the query because it most depends on the shape of that data.

[rricard]

I like it too, it makes firing mutations easier as well in a sense. My question then is: do we replace everything with this form of API or keep both forms (reducer at mutation/subscription level and query level) ? If we keep them both, how do we handle the cas where both are defined ? I find both approaches good (depending on how you want to structure your app).

[rricard]

Do you mind guys taking a look at #649 in the meantime ? I don't expect it to be merged but I'd like to know what you think about the API. Thank you!

[rricard]

So the idea I'm kind of sensing now (as also discussed in #649) is we should create an embedded actual Redux reducer for the query ( @tmeasday idea followed by @stubailo idea!). So for each query, we could attach what we can really call a reducer: (queryRes: any, reduxAction: {type: string, [payloadParts]: any}) => newQueryRes: any. All redux actions (originating from apollo or from your app will go through there!). This "form" let us actually apply stuff like combineReducers or any utility to manipulate reducers!

Here are the main steps I see in this design:

1. Apollo's reducer should dispatch each action to eachreducer associated to each query registered
2. Then we pass the reducer over the query results with the redux action
3. This will lead us to dispatch once again an APOLLO_QUERY_UPDATE action (the only action we won't let get through the query reducers)

With this system we could re-implement all of the mutation, fetchMore and subscription updateQuery systems with it!

So I'm starting back with @tmeasday proposal, the only change is we won't be constraining the data payload!

[tmeasday]

Maybe there's a lower-level reducers API and a higher level updateQuery API which is a bit of sugar for the common use case?

I can actually see 3 levels of "store alteration" that could make sense:

1. Directly updating the store.apollo.data cache based on a redux action (in such a way that affected queries are properly triggered). This is the least "natural", but most powerful.

js client.addDataReducer((data, action) => { if (action.type === "X") { // update a particular item in the store by id for some reason return assign({}, data, { '1': newValue }); } });

1. Updating a particular query's results based on a redux action, with the side-effects to other queries + the store flowing through. This could call the above API

js client.watchQuery({ query, reducer(results, action) => { if (action.type === "X") { return results.concat(newItem) } });

1. Updating a particular query's results based on a subscription or mutation result (sugar for 2):

js client.watchQuery({ query, updateQuery(results, operation) { if(operation.type === 'mutation' && operation.name === 'addComment') { return results.concat(operation.comment) } });

Is this getting to complicated?

[rricard]

@tmeasday: looks reasonable to me. You clearly split the 3 levels of logic and build on each one the other so I think it is actually really elegant and simple. The only thing I have to say is if we were to implement this kind of design, this would fundamentally change the way apollo works internally and how we use it externally. For the sake of stability, I'm trying to figure out how the current updateQueries mechanisms in mutations and in fetchMore would work with this new system. But definitely this kind of design could really benefit for the whole thing! I really like the fact we're getting closer to redux!

[stubailo]

I was thinking we can merge 2 and 3, if our actions are nicely designed?

For the sake of stability, I'm trying to figure out how the current updateQueries mechanisms in mutations and in fetchMore would work with this new system.

I think it should be super easy to migrate, and even build a backcompat shim, because the reducers themselves should be about the same. So worst case you just move them from one place to the other.

[rricard]

@stubailo I'm with you if we really take the time to rethink the actions! Not saying they are bad right now, no, but they show a lot of internal stuff for instance.

If there is a backcompat shim I'm all in! I mean at least for some time while we provide some useful warning and a clear migration path to the new system...

[stubailo]

Hmm, we could redesign our actions to have "public" and "private" fields. So something like:

{
  type: 'APOLLO_MUTATION_RESULT',
  operationName: 'CreateTodoItem',
  operationAST: ...
  result: { ... }
  private: { ... internal apollo stuff goes here ... }
}

That way, people can know which fields they are allowed to safely use as part of the "public" API.

[tmeasday]

@stubailo how does using "raw" redux actions compose with the thoughts about decoupling our use of redux so people can inject in their own preferred state management libraries?

Does the shape of an action payload remain more or less unchanged anyway?

I guess what I mean is would it feel weird for a mobX user to do action.type === "APOLLO_MUTATION_RESULT" or would that feel OK?

[stubailo]

Well there are a few things that are already redux-ey, like ngrx. In that case it will definitely be fine.

I don't know how it would feel for MobX people.

[tmeasday]

I guess it's easy to add layers of sugar later anyway.

[stubailo]

Yeah, at the end of the day it's just a POJO, which is great!

Also, I think this would let us do something cool, which is to say: "Apollo Client: Flux for GraphQL" and have that be a really concise description.

Basically, it would be all of the best parts of Flux, but with some GraphQL magic built in.

[tmeasday]

I would advocate _not_ calling it reducer due to three reasons:

1. It's not a vanilla Redux reducer, in that it operates on the denormalized version of store.apollo.data.
2. It's not a vanilla Redux reducer, in that it triggers ObservableQueries as well as changing the store.
3. I personally think the name reducer is confusing to people as it's too "functional", so it has Redux baggage.

For reasons 2+3 I would probably call the first operation updateData rather reducer too.

[rricard]

Maybe we should start working on a design document for all of this explaining the APIs, redesigning the actions, and maybe more ...

@tmeasday Reducer, for me, is a good name since it describes exactly what it does in my opinion (I would advocate for keeping it):

1. Yes it's not a Redux reducer but it's applied on a redux action and the data is ultimately extracted (even if transformed) from redux and will be reinjected in redux
2. Well, it's like having a middleware attached to redux, the point is, the Observable will fire but outside of the pure function. this reducer is not a standard redux reducer, it's an apollo reducer 😉 but for me as log as it keeps the same "mathematical" properties of a reducer, we can call it reducer
3. I don't see why that is an issue: FP is generally good practice & Redux has a great design.

In fact I only fear that something like updateData would and up being too generic and feels like too imperative to me 😆 . More seriously I fear we'll end up calling it a reducer in the doc because it seriously looks like one!

[helfer]

Implemented subscribeToMore in #797, lets's see where that gets us.