Json-api: Multiple/Hierarchical Types

[ethanresnick]

Sine the beginning, JSON API has treated each resource as having one, and only one, definitive type, which is part of the resource’s identity.

I didn’t see much debate about this design decision, and I know I never questioned it, but I think some problems with it have started to bubble up recently. I want to lay those out here, as I see them, and explore some solutions.

The first problem is that a resource can, conceptually, belong to multiple types (likely a sub-type and one or more parent types), but the spec has never defined a standard way for a resource object to communicate this multiplicity. This issue was first raised a couple years ago, but was closed without any resolution; more recently, it’s come up in issue #571.

The value of having a first-class type field is that the client can use it for a contract. But if the client doesn’t know all the types that a resource belongs to, it can’t do this. For example, if I have organizations and, as a subtype, schools resources, the client must know that schools are organizations, in order to accept them in places where organizations are accepted. (And simply returning the schools as organizations won’t work, because then there’s no way for me to tell the client that a given relationship can only hold schools.) So, the client needs to know all of a resource’s types, and the only way for that to happen right now is with manual configuration on the client-side, which is couples the client and the server much more than necessary. (E.g. the server can’t introduce a new subtype without all clients breaking.)

A second, related issue is that a resource might want to change types (in a way that wouldn’t violate any interface contract, like changing from Type A to Subtype of A), and this is impossible when type is part of resource identity. That difficulty was at the heart of #481. The resolution there was to always identify the resource by it’s parent type, and then to expose the sub-type and its fields, when they apply, in some api-specific way. This works ok except, of course, that the method for indicating the sub-type is still non-standard and, as discussed in that issue, the options for exposing the sub-type-specific fields aren’t particularly elegant.

The second issue in particular gets me thinking that type really shouldn’t have been part of resource identity. (We could have still had a type field for clients, but identity could have been established just by id; databases that scope their ids to the table/collection could have created the json-api id field just by concatenating the table/collection name with the record’s id.) But, alas, that ship has sailed. So, what can we do now?

Holding aside backward-compatibility for a second (I have some ideas for that below), I can think of two basic approaches:

1. Continue to list each resource as having one type, but then separately (at the response’s top-level, or at a separate endpoint) define the relationships between different types, such that the resource’s single type label can be used to derive all the other types that apply to it. So, for example, the resource would be listed as a schools, and some part of the response would say that all schools are organizations. This approach might be more compact than the one below, but it seems a bit brittle (i.e. I don’t think it works easily if the type system isn’t a strict hierarchy) and it requires the client to do some reasoning.
2. For each resource object, list in some array all the types that apply to it. This seems like the simplest and most robust solution. It’s a bit redundant, maybe, but that’s why we have gzip. Note that this is the approach that Siren takes with it’s class member. It’s also the basis of the neo4j data model, which has undifferentiated graph nodes that can be grouped into sets, for easier querying, through the (mutable) application of labels.

If we use the second approach, one way to implement it would be to add a "labels" member to the top-level of resource objects, which would list of all the types that apply to the resource (and would be mutable to whatever extent the server allows). This would coexist directly with the existing type attribute, which would still be valuable for making some interface guarantees to the client.

So two resource objects might look like this:

{
  "type": "schools",
  "labels": ["organizations", "schools", "kindergartens"],
  //...
}

Above, the api is guaranteeing that the resource will always be a school through type, so the client doesn't need (for example) to GET it and rely on (currently non-existing) hypermedia controls to know how to send a patch. However, the labels field lets the server specify the other types—both organizations, which (hypothetically) existed as a type at the time the resource was created, and kindergartens, which is a type the server added later as its database of schools grew.

Below, the server solves the problem from #481 by adding/removing "presenters" from the labels as the person's status changes:

{
  "type": "people",
  "labels": ["people", "presenters"],
  "attributes": { } //there are more fields here when `presenters` is among the labels
}

Thoughts?

cc @json-api/owners

[ethanresnick]

@tkellen @steveklabnik @dgeb I totally forgot to bring this up at today's meeting, but I would like some feedback on it whenever you guys have a sec, as it's proposing an amendment to the basic json api data model.

[meshuga]

A second, related issue is that a resource might want to change types (in a way that wouldn’t violate any interface contract, like changing from Type A to Subtype of A), and this is impossible when type is part of resource identity. That difficulty was at the heart of #481. The resolution there was to always identify the resource by it’s parent type, and then to expose the sub-type and its fields, when they apply, in some api-specific way. This works ok except, of course, that the method for indicating the sub-type is still non-standard and, as discussed in that issue, the options for exposing the sub-type-specific fields aren’t particularly elegant.

The case when you want to expose an API to change type of a resource for me is a sign of bad design - could smell like an RPC.

The value of having a first-class type field is that the client can use it for a contract. But if the client doesn’t know all the types that a resource belongs to, it can’t do this. For example, if I have organizations and, as a subtype, schools resources, the client must know that schools are organizations, in order to accept them in places where organizations are accepted.

I don't agree - all subtypes implement/contain basic attributes and relationships of the parent, so even if the client doesn't know anything about an inherited type, it can always process the basic fields.

Continue to list each resource as having one type, but then separately (at the response’s top-level, or at a separate endpoint) define the relationships between different types, such that the resource’s single type label can be used to derive all the other types that apply to it. So, for example, the resource would be listed as a schools, and some part of the response would say that all schools are organizations. This approach might be more compact than the one below, but it seems a bit brittle (i.e. I don’t think it works easily if the type system isn’t a strict hierarchy) and it requires the client to do some reasoning.

If you really want to put information about the type hierarchy, I think that OPTIONS method is the most suitable one. One more thing, hierarchical types always should require some advanced implementation on client side.

Btw. currently I've started implementing inheritance in Katharsis and I'm planning to support inherited types in parent types (https://github.com/katharsis-project/katharsis-core/issues/85) ;)

[masterspambot]

Considering what you've said it's rather explainable to use extra field like subtype or organisationType that will match your strategy, keeping your resource under one endpoint and making you able to still implement strategy based pattern on both front- and back-end side. And even though, if your strategy is too complex I'd find this API designing mistake and should be split into more endpoints or remodeled into different hierarchical models.

In my opinion there is no point in extending the standard to match that case and complex it unnecessarily.

:+1: @meshuga

[ethanresnick]

Responding to @meshuga's comments first:

The case when you want to expose an API to change type of a resource for me is a sign of bad design - could smell like an RPC.

Can you explain here what you mean? That is, what about it seems bad to you/what are the particular negative consequences you're worried about? It doesn't strike me as RPC (and, even if it did, RPC isn't _always_ inappropriate). Also, remember that I'm envisioning here changing to subtypes or otherwise compatible types, not changing types generally.

I don't agree - all subtypes implement/contain basic attributes and relationships of the parent, so even if the client doesn't know anything about an inherited type, it can always process the basic fields.

Right, but it can't know that the resource of the inherited type _is an instance of_ the base type. So, back to the imaginary "organizations" type with a "schools" schools sub-type. A client that encounters a schools resource can't process even the fields that are common between organizations and schools using its logic for processing organizations because it doesn't know that the schools resource is an organization.

Another example of this problem is the case of relationship contracts that I alluded to in my initial post. That is, imagine a resource has a relationship that (through some schema or hypermedia) the client knows is supposed to hold organizations resources. Now, if the client wants to show the user a list of resources the user can add to the relationship, it can't know that the schools resource is allowed, because there's no standard way for it to know that the schools is an organizations.

The opposite side of this show's why I find @masterspambot's proposal to add a subtype field insufficient: with that approach, it becomes harder to say, e.g. that a given relationship only accepts schools resources, since whether a resource is a schools is identified in a non-standard/API-specific way. Also, as I mentioned in my initial post, the options for exposing the sub-type-specific fields aren’t particularly elegant (see #481 for details).

Also, re:

If you really want to put information about the type hierarchy, I think that OPTIONS method is the most suitable one.

That's pretty much option 1 from my initial proposal, and it could be a workable solution. It does seem to simplify client side implementation, since the idea of a list of types could, I'd imagine, be challenging to handle in some statically typed languages. Still, it also seems less flexible. Maybe there's a way to combine the "list of types" approach with a description endpoint that clients can use to make sense of the list...

[dgeb]

It sounds like we all agree that it's a mistake to include hierarchical information in a type value because that information makes the value more brittle. For example, people-presenters is more brittle than either people or presenters.

It's an API design decision whether to go with the more general people or the more specific presenters for type. Another more DRY option is model each individual "presenter" with two separate resources - one of type people and another of type presenters. Relationships could be maintained between the two (e.g. roles on one side and person on the other).

Another possibility entirely is to simply use an attribute such as subtype like @masterspambot suggests. This seems fine if the fields are the same across resources.

In almost all APIs I've designed, I would want to be able to fetch the more general resource and include all the specific resources as well in a response. We have two options here:

• If "presenters" are modeled with two separate resources, we return a homogeneous collection of type people.
• If you've chosen to model presenters with type presenters and not have a separate people resource for each presenter, fetching all people could return a mixed collection of people and presenters.

I tend to favor composition of multiple resources of different types to express general / specific capabilities. The major advantage here is that a "person" resource can remain constant, even as they change roles from a "teacher" to an "administrator". This also allows for more flexibility: a single person might be both a teacher and an administrator.

These API design decisions, much like table design in a relational database, seem very application-specific. I worry about leading API designers into using a hierarchical approach when a normalized, compositional approach is often preferable.

I'm not opposed to making recommendations for handling hierarchical and compositional concepts, but I'm not convinced of the need to expand the base spec because I don't see these as "one-size-fits-all" problems.

[masterspambot]

:+1: @dgeb

[ethanresnick]

@dgeb

I tend to favor composition of multiple resources of different types to express general / specific capabilities. The major advantage here is that a "person" resource can remain constant, even as they change roles from a "teacher" to an "administrator". This also allows for more flexibility: a single person might be both a teacher and an administrator.

I agree that, in the people examples, presenter, teacher, and administrator can be seen as attachable/detachable roles, and so modeling them as separate resources will likely be the best approach (holding aside validation complications).

That's why I gave the example of organizations and schools, in which being a school isn't really a detachable role: were the organization no longer to be a school, it wouldn't be the same organization in any meaningful sense, at least in my domain.

In cases like this, the compositional approach is a big pain: it complicates ?include paths; it creates a new, addressable resource—the one holding the school specific attributes—that really shouldn't be addressable on its own (e.g. shouldn't be linkable from other relationships, shouldn't be deletable, etc) and then puts the burden on the server to enforce that; and it complicates validation, because the server has to inspect field values to determine the true type of the resource.

I worry about leading API designers into using a hierarchical approach when a normalized, compositional approach is often preferable.

I don't want to lead API designers astray either, and I think good recommendations will be key for that. At the same time, there do seem to be cases where composition isn't a great approach, and, for those cases, the spec doesn't seem to offer a suitable alternative. That's why I think we have to at least consider expanding it. Maybe, after evaluating some possibilities, we won't be able to find a good way to do so now that the type key is baked in as-is. But I don't think we can say in good faith that the base spec handles these cases well as is.

[dgeb]

@ethanresnick

That's why I gave the example of organizations and schools, in which being a school isn't really a detachable role: were the organization no longer to be a school, it wouldn't be the same organization in any meaningful sense, at least in my domain.

Fair enough. I wasn't saying that hierarchical concepts should never appear in an API, but I was taking the long way to explain how carefully we need to couch our recommendations.

In your case, you might take the mixed-type collection approach when fetching organizations. In other words, you'd return a collection of schools and other types of organizations when fetching /organizations.

So the question for clients becomes - which types might be returned at a given endpoint?

This whole issue seems very closely related to our general "discovery" problem. We need to establish a means by which clients and servers can communicate about the types that are accessible at a given endpoint. And we also need to allow servers to advertise the operations allowed.

I propose that we target discovery-related issues for v1.2 beta (Dec. 31) since we already have a full plate for v1.1beta.

[masterspambot]

@dan as for me this should appear as extension of JSON API, but after thinking over what @ethanresnick said it might be useful to know what kind of sub-types can be accepted by endpoint, because in current state rest itself is not revealing the strategy pattern behind at all.

[ethanresnick]

I wasn't saying that hierarchical concepts should never appear in an API, but I was taking the long way to explain how carefully we need to couch our recommendations.

Agreed.

We need to establish a means by which clients and servers can communicate about the types that are accessible at a given endpoint.

Right. And not just which types are available at each endpoint, but what the logical relationship between the different types is. (That's why returning the schools as type schools doesn't work now: because the client doesn't know that a school can also be used where an organization is expected.)

I propose that we target discovery-related issues for v1.2 beta (Dec. 31) since we already have a full plate for v1.1beta.

That sounds good to me! I'll milestone this for 1.2. Maybe the solution will be an extension, as @masterspambot said, or maybe it'll be as simple as supporting linking from complex attributes (which, from #481, seemed like it might offer a minimum viable solution). Regardless, I agree that this is less urgent than the 1.1 stuff (and I will try to revisit extension design later this week!); I just wanted to open this issue as a bookmark for this discussion, and to see what others thought.

[ethanresnick]

FYI, got another question on twitter from a JSON API user that went to the heart of this issue: https://gist.github.com/arches/7508c83294b623e41765

[zachdaniel]

We've taken an approach to this that models subtypes simply as "has one" relationships. An example could be users where an individual user can be a student or a teacher (or in our case both).

In our implementation, there are three resources users, students and teachers. Each one has its own api that behaves as expected. The ids of a user, and that user's student or teacher record happen to be the same, but that isn't really leveraged. Instead you would GET /users/:id?include=student,teacher and can tell the subtypes by which relationships are set. If that user is related to a teacher then they are a teacher, and if its a student then they are a student. Modeling this polymorphism in this way keeps things very simple.

[maurei]

We've taken an approach to this that models subtypes simply as "has one" relationships. An example could be users where an individual user can be a student or a teacher (or in our case both).

In our implementation, there are three resources users, students and teachers. Each one has its own api that behaves as expected. The ids of a user, and that user's student or teacher record happen to be the same, but that isn't really leveraged. Instead you would GET /users/:id?include=student,teacher and can tell the subtypes by which relationships are set. If that user is related to a teacher then they are a teacher, and if its a student then they are a student. Modeling this polymorphism in this way keeps things very simple.

I think it is pretty confusing to mix up is and has. Relationships as exposed in json:api content AFAIK are used for has relationships, where composition makes sense (eg article has one author). If an API in your example now returns a student object in the included member apart from the user object in the primary data, this suggests there is composition, which does not accurately model a is relationship. I.e. using include to side-load is relationships rather than has relationships is confusing.

I think this is a sign that one might be better of with heterogeneous responses/requests/relationships, because it takes away the responsibility of having to think about is vs has relationships, which is a concept foreign to the json:api spec anyway.