React-admin: [RFC] Make expected REST responses consistent

[fzaninotto]

I've seen a few issues raised by people who wrote their own REST clients, and who were confused by the fact that the expected response format sometimes includes a data key, and sometimes not.

This is the response format explained in the REST client documentation:

Type | Response format
-------------------- | ----------------
GET_LIST | { data: {Record[]}, total: {int} }
GET_ONE | {Record}
CREATE | {Record}
UPDATE | {Record}
DELETE | {Record}
GET_MANY | {Record[]}
GET_MANY_REFERENCE | {Record[]}

Since the HTTP route used to translate GET_MANY and GET_MANY_REFERENCE is usually the same as the one for GET_LIST, developers naturally develop a response format like that:

Type | Response format
-------------------- | ----------------
GET_LIST | { data: {Record[]}, total: {int} }
GET_ONE | {Record}
CREATE | {Record}
UPDATE | {Record}
DELETE | {Record}
GET_MANY | { data: {Record[]}, total: {int} }
GET_MANY_REFERENCE | { data: {Record[]}, total: {int} }

And this doesn't work, because the reducers don't find the data they want.

My proposal is to change the expected REST response format to match this expectation. This is a BC break - all custom REST clients will break. But I think it removes a common WTF when dealing with admin-on-rest.

Thoughts?

[supermacro]

I think Axios is a great example of standardization of responses. No matter which HTTP request you make using Axios, you always have a standard object that gets returned.

Here is the Response Schema for Axios:

{
  // `data` is the response that was provided by the server
  data: {},

  // `status` is the HTTP status code from the server response
  status: 200,

  // `statusText` is the HTTP status message from the server response
  statusText: 'OK',

  // `headers` the headers that the server responded with
  headers: {},

  // `config` is the config that was provided to `axios` for the request
  config: {}
}

I think having a data field on every response would simplify things. The custom REST client can then pass other properties to the response objects.

[MikeFair]

I've been working on adding my own Rest Clients as well (starting with a CouchDB one first).

If we're going for BC changes to bring more standardization, I'd really like to see bringing the whole "Resource" object into the restClient instead of just the resource name. This way different clients can decorate the Resources with other information the client requires for accessing/updating that Resource.

In CouchDB, the List/Many handlers are all done through views that look like this:
http://couchserver.tld/[dbName]/_design/[configuratorName]/_view/someViewName

These views have all kinds of special query parameters that satisfy the List/Many filter and sorting operations and make it really easy to slice/dice different resource sets from one larger pool of objects.

That's entirely different from the singular GET/PUT/POST references which look like this:
http://api.couchserver.tld/[dbName]/[resource_id]

The "base url" for CouchDB could be set a couple different ways, not just one, depending on the range of resources you'd like to access. What I'd like to be able to do is 1) decorate each of the resources with some added information like "dbName", "designDoc", and "viewPath", and then 2) access those properties when the resource is passed into the RestClient (e.g. resource.viewPath).

I doubt CouchDB is going to be the only RestClient that could use some "extra information" about its resources to help tailor these URL references. This will also help when you've got multiple resource sets within the same resourceType; like when the single "users" resource gets sliced up into different resources by "department".

If passing in the whole resource doesn't work for you, then perhaps you can provide an example of how to retrieve the resource from the Redux store using the resource name so we can look it up?

Thanks!
Mike

[fzaninotto]

@MikeFair I'm -1 about adding client-specific props to <Resource>, and to passing the Resource element to the client.

Admin-on-rest's mission is to handle only REST. The mapping between REST and your server is the restClient responsibility. I'm aware that this implies adding some resource-specific logic in the restClient, width a switch/case on the resource name. But keeping server/http logic out of admin-on-rest makes things much simpler to reason with.

Tip: We've gone the path you describe in ng-admin, and we soon had to worry about a whole new class of REST mapping problems. We've decided that it's not admin-on-rest's responsibility.

[MikeFair]

I was wondering if you could point to me something I can read to do this one:

... an example of how to retrieve the resource from the Redux store using the resource name so we can look it up?

I can't seem to find any examples of this. The closest I see is a mention that the state's keys are 'admin', 'form' and 'routing' in the customReducers docs but I'm not seeing anything that shows an example of retrieving the state information for a resource and/or a resource item instance.

Admin-on-rest's mission is to handle only REST.

Ok, I think I got it!

Make the "REST" metaphor more about being an internal abstraction layer for how AOR internally accesses and manipulates its own resources; and less about where those resources came from.

We can use local routes to implement this REST abstraction. Resources then directly provide their own RESTful responses to these local routes. There are a number of default hookups to make the existing use cases transparent. The RestClient abstraction started this concept, but I think providing these local routes would complete carrying it all the way through. It moves the abstraction down a level to administering a "collection of RESTful resources" replacing the current metaphor of "resources on a RESTful server".

Typically these local routes respond with something to render; but what if they provided AOR with their own REST abstraction?

A component, like <List />, now calls AOR's local /api/v1/[resourcename]?action=get_list route and provides the query details.

The function/component registered there takes ownership of building the appropriate REST response (like RestClient does now).

The code for calling out to an http server is now hidden behind a resource's implementation detail instead of being directly managed by the Admin component. If a resource's data source also happens to be hosted externally on an http server that provides a REST based interface, that's more a happy coincidence over an assumption.

Then we add an optional <RestClientConfig /> type child component to a resource. This <RestConfig /> component captures any custom data a resource's REST abstraction implementation needs to know about the resource. It could describe a SQL database, an HTTP REST server, a file system; whatever/however the resource gets connected to its source data provider goes in this component.

The AOR <Admin /> component still directly owns and controls the <Resource /> component itself and explicitly ignores any and all of thse child configuration components; they are never rendered.

Aside from serializing them to the store so they can be retrieved by other things, AOR doesn't even care they exist; modules implementing services for the Resources do, but AOR itself doesn't even see them.

This way, anything that's not an AOR "thing", but is reasonably still a "Resource Configuration" thing; is given a Resource's child <NamedModuleConfig /> component.

Anything wanting access to this configuration information can retrieve it from the store.

_I started seeing this idea because the application I am building brings together resources hosted on multiple servers. Which obviously breaks AOR's current "single source only" assumption. Some of these sources aren't anything like REST; and each Resource is definitely going to require per resource MetaData configuration to integrate. I was also running into problems rendering resource lists into other areas of the screen and tracking "context" for a current combination of resources the user selected. I really like the REST metophor, and the use of MaterialUI, React, Redux, and Saga, so it seemed easier for me to see if I could teach AOR some new tricks rather than go find another framework. There's certainly nothing in REST that says everything comes from the same place, and I thought teaching AOR to bring together many REST resources from several disparate non-REST locations would make for great additions to the tutorial materials. It certainly seems like this is what the whole RestClient idea is supposed to be about._

What do you think? Is rerouting the RestClient calls so they can run through a Resources own .js file a direction that works?

[mantis]

can we spec out an expected way to generate errors from the rest api as part of this? i.e. if we putting status=500 and errors="foo" in a response or what

[fzaninotto]

@MikeFair if you want to look at the Redux store, I advise you to install the Redux dev tools in your Bbrowser.

I didn't understand the rest of your comment, but I don't want you to try and explain your position again in a lengthy comment. I'd prefer that you try to understand mine : We will not deal with the server configuration in admin-on-rest code. It's your job to do it, and you should do it in the restClient, period.

[fzaninotto]

@mantis status codes don't mean anything in REST, but error messages do.

If case of REST error (this covers boths non-200 HTTP status codes, and an unreachable API), your REST client should throw a JavaScript Error. If this Error contains a message property, it is passed as the error property of the action, and used for the notification.

[mantis]

@fzaninotto presumably that's a single string? (or can you do an array of messages[])

use case: if creating a user with password , email - and email is invalid and password is too short - one could return a single message "your email is invalid, password too short" - but for translations etc - it would be nice to be able to return an array of messages - i.e. user could then receive multiple notifications

[fzaninotto]

It's your responsibility to bake all that logic in your restClient, and throw a unique Error aggregating the errors you get.

[mantis]

Thanks - I might consider logging a seperate CR about allowing alert functionality similar to the react-material demo @ http://mayashaddad.github.io/react-material-alert-demo/ (hence my question above)

[fzaninotto]

Fixed by #385

[cloudtracer]

@MikeFair did you ever put together a REST Client for CouchDB? I'm thinking of trying to use Admin-on-rest with one of my existing CouchDB apps and just thought I'd check and make sure I'm not about to re-invent the wheel here.

[MikeFair]

@cloudtracer

I did, but I found the AOR model so incompatible from an abstraction point of view with CouchDB's as to make a meaningful CouchDB library a non-starter. It was pretty disappointing and I've had to abandon using AOR for the time being; which really sucks IMHO.

It's different enough that it's equivalent to trying to use something like a SQL server as the data backend. The only metadata the client code really gets is "Resource Type Name". No facility enables you to provide metadata describing different table names or where clauses for different types. While Couch is still very REST like and goes over HTTP pretty easily, the URL construction is equivalent to a database name being like a table name, and a view name being like a where clause. You just can't meaningfully distinguish types on the server without those things.

I do have some code that makes some of the URL building and response collection easier, and deals a little bit with the _id and _rev properties, which was fairly straight forward.

---

The real kicker was realizing that each resource type effectively ends up needing its own Couch client, and the AOR ADMIN model prohibits the idea of having different clients to access different resources.

I did see a highly restricted way to use AOR and Couch with a single client, but it requires a Couch db designed from the start to be used with AOR apps.

Overgeneralizing the whole thing, the AOR abstraction goes something like:
"Each AOR application has a base URL called a server; that URL directly exports a number of resource types; each type has an ITEM and a LIST, and a client translates requests for ITEM/LIST to that base URL."

the Couch abstraction is way more complex, going something like:
"The Internet is littered with REST items, the same item could be replicated to many places; a database encapsulates a distinct collection of items; databases are hosted by servers (the server and database name combine to identify the base URL for an items collection); the big soup of all ITEMS, regardless of the human idea of resource "type", are accessed directly under the database URL by their _id; these items can be grouped into many LISTS called views (which are differentiated by using a different URL for each list) if a human being wants to have a 'resource type' kind of list, a view is where that happens."

So in the basic case of adding a single resource type at the same server and database, it's no problem.

And if you only use one design document and ensure each view name is standardized to something like "[AORResourceTypeName]List" so the LIST URL would always look something like:
[couchServer&Database]/_design/[dDoc]/views/[AORResourceTypeName]List

You can mostly get away with something that meets AOR's expectations; enough to write a client. And it stops there. And even with that I found edge cases when navigating between LIST pages and individual ITEM pages where the URL construction required lots of special code and attention because AOR's default assumptions about the URL relationship between items and their lists is not what Couch does.

---

I think we also must toss out any thought of using AOR to help in resolving conflicts in Couch documents. I can't see any way AOR could request a different ITEM version of the same resource id. I'm sure it _could_ be done, but it's going to be convoluted.

---

I, like you, was integrating to an existing database where the view names were not standardized like I mentioned above. As the view names were unique on a per Resource Type basis, the resource declarations were the obvious place to provide the extra metadata of "What is the name used to get a LIST for this resource type". I added this to my own code because it was the only sane thing to do, but as you see above it is against AOR resource declaration policy.

This is why, absent per resource metadata in AOR, a meaningful Couch client ends up needing to declare resources twice one way or another; once for AOR and again for the client. Whether you make a special Couch Config, or code each resource with different functions, it's the same net idea.

I liken it to AOR and Couch both understanding the idea of files, but where AOR assumes everything starts in the same place and types are directory names, Couch has these added ideas of a drive, or a file root, for files; and directory names represent filtered collections of those files. Couch requires these extra dimensions of information, and AOR excludes any facility for directly providing them.

---

And the idea of having different LIST pages for the same resource is a non-starter (like different view or subsets of the same resource type -- like an "aircraft" resource might be split into "helicopters" and "planes" with a different list of default columns meaningful to each subtype).

---

Trying to establish hierarchies for a resource is also non-trivial.
For instance, imagine you have a resource "airports", and to make life easier for the user, you'd like to use the menu navigation on the left for a country/state/city breadcrumb trail to easily filter the user's list.

You also want to make it so that as they move down the hierarchy, the list is filtered; so I wouldn't see "Toronto" and "Sydney" as possible city selections when I'm rooted in the country "Germany".

---

All is not entirely lost, there is one potential path forward that I think fixes most of these problems.

If AOR adopted the model that it was actually it's own REST server, and a client then hooked/usurped the predicted AOR paths, then a client gets higher up the request food chain and can translate the entire URL into an appropriate server request. It also requires being able to look up type specific metadata.

For example, the local paths for the above country/state/city hierarchy example can be hooked by the client (probably by hooking /q/ or /geo/ or something like that) and then determine what to do from the URL path requested.

Or imagine an application where every user has their own Couch database; Couch's normal pattern.

Every database on the server has the same types of resources, but the items themselves are unique to each user and might reuse the same _id. AOR has no easy way to deal with that at the moment; it's difficult to alter the URL for a resource's LIST view based on something contextual like "currently selected user". The userid has be to part of AOR's item id to keep the uniqueness requirement, but that item id can never make it to the real server. Having the client hook the browser path, it can more easily provide that translation before making the query, and correctly recreate the required id in the response.

So rather than thinking of the client as satisfying some request at a remote server, AOR's local storage would be its own REST server and the client is implementing the path<->JSON reply relationship.
All data is retrieved and updated to/from that local storage RESTful point of view which also fits nicely with the REACT POV.

---

Anyway, there's my $0.02 of what you'll find when you try and use AOR with Couch. I hope it gave you some insights about where the sticky points are.

[fzaninotto]

the AOR ADMIN model prohibits the idea of having different clients to access different resources

I think you've misunderstood how REST clients work, see for instance https://stackoverflow.com/questions/44773018/how-to-setup-dynamic-urls-for-resources-in-admin-on-rest

Besides, if the REST client approach can handle all the backends listed in the documentation, I really don't see why CouchDB would be out of reach.

[MikeFair]

the AOR ADMIN model prohibits the idea of having different clients to access different resources

I think you've misunderstood how REST clients work, see for instance

I didn't, at least not for that statement.
The statement was that AOR can only instantiate one REST client. And that's true.
For example, I can't declare two different resources to come from two separate server sources. (Say if I wanted to compare a resource in the "test" database to a resource in the "production" database.)

Resources can't carry metadata other than their own name into that client. And that's basically true.
When you have to write custom client code for every resource you make, you are effectively making a custom client for every resource; even though those clients all happen to sit in a single client wrapper.

As there was no algorithmic way to take the "Resource Type Name" and get the "Resource List URL" because they have no correlation; the AOR model got really difficult to work with for me.

When you must have a custom mapping between "Resource Type Name" and "Resource List Name"; absent providing it in the resource declaration for AOR; you have to compensate in the client code.

---

It just occurred to me, because of the way CouchDB is organized, it might work if declaring an AOR resource was treated only as declaring a "view" in a CouchDB sense. IOW, you don't declare a resource type, you only declare item subset lists which in AOR is called a resource type.

This is because CouchDB doesn't have the same concept of a resource type.
The only type it comprehends in that sense is "document" aka "item".
It has a completely flat id space for items/documents, and then views create subsets of that flat space.

The Couch client would always ignore the "resource type" name when getting a single item, making its type name irrelevant, which is fine for CouchDB; and only use the name when looking at the LIST.

This isn't the approach I took when writing the client originally, and it's worth attempting.

---

https://stackoverflow.com/questions/44773018/how-to-setup-dynamic-urls-for-resources-in-admin-on-rest

It's true I don't have a complete mastery and comprehension of exactly how this black magic works. :)

I believe it might hold an answer, or a big part of an answer, but I couldn't tease it out.

That said, this example code highlights exactly what I'm talking about:

if (type === 'AOR_REST_TYPE' && resource === 'BASE_RESOURCE') {
    if (getUserFromLocalStorage === usr1) {
       url = url1
    } else {
       url = url2
    }
    options.method = 'GET';
    // other options       
    }

It would be one thing if there was a line of code that was able to request more data associated to the resource type; or that the params, or other passed in variable held that data already; as generalized code could make use of that.

However, as a block like this is required for every declared resource, I consider this declaring the resource twice, or requiring customized client code for every resource.

It's a choice to use custom code in this way; and I don't think it's invalid or "wrong". It's a totally valid way of doing it given the point of view and targeted intent.

While I believe it's a misfit for this resource declaration case; that's simply my opinion.

I'm not the one taking responsibility for writing the AOR code, and I can sympathize with the reasoning.

---

Besides, if the REST client approach can handle all the backends listed in the documentation, I really don't see why CouchDB would be out of reach.

I didn't say it was out of reach, I said it wasn't obvious or easy.

What I've attempted to articulate were the disconnects/challenges I had when connecting AOR to Couch.

Primarily, the client code doesn't receive enough data to properly construct a backend URL string for accessing Couch the way Couch wants those URLs constructed; and secondarily, other Couch features that I see would likely be inaccessible (like document conflict resolution).

I looked at the PostgREST backend because that's translating to a SQL backend; which I consider has essentially the same string construction "difficulties" as Couch does and it kind of proves my point.

You need to have code as complicated as PostgREST to serve as the translation layer for mapping the AOR ITEM/LIST REST model to a backend with an arbitrary kind of item and collection names scheme.

[MikeFair]

I thought it worth saying again that one way to change this would be to have a "ClientConfig" section in the "Resource" declaration.

<Resource blahblah...nothingnewhere...blahblahblah>
  <ClientData IgnoredButPassedAlongWhenInteractingWithClient />
</Resource>

[fzaninotto]

OK this is too long to read. Can you make your point in 3 short sentences?

[MikeFair]

It's actually 3 points, the sentences I think most effectively say them are:

• Resources don't appear to have any metadata other than their own name once in the REST client code.
• All resources in an AOR app must share the same REST client
• Couch's idea of "document conflict resolution" seems "extremely foreign" and may not be easy in AOR

These aren't really new or disputed points, just "what I ran into" when making a Couch backend.

• You've already explained the metadata thing is a design axiom of AOR.

• I wanted to pull in the same resources from multiple different servers to compare them (Couch has this whole replication function where items get copied around to many servers); so different resources coming from different servers makes sense in that context.

• Couch reuses the same "ID" field for documents that have an internal conflict (multiple people updated the same object simultaneously, but differently); they are distinguished using the _rev field. But asking AOR to pull in multiple items that all have the same ID isn't doable; so it needs some method to make those IDs distinct for AOR.

The first point is the only one I found I was unable to work through.
I found that I needed add'l, resource level, metadata to make a proper "Couch Client" mapping. I might have been "doing it wrong", but for me the criticism is a meaningful and important restriction.

One can always write specific client code to map to a particular Couch database schema, I was looking to make a generalized solution that was reusable as a "Couch client" and was "configured" over "coded".

[MikeFair]

I think I can say this another way, though it's not quite as polite:

The mapping between REST and your server is the restClient responsibility. I'm aware that this implies adding some resource-specific logic in the restClient, with a switch/case on the resource name.

That it's somehow a _good design_ feature of AOR to force people to modify their REST client backend code every time they declare a new resource is shocking to me. Especially if these backend clients are code people are supposed to share with each other.

Different subsystems within a given framework will want specific metadata. Not all metadata is relevant to all subsystems, but that doesn't mean the metadata facility shouldn't exist. The metadata for various subsystems should be collected into their own configuration units so it is clear which elements belong to which subsystem.

For example, I could see having a <Menu /> tag within a resource declaration that alters its Menu behavior. In this case, the "Menu" is a subsystem within the AOR framework.

But I'm not the one writing AOR, I'm a React and web client neophyte, you have way more experience on the topic, and you made your position clear. That doesn't change my position, nor does it make me wrong. This is your domain, you get to make those calls, and I respect that.