Elasticsearch: Removing `{type}` from the index API

[clintongormley]

As part of the process of removing types (#15613), we need to remove {type} from the REST index API. In other words:

PUT  {index}/{type}/{id}   -> PUT  {index}/{id}
POST {index}/{type}        -> POST {index}

First off, I don't like POST {index} as it is too easy to confuse with PUT {index} (which creates an index). I think we should use POST {index}/_append instead, so we'd end up with:

PUT  {index}/{type}/{id}   -> PUT  {index}/{id}
POST {index}/{type}        -> POST {index}/_append

During 6.x, we need to support both old and new URLs as users will have some indices with types and some without.

The problem comes with how we defined the REST spec in index.json, where we specify POST and PUT:

    "methods": ["POST", "PUT"],
    "url": {
      "paths": ["/{index}/{type}", "/{index}/{type}/{id}"],

Actually, PUT only works with the second URL, while POST works with both of them (so the clients always use POST). This, however, leads to a conflict between POST {index}/{type} and PUT {index}/{id}. The spec has no way of saying _"use PUT with this URL and POST with the other"_. Usually these would be separated into different specs.

Below are three possible solutions:

Add an append() method

We could separate out index.json into index.json (with IDs) and append.json (without IDs), so (in 6.x) we'd have:

index.json:

    "methods": ["PUT"],
    "url": {
      "paths": ["/{index}/{id}", "/{index}/{type}/{id}"],

and

append.json:

    "methods": ["POST"],
    "url": {
      "paths": ["/{index}/_append", "/{index}/{type}"],

This would mean deprecating the index() method when used without IDs in 5.x in favour of the new append() method, so in 5.x the REST specs would look like this:

index.json:

    "methods": ["PUT"],
    "url": {
      "paths": ["/{index}/{type}", "/{index}/{type}/{id}"],

and

append.json:

    "methods": ["POST"],
    "url": {
      "paths": ["/{index}/{type}"],

and the clients would need custom code to issue a deprecation warning when the user uses index() without an ID. From 6.0 onwards, the user would need to use append() instead of index().

Default type to doc

In 6.x we could keep the old URLs only, but default type to doc (the dummy type name for 6.x indices with a single type):

index.json:

    "methods": ["POST","PUT"],
    "url": {
      "paths": ["/{index}/{type}", "/{index}/{type}/{id}", "/{index}/doc/{id}"],

but users like to upgrade the server and the client separately, and the above wouldn't work in ES v7

QS parameter

The third option would be to add a query string parameter like url_includes_type (defaults to true), which the clients can set automatically depending on whether type was passed in or not.

In 6.x, index.json would look like this:

    "methods": ["POST", "PUT"],
    "url": {
      "paths": [ /{index}/_append, /{index}/{id}, /{index}/{type}", "/{index}/{type}/{id}"],

In ES 7 (which won't have any types), we could just ignore the setting (or throw an exception if it is set to true), and remove it in v8.

Thoughts?

[clintongormley]

@elastic/es-clients please could you comment

[Mpdreamz]

Gut knee jerk reaction: not a fan of the append nomer.

How about

POST /{index}/_doc(ument)/{id}
PUT /{index}/_doc(ument)

As the new canonical for the index API? In the interim we could keep supporting /{index}/{type}/{id} and if someone really needs to store _document typed docs create a as flag for that in 6?

[spalger]

I like the idea of separating index and append operations so that the automatic-id assignment is more clear, but I'm not sold on the name. That said, the only option I don't like is the qs parameter.

[HonzaKral]

Note that with this schema we are limiting what ids are valid - anything that is a valid endpoint (_settings, _refresh, ...) will no longer be a valid document id. Currently there is no such requirement and it can even contain / character. While we are doing this it might be worth to have a discussion around what constitutes a valid document id.

I would love to see more something like PUT {index}/_auto - just use the _auto (or any other _word) in place of the ID and finally end the difference between POST and PUT that only exists with this API. Essentially just have a special value for the document id meaning elasticsearch should just make one up.

[pickypg]

If we default the _type to doc, then what's the plan for 7.x? Deprecate /{index}/doc/{id} in favor of /{index}/{id} or forever live with doc there? It seems like that would be hard to "fix" without being totally disruptive in 7.x.

In ES 7 (which won't have any types), we could just ignore the setting (or throw an exception if it is set to true)

The query string parameter seems like it might be the most favorable in terms of stripping it in 7.x and blocking requests in the future. Especially since we automatically block requests with unused parameters, this seems like the most appropriate way to get this done.

[pickypg]

To continue with that thought, in 7.x we'd have to _block_ sending {id} with a POST request so that 6.x users sending a _type can't accidentally continue to use the same API.

POST /my-index/my-type
{ }

[HonzaKral]

For sending types in 6.x I would much prefer to move the _type into a query string rather than have 2 different versions of the url with a qs parameter distinguishing them - we could even add that to the last 5.x release (if there is still going to be one) to make it fw compatible and ease the migration...

[Mpdreamz]

Can someone reply to my suggestion? Not seeing how it would not work and it feels the least obtrusive moving forward.

[clintongormley]

@Mpdreamz I think I'm coming around to your suggestion, although semantically the methods are the wrong way around, they should be:

PUT /{index}/_doc/{id}
POST /{index}/_doc

This works nicely with (eg):

PUT /{index}/_alias/{name}
PUT /{index}/_settings

etc

[nik9000]

Would we also have GET /{index}/_doc/{id}? PUT and GET should line up whatever we do.

[clintongormley]

Would we also have GET /{index}/_doc/{id}? PUT and GET should line up whatever we do.

makes sense

[clintongormley]

Closing in favour of https://github.com/elastic/elasticsearch/issues/15613