Data: Bulk operations support

[andrewfan]

From TRANSITION.md i understood that ember-data will handle bulk operations in future. In my project it is really critical to submit data to server in 1 http\s request. I take a look inside store code and looks like we can implement bulk operations right now.

store.flushPendingSave (https://github.com/emberjs/data/blob/master/packages/ember-data/lib/system/store.js#L949) methods can separate records by record.type and then by operation type and call _commit (https://github.com/emberjs/data/blob/master/packages/ember-data/lib/system/store.js#L1761) function with new operation name f.e createRecords.

after this the only thing we have to do is to add new methods to serializer that will call existing extractArray method and new methods to adapter.

should i start working on it, or my guesswork are incorrect?

[sheldonbaker]

I had a similar requirement a few months ago and whipped this up: https://gist.github.com/sheldonnbbaker/7330288

It's messy, but it worked just fine for me at the time.

[kurko]

@igorT is it something desired by the core members? I have use cases in my app where we save several resources at the same time (same type), so we have to make several requests. We'd like to be able to send an array of hashes instead (ala jsonapi.org).

If you give us green light, we could start working on it confident that it'd be at least analyzed by the core team members.

[igorT]

Yup, we already have the infrastructure using flushPendingSave, so bulk saving would be nice. Will analyze :)

[kurko]

Just to throw something here, given you push the following array:

{
  'posts': [{
    'name': 'Rails is Omakase'
  }, {
    'name': 'TDD is Dead'
  }]
}

And then response is:

{
  'posts': [{
    'id': '1',
    'name': 'Rails is Omakase',
    'created_at': '2014-10-10T10:00:00Z'
  }, {
    'id': '2',
    'name': 'TDD is Dead',
    'created_at': '2014-10-10T10:00:00Z'
  }]
}

How do you guys think ED would track which object is which? I don't think comparing by attributes is safe, given it could be changed on the server during save.

[igorT]

What does json api say? How are you supposed to track? cc @dgeb

[ernesto-jimenez]

here's what json api says:

To update multiple resources, send a PUT request to the URL that represents the multiple individual resources (NOT the entire collection of resources). The request MUST include a top-level collection of resource objects that each contain an "id" member.

For example:

PUT /articles/1,2
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "articles": [{
    "id": "1",
    "title": "To TDD or Not"
  }, {
    "id": "2",
    "title": "LOL Engineering"
  }]
}

[dgeb]

@igorT sorry, just noticed you pinged me here ...

When POSTing multiple records, JSON API offers two solutions for matching results to corresponding records on the client:

• Client-generated IDs can be sent by the client and returned in the response: http://jsonapi.org/format/#crud-creating-client-ids
• If an API supports PATCH, multiple PATCH ops could be sent and the response order would be guaranteed to match the request order. http://jsonapi.org/format/#patch

The two solutions can be combined to allow creation of multiple primary and/or related resources in a single request.

On the simpler end of the spectrum, you could rely on the order of objects POSTed matching the order of response objects. Although JSON API does not explicitly require this right now with POST, we could discuss it.

[btecu]

@igorT any news on this?
I would really like batch PATCH or at least PUT operations.

[tim-evans]

I have a bug revolving around records needing to be committed in order. I would be delighted to use bulk commit when creating these records. (The use case is adding many items to an ordered list.)

More specifically, I would like to make this an adapter configuration. (I only want to opt-in to this behavior for some requests)

[leejt489]

This is pretty important for apps with network latency. Would like very much to see this integrated!

[SeyZ]

+1

[carrotcomputer]

+1

[workmanw]

We actually do this now by implementing a custom adapter which models a fork-join queue.

<braindump>

The TL;DR; is that when updateRecord is called on the adapter we create a "work-item" and push it into a work queue. The "work-item" has the serialized record, the record type, the id and an instance of Ember.RSVP.defer(). As for the updateRecord we return the defer.promise (Defer is used because the API is more convenient).

Pushing things into a work queue starts a timer (we are using a 100ms duration). We also have a maximum batch size. Once the queue's timer expires or the max batch size is reached, the AJAX is fired. Once the AJAX returns it's results we loop back over the work item, matching the corresponding commit status with the proper workitem. We use that status to do defer.resolve or defer.reject.

From an application perspective, we are just calling record.save() on a bunch of entities and the adapter will coalesces them properly.

This approach is very naive in that it is only batches entities of a common type and it doesn't allow the application to build it's own batch groups which could be beneficial if you wanted to have parallel transactions.

</braindump>

[kurko]

@workmanw any chance you can opensource that?

[Fenntasy]

@kurko

If you're interested, I made somewhat of a mixin for my needs :

const BatchUpdatesAdapterMixin = {
  batchUpdatesTimeout: 50,
  batchUpdatesPathForType: function(type) {
    return this.pathForType(type);
  },
  batchUpdatesBuildUrl: function(type) {
    const container = this.batchUpdatesPathForType(type);
    return `${this.host}/${this.namespace}/${container}/update_all.json`;
  },
  _batchUpdates_handle: null,
  _batchUpdates_queue: [],
  _batchUpdates_resolvers: [],
  _batchUpdates(type) {
    const container = this.batchUpdatesPathForType(type);
    const records = {
      [container]: this._batchUpdates_queue
    };
    this._batchUpdates_queue = [];
    this.ajax(
      this.batchUpdatesBuildUrl(type),
      "PUT",
      {data: records}
    ).then(() => {
      this._batchUpdates_resolvers.forEach(p => p.resolve());
      this._batchUpdates_resolvers = [];
    }, () => {
      this._batchUpdates_resolvers.forEach(p => p.reject());
      this._batchUpdates_resolvers = [];
    });
  },
  updateRecord(store, type, snapshot) {
    const record = Object.assign(
      {},
      snapshot._attributes,
      { id: snapshot.id }
    );
    if (this._batchUpdates_handle) {
      clearTimeout(this._batchUpdates_handle);
    }
    this._batchUpdates_queue.push(record);
    this._batchUpdates_handle = setTimeout(
      this._batchUpdates.bind(this, type.modelName),
      this.batchUpdatesTimeout
    );

    return new Ember.RSVP.Promise((resolve, reject) => {
      this._batchUpdates_resolvers.push({resolve, reject});
    });
  }
};

You can then use it as a mixing for your adapter and eventually overwrite batchUpdatesTimeout, batchUpdatesPathForType to change the name of your endpoint (which, for now, is the same as the container of your data that will be sent to the server), and batchUpdatesBuildUrl which define your endpoint url alltogether (I personnally end mine with update_all.json, you may want to change that to your liking).

I'm sure there's lot of improvements to be made but that seems to do the job and be similar to what @workmanw was suggesting.

[workmanw]

Sorry @kurko I missed your ping months back :( . We've basically done exactly what @Fenntasy has provided.

[kosmonaffft]

So, in which state bulk operations are now?

[dcyriller]

If not implemented in ember-data, this addon seems to provide the functionnality: https://github.com/Netflix/ember-batch-request (working with json:api spec)

[runspired]

Seeing as this is a customization specific to applications vs a general abstraction, I am going to close this ticket as something we do not intend to explicitly add as a feature.

That said, we do explicitly support your ability to create custom adapters suited to this purpose, and intend to continue improving primitives within ember-data to make implementing batch requests even simpler in the future.

One recently published RFC that will help in this nature is https://github.com/emberjs/rfcs/pull/403 which adds Identifiers which will enable users to solve race and ordering conditions prevalent in batch requests. Others are coming with improvements to overall management of the request lifecycle and improvements for communicating state updates to the store.