Data: Add a public API for adding and unshifting internalModels onto an existing RecordArray

[aortbals]

In our application, there are cases where adding a new record to an existing record array is necessary, sometimes at the beginning of the array. However, the API for this is unclear. This used to be possible, but isn't any longer. It has been the most difficult change in moving to the latest Ember Data 1.13.x for us.

Adding items to a RecordArray via pushObject(s), addObject(s), unshiftObject(s), etc throws the following error, and it is difficult to debug because the backtrace doesn't show the root cause in some cases.

internalModel.getRecord is not a function

Does not work, used to work:

this.get('content').unshiftObject(item)

this.get('content').pushObjects(items)

this.get('content').pushObjects(items.toArray())

The solution is to use the internal models:

this.get('content').unshiftObject(item._internalModel)

this.get('content').pushObjects(items.content)

Obviously this isn't ideal. Can we add a recommended API for adding or unshifting objects onto an existing record array without reaching for private methods? I'm sure that if these previously-working APIs were fixed it would help many upgrades.

Thanks!

[tchak]

Hey @aortbals. Can you explain in more details your use case? Where you getting RecordArray from? store.findAll() or similar?

[aortbals]

@tchak It's typically a relationship array or an array coming from the store. The two most common use cases I can think of that we are need to mutate the array are: (1) 'load more' pagination and (2) adding a newly created item onto an array. Sometimes these additions need to be done manually for a variety of reasons. Does that answer your question?

If there are alternative public APIs for this then I'm all for it, but I haven't seen them.

[miguelcobain]

I'm using both use cases mentioned by @aortbals.

Basically:

1. Route loads a store.query() RecordArray in model() hook. We use the query method because we need to get the records ordered by date and to send pagination params
2. User creates a new record and saves
3. On success, we need to display that new record at the top

Also, pagination based on query() is a good example of needing to add objects to the current model, which is a RecordArray.

I'm using exactly the same workaround described by @aortbals.

[xenyou]

+1

[koryteg]

yeah I had this issue as well. using this._interalModelworked for me but seems hacky because it is a private method...

[workmanw]

:+1:

[grapho]

big +1 here.
90% my collection data is with query() for pagination and filtering, but also need to support of being able to add a new thing to the array (before/or after a save).

would be ideal if i did not need to call array.update() after i add/save a record.

[ghost]

+1

[greyhwndz]

This is great :+1: +1

[carlbennettnz]

+1 yes please

[quaertym]

:+1: Need this asap

[tchak]

TLDR: This is not going to happen.

What you asking for is to be able to push random stuff in to a live array. It can't possibly work. Befor glimmer, there was actually a half valid performance driven use case. In ember 2.0 you can just do store.find('item').toArray() and then add stuff to the resulting native javascript array. The proper solution is to create a service that holds your actual data, and populate it from ember data updates and from your local updates. In cases you use query (like @grapho) you don't even have a live collection so calling toArray() will change nothing to your flow. Actually, I think query should return a plain javascript array, as it is not live anyway.

[aortbals]

So the answer to this problem for most users will be to call toArray() on a "live" Ember Data array and mutate it? Creating another service that proxies between local and remove ("live") data from ember data (but only if it's a "live" result) just to mutate an array has a lot of complexity that I'm afraid most people will not consider (I didn't).

I think part of the confusion around how to do this is the fact that Ember Data's live arrays are a duck type of a normal array but they only support a small subset of the same API. There are 7 different *Array classes listed under the DS namespace, so it becomes hard to investigate further without diving into the source. Even then, it's hard to know exactly how an array is represented in a given context.

The original goal of the issue was to somehow expose the method's that are used internal for managing live arrays so that the end user has control over low level mutations when they need it, and these updates are persisted in the store.

Thanks for the explanation @tchak.

[tchak]

@aortbals I understand you pont. I agree, there is something missing from the big picture.

There are 7 different *Array classes listed under the DS namespace

Almost non of them are public though :) And not all of them are live. I think returning native arrays from query method could help a lot.

Ember Data's live arrays are a duck type of a normal array

Yep, because Ember Data started before ES6 @iterator was a thing. In my opinion, this is the only "interface" any ED collection should expose.

The question remains, how actualy mutation of internal ED objects should work? If I mutate a live array in user space, once there changes comming in, how eventual conflicts should be resolved?

[aortbals]

There are 7 different *Array classes listed under the DS namespace

Ya, my point here was that at first glance it's hard to grok the internals, what the supported public APIs are, etc.

Yep, because Ember Data started before ES6 @iterator was a thing. In my opinion, this is the only "interface" any ED collection should expose.

This is interesting, and again it makes things much more clear if it were to be the case. If ED arrays we're always immutable and the docs indicated so, you could either dup it, like with toArray(), and use that, or don't modify it. But at some point you may still want to make transforms on the state in the store.

The question remains, how actualy mutation of internal ED objects should work? If I mutate a live array in user space, once there changes comming in, how eventual conflicts should be resolved?

In my use cases I actually expect any mutations I'm doing to get clobbered. I wouldn't expect Ember Data to resolve conflicts. Creating a record in some manner, persisting it on the server, and the unshifting it onto a hasMany relationship is what comes to mind. If you were to request the array again it actually would be identical.

[tchak]

Creating a record in some manner, persisting it on the server, and the unshifting it onto a hasMany relationship is what comes to mind. If you were to request the array again it actually would be identical.

store.find('post').get('contacts').createRecord({ text: 'hello' });

or even this is supported I think

let contact = store.createRecord('contact', { text: 'hello' });
store.find('post').get('contacts').pushObject(contact);

Has many are definitly muttable.

[grapho]

@tchak i appreciate your feedback! Storing the records separately in a service (like you mentioned) was actually going to be my workaround anyway... it turns out it seems to be the currently encouraged solution. I guess at first i was too attached to the idea of a resolved model hook applying to the controller and then i can "just go for it", like in the old days. Thanks for the reply.

[workmanw]

:-1: I too am pretty disappointed by this. Maybe manipulating "a live" array isn't exactly the right answer, but IMHO there is an undeniable need here. We frequently have this use case:

User is viewing a list of items that is the result of a server query. User adds a new item via an inline add mechanism. Newly added item (which is root.loaded.created.uncommitted) needs to show up in the same list side by side with other items (even locally sorted the same).

Right now we've built a "query manager" entity that manually manages an array and also adds observers to the RecordArray that resulted from the query to keep everything sorted properly and in sync.

[grapho]

@tchak I must ask, this seems to be a semi-recent change with new ember-data, or perhaps it was always meant to work this way but was not strictly adhered to... Anyway.. are these changes in some way indicative of how the future model/route/routable-component interactions are going to be? because if so perhaps it will make more sense in that context and the rest of us users are just a little early to see the full picture?

[tchak]

@grapho manipulationg ember-data arrays was never a good idea. It happened to work out okayish before, but with recent changes with internal models it just can't work anymore.

Yes there is obviously something missing. Depending on who we ask, ember data is facing very different expectations. I am confident that it will sort out ok, but it will take some time.

In my opinion, best thing to do right now, is to write down examples and use cases in order to understand the need. I think this issue have a good start on this matter.

@workmanw the result of a server query is not a live array. Calling .toArray() on it and then manipulating it chages nothing. It seems that what you asking is some kind of mixed live/server query, right?

A bit of history. Originaly, one of the reasons ember data had to used custom arrays was because of the way array observers worked in pre glimmer times. Another reason was to add some special behaviors to the custom arrays, like reload or update methods on ManyArray and RecordArray.

I can think of 4 types of array like objects:

• AdapterPopulatedRecordArray -> a result of query call (not a live array)
• FilteredRecordArray -> a result of filter call (live array)
• RecordArray -> a result of findAll call (live array)
• ManyArray -> hasMany relationships (live array scoped to a parent)

ManyArray is already muttable, it will change the relationship and will persist the change to the server if save is called. It is also live. It means if you create a new record with given parent, it will show up in the array.

AdapterPopulatedRecordArray is not a live one and have almost nothing special. Calling .toArray() on it should not be a problem. If you want my opinion, given recent changes in glimmer, we could just return plain old javascript arrays from query.

RecordArray is the array that contains all of the records of certain type in the store. It is important to understnd, that there is only one instance of this collection for a given type and it is always live. So I really can't see the point of mutating it. Any time a record of given type is created, it will show up in the RecordArray of its type.

FilteredRecordArray is probably the most complex question. To begin with, we know there issues with filters. Like the fact thet they are leaking memory "by design". There is work done on it, and there was several RFC on how to implement a better filtering api to avoid memory leaks. On the other hend, theus arrays are also live, so again, if a record of given type is created or loaded, it will show up intere.

It seems that one common reported dificulty is some of you are trying to manualy influence the sort order in thus arrays. This is not going to work. All of theus collections are "materialized views" on the current state of the store. We could expose a mutation api on theus collections, but it would have to return a new copy (hens call toArray) on the collection. And we are back to square one...

[bmac]

I think @tchak has provided an excellent summary of why Ember Data is unlikely to add "a public API for adding and unshifting internalModels". I'm going to close this issue because the recommend workaround is to use toArray to get a copy of the collection and manipulate that copy.

Feel free to reopen if you disagree.

[rwwagner90]

How is everyone handling this issue these days?

I tried:

this.currentModel.invoices.get('content').pushObjects(records.content);

Which does update the content array, but it doesn't seem to trigger anything that the model is being passed to. None of my components update after manually pushing in content like that.

[sandstrom]

Depends a bit on the situation, put sometimes I'll do something like this:

// https://github.com/emberjs/rfcs/pull/57
myModel.hasMany('invoices').push({ /* ... data here */ });

[alizahid]

Using this.model.content.addObject(newItem._internalModel) right now.