Ecto: add after_load callback

[briksoftware]

When working on #353, I had the idea that it would be nice to add an after_load callback to the model, which would be invoked every time a model is loaded. This would give the user a chance to set the value of virtual fields in a very convenient way.

Of course, I am up for the task of implementing this if the idea is approved.

[josevalim]

@briksoftware I had thought about this too and we had this discussion when implementing callbacks. It is worth remembering this function can cause drastic performance issues because it would be invoked for every struct from Repo.all, including assocs, preloads and so on. So I would add it only if we have very good reasons to add such.

[briksoftware]

@josevalim since each model has its own callbacks, assocs and preloads are only affected if they, too have registered callbacks, right? At the end of the day, it would basically be the Ecto.Type.load function for virtual fields. Same as in Ecto.Type.load, one must avoid doing expensive computation there.

Making clear in the documentation that this can be very expensive so the computation should be quick is, of course, very important no question about that.

This would replace, for me, the virtual_field from Crudex and I have a use case for this in a project where I use it. In my case associated fields would be also affected, but the computation I do is very cheap (computing a path starting from the model id... so string concatenation). I have not noticed, with the mechanism I am currently using, any performance hit.

An alternative would be to allow defining "resolvers" which could be invoked lazily on demand for virtual fields. But then one would have to handle virtual fields differently from "real" fields.

[josevalim]

@briksoftware sounds fair. Quick question: can't you use custom types in the virtual fields?

[briksoftware]

@josevalim custom types have access only to their own value, which for virtual fields is well..nothing until you calculate/resolve them. The callback would have access to the entire model and would return the fully-loaded model populated with virtual fields.

[josevalim]

Ok, let's do this. :)

[josevalim]

@briksoftware I am planning to rollback on the after_load callbacks. After re-reading this discussion, I was wondering, why don't you simply provide a function that computes the value instead of storing it in the model? Why do you need to store it in the model at all? :)

[briksoftware]

@josevalim oh, it would be quite bad for me if you roll it back. I had already thought about providing a function there, but then I would have to handle this field in a special way when encoding to JSON for example. I mean all fields in the model can be accessed in the form mymodel.myfield, but with a function it would become mymodel.myfunc.() instead so they would need special treatment.

Is there any particular reason why you want to remove the callback? It does not hurt anybody not using it, after all...

[MSch]

Re the example I gave on IRC with access_token, one IMO good example I have comes from a Rails App I maintain where we also derive an access_token from the primary key, but then use the token both when rendering the completion page and in the email we send out.

Because the access_token changes every time the access_token is derived we store it as a "virtual field" on the ActiveRecord model, so we can easily access the same token in the mailer view and the web view.

[josevalim]

@briksoftware it wouldn't be mymodel.myfunc.() but something like Model.myfunc(mymodel) or Crudex.myfunc(mymodel).

[briksoftware]

@josevalim the idea was to put an anonymous function in the model. They way you proposed requires that I again re-introduce the concept of "virtual fields" in Crudex which do exactly the same thing as the after_load callback.

I think it is generally a quite common problem, some fields are derived from other fields and so they are not persisted in the database. However, after fetching the structure from the database you want to have the information there, it has to be transparent to the caller. Without an after_load callback, the caller has to know exactly which fields are stored in the database and which are derived from stored ones. This goes against the principle of separation of concerns, because one day a field which used to be virtual can become a regular persisted field. And then I have to find every single place in the code using the virtual field and replace the function call with a simple retrieval form the map. With the callback, well you just remove the callback and you are done when a virtual field becomes persisted (and viceversa).

[josevalim]

Ok. Agreed. I will improve the after_load docs with counter examples. Thank you all for the discussion!

[pollingj]

@josevalim just wondering how this is now handled in Ecto 2, as after_load is no longer available?

[jamonholmgren]

I'm curious as well, @josevalim .

[josevalim]

Currently you must call some function after you retrieve the data. We are looking into ways of bringing this back but we are interested in a more general purpose mechanism and something less callback heavy.

[Tica2]

@josevalim Any draft ideas here? I think to start to write some ideas/notes about subject.

[jamonholmgren]

@Tica2 @josevalim My idea is that devs are encouraged to use a "store" to retrieve data. So you might have a module like this:

defmodule UserStore do
  import Ecto.Query, only [from: 2]

  def find(id) do
    from u in User, where: u.id == ^id
    |> Repo.all
    |> callbacks
  end

  defp callbacks(users) do
    Enum.map(users, fn u ->
      Map.put u, :full_name, "#{u.first_name} #{u.last_name}"
    end)
  end
end

Then you always just use that, rather than calling Ecto directly:

UserStore.find(95)

[jfrolich]

I like this approach, I kind of had a similar solution with an abstraction between the model and the parts of the code that use the model such as channels/controllers (thus all the side-effect heavy code). To have a "store" with all side effect heavy code is an interesting idea!

[slashmili]

@jamonholmgren / @jfrolich it works most of the time but when you load the models using preload you can't make those callback(you need to move the callback logics to other models)

Actually I used this approach and now I should always check to see if the data is coming from main module(therefor already callbacks is called or it's coming from preloaded data(therefore I should run another map on that).

So eventually gave up and now I'm calling something like UserStore.full_name(user) whenever I need to load user's full name.