Ecto: inserted_at and updated_at defaulting to DateTime instead of NaiveDateTime

[lau]

By default inserted_at and updated_at are NaiveDateTimes in Ecto 2.x. It would be nice to have them be DateTime in UTC instead by default. Because they will be inserted by Ecto as UTC. This means that we know that those timestamps are in UTC. The only case where that is not true is if for some reason Ecto is used with an existing legacy database that has existing inserted_at and updated_at fields that are not UTC. That seems like a very infrequent situation and is something that can be warned about in the docs so that if there is anyone in that situation they can specify @timestamps_opts to use NaiveDateTime instead.

When the library "knows" that a datetime is in UTC it makes more sense to have it be a DateTime struct in UTC than a NaiveDateTime. Because the computer already knows that it is in UTC it can help the user of Ecto to do more with the data. For instance in order to get a unix timestamp from inserted_at it could be passed directly to DateTime.to_unix/2. Or directly to DateTime.to_iso8601 which unlike to NaiveDateTime.to_iso8601 adds a Z indicating that it is in UTC.

In Elixir 1.7 some functions in the NaiveDateTime module do not accept DateTime structs (including add, diff). However with the future Elixir 1.8 a DateTime can be used with the functions in both DateTime and NaiveDateTime. So having more information is not a disadvantage when it comes to using the functions in the modules Date, Time, NaiveDateTime, DateTime. However Ecto 3.0 will be out before Elixir 1.8 as far as I understand.

Ecto 3.0 seems like a good opportunity to do such a change. The "only" thing most people would have to do is add/change @timestamps_opts in their schema for keep having timestamps being NaiveDateTimes. An alternative is to pipe through |> DateTime.to_naive for calls to certain NaiveDateTime functions if they have them until Elixir 1.8 is out.

Another option: Changes to the NaiveDateTime functions that allow DateTime structs being passed could be put in a new 1.7.x release and the Ecto 2.x to 3.0 upgrade guide could recommend upgrading to that release before upgrading to Ecto 3.0.

Is the Phoenix 1.4 release waiting on Ecto 3.0? Are there any defaults there that should be changed in generators so that newly generated Phoenix apps have timestamps as DateTime structs instead of NaiveDateTime? For a new Phoenix app being generated with 1.4 there would be nothing existing being broken even with the current version of Elixir.

cc @josevalim

[josevalim]

We won’t do it for Ecto 3.0 because DateTime is still limited in Elixir
v1.7 and we are hoping Ecto 3.0 will be out still this month, and Elixir

v1.8 is coming only next year.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[lau]

Would it make sense to do the change in 3.1.0 or 3.2.0 and warn about that in the 3.0 release notes? Somehow warn about it/prepare for it in 3.0 at least in the release notes/docs?

[josevalim]

Note we rarely backport features to stable Elixir branches, so having it on
v1.7 is not a possibility. Keep in mind that Ecto v3.0 is about supporting
Calendar types that shipped with v1.3, which is two years old now. It takes

time until we can consistently adopt those.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[lau]

Understood. What I mean is that if the plan is to have the timestamps default for Ecto 3.1 or 3.2 be DateTime then would it make sense to warn about that already upon the Ecto 3.0 release so that while people are upgrading or if they start a new project with Ecto 3.0 recommend setting @timestamps_opts to use DateTime UTC instead of NaiveDateTime. This way when a change is out in a future version of Ecto 3.x then new projects will not have to do any changes.

[josevalim]

Unfortunately a warning doesn't make much sense. Why are we going to push people towards a solution that will feel incomplete for the next three months, in the most optimistic scenario? That will only cause frustration.

[wojtekmach]

In Elixir 1.7 some functions in the NaiveDateTime module do not accept DateTime structs (including add, diff). However with the future Elixir 1.8 a DateTime can be used with the functions in both DateTime and NaiveDateTime.

@lau out of curiosity, how having or not having these functions in Elixir concerns Ecto? I think even if NaiveDateTime.add() takes a DateTime, the return value is still %NaiveDateTime{} so it wouldn't match :utc_datetime type. Are you relying on Ecto casting NaiveDateTime to DateTime (and the other way)?

I think changing default timestamps to :utc_datetime was raised a couple of times, and I believe the 2 biggest concerns were:

1. another breaking change for users, after switching from Ecto.Date* to calendar types
2. NaiveDateTime is in general a bit more _convenient_ to work with, there's a sigil and a matching inspect implementation.

is that it?

If we are ever to change the default, 3.0 is probably the last opportunity.

Here's one real world example where pushing folks to use :utc_datetime would be beneficial: https://github.com/hexpm/hexpm/issues/697, currently we use :naive_datetime type and it's string representation doesn't have "Z". I've briefly discussed this with @michalmuskala and the cleanest solution seemed to bite the bullet and change the type which makes me think that maybe it should be the default after all.

[lau]

We won’t do it for Ecto 3.0 because DateTime is still limited in Elixir v1.7

@josevalim The way I see it is that only NaiveDateTime.add is limited. I agree that it is inconvenient to have to call DateTime.to_naive in order to use that function. Are there any other examples of DateTime being limited?

On the other hand with a DateTime you can do the following that cannot be done with a NaiveDateTime:

• Get unix timestamp for the Ecto timestamp
• Output ISO8601 timestamps to e.g. APIs and include the UTC offset
• Compare with timestamps from external APIs or other parts of the system
• Compare with the current time on the machine and know the "actual" difference, not just "nominal".
• Work with timezones

and more. So it seems more convenient to me to have a DateTime rather than a NaiveDateTime.

Those are practical concerns, but I think there are other mort important ones. Such as Ecto setting a good example of when to use DateTime over NaiveDateTime. This is a good example of when a library can use a UTC DateTime instead of NaiveDateTime. If you use Ecto and have code where the timezone data is included there is more you can do with it.

@lau out of curiosity, how having or not having these functions in Elixir concerns Ecto? I think even if NaiveDateTime.add() takes a DateTime, the return value is still %NaiveDateTime{} so it wouldn't match :utc_datetime type. Are you relying on Ecto casting NaiveDateTime to DateTime (and the other way)?

@wojtekmach For instance if you currently pass inserted_at to NaiveDateTime.add in a current project with Ecto 2.x with the timestamps options at the standard NaiveDateTime and Elixir 1.7. That code will not work in unchanged in Elixir 1.7 because the NaiveDateTime.add functions does not accept a DateTime and would raise an error. It does work in Elixir 1.8 and returns a NaiveDateTime as before. The advantage in Elixir 1.8 is that you would not have to change as much code upon upgrading without changing the default.

Although for people that simply want to upgrade to Ecto 3.0 and avoid any changes related to the timestamps could relatively easily just change the @timestamps_opts.

another breaking change for users, after switching from Ecto.Date* to calendar types

This can be handled in large existing projects by setting @timestamps_opts to use explicitly use NaiveDateTime. Later "upgrading" to DateTime can be done more more leisurely by changing @timestamp_opts to use DateTime and do any changes that might be needed for instance in the case of an inserted_at value being passed to NaiveDateTime.add.

NaiveDateTime is in general a bit more convenient to work with, there's a sigil and a matching inspect implementation.

Sigils are nice. However I don't think sigils is worth the loss of data. There are all kinds of things you can do if you know the time zone that cannot be done with a NaiveDateTime. Going to/from NaiveDateTime (with sigils) can still be done with DateTime.from_naive or DateTime.to_naive.

If you get an inserted_at from Ecto as a NaiveDateTime and pass it on to a bunch of functions then at some point you might need to know the timezone and instead of it having it there already explicitly in the struct, you have to assume it. And then maybe the NaiveDateTime that function received turns out to come from somewhere else - maybe a Phoenix form - and is not UTC as assumed.

@wojtekmach I also think that biting the bullet in 3.0 is better and more convenient for Ecto users in the long run. All existing projects could start by setting the @timestamps_opts to use NaiveDateTime upon upgrading to Ecto 3.0. Except for those that already have @timestamps_opts set to use DateTime.

The missing "Z" in https://github.com/hexpm/hexpm/issues/69 is a good example. When it comes to outputting the datetime in the API if it were a DateTime it could all be handled automatically and the involved libraries would automatically know that it is in UTC and could safely output the "Z" in the API. E.g. a JSON library sees the DateTime and includes the "Z" when serialising it. When it is a NaiveDateTime the programmer has to go through the code to be sure that it is indeed in UTC. Then explicitly "bless/upgrade" it to an UTC DateTime or in a hack-ish way manually add a "Z" to a string somewhere.

I assume that setting timestamp_opts is not a big deal for existing projects upgrading to a new major Ecto version. For new Ecto projects I see a lot of benefits with a DateTime default.

[lau]

@josevalim ping. Did you see my last question? I am trying to figure out if you see NaiveDateTime structs being more convenient in other ways than how they can be passed directly to NaiveDateTime.add/2

[josevalim]

@lau not really, that's it. To me the root issue is the same, until more recent Elixir versions, the NaiveDateTime was more convenient and Ecto has to support what is there right now.

[datafoo]

Semantically speaking, inserted_at and updated_at are points in time in the past. Using NaiveDateTime does not make sense because, lacking the time zone, it does not provide enough information to fully know when in the past those points were.

Therefore inserted_at and updated_at are up for interpretation. If they are to be implicitly interpreted as UTC, why not using DateTime to make it explicit?

Semantically speaking, it is not about making DateTime the default but making DateTime the only possible way.

[datafoo]

Same goes for https://hexdocs.pm/ecto_sql/Ecto.Migration.html :migration_timestamps option.

[wojtekmach]

I personally agree these would be the better defaults. But we can't change them without breaking existing apps and given Ecto reached stable API it's unlikely to change and we just have to live with it and I personally think that's OK too.

[datafoo]

So the question is political: do the ecto repository owners prefer to break client code and improve things or not break anything and keep awkward behavior?

[lau]

With Elixir 1.8 DateTime structs can be passed to functions in the NaiveDateTime module. So now that DateTime structs not only can be used with all of the functions in DateTime module, but also functions in NaiveDateTime it seems that the default can finally be changed.

Ecto 2.1 introduced changes to the timestamps. Users were advised to set @timestamps_opts to keep the old defaults..

Ecto 3.1 could also advice people to set @timestamps_opts if they want to keep the old defaults. inserted_at and updated_at are textbook examples of when to use DateTime instead of NaiveDateTime. hex.pm ran into problems with NaiveDateTime being the default. Probably a lot of other users will too.

[OvermindDL1]

So the question is political: do the ecto repository owners prefer to break client code and improve things or not break anything and keep awkward behavior?

Not a political thing, it's a SemVer standard.

[datafoo]

So the question is political: do the ecto repository owners prefer to break client code and improve things or not break anything and keep awkward behavior?

Not a political thing, it's a SemVer standard.

I am not advocating to break the SemVer standard. Of course we should bump the MAJOR version when making a breaking change such as the one I discuss here.

[josevalim]

As others have mentioned, Ecto became stable API as of 3.0, so we are not planning any major releases.

[lau]

First of all thank you @josevalim and others for maintaining Ecto. Another great open source project that many people benefits from and can use for free.

I try to contribute with input where I think it can help. I wish DateTime UTC was the default for timestamps in Ecto. Because I think it can help with use in general and save headaches as well as this default maybe being seen as an implicit indicator of when to use DateTime vs NaiveDateTime. However I know that there is a balance of many things to take into considerations.

I hope that if a new version one day comes out that this default can be changed. Until then I will try to spread the word about the benefits of using DateTime when you know the time zone and setting the timestamp_opts to use UTC in blog posts etc. Even though it is not the default today, luckily it is not too hard to use something other than the default if you know how to set it the module attribute.

[aptinio]

Leaving this here for anyone who comes across this discussion:
http://www.creativedeletion.com/2019/06/17/utc-timestamps-in-ecto.html

Thank you everyone for your continued efforts and concern. :heart: