Activitywatch: Timezone issues in aw-webui

[ErikBjare]

Since I made the aw-core Event model always normalize timestamps to UTC we've gotten an issue with the web UI (and possibly other places) where what is supposed to be converted to local time isn't.

I think this also holds for the starttime and endtime parameters when querying a bucket for events. We should add some integration tests for this.

• [x] Fix it for the timeline view (fixed in https://github.com/ActivityWatch/aw-webui/commit/f2ff7045022e0e2eb5c73946d367a08f2b21f7c8)
• [x] Add note to the docs that all events will be stored as UTC (9ad62521407ed58f0a70d36f45c75081dbcb3dd1)

Not sure if there are any other places. Looks right for the bucket view.

[nikanar]

Running in the same issue, also with the timeline.

Also, in the buckets view, pressing "export to JSON", the events there are not in local time either (no idea if they should).

And the alt-text of the "export to JSON" button says "Not Implemented" which is disturbing, since it seems to be implemented (I click the button and it shows something looking like a JSON). Was going to investigate but aw-qt just crashed on me...

[nikanar]

Actually, because it is wrong in the JSON export, it seems wrong also when fetching the events with aw-client. The times are correct in UTC, but the information of "in which timezone did this event happen" is lost, afaict.

When is that information last attached to the Events ? Can it be maintained forward, somehow, please ?

[johan-bjareholt]

We are using peewee as the default datastore, and the DateTimeField in peewee does not have support for timezones so we are just converting them to UTC currently and storing them, so the timezone is lost as soon as the event is inserted in aw-server.

The only way for us to fix this would be to implement our own BareField in peewee which we could call "DateTimeZoneField" or something which has support for timezones.

[ErikBjare]

Storing timezones is a mess generally and analysing them is even worse, especially when dealing with timestamps using different timezones.

It makes a lot of things easier for us to just store them as UTC and then present them to the user whichever way they want them. The timezone doesn't actually contain any important information, so the reasoning went that for the sake of simplicity we would be better off just dropping them.

Not sure about the peewee thing, but might be true.

This has been discussed before: https://github.com/ActivityWatch/activitywatch/issues/39

[nikanar]

The timezone doesn't actually contain any important information

I beg to differ. As a user, all I want to know is what the time was in my head (or environment) when I did what was logged. That is, in my timezone, in the local time of me and my peers. You can't just tell me what the UTC time was at that point, and expect me to remember where I was in the world to interpret it back. As a user, I would be very angry at the software if it did that, for as much as I understand the rationale as a dev.

For starters, you're reporting the wrong date on much of the data, as everything done past 8pm is recorded under "tomorrow". Half the software I use has the reverse feature : extend the day past midnight, that is, assume everything done between midnight and 4am (configurable) was done yesterday, because in my head-local timezone, that's how days work (from sleep to sleep).

And sometimes I travel, and sometimes DST kicks in, so I can't just hardcode "-4" in the visualizer, it would be 1h-wrong over half the data, and very wrong for overseas periods.

I don't mind the thing having to be restarted/reconfigured when adopting DST or changing timezone. Ideally, I can set a knob somewhere in a config file to tell it my timezone (bwim "UTC+X", not the historical subtleties that go with Europe/Vienna), and I will adjust it when it changes, incl. DST. After all, I already have to re-setup f.lux with lat/lon when I move - and it reminds me to, by being broken in my face until I do.

---

It makes a lot of things easier for us to just store them as UTC

Yes, that sounds right...

and then present them to the user whichever way they want them.

... and you need to timezone info to do that.

As an end- power user, I want extend the day past midnight (midnight is local time).

As someone who is hacking at his tools, I'd be satisfied with "what did my local calendar+clock display at that point", and I can live with there being two 2:30 moments on certain days, or none, and all the related madness.

As a dev, I can deal with all-UTC times, and the localtime delta being stored elsewhere, and having to add them at some point in my visualizer. I really need it to be stored somewhere, somehow, pretty please.

Your source says :

If you need to record where the time was taken, store that separately.

My point is you need it. Else, how are you going to solve the current issue (timeline is broken) ?

[ErikBjare]

Alright, I can sympathize with much of that.

Here's my attempt at dissecting your concerns:

1. Visualization is broken. This is an easy fix, we don't need to store the timezone to fix this issue. We can get the users current timezone from within the browser, set up the query accordingly and then change the UTC timestamps into local timestamps before visualizing.
2. You want to extend the visualized day past midnight. This could easily be done and I consider it worthwhile. I'd just want a good default as setting up configuration for it is a lot of extra work. 4h past local midnight sounds good to me, do you agree?
3. When traveling, you want the visualization view to still be coherent and want the logging itself to be timezone-aware. RescueTime has a timezone setting that presumably solves this in a manner similar to the hard-coding you describe. I think this is the tricky one, and while it's certainly something we could pour effort into I don't think it's worth it at this point (unless you have a good idea on how to fix it).

As a dev, I can deal with all-UTC times, and the localtime delta being stored elsewhere, and having to add them at some point in my visualizer. I really need it to be stored somewhere, somehow, pretty please.

Writing a watcher that logs your current timezone would be trivial, I'd give you a code example if I was in front of the computer. But basically just send regular heartbeats with the timezone stored as the data.

[nikanar]

Your dissection skills are spot on and useful :)

I think we can leave 2. aside in this thread, as it really is a separate enhancement. I think most users would be surprised to see non-midnight day ends, so this feature is usually on a sign-in basis. Midnight default sounds right. With the feature on, yes, 4am is good.

I think this issue should focus on 1 - visualization is broken. I believe logging should be timezone-aware (times stored in UTC, timezone stored in separate field) - so 3, because if timezone data is forgotten when logging, then visualisation cannot be fixed.

Because, right now I'm in America, UTC-5, I eat lunch at noon, it's recorded as lunch 17h UTC. Next year I'll be in Europe, UTC+1, and I'll visualize today's data. 17h UTC will get interpreted as 18h local time, and then "you had lunch at 18h" makes no sense in the visualizer. Worse, I'll eat lunch at noon again, it will be recorded as lunch 11h UTC, and my lunchtimes will be scattered in the data, while my habits stayed exactly the same. Once the database contains lunch 17h and lunch 11h, how to recover that I really lived through the same event ?

---

Ah, no need to watch it, really, if I really can't get its localtime offset from the event itself, I'll just hardcode my travel dates into the data interpreter/visualizer (offset AW-data with the following dict {from <day> to <day> : UTC-5, from <day> to <day> : UTC+1, DST, etc.}), there are few enough events to do this manually. I just don't think you want to produce the seemingly inconsistent databases that I describe above, and that I think are bound to happen with the current timezone-forgetting design.

[ErikBjare]

I'd be open to adding a column containing the UTC offset to the database.

That would require writing a database migration however (which we've actually never done, we've just discarded the old databases until now). Would probably be a good exercise, but I want someone else to chime in and make a good argument in favor of it.

Would also require some changes to the Event class which would lead to some added complexity.

[johan-bjareholt]

An alternative could be to simply discard the timezone instead of converting to UTC. That way we wouldn't need to care about the timezone at all, and the only flaw would be that there could be more or less than 24hrs in a day and order could be messed up when traveling but that's an acceptable flaw in my opinion.

[nikanar]

At first sight, I love this alternative.
I goes straight up against all well-established programming good-practices though.

Looking for reasons why established practices would trump our local use case, I find :

• This YCombinator post (the first comment) makes the same point as I was making above.
• This SO community wiki covers the topic in depth, and of course recommends to store UTC + offset.

• This SO question is most relevant for us, and I take away from it that the alternative is viable.

  If we go this way, then during DST-transition (and travel), some period of time (1h / several) will be watched twice, or not at all. I think this is fine, as it is indeed what the user lives through.

We would lose the ability to compare one's history to universal time (or other users), i.e. "The opening ceremony of the 2016 Summer Olympics took place on Friday 5 August 2016 starting at 20:00 BRT, what was I doing back then ?" cannot be accurately answered (one can only guess what his offset may have been back then). I don't think it matters too much for the purpose of AW.

[vith]

The temptation may be to forget the timezone/offset so that the problem of choosing when a new day starts seems simpler. But I think it would cause even more problems.

Storing local time without zone/offset info would mean you can no longer ascertain the duration of an event with confidence. Work periods could end before they begin and have negative duration, or be exaggerated in duration, depending on which direction you're traveling. Even if you're bolted to the ground, timezones will change out from under you. This would be dangerous for anything related to billing or contractual obligations.

Even if you stored the duration of the event explicitly, the visualization would still be nonsense, showing the same event overlapping on top of itself multiple times or overlapping other events that really didn't happen simultaneously.

If you work while riding a bus or plane, you don't want to see your activity time compressed or expanded like this whether with gaps/overlaps or by misrepresenting durations. This could be a rare event for some but a common occurrence for others.

So I think this option should be ruled out. Keep UTC time, add offset or zone info.

I do think it's absolutely necessary to allow the user to specify their "start of day" time, but even that won't be so simple, if you consider that a user might move to the other side of the planet or just change their work/sleep schedule and want to be able to have their old data and new data both still make sense. The preference would probably need to be stored with a datetime from which it applies going forward, with the ability to have different values for it apply to different segments of the data records. The start of day would then need to be determined separately for each day (as a UTC timestamp).

Visualizations should probably just use the UTC clock as their underlying coordinate system regardless of how times are labeled for display in the UI or day divisions work.

[ErikBjare]

Yeah, storing as UTC is definitely the way to go, I'm more sure about that now than ever.

Adding a column for the timezone offset is still the proposed solution to the problem of figuring out "head-local" time. This will require a database migration, so will probably not land in v0.7.

Workarounds for the web UI will be in place soon, they will work roughly as I've described before. There will be a PR I welcome you all to review where you can raise specific concerns with my solution, I think it will be easier to discuss issues over a proposed changeset. If it turns out that there will be issues that can't easily be resolved in the aw-webui PR it will have to be the subject of another issue.

I'm starting to have difficulty following the conversation so please keep your responses short and concise going forward in this issue. I think we've said most things there are to be said, so if you feel a point needs repeating, please refer back to it instead of restating it.

And welcome @vith, very happy to see you here. Your input summarized important aspects of the issue well, so thanks a bunch.

[ErikBjare]

A fix for the timeline visualization was made here: https://github.com/ActivityWatch/aw-webui/commit/f2ff7045022e0e2eb5c73946d367a08f2b21f7c8

[ErikBjare]

Added caveat regarding datastore normalizing to UTC and stripping offset here: 9ad62521407ed58f0a70d36f45c75081dbcb3dd1

We're done here for now I think. Will keep this discussion in mind for when we add UTC offset to the database.

Thank you @nikanar, @johan-bjareholt and @vith for helping out!