Deltachat-android: if message time is in the future, Delta chat uses device time not message time to sort messages

[nutpantz]

Delta chat uses device time to mark sort new messages. After a reset my phone time went to Jan 14 2014 and for 2 hours all new messages were marked with that time and sorted to the very front of my message list.. Needless to say I did not see a one.After I fixed the time on my device the messages stayed at the wrong time. Messages should be marked new and put at the end of the list where they are visible and the message sent time and received time should be checked in some way to notify of newly acquired messages with unusual time stamps.

[r10s]

in general, delta chat tries to sort messages by the _sending_ time, however, due to its chat interface, "fresh" messages are always sorted to the end of the list and must nor be newer as the device time - otherwise, they're set to the device time.

if the device time is set wrongly, strange things may happen. as well as when bad times are set from the sending or transmitting servers. however, i do not see an easy way to fix bad device times from the delta chat side.

[testbird]

Isn't there a database index with chronological sorting order that could be used?
Using that could also remove the need to overwrite the sending time.

[r10s]

i think the problem is not a lack of time information as such.
it's more about bad timestamps.

Delta Chat uses the "effective date" from Autocrypt as defined at https://autocrypt.org/level1.html#effective-date - in short, if a timestamp from a received messages cannot be in the future. this makes some sense and protects against different issues. however, if the device data is bad, this check will fail - but if will protect _other_ users eg. if your device time is in the future ...

i think the simple fix should be just to fix the time of the device as soon as possible :)

doing other things here _may_ work, but are hard and may also have other side effect - esp. when delta chat changes the database based on changed system times. what if the new time is even worse?

maybe we could calculate the "effective date" on the fly or so, however, this would have a bad performance impact and cause probably lot of more problems.

so, currently, i cannot see what can be done with reasonable effort on this issue.
beside setting the device clock correctly (which, btw. is done in the very most cases automatically by default)

[testbird]

What I meant wasn't a lack of time information, but to avoid the error entirely by using a local DB index column instead.

Sqlite seems to use a "rowid" index column, and allow to have own index (counter) colums. As an index is increased with every entry saved, it should serve well to sort the messages in the order in which they arrived, independently of local or remote system times.

If using the index to sort the messages when displaying, the "effective date" set in the message may then possibly be curious, but irrelevant. (Also no need to overwrite it anymore, which is a bad idea with time zone differences. I think there was also already a bug report with time zone problems. )

[r10s]

but this would not change the message order if the system time changes, imu this was the gist of the original post.

wrt to the id: in fact, we're using sth. similar, the sort_timestamp - in contrast to a uid, this keeps the messages in order. eg. when forwarding 10 messages, they may arrive in any order at the recipient - but if this is a conversation, they should appear in the correct order. this is ensured by the sort_timestamp and so called "time smearing", however, all this requires a correct date on the system (if works with a bad date, however, as mentioned, things are not resorted when the system date change, for the reasons mentioned above)

see https://github.com/deltachat/deltachat-core/blob/master/src/dc_receive_imf.c#L268 for more details about the calculation

[testbird]

but this would not change the message order if the system time changes, imu this was the gist of the original post.

But that's the idea. If the messages are stored to the database in the order in which they arrive, they get subsequent index values. That order always remains the same then (order of arrival) regardless of any local or remote time value, their correctness, or time zone or error corrections. That's why I think using such an index for the display order could solve the issue.

In the case of forwarded messages, when an earlier message arrives later due to some transport issue, the message timestamps could still be used to reorder the messages in the database, e.g. switch the index values (during message saving).

[r10s]

That's why I think using such an index for the display order could solve the issue.

it may solve one issue but raise different ones, maybe worse ones - as mentioned above.

[testbird]

Which problem in particular do you see unsolved?

[r10s]

ordering of messages. forwarded ones, but also if it comes to restore old messages from imap. how should this be solved when _only_ sorting by incoming date? (in fact, using the uid is similar to that) (apart from that the uid would need some tweaking as unused ids may be recycled, eg.)

[r10s]

@testbird and how does sorting by uid help on the initial issue reported by @nutpantz?

[testbird]

I've never said anything about using UIDs (Unique Identifiers) or about only sorting by incoming date.

the messages are stored to the database in the order in which they arrive, they get subsequent index values.

Sqlite seems to automatically add a rowid index value which is automatically incremented by one with every record saved. It also seems to support custom index columns if needed.
The index values then always represent the order of arrival at the device, regardless of any timestamps in the messages. (If the autoincrementing index values of the database are used for the message displaying order in a chat, nutzpantz would always see the messages in the correct order of local arrival, regardless of the timestamps in the messages.)

The message timestamps are still useful, to detect if the order of forwarded messages has changed during transport. Thus if an earlier forwarded message actually arrives later, deltachat may read, delete, and save the messages to the database again in the corrected order, or directly adjust the index values in this case (subsequent forwarded messages).

---

BTW: I think the "time smearing" manipulation of message header times from the future is not a good idea, and not necessary when using the order of arrival index for displaying. The reason is that all messages from eastern time zones appear to come from the future, if local times are used. Thus timestamp adjustments can lead to errors and misinterpretations.

Timestamps from future are also no problem when using the index value for the displaying order of messages. Subsequent messages will get an incremented index, and show up after prior messages with timestamps still seaming to be in the future.

[testbird]

https://stackoverflow.com/questions/7905859/is-there-an-auto-increment-in-sqlite
http://www.sqlitetutorial.net/sqlite-autoincrement/

[r10s]

i think, things get mixed up a bit here.

the _uid_ mentioned above is more or less the same as the _subsequent index value_ or the _auto-increment_ mentioned by you.

however, again, how should solve all this the initial issue? and how could removing "time smearing" (which is more for outgoing messages, btw) help?

[testbird]

The incoming timestamp depends on the local system's time setting (which may be wrong, corrected, or regulary change when switching time zones). That's the cause for this issues' problems.

Simply saving received messages to the database in the order in which they arrive does not depend on any time setting. Displaying the messages in the order in which they were saved to the database will always result in the same order, the order of arrival as wanted for diplaying the messages.

As said, the order of incoming forwarded messages may still be checked by their sending timestamps, and if there were changes from the sending order they can be (re)written to/in the database so they show in the same order as sent by the forwarding party (which would be the order in which the forwarding party originally received and answered the messages).

Limiting the sending time to the local reception time ("time smearing") would not be required anymore. (It won't lead to messages that are received later being shown before the messages from the "future".) It would thus be possible to actually provide the correct information about the sending time, as it was effective/given at the sender. This is especially useful and important across time zones. If the timestamp contains information about the time zone, the sending times from other time zones should of course be displayed converted to the correct local reception time zone.

[r10s]

let us put time smearing away, this is not really related.

so, let me collect some facts wrt the original issue:

• Delta Chat _uses_ the message-time _unless_ it is newer than device-time
• this is not a delta-thingie, this is from the Autocrypt-spec, we _cannot_ change this without breaking the spec
• the truncating to the current time has good reasons - it protects against malformed incoming mails
• _fresh_ messages are, as you suggest, already always sorted always to the end of the list (see https://github.com/deltachat/deltachat-core/blob/master/src/dc_receive_imf.c#L268) - so for _fresh_ messages, the initial issue should be fixed wrt sorting at least
• downsides are - if the Device time is bad, things stop working

maybe we could get around this by using a fixed increment instead the sort_timestamp, however, this raises problems when mails come out-of-order and also we could never download _old_ mails then. or react to IMAP giving us, for any reasons, old mails.

the out-of-order could probably be ignored, what's remaining are download of older message. if we decide that we really do not need this, we could just remove the is_fresh_msg check from calc_timestamps() and we would have the same effect as you described above.

so, not relying on the device time may solve some problems, but will raise a bunch of others and/or make things more complicated (which is also an issue then).

i do not think it makes much sense to adjust many things here - esp. as the problems only arise if the device time is not set properly - so there is already a very easy fix ;)

[testbird]

Thanks for the summary, we agree in the analysis. (I had mixed up effective and smearing time before,)

problems when mails come out-of-order and also we could never download old mails

Yes, because I think being able to insert old emails is important, that's where I was thinking of re-inserting (copying) the chat to a new temporal chat_id in the corrected order and then replace the old chat_id with it, if older messages are to be inserted in between existing ones.

Hm, but if that is too complicated, maybe this issue can also be solved simply by adding a lower bound to the effective date, i.e. to set timestamps from the future back to the currents system time, but not before the last message in the thread?

[r10s]

but not before the last message in the thread?

yes, this makes some sense.
or even better an algo that validates the current time in a way.
however, both approaches can become hard for traveling around timezones and if the current time is correct but the reference time was not. it's complicated, though.

[testbird]

From what I read in the autocrypt spec now I think all timestamps should be UTC, thus because times should only be displayed in the current local time zone there should be no time zone traveling problems.

The only problem I currently see with this fix for system time resets would be if the system time would initially be set to a future time and a first (no prior messages) and also wrongly dated message arrives. This would leave behind a last message with a future date.

But as devices usually initialize with a past date that's a quite unlikely case.

[testbird]

BTW I've also seen this "phone reset moves message into the past" thing.

[testbird]

Another confirmation of this issue: https://support.delta.chat/t/desktop-app-and-previous-chats-not-showing-up/174/9

[r10s]

i think the fix is to set the correct device time. at least for now i do not think someone is working on other solutions.

however, as the discussion is already shifted to the forum, i'll close this issue. the result of the forum discussion might be a concrete actionable and reasonable item then.

[testbird]

Device clocks are and will always get adjusted, corrected and resetted. The "sorting before older messages" problem occurs [even with automatic clock adjustment] in _the meantimes_, because the sorting timestamp for incoming messages is only cut-off (limits future dates) by the current system time, and not also by the sorting time of the last already received message (limits past dates).

I think this is about as actionable as a user reported, and repetedly confirmed issue can get. My clumsy pull request draft would of course be in need of some developer insight to only do the cut-off for fresh messages in the code without affecting the other functions. https://github.com/deltachat/deltachat-core/pull/435#issuecomment-447657221

[r10s]

for sure - therefore the discussion can continue in the forum.

it's just nothing the android-ui can do in this item imu.

and also for the core it's not _that_ clear, as the comments in this thread and in the pr - that raised other problems - pointed out.

[testbird]

message time is in the future, Delta chat uses [a transient past] device time not [bound by the last] message time to [correctly] sort messages

[adbenitez]

if there are messages from date X, and for some reason(ex. factory reset) I send a new message with a wrong date X-1, following what r10s said that it would be better to fix the date as soon as possible, maybe some warning can be shown to help users, telling them something like "hey! seems like your phone date is wrong" would be helpful.
related #749

[testbird]

It's a good idea to provide a hint to the user that the system time has been set back, it may be wrong but might also be properly set back to another timezone.

When this happens, however, it is already too late to correct the system time before a message arrives.
And it would only be possible to correct the system time, if it wasn't actually correctly set back to another timezone.

Thus, the test for messages with timestamps before the last sort_timestamp should probably still cause a lower bound for the new message's sort_timestamp, plus the hint to the user in case of a system reset.

[nutpantz]

My berry has a rtc so that may not be applicable.

On April 3, 2019 9:52:07 AM PDT, testbird [e-mail] wrote:

It's a good idea to provide a hint to the user that the system time has
been set back, it may be wrong but might also be properly set back
to another timezone.

When this happens, however, it is already too late to correct the
system time before a message arrives.
And it would only be possible to correct the time, if it wasn't
actually correctly set back to another timezone.

Thus, the test for messages with timestamps before the last
sort_timestamp should probably still cause in a lower bound for the new
message's sort_timestamp, plus the hint to the user in case of a
system reset.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/deltachat/deltachat-android/issues/398#issuecomment-479571270

[testbird]

If you travel back in timezones, I think even with a radio controlled rtc new messages would need to get sorted behind the last message's sort_timestamp ("from the future").