K-9: Notifications are generated for old mails from time to time

[IzzySoft]

Expected behavior

There should be no new notifications for old mails

Actual behavior

From time to time I receive notifications for old mails. The only thing they have in common is they are (still) marked unread. The notification shows them as "just received". For example, tonight I had a notification for a new mail received at 1:39 am. I cross-checked with the mail, and it was 3 days old, from 8:34 pm.

Steps to reproduce

I cannot reproduce it. It doesn't happen regularly – but often enough to be annoying. I'm not sure if it always happens for the same account, but AFAIR that's what it is. I have 3 accounts configured with K9, all IMAP. Same (physical) server, but different accounts on different domains.

Environment

K-9 Mail version: 5.202 (F-Droid)

Android version: 5.1.1 (CyanogenMod)

Account type (IMAP, POP3, WebDAV/Exchange): IMAP

[alex-pl]

I have the same issue, but in my case it _is_ reproducible. I have three mail accounts configured (completely different servers and configurations, but all IMAP) and it happens on all of them.

Steps to reproduce

It happens when I delete mails (via Android notification, in k-9 mail or on another device) that are part of the list of the last _n_ (let's say 25) mails of the INBOX folder, that are downloaded into k-9 mail. Because of the deletion, the list shrinks and has less then 25 entries. After a while (several minutes, that's probably a setting) the phone updates the list and loads old messages until the list again has the length of 25 mails. If one of those old messages is marked as unread, a new notification pops up. (And I'm wondering whether there's something new on my phone or it's just an old unread mail ;).)

Environment

K-9 Mail version: 5.203
Android version: 6.0
Account type: IMAP

Would be great to have that fixed in any way :).

[IzzySoft]

@alecpl that sounds like a plausible trigger indeed. I'll keep an eye on whether something like that is behind here as well, though I doubt it: I mostly only dismiss the notifications. Cases where I delete messages via K9 are rare (except when on vacation).

A raw idea would be not to "announce" unread messages if they are "preceded" by other messages already marked read. That would catch your case and mine as well.

[IzzySoft]

PS: I can confirm now what @alex-pl wrote – with the exception that the other message has not necessarily to be deleted via k9. I just got the very same old mail announced today 4 times, and got curious, so I checked:

• ordered by timestamp newest first, that mail sits on position 26
• k9 had just sync'd, so all fine
• deleted one of the first 25 on my desktop, so it got purged from IMAP
• k9 refresh: Pling! Old mail notification
• a new mail comes in ("pling"), and gets deleted via desktop → "Pling" the old one again

Suggestion: k9 should remember the timestamp of the last mail announced and only notify for mails newer than that. Maybe give it a configurable frame – but everything older than 24h should definitely be kept out.

[alex-pl]

with the exception that the other message has not necessarily to be deleted via k9

Sorry, maybe I didn't express myself clearly enough. Of course it's the same in my case. As I said it also happens when I delete mails "on another device", which may be running Thunderbird or anything else.

Suggestion: k9 should remember the timestamp of the last mail announced and only notify for mails newer than that. Maybe give it a configurable frame – but everything older than 24h should definitely be kept out.

Generally a good idea, but I think it's not applicable, because some mails might be delivered delayed or with a wrong date. In those cases you would never get notified.

k-9 mail has a setting _Account settings_ -> _Fetch messages_ -> _Synchronize messages_ (translated from German: _Kontoeinstellungen_ -> _Nachrichten abrufen_ -> _Nachrichten synchronisieren_) and I'm wondering if this is the functionality that's normally responsible for not notifying about old mails. I've selected "1 year", but probably the synchronization doesn't work properly.

Does someone know what's the purpose of this setting?

[philipwhiuk]

Here's a brief overview of how I believe K-9 works in this area.

We only retrieve and retain a certain number of emails. This number is based on a combination of the date the last email was received by the email server and the maximum allowed.

However each email also has an ID. These IDs we keep regardless.

We notify based on whether we have seen the emails ID.

For the most part this works fine.

However, if you have 50 emails in your inbox and are viewing 25, if you delete or move some emails we will notify for the older emails we are now allowing ourself to pull.

Probably what we should do is something like get a list of all email IDs when we first sync a folder and then after that we'll have all the older IDs in our database.

[IzzySoft]

Generally a good idea, but I think it's not applicable, because some mails might be delivered delayed or with a wrong date. In those cases you would never get notified.

That's why I suggested to "give it a configurable frame". Mails with a wrong date more than 24h off are rare (all of those I saw until now where spam anyway), so these could either be ignored anyway – or the "do not announce those" feature be made optional, even with "default=off", though I feel "default=24h" or, if skeptical, "default=48h" should be fine. This gives us the chance to fine-tune based on our experience, while no one is being forced into it.

Probably what we should do is something like get a list of all email IDs when we first sync a folder and then after that we'll have all the older IDs in our database.

Than this sync might need to be repeated on a regular base, to purge out deleted ones. And while this might be the technically most accurate approach, it has it's drawbacks: some folks have folders (including INBOX) with thousands of mails therein. Not sure at which point this might get unhandy/imperformant, and cause unwanted "data plan (ab)use". My suggestion wouldn't cause any additional data being transferred, nor any additional space needed – while still covering the issue almost fully.

[danielegobbetti]

So if I understand it correctly there is a list of X synchronized emails per folder, which are the most recent.
A second data structure might be added per folder, that stores the unique ids of each unread email, regardless if it is in the first list or not. I believe this list could contain only the email ID and if the user was notified already or not. I also believe this list doesn't need to be bigger than X*2.
The notification logic would then work on this second list only:

• if a new email is received, its ID is fetched and goes in the notification list, since the user was never notified a notification is generated;
• if an old email falls back into the "synchronized emails window", the notification is not generated again since it was already done

I believe this approach is a cleaner solution than relying on the emails' timestamps or other metadata; Further, it doesn't require to memorize all the emails' IDs per folder. But of course, all of this is IMHO and I might have overseen some pretty obvious problem :-)

[IzzySoft]

@danielegobbetti that's basically the approach suggested by a k9 member. I agree it's probably the cleanest approach, but it has some drawbacks: some folks have thousands of mails in their INBOX (size-issue), and the entire list might be needed to be sync'd in intervals (to purge out deleted mails; data-usage issue). Mails with a far-off timestamp are rare, and in most cases spam anyway. I suggested a configurable time-span, so e.g. mails with a "received" time stamp (which IMHO is set by the server anyway) older than X are ignored.

Anyhow, if this really is the "root of the evil" (as it very much looks like), it's something that should be solved on the k9 end, not by GB :stuck_out_tongue_winking_eye:

[alex-pl]

@philipwhiuk Thanks, now I understand the "Sync messages from" setting ;).

Currently I'd prefer the way @danielegobbetti and @philipwhiuk are describing. As a default I would probably sync all mails. Maybe with a user preference for people who have a _huge_ amount of mails. This looks like a very clean solution that, per default, works regardless of how many mails you are deleting and it doesn't need much space either. Let's say the message ID has a length of 50 Bytes. If you have 10.000 mails that's 0,5 MB for a very useful functionality. Maybe plus a little overhead, but even 1 MB for an amount of 10.000 mails should be okay.

For people that want to adjust the time window of how many mails are synchronized, I think a setting like the one for "Sync messages from" would be useful. Previously I thought this setting is exactly for what we are talking about now ;). The new settings could be something like "_Download messages from_" and "_Save notification state from_".

A thing one must think about is whether moving an unread message from one folder into another should trigger a new notification or not. Depending on that decision, there should be folder specific or mailbox specific ID "tables".

One problematic thing I'm thinking about now, are missing message ids. Normally about every mail has one, but it's not forced by the RFC. It should be present, but isn't forced. In those cases one could use a combined strategy, for example a date as a replacement or addition to the message ID or those messages simply are ignored.

[IzzySoft]

Thanks @alex-pl – I had no ideas about storage space needed. 500k sounds reasonable for 10k mail, so full ack. Same goes for the "window".

As for messages moved: I see no reason to announce them twice. Two cases to distinguish here:

• user moved manually: No need for a second announcement
• Sieve rule or the like (i.e. server moved automatically): this usually was not announced in the first place, so it would only get announced once anyway.

Missing IDs: yupp, sounds good.

OK then, let's look forward to how @philipwhiuk and the k9 team decide. Any solution will be better than none :stuck_out_tongue_winking_eye:

[IzzySoft]

Almost a year later: May I gently ask if this is worked on? Though it's certainly nothing "major", it's quite annoying. Even a "don't create notification if mail is older than X days/hours" would help a lot.

[reitzig]

I never noticed this before, but since the latest release such "false" notifications have been coming in.

The scenario is that I get notification for new mails which are later (without me having read them) a desktop client filtering them away into different folders. Then, I get notifications about new mails again from K9, for those (unread) mails "unearthed" by removing the latest ones. That happens even though the doubly-notified mails are among the 10 shown during all of it (as far as I can tell).

[tomome]

I need to click "mark all as read" every now or then and it solves it atleast for me..

[IzzySoft]

@tomome That's no solution to me as often I've left mails unread incidentally, as I need to process them later. My situation is comparable to what @reitzig described.

I really hope this issue will be addressed soon. It's really annoying.

[IzzySoft]

It's almost 2 years now, and this is still driving me crazy. Any chance to see the issue addressed? Any ETA?

[danielegobbetti]

@IzzySoft it might be that current master of Gadgetbridge mitigates the issue, since we are working around other issues and are now using the when field of a notification to exclude old messages.
So if k9 is already populating the when field with the timestamp of when the email was originally received, next release of Gadgetbridge could work without further changes.

[IzzySoft]

Thanks, @danielegobbetti – that's good news! So less vibrations on my arm. But still noises from my pocket (and me wondering as GB didn't vibrate my Pebble). But chances are I won't hear the noise due to "background noise" :wink:

[IzzySoft]

As the issue still persists: any chance to get it fixed? @cketti? It just had its second birthday a month ago. I don't want it going to school :wink:

[bigretromike]

Hello,
I have this same issue, from time to time I get notification about old email that is UNREAD. Also k9 try to auto-fix it from time to time by removing the notification and stopping the notification sound. So in a brief moment I see notification, and hear the sound for it but it get cut in middle and notification disappear. It would be awesome if this would be fix or if k9 try to hide/clean it then make k9 delay the notification so there is no phantom notification. Its super annoying when I get notification, unlock my phone and see nothing there.

[reitzig]

That, and I've also seen notifications for _new_ messages removed this way. Maybe because an old message was part of the notification? Not sure, hard to check.

[oliv3]

I also have this issue on 5.600 (F-Droid).
To me it seems like a K9 issue, but just in case, the server is
ii dovecot-imapd 1:2.2.27-3+deb9u3 amd64 secure POP3/IMAP server - IMAP daemon
All other clients (thunderbird, etc) are working fine.

[IzzySoft]

@oliv3 it's not Dovecot (though I use that as well). Reason behind the issue is that k9 only holds a certain amount of mails per mailbox (25 by default). As soon as you delete something "on top", older messages get pulled on the next sync. If one of those is still marked "unread", you get a new notification (see the first 4..5 replies for details).

Solution would be to either remember all msg ids already notified – or have a "notification interval limit", best user-configurable (e.g. do not notify for messages older than 48h). Again, see replies 3ff for details.

It's now more than 2 years this was filed, and confirmed by several users. Hope it gets fixed before the 3rd year is over (no demands of course, but it'd be nice). Hits me about once a week.

[oliv3]

@IzzySoft Excuse me in advance if it's a silly question, but what about the IMAP \Recent flag ?

[bigretromike]

@oliv3 it's not Dovecot (though I use that as well). Reason behind the issue is that k9 only holds a certain amount of mails per mailbox (25 by default). As soon as you delete something "on top", older messages get pulled on the next sync. If one of those is still marked "unread", you get a new notification (see the first 4..5 replies for details).

I'm not sure if that's is the real (full) problem. Because I have mine setup to have 200 messages, when I delete lets say 20, it does notify me about old message, but then in middle of day it notify me even when the message is still on the list (while me getting 1 new message). Also when I don't get message I still get popup.
Like I wrote before k9 (or android) try to clean/hide them, so I end up unlocking phone and not seeing the notification at all - but sometimes when I have unlocked phone looking at it (ex. watching yt) I see the notification about old email and then it vanishes.

Really don't know, its really nice email client but those "phantoms notification" are bad.

[IzzySoft]

@oliv3 I don't know which flag k9 evaluates. And I didn't dig when the \Recent flag would be removed by Dovecot (or other IMAP servers at that). By time? By setting a mail to read (and to unread afterwards)?

OK, I've checked. No, that wouldn't fit. The first session seeing it would clear it. That would be my client at home, which is permanently on and would long have seen such a message that was received a day (or more) ago (as it uses IMAP IDLE, which I don't use on mobile where k9 checks hourly). And for a message received half an hour ago, I'd never get a notification from k9 as the desktop client would already have cleared the flag.

@bigretromike there might be multiple issues then. I can just tell from my experience – and that's what I described.

[mildmojo]

I've been living with this issue for two years; delete some messages, new unreads get sucked in from the bottom, my wrist buzzes with two-month-old unreads (but doesn't show the dates), I get out my phone and open K-9, but there's nothing new. My inbox is peppered with unreads across its entire history, so I get bogus notifications from K-9 several times daily, almost every time I delete emails.

Keeping all email IDs from all of history sounds great. If that's too many (could be tens of thousands), maybe keep twice the size of the inbox window? If K-9's configured to show 25 emails, keep 50 IDs. Showing 50 emails, keep 100 IDs.

Fixing this issue would be a huuuuge quality of life improvement for me and others. ❤️

[Kaligula0]

Mine K-9 behaves similarly to what @bigretromike said - "Also when I don't get message I still get popup". That's very annoying :(

[IzzySoft]

The easiest work-around would be to remember the time of the last N successfully refreshes – and don't notify for any mail that has a time stamp ("received") older than that. Make N configurable, set the default to something reasonable (3 or 5 would fit fine), et voila: no mails from last week should make any noise unless you just returned from a (net-free) vacation. No need to keep a huge database with "all IDs ever seen". Just N timestamps.

@cketti begging you, have mercy on us!

[bigretromike]

The easiest work-around would be to remember the time of the last N successfully refreshes – and don't notify for any mail that has a time stamp ("received") older than that. Make N configurable, set the default to something reasonable (3 or 5 would fit fine), et voila: no mails from last week should make any noise unless you just returned from a (net-free) vacation. No need to keep a huge database with "all IDs ever seen". Just N timestamps.

@cketti begging you, have mercy on us!

That or just a N*3 (or 2) big list of already received email IDs where N is the number that you want to check messages (already there). In normal usage case you shouldn't get more than half of the emails you have - in rare cases (spam, newsletters yes).

[IzzySoft]

@bigretromike anything would be an improvement. But as on one of my mailboxes gets a lot of traffic some days (Github/GitLab notifications mainly; I'm maintainer in some bigger projects like F-Droid), I'd prefer the time-based approach. Also consider some people keep all (or most) their mails in the Inbox: in one of mine, your formula would mean several thousand entries :scream: In all those cases, my formula would work fine :smile:

But as I wrote: anything would be an improvement…

[bigretromike]

@bigretromike anything would be an improvement. But as on one of my mailboxes gets a lot of traffic some days (Github/GitLab notifications mainly; I'm maintainer in some bigger projects like F-Droid), I'd prefer the time-based approach. Also consider some people keep all (or most) their mails in the Inbox: in one of mine, your formula would mean several thousand entries 😱 In all those cases, my formula would work fine 😄

But as I wrote: anything would be an improvement…

But I'm reffering to `Local folder size' as N, so when you have it set to 25 the value to store (array) would be 253=75. if You fetch 100 then cache value would be 300. Even if array value would be 2N that would make those phantoms popups pop only once in long time. With this approach client don't have to configure any additional value. But some spoofed emails or badly configured servers send emails with wrong data so your timestamp should use the server received time.

But you are true that any improvement would be great.

[Kaligula0]

What about adding a flag "ALREADY_BEEN_NOTIFIED"?
Look at the procedure…

notification creation triggered (e.g. after refresh or e-mail deletion) -> check if there are any e-mails with "ALREADY_BEEN_NOTIFIED"=FALSE -> if any -> push them to notification -> if creating notification succeeded -> mark them "ALREADY_BEEN_NOTIFIED"=TRUE

[IzzySoft]

@Vinne2 where do you want to add that flag? On the client? Won't work: once the message leaves the window, it's gone. On the server? Won't work: consider one uses multiple clients, so only one would ever be notified.

But it's moot discussing this here in depth as long as @cketti isn't picking it up :cry:

May I suggest all in favor of this give the original question a :+1: so the number accumulates, indicating "priority"?

[bigretromike]

It's been 2 and a half year. This issue is in top10 of most commented on. If that's not a 'priority' itself then I don't know what is it.
But you have my 👍

[IzzySoft]

Thanks, @bigretromike – and I fully agree. I just thought of how to "increase visibility", wondering why it still did not get picked up after this long time and those much comments – even with several approaches mentioned (one of those shouldn't be too hard to implement I hope). There are other issues with more comments (this one is just at position 9 – and going by :+1: not even on the first page anymore), and with more than 500 open issues quite a lot of things to consider.

[bigretromike]

You know, in the end its an opensource project. Any one can contribute.
Fingers cross that someone will pick it up eventually.

[IzzySoft]

If I were an Android dev, believe me, I had already submitted a PR… And yes, right you are. :crossed_fingers:

[justinacolmena]

On Thu, Jun 20, 2019, 08:26 Tim [e-mail] wrote:

Keeping all email IDs from all of history sounds great.

That's gmail. It's not the be-all and end-all of hreat policy.

[kwisatz]

I don't know about you, but since last week, this problem has become a curse for me. I receive them all the time, I don't even have to delete older mail.

[IzzySoft]

@kwisatz that, luckily, not yet. Annoying enough without that. And soon, this issue celebrates its 3rd birthday :cry: