K-9: K9 not showing all emails in IMAP folder

[EliseSpPrj]

Expected behaviour

It should show all messages in the folder.

Actual behaviour

I have 6 emails in my inbox but K9 only shows 4 of them.
The two missing mails are mixed in with the others date wise, as in they're not old or last in the list but in the middle. I've tried other email clients and they show all emails (see attached picture).
I've tried telnetting into the IMAP server and the server says EXISTS 6 and UNSEEN 2. The two unseen are the two not shown in K9, but they're perfectly readable by the other clients.

I tried the "Clear messages" which cleared the 4 messages, after syncing the 4 came back, still missing the other two.

Steps to reproduce

1. Have an email account with emails in it, including emails that K9 doesn't like to display.
2. Open account in K9, look at emails, notice missing emails.
3. Refresh account, delete account, redo account, still see missing emails.

No matter what I do K9 refuses to display those 2 emails, why those 2? I have no idea, they're normal emails; not too big, nothing weird with them, valid date, not old, not at the end of the list, etc.

Environment

K-9 Mail version:
5.010

Android version:
5.1.1 (14.6.A.1.236)

Account type (IMAP, POP3, WebDAV/Exchange):
IMAP (with SSL/TLS)

[cketti]

Can you please use "clear messages" again and then record a debug log while refreshing the folder. https://github.com/k9mail/k-9/wiki/LoggingErrors

[EliseSpPrj]

Done, attached.
Still showing only 4, I think I saw in the log that K9 saw 6 total on the server, dunno why it only shows the 4.

k9-log.txt

[Valodim]

log says that there are 6 messages, but then says there's only 4 unsynced ones which will be fetched. so the error doesn't happen in the process of downloading, parsing or saving the messages, the messages are never considered for download.

[OpenAai]

I've got about the same problem, as described here. After moving old messages into my inbox, K9 only displays these older messages, and newly received messages. I'm looking forward to a solution.

[gpe5280]

I am having this same issue on two different Android devices (phone and tablet). The issue appeared exactly at the same moment I updated the SSL certs on my email server (they were due to expire in a few days.) K9 recognized the new certs and prompted whether to accept or deny them. I've seen this prompt before each time the certs have been updated (every 90 days with Let's Encrypt). After accepting the certs the issue appeared. Not all messages are displaying in K9 but are in other email apps.

[eucalypto]

I have the same problem as @OpenAai. I moved old messages from an old imap account to a new one with Thunderbird and K9 shows those copied emails and emails received after copying. Even deleting this account and setting it up again in K9 did not help.

Then I had a look at the email list in Thunderbird and found the sorting criterion Order Received, (hidden by default). And this seems to be the root of the problem. This Order Received looks like an integer counting the emails that "arrived" on this account.

K9 seems to use this number to sync the most recent mails according to this Order Received; because ordering the emails according to this number in Thunderbird gives _exactly_ the list of emails that K9 displays!

And since K9 only syncs a finite amount of emails, it does not "know" about the other new emails because they have too small Order Received numbers and K9 does not load them.

I don't have a solution, but at least this explains K9's behavior.

[olovm]

I have recently moved my mail to a Kolab server and ran into a similar problem, only new mail and some older ones showed up in my inbox. I get this problem if i change the account settings under Fetching mail. If I change "Sync messages up to" from "any time limit" to some shorter interval (try the last week).

Two settings determine what mail is shown in my inbox, "Sync messages from" and "Local folder size". I think that K9 fetches mail from the server based on a combination of mail modified date/time, and "sync messages from" date.

I moved my messages in chunks (50 or so) first moving the newest mail, making them get an older modified datetime than the older mail moved a few minutes later. So when K9 fetches mail and my settings are sync mail from last week, and a small "Local folder size" (25), only the mails that have arrived after I moved mail into my inbox show up. If I change "Local folder size" to 1000, all mails for the last week are shown (I have less than 1000 mail in my inbox on the server).

So this is what I think happens, when the mail are fetched, the server has a list of mail based on modified date(or similar). From this list the 25 from the top (Local folder size setting) is taken, THEN is the "sync messages from" setting applied. The result of this is that only mails that has one of the last 25 modified AND has a (send/recieved) date that is in the last week is returned from the server.

( This problem might also be related to #1881 )

[rnc]

Interestingly I think I've just hit this. After a mail transfer (involving gmail) I was wondering why on some labels/folders only certain mails were showing up. I had some recent and some much older but was missing the most recent and the ordering seemed strange. Using this hint and changing the local folder size to 1000 fixed the issue - all the recent mail are there which matches the web interface and what thunderbird sees. Therefore I guess mail was being migrated in similar 'blocks' affecting the modified time

[tvld]

I have same issue: I have only about 20 messages in my Inbox (on the server) and about 12 in the Inbox in K9. (Sync is to any time and limit to 500 messages.)

There is another issue at play...

[Cactus-World]

Okay,
recently I ran into that problem as well. Even new messages arriving are not synched by K-9 with my IMAP server. I can see them in Thundebird which downloads them correctly. Maybe it's related to synching with K-9, synching with Thunderbird then, Thunderbird sorts out spam and moves some messages to other folders on IMAP and afterwards K-9 has a higher number for already synched messages in inbox stored somewhere than exist on server thus leaving out everything below that number even if new mails arrives?
Best,
Cactus

[laclaro]

I ran into this issue after moving a bunch of emails to a new folder. Now k9 leaves out messages. And syncs very recent ones then leaves out very many and syncs again email that is a couple of months old.
I think there could be this mentioned modification date issue.

I'm on android and use a recent version of k9

[a-keil]

@eucalypto wrote:

I have the same problem as @OpenAai. I moved old messages from an old imap account to a new one with Thunderbird and K9 shows those copied emails and emails received after copying. Even deleting this account and setting it up again in K9 did not help.

Then I had a look at the email list in Thunderbird and found the sorting criterion Order Received, (hidden by default). And this seems to be the root of the problem. This Order Received looks like an integer counting the emails that "arrived" on this account.

K9 seems to use this number to sync the most recent mails according to this Order Received; because ordering the emails according to this number in Thunderbird gives _exactly_ the list of emails that K9 displays!

And since K9 only syncs a finite amount of emails, it does not "know" about the other new emails because they have too small Order Received numbers and K9 does not load them.

I don't have a solution, but at least this explains K9's behavior.

Same here. I think @eucalypto 's analysis is on the spot. This problem just recently exacerbated for me because Gmail decided to re-sort my sent emails. (I used to organize my sent emails so that they would not all gather in the "Sent" folder but rather stick with the received emails of the same conversations in the respective folders (i.e. the Inbox or subfolders thereof). Since Gmail now copied all my old sent emails into the Sent folder, these apparently received new "Order Received" integers. All this results in my "Sent" folder in K-9 mail only showing emails that I've sent ages ago.

I am guessing that the K-9 mail developers were seeking a way to limit request to the mail server by using the "Order Received" attribute. However, this seems to be a non-optimal choice in quite a few scenarios. Since there seems to be no workaround that users could apply here, I am wondering if the developers could re-think the way they are fetching mail. This would be my suggestion:

• Use the two parameters in the Account settings -> Fetching mail as follows: The "Local folder size" parameter should not be related to the "Order received" attribute but should only determine the number of emails that are fetched at a time. (The latter meaning is already connected to this parameter. But the relation to "Order received" is a hidden functionality.)
• Use the "Sync messages from" parameter as-is (determining the time frame from which to fetch emails) or remove this parameter altogether (since the "Local folder size" already limits the amount of emails requested at any time).

I would very much appreciate a comment from the developers and I am eager to help resolving this issue.

Thanks for your great work!

[natrius]

Encountered the exact same problem now.
Uploaded old e-mails from locally to my online account via thunderbird and now k-9 does not show up e-mails anymore.
Just new received messages does show up - i have to check more.

A solution would be nice.

[natrius]

* Workaround *
Create a folder (example: Archive) and move all the old e-mails there. Suddenly more e-mails show up in the
inbox.

[Hetti]

I'm also encountering this bug frequently. Would be great to see a fix for this problem

[troglodyt]

+1 here; while I generally enjoy using K9 and also like its overall maturity, this issue has already caused me some trouble.

[laclaro]

For my taste this issue is the most important one on K9.
Are you at it and it is difficult to solve
or
do you not think it is important?

Would be nice to hear from you.

Best,

Henning

[va1entin]

Not sure if this is the same bug at work here but in my K9 months of mails are missing in several folders across several mail servers. I can see them in various Thunderbird installations and the respective web mailers. Since this also affects the search feature (at least for me), it renders K9 unusable for me at this point, which is why I'm switching to another client until this is resolved. I couldn't see a pattern by which mails go missing so far unfortunately, so I can't help with that.

[idanoo]

Also having this issue. Happy to provide any debug logs/etc to help. I have migrated emails from a different IMAP account in the past.

Update: Okay so clearing app data + cache and setting it up agains pulls all emails, even the ones that didn't show before.

I had changed from self-signed SSL cert to LetsEncrypt, as well as migrated from an old IMAP inbox around the time I noticed there was a few hidden emails - Unsure how related either one of those is.

[Cupora]

Similar issue for me with googlemail in the Inbox. Thunderbird over IMAP works great. K9 skips some messages.

[Cactus-World]

I'd like to know -as this is a central requirement for a mail client, i.e. to display all messages correctly- if somebody is working on this -now two years old- issue?

[natrius]

I just switched to SimpleEmail, looks nice with some minor flaws but shows all e-mails.

[nuess0r]

My girlfriend is also effected by this bug. Interestingly it does not affect myself.

Working setup:

• K-9 Mail from f-droid, 5.600
• courier IMAP with Let's encrypt
• Claws as client on the desktop
• Android 5.1.1, Lineage 12.1 based custom ROM

Not working setup:

• K-9 Mail from f-droid, 5.600
• courier IMAP with Let's encrypt (the same as above)
• Roundcube as webmail client
• Android 6.0.1, CAT S60 stock ROM

[danielgrant]

Just ran into this one myself. I've migrated around 11,000 emails from an old provider to a new provider using Mac's mail app (by selecting all emails in a folder, right clicking and choosing "Copy to...").

In Mac's mail app, the emails post-migration are correct and they're correct in the web UI of my new provider. However, K9 is exhibiting the same behaviour as above whereby emails that should be showing in my recent list aren't. The list is full of missing emails.

[mdvthu]

I'm having this problem.

K-9 Mail 5.705
Zoho Mail IMAP
Android 9 (OxygenOS 9.0.10)
OnePlus A5000 (5)

It persists despite clearing the application cache.

Moving the mails to temporary folder then moving them back to the Inbox seems to partially fix the problem but the message bodies are still empty.

[jjakob]

Same issue here. I've created some new folders and moved messages from the Inbox to them. After refreshing the folder list, only old messages were synced - none from this year!
Clicking the "Show up to 25 more" button fetches 25 messages, but of a random date so they get inserted at random places in the list not at the bottom as expected if they were correctly sorted and fetched.
I was able to get all messages displayed correctly by clicking the "display up to 25 more" button until all messages in the folder were downloaded, then I could see all of the new ones at the top as they should be.
So it seems like K9 is sending a wrong IMAP request to the server, instead of fetching the recent messages by date it gets random ones.

[va1entin]

A hint for everyone looking for an open source alternative: Due to this issue I have switched to FairMail and am very satisfied with it so far.

[jjakob]

@va1entin thanks for the suggestion, I've switched to FairEmail. I would still like to switch back to k-9 when (if) this is solved, so I have it installed but with syncing disabled, so I can check a new version's behaviour. So far it still isn't fixed in 5.709.

Correction: after refreshing all the folders (by clicking "load up to 25 more a few times) it seems that all mails are displayed now - I'll try using k-9 for a bit now to see if it's any better.

[jjakob]

I used 5.709 since the last comment and I haven't encountered the issue yet. It may still be present, but it doesn't seem to be manifesting now.

[coderonline]

I can confirm, that this can be fixed by downloading more messages. For that I have changed the local folder size to all messages instead of downloading the box in 25 message steps and it works as well. The mails are then received in a strange order and I may have this problem, because I recently restored a backup of my mails. I hope that this can help with the debugging.

@va1entin FairMails looks really good as well. Thanks for that suggestion.

[Volker-Weissmann]

I had the same problem, but "Refresh fodler list" solved it for me.

[cketti]

This should be fixed in recent beta versions (5.7xx). Please create a new issue if it's still happening there.