Deltachat-android: IMAP folder ignore list: additional defaults, handling & custom string

[testbird]

DC had been installed and updated for a while.

Tapping on "+" now shows a long list of strangers (and the few really known contacts). Anybody seeing this as well?

I guess spam "From" addresses could falsely have been picked up as "known contacts". But spam is supposed to get filtered into a different folder.

---

Summary from comments:

• The MR_ORIGIN_ states do not allow tracing back and reverting inconsistencies. (now https://github.com/deltachat/deltachat-core/issues/124)

• Additionally needed folder names for the default ignore list: Spambox, Unwanted. Note that the english names are also used untranslated at providers of other languages!

• ...thus, add an additional, comma separated translatable and localizable string with typical names for the locales. This should then for exampe also add Unerwünscht, while the english "Unwanted" etc. from the default english list still remain in effect.

• Yet another user customizable ignore lists is necessary, e.g. to allow ignoring folders with support threads or simmilar, and avoiding unconstructive flooding of the contact list.

• The custom-folders-to-ignore string may contain "*", to disable all checking of custom (not-hardcoded like "Sent" etc.) subfolders.

• also ignore the ignored folder names if they exist below the INBOX folder (as some providers only support subfolders there, as in #108)

---

Additions:

Changing the custom-folders-to-ignore string should also (only offer to?) reset the list of email-approved contacts and the last seen states of the not-hardcoded subfolders, so that the list of email-approved contacts gets build up again with a full re-scan of the server subfolders using the new settings?

[Ampli-fier]

I noticed this also, see https://github.com/deltachat/deltachat-core/issues/32.
DC is collecting addresses from mails that you received from known contacts with other recipients on CC for example. This should make it easier for you to contact others (contacts of your contacts might be known by you as well). This is handled by Core.

[testbird]

Thanks, from https://delta.chat/en/help#which-messages-do-appear-in-delta-chat :

Delta Chat automatically shows …

• Messages from contacts in you address book
• Messages from contacts contacted by you
• Messages from known contacts of your contacts
• Replies to messages send by you

Since the DC contact list is getting populated with unknown adresses, the question is how does DC get to know the "known contacts of my contacts"(point 3 from above)?

• As messages from completely unknown contacts do not pop up automatically, normally, there’s no spam.

Is this still true after DC prompts about all incoming messages?

Now I recognized some of the unknown addresses where receivers (To or CC) of spam that that I had seen, because it made it through the spamfilter. But other addresses I have never seen. Maybe the latter could be avoided if DC could make use of X-SPAM-STATUS, X-*SPAM-STATE et.al. headers not read any spam folder?

And to avoid adding contacts from false negative spam detection, is DC only doing address processing after the user agreed to the prompt?

[r10s]

Is this still true after DC prompts about all incoming messages?

Yes. Even if you decide not to start a chat, these "known contacts of your contacts" are used to fill the contact list.

So, if one of your friends sends a mail to you and Cc'ed spam addresses, these spam addresses will be shown as known contacts. Ugly, but true.
I do not see a simple method to distinguish these spam addresses from normal Cc'ed. Note, that this won't happen for "normal" spam but only for spam send in the name of a friend.

Regarding the spam-folders: Typically, they are already ignored, see https://github.com/deltachat/deltachat-core/blob/master/src/mrimap.c#L82

[testbird]

OK thanks! So this is tracked down now to the spam folder name used by the provider then.

I asked the user and will report back with the specific name.

Another I recall from the back of my head from a known provider that I did not see in the source is "Unerwünscht" if that is spelled correctly.

In the end the list needs to be user configurable (folder selection using the current defaults?) to allow for all sorts of providers and languages.

[r10s]

In the end the list needs to be user configurable (folder selection using the current defaults?) to allow for all sorts of providers and languages.

Maybe it is sufficient to make the strings localizable, maybe as a comma separated list with typical names.

[testbird]

A translatable, comma separated list with typical names sounds good for the defaults.

Still another such simple individual ignore lists seems still necessary, e.g. to ignore folders with support threads or simmilar, to avoid unconstructive flooding of the contact list.

[testbird]

Also think of a folder holding messages forwarded from another address.

[testbird]

These were folders that were reported as still missing in the default-ignore-folder-list:
(Note that english names are used untranslated.)
Unerwünscht, Spambox, Unwanted

(at some provider prepended with separator INBOX.* as in #108)

[Ampli-fier]

Regarding folder to be ignored, I'd like to share some details:

What is stated by @r10s is unfortunately not always working

Regarding the spam-folders: Typically, they are already ignored, see https://github.com/deltachat/deltachat-core/blob/master/src/mrimap.c#L82

The folder meanings are (I think in most cases - this depends on the LIST capabilities provided by the IMAP server) taken from the Flag names and not from the Folder names.

This is an example (connected to IMAP server with openssl s_client -host imap.strato.de -port 993):

. xlist "" "*"
* XLIST (\Archive \HasNoChildren) "." "Archive"
* XLIST (\HasNoChildren) "." "Chats"
* XLIST (\Drafts \HasNoChildren) "." "Drafts"
* XLIST (\HasNoChildren) "." "Sent"
* XLIST (\HasNoChildren) "." "Spambox"
* XLIST (\HasNoChildren) "." "INBOX"
* XLIST (\HasNoChildren) "." "Papierkorb"
* XLIST (\Sent \HasNoChildren) "." "Sent Items"
* XLIST (\Spam \HasNoChildren) "." "Spam"
* XLIST (\Trash \HasNoChildren) "." "Trash"
. OK XLIST completed

The format is:

* XLIST (Flag list) "." "Folder name"

DC is evaluating all flags for all folder in a loop and will return the MEANING based on this evaluation.
Values used for evaluation are: _spam_, _trash_, _drafts_, _junk_, _sent_, _inbox_ (case insensitive).

When the folder "Spam" is evaluated, the MEANING will be: IGNORED, as the flag list "\Spam \HasNoChildren" contains the string "spam". -> Ok

When the folder "Spambox" is evaluated, the MEANING will be: NORMAL, as the flag list "\HasNoChildren" does not contain the string "spam". -> Not Ok

When the folder "Sent Items" is evaluated, the MEANING will be: SENT_OBJECTS, as the flag list "\Sent \HasNoChildren" contains the string "sent". -> Ok

When the folder "Sent" is evaluated, the MEANING will be: NORMAL, as the flag list "\HasNoChildren" does not contain the string "sent". -> Not Ok

The folder "INBOX" is not detected correctly. The MEANING will be NORMAL and not INBOX. Just it does not harm as the intention is to read this folder type first. Now it is read at a later stage - but is not correct behaviour -> Not Ok.

Folder not "_properly_" flagged can originate from another client, e.g. from K9-Mail. Or be manually created by the user.

The idea to evaluated those flags is good, but does not consider the above. I wonder if DC needs some kind of "Flag Handling" procedure. For example scan the folder list and ask the user about the wanted flag when the flag list is not informative - for the folder in question.

[testbird]

From reading around I thing INBOX (case insesitive) would not need any evaluation, I think this folder name is a defined requirement. From looking at your example I get the impression the evaluation could just work with a joined list of flags and folder name, without any user interaction, or not?

[Ampli-fier]

No question for INBOX. DC could even do the tagging based on the name.

Flags seem to be in english ... always, but folder names are Language specific. In Core they are partly collected for "Fallback" scenario, see issue 41.
DC would need to have input for many languages and many variants ... see mrimap.c.

[testbird]

As r10s wrote, localizable strings as a comma separated list with typical names should work fine. One for the translated DC defaults, one for the user editable ignore list. Plus the always used english defaults list.

If there is a fallback, when no flags are available, then why are the folders in your example not identified correctly? Doesn't seem to fall back where required (e.g. folders Sent and Spambox).

[Ampli-fier]

The fallback is used, when the XLIST command is not available, not when a single flag is missing. But this could be detected and the fallback could be used in such case. A combination of these two "single methods".

My idea with a "Flag handling" procedure is that the majority of user does not even know what folder are existing on the server. A user-configurable list only works when the user has an idea about existing folder.
DC could present folder names to the user that are a) not flagged and b) not detected by the fallback. Then the user marks the folder that shall be ignored (not only because they contain spam, maybe also because of personal preference). With this input DC can set the flag that means "Ignore" for these folders and set the flag "Normal" for the rest. Imagine the folder list grows over time. This would be detected by DC - new folder that are not tagged get presented.

I see the benefit for Multi-client usage that the information is stored on the server (as flags). One does not need to fill in the same information into another DC-Client installation.

I'm unsure about the "Sent" folder. If there must only be one folder flagged as Sent folder.
I have a german Sent folder ("Gesendet") that is created by K9 and not flagged.
Shall this be another marked "Sent" folder, or "Ignored" (Don't care about K9-Messages) or treated as "Normal" (also collect addresses from K9-usage and display these mails in existing Chats)?

[testbird]

Ah, you are already thinking of the UI for the user-folder-ignore list (as flaged on the server). Good. One thought, may be flag with "collect_adr" or somthing better instead of an ambiguous "ignore".

Your unflaged sent folder would then trigger a question, and you could decide.
Even if no questions are asked automatically, making the folder list (with usage flags) available in the preferences may be sufficient, or not?

[Ampli-fier]

I cannot figure out, how to change folder flags. Maybe this is not possible ... anyone knows?

[Ampli-fier]

It is possible as described in RFC 6154.

. SETMETADATA "Test" (/private/specialuse "\\Drafts")

It seems only these attributes are allowed (for specialuse):

use-attr        =  "\All" / "\Archive" / "\Drafts" / "\Flagged" /
                    "\Junk" / "\Sent" / "\Trash"

I do not find this supported in libetpan :(

[Ampli-fier]

On top, these attributes (our flags) can only be used for one folder at a time (at least for my Strato mailbox).

From the RFC:

In most cases, there will likely be at most one mailbox with a
given attribute for a given user, but in some server or message
store implementations it might be possible for multiple mailboxes
to have the same special-use attribute.

:(

[testbird]

Too bad. Sounds like a case of over-engineered, unnecessary restrictions, unfortunately.
What about the "\All" flag, is that also restricted? Maybe that could work as a "collect contacts from here" flag?

[Ampli-fier]

I stopped guessing :) Don't think this can be done as I was suggesting (flags on the server).

Presenting the folder list to the user is out of question, when the list becomes too long. Inbox's with a couple of hundred folder would create problems here, and also with collecting addresses and seeking for potential conversations with known contacts.

Looking at K9, there are 45 languages defined for spam, sent, trash, etc. Maybe this is a good start to insert them into mrimap.c.

[testbird]

Presenting the folder list to the user is out of question, when the list becomes too long.

Why? I don't think is likely to become too long for the user, these are the user's own folders, and isn't there some list widget with index bar thats able to traverse a hierachical tree of folders?

Of course, with well chosen defaults there might not be much reason for an average user to ever open the folder list, but users that are sorting emails into many folders are more likely to edit the ignore flags and others.

So for now it stays, an advanced DC settings string for a user-defined coma separated list of additional folders to ignore, would be just as fine.

[testbird]

Note that without a folder list that shows the folder types ("meanings") nobody can notice and report detection errors.

[r10s]

NB: another idea is to show only INBOX and DeltaChat folders by default, plus manually selected folders.

[testbird]

The custom-folders-to-ignore string may contain "*", to disable checking of any custom (not-hardcoded) subfolders.

[r10s]

might be targeted by https://github.com/deltachat/deltachat-core/milestone/2

closing this for now here, of course, the discussion can go on in the new support forum at https://support.delta.chat