Deltachat-android: Emails from other MUA are shown in the chat

[nexx512]

In my MUA I forwarded a normal email that had nothing to do with a chat to another person having delta chat installed. The email showed up in Delta Chat in a conversation.
I would expect Delta to only show chat related emails.

How does Delta decide if an email is a chat message or not?

If you need more details about the email please let me know.

[rizzopablo]

If I'm not wrong, today the sender listed in the Delta contacts list determine if this is considered a chat.

[nexx512]

If that is true then I think it's a bit too optimistic.
What about using the "Chat" keyword in the subject to determine if it's considered a chat message.
Maybe even in addition to the already existing algorithms.

[r10s]

If you decide to start a chat in Delta Chat (either by sending a message to another person or by hitting "reply" in the mailbox), all subsequent messages from this person will be shown in Delta Chat - independently of whether they're send by Delta Chat or by another e-mail-client.

This is because the receiver may not make such a hard difference between the messages "types" and may answer to your chat message from within an e-mail-client, if he's just using it. If this answer would not appear in Delta Chat, this will be even more irritating.

Moreover, in many conversations there are _many_ chat messages but only _few_ normal mails, so, there is not much noise added this way.

There is a minor difference in the server handling: Messages received from a Delta Chat messages go to the "chats" folder, normal mails stay in the inbox.

So, in short, Delta Chat shows all messages from your chat partners independently of the software the partner uses.

[nexx512]

OK, I see your point in making a chat independent of the way of how someone replies to a message.

But it doesn't cover the case when someone I normally chat with sends an email with content that is totally unrelated to a chat as it was the case my scenario.
I have to see whether this creates too much noise for me or if it is OK this way.

[nexx512]

I tested Delta for a few weeks now. I really like the idea using the mailing infrastructure for messaging. But for me it creates too much noise if "real" emails and chat messages are not separated if they come from the same person.
So for now I have to switch back to another messenger, but I would like to switch back once delta is a bit more advanced and hopefully there is a solution to this issue.

[r10s]

Again, how much noise does a few mails produce if they are in a chat of hundreds of messages? I still see larger disadvantages to _not_ show the "normal" mail. However, maybe someone adds an option on this point.

An alternative, to force separation between e-mails and chats, of course, you can simply use an extra account for Delta Chat.

[ghost]

Sorry to spam the thread, but I also find this very confusing, especially with very long mails. Having an option to only see actual chat messages would be awesome!

[ghost]

I was expecting this to work the way it does now.

Maybe there could be an check-box to show only Delta Chat messages, whilst Delta Chat always tags it's messages with a marker.

[svictor9]

Just adding my "vote" here : I also found Delta Chat too noisy because of this. I normally don’t retrieve emails automatically on my phone. I prefer to attend to mails from a computer. I do want notifications for "urgent" things though. As I understand them, SMS and chats are for quick "urgent" things (or, well, for family pictures). That’s why I’d prefer not to receive regular emails in Delta Chat, even from contacts with whom I’ve chatted before. Regarding the risk to not receive a reply to a chat, couldn’t Delta Chat use the in-reply-to header to determine whether an email was sent as a reply to a chat?

[nebulon42]

I also do not automatically check mails on my phone and would be very irritated if I got notified by a message that is not related to a chat I had with this contact. I prefer the asynchronity of e-mail but the synchronity of chats for some cases.

Regarding the use case that somebody replies to chat messages via another mail client: Won't this only be possible in the minority of cases? If the messages are encrypted and you might use your online mail interface or another client where you don't have the private key you couldn't even read the message hence not comfortably reply to it. But maybe this only applies to autocrypt-incapable clients, which are still the majority out there.

[STPKITT]

For me this is also a problem that needs to be adressed.

[chr15m]

Great software!

I also find this behaviour annoying. What I'd expect is that only if I have started a Delta Chat with somebody do their emails show up in Delta Chat. If I have never used Delta Chat to contact them before and they don't use it then I would not expect to see their messages in the application.

Maybe there could be an check-box to show only Delta Chat messages

This would solve the issue for me. Thanks again for your work on this software!

[r10s]

[...] What I'd expect is that only if I have started a Delta Chat with somebody do their emails show up in Delta Chat. If I have never used Delta Chat to contact them before and they don't use it then I would not expect to see their messages in the application.

This is a good summary. Thinking over this issue, we came to the same result and we will implement this in the next version this way (beside some similar things, in fact, the next version probably gets subtitle _Reduce Noise Version_ :)

However, as mentioned above, for a given and wanted chat, Delta Chat will not make a difference between the MUA used by the sender.

[csb0730]

IMHO it's important NOT making a difference between MUA used by sender to keep possibility open to have communication to users which are not using DC (and this is still the majority in my use case).

But on the other hand: Would it be much effort to add an option to show only DC sent messages?

[r10s]

IMHO it's important NOT making a difference between MUA used by sender to keep possibility open to have communication to users which are not using DC (and this is still the majority in my use case).

That's what we're currently doing and we are not about to change this.
But we want to avoid _new chats_ popping up unexpectedly.

But on the other hand: Would it be much effort to add an option to show only DC sent messages?

Every option always brings more complexity than expected ;)
But I still do not get the point on this:

• As mentioned above, In a typical chat, most messages are from Delta, not from normal MUAs
• If the sender switches from Delta Chat to normal Mail for an answer - maybe for low battery on the mobile - why should this answer not appear in Delta Chat? And how can the receiver know before?

[elfrinjo]

I'd suggest a simple subject prefix like [d-chat]

This would make it easy to understand for every user why conversations Show up and why Not.

[r10s]

We have the subject prefix Chat:, not sure what you mean with [d-chat]

Do you want the sender of messages from normal MUAs to decide whether a message pops up in Delta Chat or not by adding (or skipping) a special prefix? This could work, however, this would require some learning for the users of normal MUAs.

[elfrinjo]

Hey Björn, sorry my last reply was a bit short, as being on mobile.

I basicly think, that there should be a obvious and simple filter for what is showing up on delta chat and what not. The MUA s quite useful, but I think it would become kind of unpredictable as most users can not modify or even see this header. The Chat: prefix in the subjectline might not be distinctive enough to select the right conversations.

Let me elaborate on the different types of users I do expect:

1. Users with a distinctive email account for deltachat.
   Those users won't need any filtering and be happy with anything showing up in delta-chat.

2. Users with a dislike for traditional email clients
   Those do not need filtering as well and will be happy to use deltachat to respond or create all of their messages.

3. email only users
   Those do not know anything about deltachat. They will not initiate chats, but they might receive and respond to them using their usual mail client. For those users, an explanations might be included inside the email-message which is not displayed inside deltachat.

4. Mixed mode user (my personal usecase)
   I'd like to connect deltachat to my general-purpose email account and have chats showing up in deltachat while other mails are exclusively in the usual email client. Chats could live inside a special folder.
   To achieve this, two features are needed: a) a distinctive mark to filter messages and move them to a folder (inbox rules of my mail system). b) deltachat monitoring more folders than just inbox

Soo... please excuse the rambling on my personal expectations and opinions :)
I hope, it contains some ideas you like as well!

Cheers, Jörg

[r10s]

Okay, I understand your use case better now :)
However, for the default, I would still prefer showing all messages in a chat (I would prefer showing one message too much than to miss one - esp. as most messages are normally wanted chat messages; I mentioned this :)
Only if the user is quite sure, he does not miss a mail, I could imaging an option not showing up messages from normal MUAs even if there is a chat with the person active. But things get complicated then - what eg. about replies of normal MUAs to chats initiated by Delta ...
(disclaimer: please forget me if I mess up things, I'm tiered and I think I will go to bed soon)

[chr15m]

My understanding of the change that is being proposed:

New option "deltachat messages only" or "reduced noise".

With the option turned off (default)

No change from current behaviour.

With the option turned on

• Message from another deltachat client: shown.
• Message in a thread with existing deltachat messages (even if from another MUA): shown.
• Message in a thread with no deltachat messages in it: not shown.

So basically if we've got an existing deltachat communication happening with some user then it shows up. If the user is not using deltachat and we have not used deltachat to communicate with them, then the message is ignored.

[testbird]

(edited)

Two things to decide: Chat / E-Mail (display) and Alert / Mute (attention paging)

Thread based decisions.
Answers to a chat thread (in-reply-to ?) must always show up in the chat. (was that implemented already?) This is a bypass to the below chat vs. email decision rules.

But it is not possible to always know whether a new thread will be a chat or email.

A problem arises after one has chatted with a contact, and then the contact or oneself initiates a new more elaborated email correspondence (possibly in another MUA).

Defaults
(see https://github.com/deltachat/deltachat-android/issues/49#issuecomment-338352424)
Basicly, chat messages trigger alerts, emails don't, and we allow answering chats from other MUA (for full interoperability).
But we need some options to handle the special cases / circumstances.

Custom options

On the receiver side:

• make it real easy to switch new messages (possibly large or small, e.g. the new monthly report) that were not categorized according to the users wishes between chat/email and alert/mute mode. (Even during a running chat.). It is important to avoid working against the users wants. Simply offer to adjust the categorization right in an option for the message:

per contact options

• per contact option to "default short emails to emails" (even new messages shorter then 240? characters)
• per contact option to "default long emails to chat" (even long ones)
• per contact option to "always alert" (allow your server's watchdog to override global mute)
• per contact option to "always mute" (e.g. your daily server stats) (alredy implemented)

global options

• "consider short messages as chat and long messages as emails" (the default)
• "default short emails to emails"
• "default long emails to chat"
• max. length of "short message" ( 240? 300? 400?)

Exceptions (allowances for the sender side):

• Allow known contacts that are set to "default short emails to email" to still start a chat (initiate) using a chat flag. It could be enough to manually choosing a subject that begins with "Chat:" (case insensitive).

• Allow known contacts to send a message beginning with an "ALERT" or "ALARM" subject (case insesitive) to override the general muting (but not the per contact muting). To allow paging by really important messages but blocking spammers.

• Allow the sender to suppress alerting the receiver for sending low priority (status) messages. Any idea for an intuitive method here, anyone? Maybe using braces "(Chat:)" or a subject not beginning with "Chat:" from a known chat contact sent with a chat program? "Silent Chat:"?

[csb0730]

Now it seems to get complicated in some way?!

I'm with r10s to keep message handling into DC as simple as possible and it would be more important not to miss displaying some messages than suppressing them.

Another hint from r10s:

"But we want to avoid new chats popping up unexpectedly."

Are messages from unknown user really lead to a new chat automatically?
I know only the behaviour in this case to dispatch such messages to "Kontaktanfragen" and I find this really ok in this manner. Isn't it?

Ok, these are only a few thought's about this issue...

[r10s]

One part of this issue - the less controversy :) - will be addressed by the commit https://github.com/deltachat/deltachat-core/commit/c1a852ce01c7f9ae890d75a2ce94473d522192fe which will be rolled out in 0.9.7 then:

• Delta Chat then does no longer create chats from messages he "thinks" that may be known in some way (eg. if the sender is known through a CC: of a known contact or by the address book)
• all this creates too much confusion as it was never clear why a chat was opened - and there are many situations a chat would be expected but not started
• so, things are very clear now: chats are created by _you_ - not by Delta Chat
• Delta Chat only shows up messages of chats started by you, this includes groups (group chats are created if there is a normal chat with the sender)

This fix does _not_ affect the other point in this thread, _which_ messages appear in a chat, Delta Chat shows both, messages of normal MUAs and of Delta Chat in the chats started by you.

[r10s]

Upps, temporary closed accidentally ... it's late ...

[r10s]

... and for the rest I plan to add some options in the advanced settings ... :)

[testbird]

Good, the new rules are better to understand.

One thing you probably just forgot to mention above: Incoming messages sent with delta chat (clearly a chat) from an address that is known, but to whom no chat has been initiated yet, must open a new chat. (otherwise a chat could never be established)

Another reasonable exception may be to allow chat initiations from known addresses even if using a non-delta chat client when the message actually is short (=<240 characters?).

...for the rest I think, some settings from my message above are already there, I will edit and update my message above...

[r10s]

Incoming messages sent with delta chat (clearly a chat) from an address that is known, but to whom no chat has been initiated yet, must open a new chat. (otherwise a chat could never be established)

Well, this is what was exactly done up to now.

This approach has the problem, that addresses that are _known_ not always match the addresses you _want to have_ as a chat. And, the other way round, you may want a chat of an address that is not yet known.

So, finally, it might be more clear to say, you have to initiate all chats on yourself. The way to do this, are the contact requrests.

There is no perfect solution for this up to now; for now, I'm working on a switch to say "create chats with known addresses automatically" which restores the old bahaviour then.

An good solution might be to show up a contact request in some non-disturbing way in the chat list; however, doing so, we got the spam problem at once. Any ideas on this point are very welcome.

[testbird]

User option systematics updated: https://github.com/deltachat/deltachat-android/issues/49#issuecomment-335565565

[testbird]

finally, it might be more clear to say, you have to initiate all chats on yourself.

This can not work for the other one (the receiver), because he is then not initiating himself. :-)

And really, I think it would make delta chat way too cumbersome, if one is not reachable right away by known contacts (i.e. adress book). So as you write we need to find the perfect solution.

This needs to be identified and fixed:

it was never clear why a chat was opened - and there are many situations a chat would be expected but not started

1) clear basic rules
All replies, and known contacts (from messages and address book) that use a chat client, or write a short email open in the chat UI.

2) Way to reduce/expand the chats to the wanted degree. (see above)

to show up a contact request in some non-disturbing way in the chat list;

Don't we have that already, with the option "show (muted) contact requests in chat list"?

BTW: Maybe default to always show this "chat interface" for email (named "E-Mails") at the top, with a "?" in its icon.

3) Way to convert received email into chat.
A "start chat" option in the muted Emails "chat interface".

For the original problem of this thread I propose a max. message length for regular MUAs not in-reply-to for limiting the default chat message appearances, and making it easy for the receiver to express his sorting wishes when the defaults are not working exactly as he wants.

[testbird]

The problem has its complexities, but that does not mean the UI needs to be complicated!

For the user it may be as easy as finding an email message and selecting "long emails should also be chats" and then "only from this contact or generally" (other cases similarily)

[testbird]

But make the selection with something like two drop down menus in a single screen, not series of questions about which the user has no oversight.

[testbird]

In the last example for an uncomplicated interface, when the user finds an email and chooses to "adjust filter rules" the selection (long emails should be chats) would then not only change the filter rules but also open the message in the chat, ready to be answered.

[testbird]

In the original case of the issue reporter, the message should not have been considered a chat message anyway (too long and not in-reply-to). But suppose it would have been a shorter new, unrelated message, when he selects "adjust filter rules" and chooses "from this contact | short emails should be emails", the message should automatically be moved from this contacts chat history into the "(?) E-Mail" history.

[testbird]

Note that global options can not solve this. With some contacts one may want to use delta chat to chat and other MUA to email, with others one may want to even quickly answer long emails, and any other combination.

[r10s]

@testbird thank you very much for the high-qualified input, sounds very reasonable. I will get into details these days then. If you find the time, can you mail me at [email protected] directly - I have a private question.

[testbird]

Your're welcome. Since the logic is important for all chat clients, hopefully it can be part of deltachat-core for it to be usefull for all clients.

[r10s]

The logic will go to the deltachat-core, so changes discussed here in reality affect the core.

[r10s]

My ideas a roughly the following:

• chats are no longer created automatically, as mentioned above
• instead, if a potential chat partner wants to initiate a chat, a prominent message appears atop of the chat list. sth. like that:

• if the user click on it, a chat with the sender is started.

Intersting questions now are:

• which mail produce a chat request - Delta, E-Mail, from known users, possibly with length limit
• if the user does not want to start a chat, and another mail arrives, will it produce another request? I would say no ... but there will be also many cases to handle

However, all this seems to be minor things that can be tweaked - what do you think about the general approach of showing a request directly?

[testbird]

Good idea. Prompting with an item (without playing a tone) could be a nice middle way between paging and having to manually check the incoming emails for Chats. Of course it should be possible to disable it, for those that prefer to go through, or have to go through the incoming emails anyway, whenever they decide to do so. So some sort of "(?) E-Mail" list and viewing UI would still be necessary. (Simply a chat view showing the sender addresses but without the text entry bar at the bottom?) In the list of threads, the () E-Mail entry could have a message count over the total unread e-mails, and maybe list the last senders instead of the message text, so that the user can see when there was a (maybe slightly to long) message that slipped into the E-Mails and is worth an answer.

In order to inform the user about what the prompt is all about, it could ask "New Chat or E-Mail" instead of talking about requests that nobody made. The App is a tool that allows to work with e-mail correspondence. Maybe have a two buttons (or a slide/drag thing like answering a call) to decide, and a "adjust filer rules" entry in the options?

which mail produce a chat request - Delta, e-mail, from known users, possibly with length limit

Yep, that should do.

if the user does not want to start a chat, and another mail arrives, will it produce another request? I would say no ... but

Exactly, these are the cases that depend on the sender and receiver (=> adjust filter rules).
My take is that by default (with a length limit) the next short email should be shown again. But give the user the possibility to adjust the filter rules, and select "from this contact | short messages | are emails".

Or, maybe the prompt could show the current rule right away:

                                  New Chat or E-Mail

  [E-Mail]     **Bob** ([email protected]): What do you think...    [Chat]

              ( by default | e-mails < 300 characters | get prompted )

[testbird]

Ticking the prompt (not the buttons) could show the full email.

[nebulon42]

Is this resolved? Sorry for asking that, but I suspected that "Show messages only for explicitly wanted chats" in the changelog of 0.9.7 has something to do with this issue. But I could be mistaken.

With 0.9.9 I had the use case that I started a chat to an address and received an encrypted message from this address (unrelated to Delta Chat). The message showed up in Delta Chat.

[r10s]

Well there are (at least :) two issues discussed here:

• _when will a new chat be created?_ This is - more or less - addressed by https://github.com/deltachat/deltachat-android/issues/49#issuecomment-338352424

• _which messages are shown in a chat?_ This is not changed.

Maybe we should close this issue and create a more focused one ...

[testbird]

which messages are shown in a chat? This is not changed.

And it matches the this issue, to distinguish between emails and chats, and to let the user decide to handle emails within DC or not (per default and per user).

[testbird]

which messages are shown in a chat? This is not changed.

Seems it has changed after all. Isn't the new prompt now bugging the user about every email that arrived (one after the other), as if assuming the delta chat to be responsible for all email (Delta chat as single MUA) instead of just short message chats? It seems to ask about not-in-reply-to, long emails from unknown contacts.

[csb0730]

I think, at first glance DC should handle all incoming messages. So that You can use it as a single MUA if You want (IMHO).
Special options? Maybe could be useful...

[testbird]

The "as single MUA" would also be my personal use-case currently, but I would prefer a detailed browsable (chat like) list as it was previously available.

Spam is a problem for the adoption of Delta Chat that was nicely avoided by only allowing opening of chats for newly incoming messages of known contacts by default and hiding not admitted messages away as "contact requests".

I could not find where I posted my use-case thoughts. I think for the "parallel chat and email clients" use-case (current default) a max. message length for prompting might still be needed.

[testbird]

I consolidated the concepts at https://github.com/deltachat/deltachat-core/wiki/Use-Cases

[r10s]

@testbird thank your for the nice overview.

Maybe we should close this issue and create a more focused one ...

i close this one now, I think we can discuss this clearer in #229 , or, if sth. is missing there, anyone is welcome to open a more focused issue.

[isheff]

For what it's worth, as a random user who decided to try DeltaChat for kicks, having a non-delta-chat email show up in DeltaChat at all was not expected behavior. Likewise, it would be nice if somehow my other email clients did not show me DeltaChat messages ever. I could pretend they're totally different protocols that just happen to use the same servers.

One problem the current behavior creates is this: a new email arrives unencrypted into my chat stream. Then my next message (in DeltaChat) is by default unencrypted. From the other person's perspective (not seeing the email as part of the chat), there's an arbitrarily unencrypted message for some reason. This is not good.

[r10s]

we try to target this and make things clearer to the user by the single-folder-approch, see https://github.com/deltachat/deltachat-core/milestone/2

"single-folder" means that there is _one_ configurable folder that is used by DeltaChat for sending and receiving messages. by default, this is the INBOX folder.

this change will only affect a multi-client-setup sharing the same account between DeltaChat and a normal MUA.

[testbird]

Hi, isheff, sorry for your bad experience well over a year after this bug has been filed, and closed without solving the problem.

Would one of the simple measures for that were proposed above have prevented your bad experience?:

• Default to only show messages shorter then say 300 characters directly within chats. For longer emails from the contact, only show an email envelope icon in the chat (as a link to the "incoming emails").
• Optionally, have a setting to only show Email-chat compliant messages in chats.

this change will only affect a multi-client-setup sharing the same account between DeltaChat and a normal MUA.

I'd expect it to be the same for any email account used by deltachat, no matter how one calls the account. Without fixing the problem appropriately, every longer unencrypted email message will show up in the chat, even if the account is named "dedicated".

[testbird]

@isheff See https://support.delta.chat/t/wiki-use-cases-chat-rules-and-configuration-options/122 for more about finding the best default use-case and adaption options.

[isheff]

@testbird Thanks! I wouldn't say it was a "bad experience" so much as me trying to contribute some feedback on an (admittedly old at this point) discussion in which I have opinions.

In any case, thanks for the suggestions. I had taken a look at https://support.delta.chat/t/wiki-use-cases-chat-rules-and-configuration-options/122 very briefly, but got the impression these were hypothesized use cases and suggestions for future settings, rather than actually available yet.

To answer your questions:

Personally, by default, I would only want to show Email-chat compliant messages in a chat. If not by default, then I'd personally like that as a setting option.

In response to @r10s: Can the DeltaChat sending & receiving folder be set to something other than INBOX (e.g. DeltaChat folder) ? Don't receiving email servers just always put all incoming emails in INBOX, and so without their cooperation?

[testbird]

I would only want to show Email-chat compliant messages in a chat.

Yes, that is a preference considered in the use-case document. It was not selected for the default proposal, because it would not allow contacts to answer using another email client.

[testbird]

What about a max. chat message length for not Email-chat compliant messages. Would it have worked to filter out the email in your case?