Deltachat-android: MUA replies left behind in INBOX

[gerroon]

Hi

Thanks for this app. It works well when sending messages meaning that all messages are sent to the chat folder on my server. But the replies to my messages are delivered to Inbox. The friend who replies does not use DC, using regular email client . Is that how it works? If so, seems convoluted.

I am using it
Android 0.17.1 Lollipop
Server Exim, Dovecot on Debian Wheezy.

thanks

[r10s]

Is that how it works? If so, seems convoluted.

No, the idea is that replies should go to the "Chats" folder, too. However, this only works if Delta Chat is online and got the mail before.
Can you check, if this happened? Is the reply visible in Delta Chat?

[testbird]

Here, I see a qr related message (according to the time sent) that stayed in the inbox.

[testbird]

After the qr test last night, and now receiving an unrelated non-encrypted message from a former deltachat user (correctly in a chat): The old qr message is now in the chats folder, but the new unencrypted message stayed in the server's INBOX.

[gerroon]

@r10s

DC is running in the background constantly, I see it is in the status bar of my phone. I am sure DC gets the message first because the chat (ones with Re:Chat:) messages (in the Inbox) in the email client (Claws mail) on my desktop shows that the message is already read.

The replies are notified by DC to me on my phone and I read the replies in DC first.

[gerroon]

So can you guys guide me on this? This is driving me nuts here :( The whole reason I use DC was to keep my inbox clean now it is flooded.

the weird thing is that I remember earlier versions did not have this issue, I was still using the same email and email server.

Is this a bug?

[testbird]

It looks I had wrongly thought the message from a qr contact verification was later moved to the chats folder. (Due to different time formats (hours/minutes hours/minutes/seconds) in the MUA.) The messages have not been moved later.

The situation is rather that:

1) As gerroon, I see a plain-text email message that appears in the chat but is not moved to the chats folder.

In my case the message received seems to be a reply to a message not sent with deltachat.

2) I see two deltachat messages staying in the inbox from using the experimental qr-code based contact verification process that can be enabled through the advanced options. The messages seem to be the messages send by the qr-code scanning party (received by the qr-code displaying party) during two scans with about ten minutes in between.
(In the inbox of the scanning sender, strangely, only the first one is still present, but marked for deletion. May that be due to having the folder open and attempting to view the message with the MUA?)

[testbird]

Here is an idea to solve the problem of flooding the inbox with plain-text chat messages that chat contacts may send not in-reply-to a chat message (which was true in my case):

• Use the proposed option 4) a max. chat message length configuration from https://github.com/deltachat/deltachat-core/wiki/Use-Cases, not just to ignore longer emails, but also to move all the shorter email messages to the chats folder, by default.

Gerroon, could you check if the messages you get have a proper in-reply-to header?

[gerroon]

@testbird,

Thanks for looking into this.

I do not have DC to DC chat use case. My friends are all using their own emails/clients.

here is the portion of the header

Subject: Re: Chat: interesting that ...
In-Reply-To: <[email protected]>
References: <[email protected]>
 <[email protected]>
 <[email protected]>
 <[email protected]>
X-Priority: 1 (Highest)
Message-ID: <[email protected]>
X-Sender: [email protected]
User-Agent: Roundcube Webmail/1.2.7
X-Originating-IP: xx.xx.xx.xx

[testbird]

From my naive view it looks as if References: even contains a deltachat reference (mr.thread), so is it a bug @r10s?

[testbird]

Is [email protected] the Message ID of the deltachat message you've sent?

[gerroon]

Yess all XXXX are my domain

[testbird]

OK, thanks, could you please edit the title to something like "MUA replies left behind in INBOX" for later recognition?

[gerroon]

So is this a bug?

[testbird]

I guess so. Hopefully, @r10s will find it.

[gerroon]

I uninstalled and reinstalled Deltachat (Fdroid 0.7.3) and I added my account from scratch. It is still broken with message replies staying in Inbox.

[violoncelloCH]

0.7.3 or 0.17.3?

[gerroon]

Sorry It is is actually "v0.17.3"

[gerroon]

Is this going to get fixed or any ideas?

I am thinking to stop using this app just because of this. If this issue is not concern to others, that is fine. I am not a dev, I can't do much about it. It is just that it is quite a bit of hassle for me to keep cleaning my inbox, not worth the trouble really. And I do not know how to make it work.

thanks

[webratte]

@DC-deveteam

Maybe there is a way everytime DC connect to the server it could run a scan for DS message replies in the inbox.

And if it found a DC message or a a reply of a DC message (readed or not readed) it could move it into the chat folder.
Maybe like a Sieve filter (I hope I use the correct word - my bad english ;-))

I don't know how you mark a chat message to seperate it from normal mails (by subject I suppose).

I hope you understand what I mean.

[gerroon]

I thought that is what it ws doing but appearantly it is not the case. It seems like this works fine for the most people but not for me :(

[Ji-eF]

Maybe like a Sieve filter

It's planned for a future release.
In the meantime, you can create a filter that move messages in a Chat folder on your Inbox.
I created these rules on my Seamonkey Mail client (Tools > Messages Filters) for sorting out my DC messages :
Match all of the following :

• Custom header x-mailer, begins with, Delta Chat
• Custom header chat-version, contains, 1.0
  Actions
• Move messages to, Chats on [email protected]

[gerroon]

@Ji-eF

Are you saying that replies goes to Inbox is by design? I mean sounds like you have this issue as well?

I personally would like this issue to be resolved and I am not running an email clinet on my computer so that some messgaes are sorted propery, in reality DC should have been taking care of this properly.

Also it is a bit embarrasing when I recommend this app to freiends, and they also get confused by their inboxes are littered with "Chat:" messages. Am I supposed to tutorial them about how to set up emails filters on theor desktops?

Do not get me wrong, but as @r10s indicated in his reply to my post, this sounds like a bug and to me an important bug, because it just destroys the experience and I stopped recommending to my friends :(

[webratte]

I have now created a filter in my mail account and it works.

But I think @gerroon is right. This job should be done by DC.

As I said before.
If DC is runnig it should move the mail immediately after receiving a new mail.

If DC is off-line it should check the inbox for new mails if it comes online an move it into the chat folder.
Of course in the meanwhile the mail stay in the inbox until DC comes online.

[violoncelloCH]

for me delta chat moves all sent messages as well as replies to them to the Chats folder as soon as it is active...

[gerroon]

@violoncelloch

This is quite a dilemma :( What email server are you using?

[violoncelloCH]

@gerroon not quite sure, but I think my hoster uses exim

[Ji-eF]

Guys :) Don't forget there's no stable release yet. The 1.0 Milestone should give you a hint on what to expect from Delta Chat when it's released to the general public.
Meanwhile, kindly explain to your friends that you found a new cool Chat software to test with them, but also tell them that it still is under development, so they should expect that few things go wrong sometimes.
( Also take a look at #108 )

[gerroon]

@Ji-eF

Thanks for the link, it seems like it was broken for a while but I do not understand why this is broken for this long? Is it like a mysterious edge case (seems like many users experience it)? is it that the devs cant replicate it? Is it going to be a very hard to fix?

To me delta chat has two serious issues.

1 - This bug already

2 - It is not poling messages fast enough, sometimes I have to kill the app and restart it to make it force get it. Maybe it should offer press and pull down in the message window for a refresh like K-9 does

These two makes this app more like a buggy email client than a chat app to be honest. Sorry for the negative tone in my reply, but sometimes hard truths have to come out hard to be useful.

[webratte]

Meanwhile, kindly explain to your friends that you found a new cool Chat software to test with them, but also tell them that it still is under development, so they should expect that few things go wrong sometimes.

I will do that if there is a Beta version in Google Playstore ;-)

[csb0730]

DC v0.17.1

I can confirm a similar behaviour in the moment. Messages which should be moved to Chats folder cannot be moved because connection to IMAP server is lost after each try to move.
Please see or compare core 170 issue.
I don't saw this with v0.16.0 release as far as I'm able to estimate this. But this is only an assumption.

What I see too, sometimes DC seems not to be able to send any message. Only a restart by application manager helps to get it working again. But then adhoc it works properly again.

I suspect, that there are some (two) issues in the air!

Please check imap/smtp logging for errors (settings->Info->Info). Maybe there are some hints?!

[gerroon]

@csb0730

I do not see anything beside DC constantly checking some empty or non existing folders on my server with "0 mails read from _ORDER/[email protected] " and some others in that folder :(

For instance I have "_ORDER" (which I created back then) folder but it is empty in my server. Somehow DC looking for _ORDER/[email protected] (Something is not te real name there)> The _ORDER folder is empty according to my desktop email client (after subscribing to everything in the folder).

Beside those I am wondering why Dc is parsing all the folders, sshould not be just looking at Inbox and Chat folders only in the first place?

Maybe you should give us an option to choose where to parse the new emails as well, that way DC does not keep the email server busy constantly..

[gerroon]

So is there any rough estimation about fixing this? Are we going to wait for the release to get this working properly?

[t-oster]

@gerroon for estimating and fixing we have to reproduce the issue fist. In my case it works as expected on DomainFactory. So what provider are you using? Can you provide a test-account? Does this happen to all replys or just randomly?

[Ji-eF]

@t-oster @r10s : I can provide you with an e-mail address from my host if you want to fiddle with.

[t-oster]

yes please. You can send the details mail at thomas-oster.de

[Ji-eF]

Sent (Hope I got your mail correctly :p )

[gerroon]

yay progress!

[t-oster]

didn't receive it yet ;(

[Ji-eF]

Did my parser failed me ? mail<at>thomas-oster.de
Also, check your Junk mail folder I guess...

[t-oster]

Well. I can reproduce this issue. IMHO this belongs to core instead of android. But I am still searching for the code which is responsible for moving the message. Here is a sample log:

• Mail was first sent by a non-DC client. Then in DC answered and then again answered by the non DC client. The answer appears in DC but was not moved to DeltaChat folder:
  ```
  I/DeltaChat: Receiving message INBOX/4...
  I/DeltaChat: Protected headers found in MIME header: Will be used.
  I/DeltaChat: Protected headers: Overwriting "From".
  I/DeltaChat: Protected headers: Overwriting "To".
  I/DeltaChat: Protected headers: Overwriting "Message-ID".
  I/DeltaChat: Protected headers: Overwriting "Subject".
  I/DeltaChat: Protected headers: Overwriting "References".
  I/DeltaChat: Protected headers: Overwriting "In-Reply-To".
  I/DeltaChat: Message is a reply to a messenger message.
  I/DeltaChat: Message has 1 parts and is moved to chat #10.
  I/DeltaChat: 1 mails read from "INBOX".
  I/DeltaChat: Chatlist for search "" created in 0.578 ms.
  I/DeltaChat: Chatlist for search "" created in 0.267 ms.
  I/DeltaChat: Message list for search "(null)" in chat #0 created in 0.031 ms.
  D/OpenGLRenderer: CacheTexture 8 upload: x, y, width height = 96, 99, 107, 152
  I/DeltaChat: IDLE start...

[t-oster]

@r10s the message is generated here https://github.com/deltachat/deltachat-core/blob/master/src/mrmailbox_receive_imf.c#L1334 can you point me to the code, where the message should be moved afterwards? Or should it already be moved then? I think it should be moved in the markseen function, but I only see this call (https://github.com/deltachat/deltachat-core/blob/master/src/mrmailbox_receive_imf.c#L1447) which is only for MDNs if I see correctly?

[r10s]

@t-oster the log line is a bit misleading, when Message has .. parts and is moved to chat #..., the message is already "moved" there by the INSERT INTO msgs command some lines above. maybe we should write is assigned to chat #....

[r10s]

this call (https://github.com/deltachat/deltachat-core/blob/master/src/mrmailbox_receive_imf.c#L1447) which is only for MDNs if I see correctly?

yes, for received MDNs, this adds the job MRJ_MARKSEEN_MDN_ON_IMAP to the database which results in a call to mrmailbox_markseen_mdn_on_imap() { .. mrimap_markseen_msg(.. MR_MS_ALSO_MOVE ..) .. }

however, it might really be that the overall logic is not always working ... great that you have a look on this :)

[t-oster]

so where should the E-mail be (imap)-moved to the DeltaChat folder in this case? Or be marked as seen?

[r10s]

so where should the E-mail be (imap)-moved to the DeltaChat folder in this case? Or be marked as seen?

i do not understand the question, what does the "where" refer to?
the "real" moving on the imap server is done in this line https://github.com/deltachat/deltachat-core/blob/master/src/mrimap.c#L1709
however, i am not sure if the problem is the _"real" moving_ - or if the mess of conditions whether to add the move job or not is just wrong :) (i will have a closer look on this and trying to understand :)

[t-oster]

Is it correct that the messages are only moved AFTER they are read in deltachat? Maybe we should change this so they are moved on receive.

[gerroon]

I say they should be moved on receibe if not moved by some other server/client side filter.

[r10s]

Is it correct that the messages are only moved AFTER they are read in deltachat?

i'm only half here currently, however, yes, iirc, this is true.

Maybe we should change this so they are moved on receive.

iirc, then reason for this is multi-client (only the INBOX is watched via push and if it is moved away by a device currently not in use (eg. the computer), the device in use (eg. the handy) may get the message delayed)

[t-oster]

So we should monitor inbox and DeltaChats folder via push.we can still use the Seen flag for tracking if the msg is read by another client

[r10s]

So we should monitor inbox and DeltaChats folder via push.

yes, this could be a solution.
unfortunately, it's not easily possible to monitor multiple folders, see here: https://github.com/deltachat/deltachat-core/issues/180#issuecomment-394720059

[gerroon]

If you guys want me to test it, hit me witha debug build, I will give it a go.

[gerroon]

@t-oster @r10s

Btw in my case whether the first message was from a non-DC client or I created the first message in DC, replies to my messages stays in INBOX and never moved to the Chats folder.

[t-oster]

Coming closer:

```
20:57:20 0 mails read from "INBOX.Sent".
20:57:20 Marking message INBOX/8 as seen...
20:57:20 Message marked as seen.
20:57:20 Moving message INBOX/8 to INBOX.DeltaChat...
20:57:20 Cannot move message.
20:57:20 Job #1 done and deleted from database

[t-oster]

Well it seems as if on your server the MOVE does not work. As stated in the source here: https://github.com/deltachat/deltachat-core/blob/master/src/mrimap.c#L1708 we should fallback to Copy+Delete. I have the copy part working, but since I am not into that IMAP stuff maybe @r10s can help out? I think for deletion I need that rfc724_mid, but I don't know where to get it in the mrimap_markseen_msg function.

[gerroon]

@t-oster thanks for the update. It would be nice if the date information (of the copy) is kept intact when doing copy and delete.

[t-oster]

@gerroon here are multiple problems. First: the chat server of at least @Ji-eF does not support MOVE. Thus all messages received stay in the inbox. This will be fixed by deltachat/deltachat-core/pull/181.

There is another Bug(?) though: If I receive a non-DC message from an unknown contact and then open a DC chat from that message, replies to that message are not detected as replies to a DC message and thus are not even tried to be moved...
I will look into that

[t-oster]

Nope: false alarm. The other problem was just becaused I answered on a message which was sent before a reinstall of DC.
So I think this issue is fixed by deltachat/deltachat-core/pull/181

[gerroon]

@t-oster
thnaks for the update. Can you guys make a release soon so we can test #181 ?

thanks

[t-oster]

@gerroon here you are https://cloud.thomas-oster.de/index.php/s/Alf3813p4pUCySK/download (will be valid a week)

[gerroon]

@t-oster

thanks, is this using the same key as the github rrelease or your own key?

[t-oster]

It is an unsigned debug build. You should of cause create a backup but maybe it picks up your existing data

[gerroon]

Ok thanks, it would be even nicer if the app was niced with "debug" so I can differentiate it from the original. it is ok I installed and start using it, I will update the bug once i have some case.

[gerroon]

Two issues

• It does run in the background, no tray visibility

• It is not polling emails (or not fast enough), I am still waiting for the reply to an email I sent while all other emails clients already got the result (did not open it in them yet). Since there is no custom refresh, I seem to be able to force, if I kill the ap and restart.

I am using a lollipop device

[gerroon]

Ok after some initial test it seems to work except the issues I mentioned above. I have not tested extensively yet.

Also It did not put the emails in "Chats" it put them in "DeltaChat". It took a while to figure it out :)

@t-oster

So this is quite a progress, thanks for working on the issue.

[Ji-eF]

@gerroon :

It does run in the background, no tray visibility

Search Settings > Advanced for "Fetch Mode"

It is not polling emails (or not fast enough), I am still waiting for the reply to an email I sent while all other emails clients already got the result (did not open it in them yet). Since there is no custom refresh, I seem to be able to force, if I kill the ap and restart.

Same as above I guess ^^

[gerroon]

@Ji-eF

Cool looks like this is a new thingy.

[testbird]

Also It did not put the emails in "Chats" it put them in "DeltaChat".

@t-oster do you move the old folder? Maybe see/improve/update: https://github.com/deltachat/deltachat-core/wiki/IMAP-strategy#setup-first-start-of-new-release

[testbird]

Since there is no custom refresh, I seem to be able to force, if I kill the ap and restart.

Shouldn't foregrounding refresh and switch to push?

[gerroon]

@Ji-eF

I cant seem to be able to revert yo battery save mode one the persistent is enabled?

[Ji-eF]

No, I have not this bug on my end :/
I can switch back and forth from one setting to the other with no problem.

[gerroon]

@Ji-eF
Bizarre. How do you switch? I cant do it on my Lollipop Touchwiz rom

[Ji-eF]

How do you switch?

Well like you I suppose : Click "Fetch Mode" > Clich either "Persistent connection" or "Save battery"

[gerroon]

@Ji-eF
WHen I press fetch mode all I see is "Cancel - Persisten Connection" buttons

[Ji-eF]

Oh, I suppose your screen is not long enough . "Save battery" is right of "Persistent connection"
Try turning 90° your phone to landscape mode

[t-oster]

See https://github.com/deltachat/deltachat-core/issues/47 about the Filder name

[gerroon]

@Ji-eF

Yup that was it, the 3rd option is not shown in landscape on my phone.

[testbird]

@t-oster I knew about the name change, what I wanted to know is whether your patch smoothly migrates the Chat folder into to DeltaChat (move or copy+delete) as in the link.

Either way, its a beta no prob.

[t-oster]

My first attempt did but we decided to not complicate the code

[t-oster]

So this can bei closed?

[gerroon]

@t-oster
I say keep it open until miltiple people and multiple tests can confim the expected behavior.

[gerroon]

@t-oster @Ji-eF

This is based on t- your build. I have the persistent connection is enabled and a new chat message is landed in the inbox for 15 mins and it was still not in the app. So I killed the app and started again, it grabbed the new chat messgae right away.

Does this sound normal? I mean for a chat app to wait more than 15 min for grabbing mesage does not sound proper.

thanks

[t-oster]

Well my build uses the latest code which may have new Bugs, but the change i Made for this issue cannot cause a delay

[t-oster]

But you could provide logs

[Ji-eF]

@r10s @t-oster : I forgot to mention : you have an access to a roundcube page along with your test mails from my host : https://mail.ovh.net
I mainly use Seamonkey for my mails operations, but I decided to go take a look at my account from there... You really should go see your accounts there : It's quite messy :/
On one account, I had a Trash folder inside Chat folder.
On another, I had a Chat folder inside Chat folder.
Also, while on this second account from Seamonkey, I even managed to "Refresh" INBOX.INBOX.Chats ( <- Yes, twice INBOX) o_O

[gerroon]

Hi

Can we please get a regular release with this fix? It seems to work better than the last release as far as this bug goes.

@r10s @t-oster
No reason to make people suffer from this bug any further in my view.

thanks

[r10s]

@gerroon we'll make a new release as soon as possible, however, it may still take some days as we've also established a new more reliable background-method (see mailinglist) that needs some more adaptions.

[gerroon]

@r10s thanks, keep up the great work guys

[gerroon]

Ok I thought that this bug was fully crushed, actually it is not :(

There is one issue is that if the original email initially sent by a non Deltachat user, all the replies by the original sender stay in inbox still :( My replies to the conversation seems to be sent to "DeltaChat" folder

[gerroon]

Any thoughts on the comment I left above? This bug seems to be half way there.

[t-oster]

@gerroon i can confirm this

[r10s]

this issue will be targeted by the re-implementing the moving in https://github.com/deltachat/deltachat-core/issues/422