Mail: Large inbox folder causes the vanished ID check to send too large of a UID SEARCH command.

[MJDSys]

Expected behavior

My background sync to happen normally without exceptions. Also a normal (non-forced) sync using the occ command to complete normally.

Actual behavior

During the normal sync I get an exception thrown: Sync failed for 1:INBOX: IMAP error reported by server.. Enabling debugging and digging deeper, it appears during the deleted message partial sync, some ~20000 UIDS are checked for existence. Horde sends the complete list to my IMAP server, which responds with an error saying argument too long (error from horde_log: S: 13 BAD Error in IMAP command UID SEARCH: Too long argument (0.001 + 0.001 + 0.001 secs).). This causes an exception to be thrown, cancelling my sync.

Mail app

Mail app version: (see apps admin page, e.g. 0.5.3)
1.3.4
Mailserver or service: (e.g. Outlook, Yahoo, Gmail, Exchange,...)
Custom setup using Dovecot 2.3.4.1 from Debian.

Server configuration

Operating system: (e.g. Debian 8)
Debian 10
Web server: (e.g. Apache, Nginx,...)
Nginx
Database: (e.g. MariaDB, SQLite or PostgreSQL)
PostgreSQL
PHP version: (e.g. 7.0)
7.3
Nextcloud Version: (see admin page, e.g. 13.0.2)
18.0.4

Client configuration

Browser: (e.g. Firefox 48)
Command line
Operating system: (e.g. Arch Linux)
Debian 10

I believe the issue is with the vanished UID check using the full list of UIDs from my mailbox. I think it would be better to chunk the list for each request. I can try to whip up a PR if this solution is wanted. A quick workaround I tested was to slice the UID list to 2000 elements to test, and that did allow the sync to progress normally (I realize of course that breaks the deleted mail check).

[doc75]

Seems pretty similar to #3024 (even if error message raised by horde is slightly different)

[MJDSys]

@doc75 Maybe? I'll try to whip up the PR (need to get my PHP dev environment setup again), maybe it will help fix several of these issues. #2929 Has a similar comment as well.

Please close this issue if there is a similar you rather track this bug with. I thought I managed to check the issues with my searching.

[doc75]

@MJDSys as there is a doubt I propose to let them open. I'll be able to test the PR to check if it fixes the other issue.

[MJDSys]

I dug further into this, and what I thought as an original idea I'm not sure will work any more. I originally was going to split the uids sent into check for vanished uids, and then process over the chunks updating the database. However, it appears Horde expects the uids to be a complete list, so I'm not sure chunking this data before calling Horde is a reasonable option at this point. It might work now, but I worry this is a trap waiting for this app if Horde every changes.

As an alternate, I could reimplement the same check myself inside the app, and chunk the UIDs. This would complicate the code, but would avoid breakage if Horde changes its detection routines.

Alternately, there is support in Horde for using the new QRESYNC extension, which would reduce both the communication overhead and fix this issue. However, Horde requires a Horde_Cache object to use this extension, which would make this a bigger project.

Since those alternatives are not easily achieved, one note for people hitting this now (at least with dovecot) is to increase the maximum size of the allowed command. Unfortunately that doesn't help you, @doc75.

If there is consensus on a way forward, I can take a stab at implementing it now.

[doc75]

@MJDSys could be interesting to point out the dovecot setting to unblock the situation for others (I am not a specialist, but was not able to find the setting you mention easily). Too bad it does not help me ;-)

[MJDSys]

@doc75 Good point.

The temporary fix for dovecot is to increase the imap_max_line_length parameter in the configuration file. For my setup with a ~24000 email inbox, I've increased the parameter to 1M and that seems to work, YMMV.

[ChristophWurst]

Hey :)

Thanks a lot for raising this issue and the work you did on debugging this. This is a scenario I had not run into before but I totally see that a mail server might have limits on the command size.

Now, ideally this should be fixed in Horde. In my understanding of how IMAP sync works it should not really matter if you do the vanished UIDs check in chunks.

Given that Horde is still being used by many people I assume this won't be the first time someone ran into this. There will be a bug report or mailing list entry somewhere and we should probably check that before we go our own way.

Alternately, there is support in Horde for using the new QRESYNC extension, which would reduce both the communication overhead and fix this issue. However, Horde requires a Horde_Cache object to use this extension, which would make this a bigger project.

As with any IMAP capability this will fix the problem only for some setups. We do have a Horde_Cache already that connects to Nextcloud's cache service.

[ChristophWurst]

@MJDSys you're also very welcome to join the dev discussion at https://cloud.nextcloud.com/call/89474m7g :)

[ChristophWurst]

As an alternate, I could reimplement the same check myself inside the app, and chunk the UIDs. This would complicate the code, but would avoid breakage if Horde changes its detection routines.

I don't know if you had the same in mind but we could do that at https://github.com/nextcloud/mail/blob/f4dd491c33d257f4e213848c517e51123e76f096/lib/IMAP/Sync/Synchronizer.php#L60-L63.

[ovv]

We are hitting https://tools.ietf.org/html/rfc2683#section-3.2.1.5 , they mention using ranges when making queries.

[ChristophWurst]

Yes, good point. We could try to compress the list with ranges. But in the worst case this does not get amount of data down much. Let me see what I can do …

[MJDSys]

@ovv @ChristophWurst Horde is already trying to compress the list with ranges. My mailbox at least is relatively sparse, so there is only so much Horde can do in this case. I think it's trying to avoid non-existent uids in this case, I'm not exactly why though.

[ChristophWurst]

It does? Damn, just implemented my own version of the compression :grimacing:

[ChristophWurst]

See https://github.com/nextcloud/mail/pull/3103

[ChristophWurst]

I think it's trying to avoid non-existent uids in this case, I'm not exactly why though.

Because it would return *message X vanished" if X did not exist before but was added to fill a gap

[ChristophWurst]

@ovv @ChristophWurst Horde is already trying to compress the list with ranges.

Could you point me to that code, please?

[MJDSys]

@ovv @ChristophWurst Horde is already trying to compress the list with ranges.

Could you point me to that code, please?

I don't have the code, but I saw in the horde_imap.log file. A sample looks like this:

C: 23 UID SEARCH RETURN (ALL COUNT) UID 1,3:5,7,10,12,14:15,18:19,23:24,26:27,29:31

This is from one of the long UID SEARCH lines, but truncated. It does have the range optimization in it though.

I think it's trying to avoid non-existent uids in this case, I'm not exactly why though.

Because it would return *message X vanished" if X did not exist before but was added to fill a gap

I assumed so, but I didn't want to assume that in case something else interesting is going on in Horde.