Mail: Stuck at "loading messages" at 1.5.1

Created on 29 Oct 2020 · 22Comments · Source: nextcloud/mail

Expected behavior

Load Mails

Actual behavior

Stuck on Loading Messages

Mail app

Mail app version: 1.5.1 (see apps admin page, e.g. 0.5.3)

Mailserver or service: @Outlook.com (e.g. Outlook, Yahoo, Gmail, Exchange,...)

Server configuration

Operating system: Unraid 6.8.3, Docker 19.03.5 (e.g. Debian 8)

Web server: nginx (e.g. Apache, Nginx,...)

Database: PostgreSQL (e.g. MariaDB, SQLite or PostgreSQL)

PHP version: 7.3.23 (e.g. 7.0)

Nextcloud Version: 20.0.1 (see admin page, e.g. 13.0.2)

Client configuration

Browser: Chrome 86.0.4240.111 (e.g. Firefox 48)

Operating system: Windows 10 (e.g. Arch Linux)

After updating to 1.5.1, two servers (same machine, different docker) promted me to reenter mail account password. After that it stuck at "Loading Messages", They have pulled the Folder structure, if i click any folder, it displays how many mails are inside.

I tried the following:

Deleted the mail account
run occ mail:clean-up
disabled and removed app
reinstalled, added the account again
but i still get “Loading messages”.

Opened the database with pgAdmin
the account is added, but no new messages appear on the db.

Tried running occ mail:account:sync 10 -f (my current account id) and got:

[debug] account is up to date, skipping mailbox sync

In MailboxLockedException.php line 33:

331 is already being synced

After 3 hours i still get “331 is already being synced” so it doesn’t download anything…

Nextcloud log level to 0, doesn’t show anything for the mail app
On chromes network dev tab i see a tof on “sync” in the list and the preview is:

{status: “fail”,…}

data: {message: “401 is already being synced”, type: “OCA\Mail\Exception\MailboxLockedException”}

message: "401 is already being synced"

type: "OCA\Mail\Exception\MailboxLockedException"
status: “fail”

Looks like it affects other people too
https://help.nextcloud.com/t/mail-app-v1-5-1-stucks-on-loading-messages/96535/2

If you need any more info let me know

0. to triage bug

Source

stavros-k

👍4

Most helpful comment

Yes, i know, for now i'm applying the "patch" mentioned above, until LSIO updates the container

stavros-k on 18 Nov 2020

👍3

All 22 comments

as noted on the forums the locked mailbox is not an error

anyhting in your nextcloud.log that indicates an error?

ChristophWurst on 30 Oct 2020

I get the same stuck loading messages behavior.

Error log shows:

[mail] Error: OCA\Mail\Exception\ServiceException: Sync failed for 1:INBOX: Return value of OCA\Mail\Address::getEmail() must be of the type string or null, bool returned at <<closure>>

0. /var/www/html/custom_apps/mail/lib/Service/Sync/SyncService.php line 121
   OCA\Mail\Service\Sync\ImapToDbSynchronizer->sync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox {id: 15}, OC\AppFramework\ScopedPsrLogger {}, 42, [], true)
1. /var/www/html/custom_apps/mail/lib/Controller/MailboxesController.php line 161
   OCA\Mail\Service\Sync\SyncService->syncMailbox(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox {id: 15}, 42, [], false, "is:important")
2. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 169
   OCA\Mail\Controller\MailboxesController->sync(15, [], true, "is:important")
3. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 100
   OC\AppFramework\Http\Dispatcher->executeController(OCA\Mail\Control ... {}, "sync")
4. /var/www/html/lib/private/AppFramework/App.php line 152
   OC\AppFramework\Http\Dispatcher->dispatch(OCA\Mail\Control ... {}, "sync")
5. /var/www/html/lib/private/Route/Router.php line 308
   OC\AppFramework\App::main("OCA\\Mail\\Cont ... r", "sync", OC\AppFramework\ ... {}, {action: null,id ... "})
6. /var/www/html/lib/base.php line 1009
   OC\Route\Router->match("/apps/mail/api/mailboxes/15/sync")
7. /var/www/html/index.php line 37
   OC::handleRequest()

POST /apps/mail/api/mailboxes/15/sync
from xxx.xxx.xxx.xxx by user at 2020-10-30T21:23:39+00:00

and

[PHP] Error: Error: iconv(): Wrong charset, conversion from `UTF-8' to `UTF-8//IGNORE' is not allowed at /var/www/html/custom_apps/mail/lib/Address.php#70 at <<closure>>

 0. <<closure>>
    OC\Log\ErrorHandler::onError(8, "iconv(): Wrong  ... d", "/var/www/html/c ... p", 70, [])
 1. /var/www/html/custom_apps/mail/lib/Address.php line 70
    iconv("UTF-8", "UTF-8//IGNORE", "[email protected]")
 2. /var/www/html/custom_apps/mail/lib/Db/MessageMapper.php line 292
    OCA\Mail\Address->getEmail()
 3. /var/www/html/custom_apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php line 280
    OCA\Mail\Db\MessageMapper->insertBulk(OCA\Mail\Db\Message {id: null}, OCA\Mail\Db\Message {id: null}, OCA\Mail\Db\Message {id: null}, OCA\Mail\Db\Message {id: null}, ... OCA\Mail\Db\Message {id: null})
 4. /var/www/html/custom_apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php line 214
    OCA\Mail\Service\Sync\ImapToDbSynchronizer->runInitialSync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox {id: 15}, OC\AppFramework\ScopedPsrLogger {})
 5. /var/www/html/custom_apps/mail/lib/Service/Sync/SyncService.php line 121
    OCA\Mail\Service\Sync\ImapToDbSynchronizer->sync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox {id: 15}, OC\AppFramework\ScopedPsrLogger {}, 42, [], true)
 6. /var/www/html/custom_apps/mail/lib/Controller/MailboxesController.php line 161
    OCA\Mail\Service\Sync\SyncService->syncMailbox(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox {id: 15}, 42, [], false, "is:important")
 7. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 169
    OCA\Mail\Controller\MailboxesController->sync(15, [], true, "is:important")
 8. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 100
    OC\AppFramework\Http\Dispatcher->executeController(OCA\Mail\Control ... {}, "sync")
 9. /var/www/html/lib/private/AppFramework/App.php line 152
    OC\AppFramework\Http\Dispatcher->dispatch(OCA\Mail\Control ... {}, "sync")
10. /var/www/html/lib/private/Route/Router.php line 308
    OC\AppFramework\App::main("OCA\\Mail\\Cont ... r", "sync", OC\AppFramework\ ... {}, {action: null,id ... "})
11. /var/www/html/lib/base.php line 1009
    OC\Route\Router->match("/apps/mail/api/mailboxes/15/sync")
12. /var/www/html/index.php line 37
    OC::handleRequest()

POST /apps/mail/api/mailboxes/15/sync

maurerle on 30 Oct 2020

👍1

as noted on the forums the locked mailbox is not an error

anyhting in your nextcloud.log that indicates an error?

After doing more cleaning,
Deleting all oc_mail* columns
and deleting all rows specific to mail app from oc_migrations
deleting /data/appdata*/mail and /config/www/nextcloud/apps/mail
Also manualy patched this https://github.com/nextcloud/server/pull/23606 to cleanup the error log.
Reinstalled the mail app and i got the same errors as above.

On database i can see the account created but 0 messages downloaded
EDIT* the Error connecting to mail server might not be related (probably server problem?) After 2 Error connecting to mail server i got the same errors again (iconv and ServiceException))

Error: iconv(): Wrong charset, conversion from `UTF-8' to `UTF-8//IGNORE' is not allowed at /config/www/nextcloud/apps/mail/lib/Address.php#70
<<closure>>

OC\Log\ErrorHandler::onError(8, "iconv(): Wr ... d", "/config/www ... p", 70, [])

/config/www/nextcloud/apps/mail/lib/Address.php - line 70:

iconv("UTF-8", "UTF-8//IGNORE", "[email protected]")

/config/www/nextcloud/apps/mail/lib/Db/MessageMapper.php - line 292:

OCA\Mail\Address->getEmail()

/config/www/nextcloud/apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php - line 280:

OCA\Mail\Db\MessageMapper->insertBulk(OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null})

/config/www/nextcloud/apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php - line 214:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->runInitialSync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {})

/config/www/nextcloud/apps/mail/lib/Service/Sync/SyncService.php - line 121:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->sync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {}, 42, [], true)

/config/www/nextcloud/apps/mail/lib/Controller/MailboxesController.php - line 161:

OCA\Mail\Service\Sync\SyncService->syncMailbox(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, 42, [], false, "is:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 169:

OCA\Mail\Controller\MailboxesController->sync(6, [], true, "is:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 100:

OC\AppFramework\Http\Dispatcher->executeController(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/AppFramework/App.php - line 152:

OC\AppFramework\Http\Dispatcher->dispatch(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/Route/Router.php - line 308:

OC\AppFramework\App::main("OCA\\Mail\\ ... r", "sync", OC\AppFramew ... {}, { action: nu ... "})

/config/www/nextcloud/lib/base.php - line 1009:

OC\Route\Router->match("/apps/mail/ ... c")

/config/www/nextcloud/index.php - line 37:

OC::handleRequest()

OCA\Mail\Exception\ServiceException: Sync failed for 1:INBOX: Return value of OCA\Mail\Address::getEmail() must be of the type string or null, bool returned
/config/www/nextcloud/apps/mail/lib/Service/Sync/SyncService.php - line 121:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->sync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {}, 42, [], true)

/config/www/nextcloud/apps/mail/lib/Controller/MailboxesController.php - line 161:

OCA\Mail\Service\Sync\SyncService->syncMailbox(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, 42, [], false, "is:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 169:

OCA\Mail\Controller\MailboxesController->sync(6, [], true, "is:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 100:

OC\AppFramework\Http\Dispatcher->executeController(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/AppFramework/App.php - line 152:

OC\AppFramework\Http\Dispatcher->dispatch(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/Route/Router.php - line 308:

OC\AppFramework\App::main("OCA\\Mail\\ ... r", "sync", OC\AppFramew ... {}, { action: nu ... "})

/config/www/nextcloud/lib/base.php - line 1009:

OC\Route\Router->match("/apps/mail/ ... c")

/config/www/nextcloud/index.php - line 37:

OC::handleRequest()

Caused by TypeError: Return value of OCA\Mail\Address::getEmail() must be of the type string or null, bool returned
/config/www/nextcloud/apps/mail/lib/Db/MessageMapper.php - line 292:

OCA\Mail\Address->getEmail()

/config/www/nextcloud/apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php - line 280:

OCA\Mail\Db\MessageMapper->insertBulk(OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null}, OCA\Mail\Db\Message { id: null})

/config/www/nextcloud/apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php - line 214:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->runInitialSync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {})

/config/www/nextcloud/apps/mail/lib/Service/Sync/SyncService.php - line 121:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->sync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {}, 42, [], true)

/config/www/nextcloud/apps/mail/lib/Controller/MailboxesController.php - line 161:

OCA\Mail\Service\Sync\SyncService->syncMailbox(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, 42, [], false, "is:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 169:

OCA\Mail\Controller\MailboxesController->sync(6, [], true, "is:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 100:

OC\AppFramework\Http\Dispatcher->executeController(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/AppFramework/App.php - line 152:

OC\AppFramework\Http\Dispatcher->dispatch(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/Route/Router.php - line 308:

OC\AppFramework\App::main("OCA\\Mail\\ ... r", "sync", OC\AppFramew ... {}, { action: nu ... "})

/config/www/nextcloud/lib/base.php - line 1009:

OC\Route\Router->match("/apps/mail/ ... c")

/config/www/nextcloud/index.php - line 37:

OC::handleRequest()

OCA\Mail\Exception\ServiceException: Can not get messages from mailbox INBOX: Error connecting to mail server.
/config/www/nextcloud/apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php - line 214:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->runInitialSync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {})

/config/www/nextcloud/apps/mail/lib/Service/Sync/SyncService.php - line 121:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->sync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {}, 42, [], true)

/config/www/nextcloud/apps/mail/lib/Controller/MailboxesController.php - line 161:

OCA\Mail\Service\Sync\SyncService->syncMailbox(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, 42, [], false, "not:starred not:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 169:

OCA\Mail\Controller\MailboxesController->sync(6, [], true, "not:starred not:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 100:

OC\AppFramework\Http\Dispatcher->executeController(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/AppFramework/App.php - line 152:

OC\AppFramework\Http\Dispatcher->dispatch(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/Route/Router.php - line 308:

OC\AppFramework\App::main("OCA\\Mail\\ ... r", "sync", OC\AppFramew ... {}, { action: nu ... "})

/config/www/nextcloud/lib/base.php - line 1009:

OC\Route\Router->match("/apps/mail/ ... c")

/config/www/nextcloud/index.php - line 37:

OC::handleRequest()

Caused by Horde_Imap_Client_Exception: Error connecting to mail server.
/config/www/nextcloud/apps/mail/vendor/pear-pear.horde.org/Horde_Imap_Client/Horde/Imap/Client/Socket.php - line 375:

Horde_Imap_Client_Socket->_connect()

/config/www/nextcloud/apps/mail/vendor/pear-pear.horde.org/Horde_Imap_Client/Horde/Imap/Client/Base.php - line 838:

Horde_Imap_Client_Socket->_login("*** sensiti ... *")

/config/www/nextcloud/apps/mail/vendor/pear-pear.horde.org/Horde_Imap_Client/Horde/Imap/Client/Base.php - line 2110:

Horde_Imap_Client_Base->login("*** sensiti ... *")

/config/www/nextcloud/apps/mail/lib/IMAP/MessageMapper.php - line 103:

Horde_Imap_Client_Base->search("INBOX", null, { results: [ 4,3,1]})

/config/www/nextcloud/apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php - line 270:

OCA\Mail\IMAP\MessageMapper->findAll(Horde_Imap_C ... e}, "INBOX", 5000, 0)

/config/www/nextcloud/apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php - line 214:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->runInitialSync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {})

/config/www/nextcloud/apps/mail/lib/Service/Sync/SyncService.php - line 121:

OCA\Mail\Service\Sync\ImapToDbSynchronizer->sync(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, OC\AppFramew ... {}, 42, [], true)

/config/www/nextcloud/apps/mail/lib/Controller/MailboxesController.php - line 161:

OCA\Mail\Service\Sync\SyncService->syncMailbox(OCA\Mail\Account {}, OCA\Mail\Db\Mailbox { id: 6}, 42, [], false, "not:starred not:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 169:

OCA\Mail\Controller\MailboxesController->sync(6, [], true, "not:starred not:important")

/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 100:

OC\AppFramework\Http\Dispatcher->executeController(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/AppFramework/App.php - line 152:

OC\AppFramework\Http\Dispatcher->dispatch(OCA\Mail\Con ... {}, "sync")

/config/www/nextcloud/lib/private/Route/Router.php - line 308:

OC\AppFramework\App::main("OCA\\Mail\\ ... r", "sync", OC\AppFramew ... {}, { action: nu ... "})

/config/www/nextcloud/lib/base.php - line 1009:

OC\Route\Router->match("/apps/mail/ ... c")

/config/www/nextcloud/index.php - line 37:

OC::handleRequest()

stavros-k on 31 Oct 2020

After looking around found that probably php iconv has an issue on alpine images. Applied this fix
https://github.com/docker-library/php/issues/240#issuecomment-373723043 (changed php version to 7.3.23), and it started downloading messages. But after i had warning when trying to open specific folders, and an error coming up without touching the mail app.
The folders and mails are opening, even those that give warnings.

Error Module 'iconv' already loaded at Unknown#0
Waring KItinerary binary adapter is not available, can't extract information

The error seems right, since i have installed another iconv version/package. But i dont know what to look next.

I Don't know if this help pin point my problem, i'd call myself a newbie on linux and php. But i figured i should post my findings.

stavros-k on 1 Nov 2020

👀1

OCA\Mail\Exception\ServiceException: Sync failed for 1:INBOX: Return value of OCA\Mail\Address::getEmail() must be of the type string or null, bool returned
/config/www/nextcloud/apps/mail/lib/Service/Sync/SyncService.php - line 121:

Yeah it looks like an issue with the iconv conversion indeed. On the other hand iconv can legitimately return false if the conversion fails. @rullzer a case we didn't consider in your PR ;)

ChristophWurst on 3 Nov 2020

iconv("UTF-8", "UTF-8//IGNORE", "[email protected]")

@stavros-k is that the original email that cause this? Because then I'm quite confused and it might indeed be the alpine issue you posted.

ChristophWurst on 3 Nov 2020

@ChristophWurst I had "[email protected]" in the error message (yes people are still using delphi).
Also running on 20-fpm-alpine. No issues with 19-fpm-alpine.
Can I savely try the docker:20-fpm without alpine?

maurerle on 3 Nov 2020

I'm not familiar with the docker setups, so I can't give any recommendations on that, sorry.

ChristophWurst on 3 Nov 2020

I just tried and the standard user seems to be different between 20-fpm-alpine and 20-fpm.

I added
user: "82:33"
to the nextcloud docker-container in the compose file.
Also had to add the userid to the cron execution.

But now it is working and fetching mail.

maurerle on 3 Nov 2020

But now it is working and fetching mail.

Cool. I'm concluding the iconv is the only issue of this specific problem, but this is out of scope for us and needs to be fixed externally.

ChristophWurst on 3 Nov 2020

:clown_face: https://stackoverflow.com/a/63747282/2239067

ChristophWurst on 5 Nov 2020

I fixed it with the following script. There might be easier way out there. But this script i found somewhere (i dont really remember sorry) fixed it.

PHP_VER=$(php -v | grep ^PHP | cut -d' ' -f2) && \
BUILD_PACKAGES="wget build-base php7-dev" && \

apk add --no-cache --virtual .php-build-dependencies $BUILD_PACKAGES && \
apk add --no-cache --repository https://dl-3.alpinelinux.org/alpine/edge/testing/ gnu-libiconv-dev && \
(mv /usr/bin/gnu-iconv /usr/bin/iconv; mv /usr/include/gnu-libiconv/*.h /usr/include; rm -rf /usr/include/gnu-libiconv) && \
mkdir -p /opt && \
cd /opt && \
wget https://secure.php.net/distributions/php-$PHP_VER.tar.gz && \
tar xzf php-$PHP_VER.tar.gz && \
cd php-$PHP_VER/ext/iconv && \
phpize && \
./configure --with-iconv=/usr && \
make && \
make install && \
mkdir -p /etc/php7/conf.d && \
#next command not needed in LSIO Docker
#echo "extension=iconv.so" >> /etc/php7/conf.d/iconv.ini && \

apk del .php-build-dependencies && \
rm -rf /opt/*

stavros-k on 5 Nov 2020

I had the same issue and I was able to trace it back to encoding problems as well, however, mainly in nextcloud and the database it uses. For me , changing the format from utf8 to utf8mb4 solved it. I noticed that running php -f occ mail:account:sync 1 always failed at some messages (assume accountid is 1):

Sync failed for 1:INBOX: An exception occurred while executing 'INSERT INTO oc_mail_messages (uid, message_id, mailbox_id, subject, sent_at, flag_answered, flag_deleted, flag_draft, flag_flagged, flag_seen, flag_forwarded, flag_junk, flag_notjunk,flag_important) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' with params [27566, "<redacted by me>", 110, "\u00fc \u00fc \ud83d\udc9a", 1600446988, false, false, false, false, true, false, false, false, false]: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\x92\x9A' for column 'subject' at row 1

(I removed the unimportant parts of the subject).

So I googled a bit and found how you can enable mysql4 byte support in nextcloud.

For my ~~postgresql~~ mysql database it boiled down to steps 4,5 and 6 in the mysql subsection:

Change your databases character set and collation:
ALTER DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
Set the mysql.utf8mb4 config to true in your config.php:
$ sudo -u www-data php occ config:system:set mysql.utf8mb4 --type boolean --value="true"
Convert all existing tables to the new collation by running the repair step:
$ sudo -u www-data php occ maintenance:repair

After that, the messages started showing up.

johanneshorak on 18 Nov 2020

Good finding @johanneshorak. MySQL 4Byte support is indeed required. Yet it's a different problem than the iconv one. But thanks for the info.

For my postgresql database it boiled down to steps 4,5 and 6 in the mysql subsection:

Postgres? Are you sure?

ChristophWurst on 18 Nov 2020

@ChristophWurst Thank you!

About postgres: My mistake, it's a mysql database of course. I'll correct it in the post above to avoid further confusion.

johanneshorak on 18 Nov 2020

👍1

I had the same issue and I was able to trace it back to encoding problems as well, however, mainly in nextcloud and the database it uses. For me , changing the format from utf8 to utf8mb4 solved it. I noticed that running php -f occ mail:account:sync 1 always failed at some messages (assume accountid is 1):

Sync failed for 1:INBOX: An exception occurred while executing 'INSERT INTO oc_mail_messages (uid, message_id, mailbox_id, subject, sent_at, flag_answered, flag_deleted, flag_draft, flag_flagged, flag_seen, flag_forwarded, flag_junk, flag_notjunk,flag_important) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' with params [27566, "<redacted by me>", 110, "\u00fc \u00fc \ud83d\udc9a", 1600446988, false, false, false, false, true, false, false, false, false]: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\x92\x9A' for column 'subject' at row 1

(I removed the unimportant parts of the subject).

So I googled a bit and found how you can enable mysql4 byte support in nextcloud.

For my ~postgresql~ mysql database it boiled down to steps 4,5 and 6 in the mysql subsection:

Change your databases character set and collation:
ALTER DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

Set the mysql.utf8mb4 config to true in your config.php:
$ sudo -u www-data php occ config:system:set mysql.utf8mb4 --type boolean --value="true"

Convert all existing tables to the new collation by running the repair step:
$ sudo -u www-data php occ maintenance:repair

After that, the messages started showing up.

Im using postgre, so i dont think this is the problem! It's how the iconv in the alpine image i'm using is compiled.
It's already discussed on LSIO github and i think they have addressed a fix or sometihng.

stavros-k on 18 Nov 2020

It's how the iconv in the alpine image i'm using is compiled.

Indeed. But this is nothing we can fix inside our app.

ChristophWurst on 18 Nov 2020

Yes, i know, for now i'm applying the "patch" mentioned above, until LSIO updates the container

stavros-k on 18 Nov 2020

👍3

Hello, I'm running nextcloud in an Alpine Linux LXC container and I'm hit by this issue too. Looking at the Alpine bug, it seems to be linked to the fact that Alpine use musl and busybox which have their own implementation of iconv. The posix manpage for iconv_open doesn't mention the //IGNORE extension provided in the gnu libc iconv, so it seems this is not really a bug in Alpine neither.

Assuming the most common case is the email address being valid utf-8, you could first do a conversion to UTF-8 and check the return value to see if it is indeed valid utf-8. If it returns false you could then fallback on using UTF-8//IGNORE to try to fix things up, but I wonder why you would accept any input in that case. It means the address is somehow borked so it is going to be wrong anyway.

canatella on 18 Jan 2021

Assuming the most common case is the email address being valid utf-8, you could first do a conversion to UTF-8 and check the return value to see if it is indeed valid utf-8. If it returns false you could then fallback on using UTF-8//IGNORE to try to fix things up, but I wonder why you would accept any input in that case. It means the address is somehow borked so it is going to be wrong anyway.

That's basically why that statement is there. How can we check the return value to see if it is indeed valid utf-8?

ChristophWurst on 18 Jan 2021

Assuming the most common case is the email address being valid utf-8, you could first do a conversion to UTF-8 and check the return value to see if it is indeed valid utf-8. If it returns false you could then fallback on using UTF-8//IGNORE to try to fix things up, but I wonder why you would accept any input in that case. It means the address is somehow borked so it is going to be wrong anyway.

That's basically why that statement is there. How can we _check the return value to see if it is indeed valid utf-8_?

From my understanding and looking at man 3 iconv and the musl code: https://git.musl-libc.org/cgit/musl/tree/src/locale/iconv.c#n257, is that if you run with UTF-8 it will fail when stumbling upon an invalid input byte sequence:

The conversion can stop for four reasons:

       1. An invalid multibyte sequence is encountered in the input.  In this 
          case, it sets errno to EILSEQ and returns  (size_t) -1.   *inbuf
          is left pointing to the beginning of the invalid multibyte sequence.

From man 3 iconv_open, if you pass //IGNORE:

       //IGNORE
              When  the  string  "//IGNORE"  is  appended to tocode, characters
              that cannot be represented in the target character set will be
              silently discarded.

So using UTF-8 would make it fail on invalid utf-8, using UTF-8//IGNORE would make ignore and drop the invalid utf-8 bytes.

canatella on 18 Jan 2021

That said, I'm looking at the current code and it seems that the error is handled now: https://github.com/nextcloud/mail/blob/51504e3ead8788a13710bf93ed47186ddc3ace63/lib/Address.php#L81 . I still think that the //IGNORE serves no purpose here if what's wanted is just to ensure that the address is valid utf-8. The //IGNORE would make it ignore those case...

EDIT: And the current code will still fail on Alpine, so it I think the //IGNORE should really be removed here.

canatella on 18 Jan 2021

Was this page helpful?

0 / 5 - 0 ratings