Mailcow-dockerized: SoGO Search (with SOLR) not working correctly

[phipag]

Hello,

I am running a mailcow installation with ~200 mailboxes. Some users complain that their Sogo search does not work anymore (it always shows "no results found"). When I repair the indeces in Solr with docker-compose exec dovecot-mailcow doveadm fts rescan -A it works again but stops working after a few hours.

Mailcow is running on a qemu vm with 12GB ram.

[andryyy]

Solr could be crashed. But nevertheless, we cannot help without logs from Dovecot and Solr (when users search and fail to find something).

[extremeshok]

You will need to set the SOLR_HEAP to more than 4096 in the .env (mailcow.conf)

SOLR requires a HEAP size of 2GB per every 1 million objects.

12GB ram is super low for a SOLR+SOGO and 200+users..

[phipag]

Okay, thank you already for the information. I will provide logs asap.

[chriscroome]

I have a Mailcow server with 246 domains and 1102 mailboxes, it has 36GB of RAM and I found that when SOLR was added to Mailcow the PHP container no longer worked, so I disabled it, would it be possible for the SOLR HEAP size to be dynamically generated? I'm willing to experiment with re-enabling SOLR on this server at a not very busy time.

[andryyy]

PHP is not affected, but it might have run oom when Solr ate all the RAM.

You will need to find the fitting heap size for your setup.

[extremeshok]

@chriscroome im suprised SOGO is working with activesync.
According to SOGO dev, it requires a minimum of 30mbyte ram per a mailbox, that excludes space for the actual object which is being transfered, ie +128mbyte per a connection

Wow, cant believe how ram limited your setups are and here I was complaining with 2 servers of 16cores and 256GB ram

[chriscroome]

@extremeshok I have never use ActiveSync personally, the only MUA I use is Mutt... so I'm not sure if it is working or not, how could I find out?

[chriscroome]

I wasn't waiting long enough for the PHP container to come up, with SOLR enabled it seems to take longer.

[chriscroome]

I have started to receive complaints that emails are not returned in Solr search results and I have been able to confirm this, the Mailcow instance has 48GB of RAM, SOLR_HEAP is set to 16384, the admin interface reports:

I have been receiving all GitHub issues for Mailcow as email for a while, in Mutt if I limit the INBOX display to only email with the word "André" I have about 10 screens of emails, when I use the Solr search interface I have 8 messages:

This is following rebuilding the index earlier today:

docker-compose exec dovecot-mailcow doveadm index -A '*'

Any suggestions for any further debugging I could do to try to find out why the Solr index appears to have so many emails missing from it?

[chriscroome]

After editing mailcow.conf to edit the SKIP_SOLR to set it to y and running docker-compose down and docker-compose up -d a search on my INBOX for "André" returns 4336 results and it only took a few seconds:

With Solr enabled 8 results are returned.

[andryyy]

Perhaps it doesn't like "é". You could try a reindex.

[chriscroome]

I had similar results without accented characters and I did do two reindexes before deciding to disable Solr and the only errors thrown up with the last reindex were the errors I posted to issue #2492.

[andryyy]

Ok, don't know, cannot replicate and enotime.

[chriscroome]

No problem, the search is fast enough without Solr and the resource usage is better so I don't see any drawbacks from disabling it. Perhaps leave this issue open for now in case anyone else is able to replicate this issue?

[hachre]

Try running both of these:

docker-compose exec dovecot-mailcow doveadm fts rescan -A
docker-compose exec dovecot-mailcow doveadm -v index -A '*'

This does a full rescan and has solved problems for me in the past which weren't solved by running just index alone.

[chriscroome]

Thanks for the suggestion @hachre, I have set that running.

Is there a way of knowing if the rescan fails or generates any errors?

[hachre]

No, I haven't found a way personally. I needed to do it after the recent Solr-related updates but haven't done it since and everything is working fine right now.

[chriscroome]

Thanks @hachre, your suggestion:

Try running both of these:

docker-compose exec dovecot-mailcow doveadm fts rescan -A
docker-compose exec dovecot-mailcow doveadm -v index -A '*'

This solved the problem, the search for "André" now returns 4346 messages, I guess the initial indexing, when Solr was first added, to Mailcow must have failed and this was the cause of the incomplete results.

[hachre]

Very nice! 👍