Docker-mailserver: Dovecot with Full Text Search

[bityoda]

How about adding FullTextSearch support to dovecot?

Context

I'm currently trying out your great project. Beside from the fact, that i have it already running, i'm thinking about to improve it using a better fulltextsearch support.
To be more detailed, i would like to enable this within your container:
https://www.howtoforge.com/community/threads/dovecot-with-fts-lucene.76812/

My first experiences to directly modify the container has failed, which is for sure a lack of knowledge how about the thinks work together there.

Perhaps it are small and fast enhancements for you, from which everyone could improve afterwards.

Expected Behavior

Searching mails within bigger mailboxes should be fast.

[johansmitsnl]

This sounds good, what about the impact on disk and CPU when adding this?

[17Halbe]

What about fts-solar? Lucene is a bit discouraged by the dovecot team. See this note

[bityoda]

Solar uses more resources, lucene is based on c and smaller in comparison to solr. But also with solr i would be happy;-)

[17Halbe]

I haven't used either. But if we implement something, we should choose something which is still maintained, what is your experience/suggestion?

[bityoda]

My impression is that both have their right to exist within dovecot and for both someone could find use cases that makes sense together with dovecot.

Targeting this project, supporting lucence would have less impact on resource usage and changes within the docker containers. Solr runs within his own jvm, with web frontend and so on. It is probably better yes, but also much bigger.

If you ask about the future of dovecot and lucene, i'm the wrong person to ask. I also saw this remarks from the dovecot team, but i also saw still many persons using lucene.

[johansmitsnl]

I think after reading the post Solr is the way to go.
You can use https://hub.docker.com/_/solr/ for a server and make configuration to use the docker (like ldap).

[17Halbe]

What about a wiki entry if/when you got it running?

[cyrinux]

Hi, Must adapt this one, https://extremeshok.com/6622/enabling-apache-solr-4-10-using-jetty-with-dovecot-2-2-for-fulltext-search-results-on-centos-6-iredmail-compatible/

Allready tried by the past on another mail server, try well

[Thamster]

I got solr working manually with dovecot from this docker image, maybe someone can put together a pull request to incorporate into this project.

Just run a separate container to host the solr db. This one is preconfigured for dovecot-solr
docker run --name my_solr -d -p 8983:8983 -t lmmdock/dovecot-solr:latest

Then on the mail docker container simply run:

apt update && apt install dovecot-solr
cd /etc/dovecot/conf.d/

vi 10-mail.conf 
mail_plugins = fts fts_solr

vi 90-plugin.conf 
plugin {
  #setting_name = value
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search debug
}

/etc/init.d/dovecot restart
doveadm -v index -u username Inbox
curl http://localhost:8983/solr/dovecot/update?optimize=true
curl http://localhost:8983/solr/dovecot/update?commit=true
doveadm fts rescan -u username

That's it

[cyrinux]

Thx @Thamster, works very well. Should be added as a offcial feature :)

[Thamster]

Glad it worked for you too @cyrinux .

I created a fork with the required changes:
https://github.com/Thamster/docker-mailserver/commit/9e08cb45e4eac6350dbce7a9efd868800d96289e

Though I am not sure if it is the best way to fit this into the existing code,
maybe someone can suggest a better way, before I make a pull request.

[time4tea]

This worked for me:

docker-compose.yaml changes

version: '2'

services:
  mail:
    build: .
    volumes:
    - ${PWD}/dovecot-conf.d/10-mail.conf:/etc/dovecot/conf.d/10-mail.conf
    - ${PWD}/dovecot-conf.d/90-plugin.conf:/etc/dovecot/conf.d/90-plugin.conf
    links:
    - solr

  solr:
    image: solr:8.1.1
    volumes:
      - /data/solr/dovecot:/var/solr/data/dovecot
    ports:
      - "8983:8983"

Adding dovecot-solr is a non-destructive change - it could be added into the base image - it just adds a shared library.
https://packages.debian.org/en/stretch-backports/amd64/dovecot-solr/filelist

Dockerfile

FROM tvial/docker-mailserver@sha256:26403a49ed8a22135baaf8d4c326ffe21033fd8b69c2dcaf959b71ffab5ecabd

RUN apt-get update && \
    apt-get -t stretch-backports -y install --no-install-recommends dovecot-solr && \
    apt-get autoclean && \
    rm -rf /var/lib/apt/lists/* && \
    rm -rf /usr/share/locale/* && \
    rm -rf /usr/share/man/* && \
    rm -rf /usr/share/doc/*

Then you can clone the repo above, and put the files into your host machine in the right place. You'll need to delete the "data" directory, as this changes between solr versions, however will be created on demand.

[erik-wramner]

Looks promising! We could add dovecot-solr to the Dockerfile for sure. Then I think we should use an environment variable for enabling the feature and fix start-mailserver.sh so that it patches the configuration files on startup when the feature is on. That is consistent with how other features work.

I'm not sure about the doveadm and curl commands as I would guess that both the mail directory and the sol database are on persistent volumes? They probably don't hurt, but they sound expensive so it may be better to run them on demand.

The documentation needs to be updated and yaml files for docker compose added. Finally it is always nice with tests.

[lukecyca]

As a first step lets put dovecot-solr into the base image so this can be done without modifying the container. The config files can be adjusted as @time4tea has done.

Personally I have just gone into my container and made the appropriate changes, but I feel dirty for doing so. I know I'll have to redo it each time I rebuild the container.

[erik-wramner]

See #1319.

[lukecyca]

Thanks @erik-wramner!

With the new image, here is the necessary configuration that I am using:

1. docker-compose.yml:

  solr:
    image: lmmdock/dovecot-solr:latest
    restart: always

  mailserver:
    image: tvial/docker-mailserver:latest
    ...
    volumes:
      ...
      - ./etc/dovecot/conf.d/10-plugin.conf:/etc/dovecot/conf.d/10-plugin.conf:ro
    ...

1. etc/dovecot/conf.d/10-plugin.conf:

mail_plugins = fts fts_solr

plugin {
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://solr:8983/solr/dovecot/ 
}

1. Restart the mailserver container: docker-compose restart mailserver

2. And then run some commands inside docker-compose exec mailserver /bin/bash:

doveadm -v index -u username Inbox
curl http://solr:8983/solr/dovecot/update?optimize=true
curl http://solr:8983/solr/dovecot/update?commit=true
doveadm fts rescan -u username

[erik-wramner]

The previous example has a persistent volume in the solr container plus a link. You don't have either? Surely we need the volume at least or everything has to be re-indexed after down/up?

[lukecyca]

I've based it mostly on @Thamster's example above which uses lmmdock/dovecot-solr.

I'm not sure which directory within the solr container should be made a volume. I suspect /opt/solr/server/solr/dovecot/ or one of its subdirectories but I'm not 100% sure where solr keeps its state. The problem is that this directory comes pre-populated in the image, so we can't just mount a blank directory over top of it.

The link and port mapping are not necessary because I changed the url to http://solr:8983/..., although the link is probably still a good idea so that solr is started before the mailserver container.

[erik-wramner]

A blank directory won't work, but a docker volume will be initialized with whatever files are in the image when it is created and would persist between restarts.

The lmmdock image (and associated Github repo) has not changed since 2017, so perhaps it would be better to use the official image for this? It changed 23 days ago.

[aendeavor]

This seems to have stalled, which is a shame. Anyone up to this?

[lukecyca]

@aendeavor I've been running it with no issues using the steps I posted a few messages up. There is not really any further modification required to docker-mailserver to make this work.

That said, there is still an improvement that could be made by confirming which solr directory holds the necessary state, and make that a docker volume. Otherwise you must run the commands (my step 4 above) anytime you recreate the solr container.

[aendeavor]

Thanks for the update. Is there already a wiki entr for this? If not, could you create one @lukecyca?

That said, there is still an improvement that could be made by confirming which solr directory holds the necessary state, and make that a docker volume. Otherwise you must run the commands (my step 4 above) anytime you recreate the solr container.

Is a PR necessary for this? Could you provide one if so?

[lukecyca]

New and improved.....

Setup Steps

1. docker-compose.yml:

  solr:
    image: lmmdock/dovecot-solr:latest
    volumes:
     - solr-dovecot:/opt/solr/server/solr/dovecot
    restart: always

  mailserver:
    image: tvial/docker-mailserver:latest
    ...
    volumes:
      ...
      - ./etc/dovecot/conf.d/10-plugin.conf:/etc/dovecot/conf.d/10-plugin.conf:ro
    ...

volumes:
  solr-dovecot:
    driver: local

1. etc/dovecot/conf.d/10-plugin.conf:

mail_plugins = fts fts_solr

plugin {
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://solr:8983/solr/dovecot/ 
}

1. Start the solr container: docker-compose up -d --remove-orphans solr

2. Restart the mailserver container: docker-compose restart mailserver

3. Flag all user mailbox FTS indexes as invalid, so they are rescanned on demand when they are next searched

docker-compose exec mailserver doveadm fts rescan -A

Discussion

I'm pretty happy with the above. I've blown it away and recreated it several times to test, and these instructions seem stable. The solr index persists even when bouncing the solr container.

The previous example has a persistent volume in the solr container plus a link.

Persistence is fixed. I decided not to use a link, as Solr is not strictly required for docker-mailserver to start. Nor does it need to be started first.

The lmmdock image (and associated Github repo) has not changed since 2017, so perhaps it would be better to use the official image for this? It changed 23 days ago.

I looked into upgrading, however the latest official Solr 8 docker image doesn't have the dovecot schema, so I would have to build a new image (based off the official solr one) or some other non-trivial preconfiguration steps. The dovecot solr instructions still reference Solr 7, so it's not even clear what would need to be done to make use of the newer Solr 8 versions (the schema seems version-dependent, judging by its filename). I think it's best that we stick with Solr 7.3.0.

Is a PR necessary for this? Could you provide one if so?

I don't think so. If everyone is happy with these instructions, I'll move them onto a wiki page and we can close this issue.

[erik-wramner]

It sounds simple enough to me. Please create a wiki page, and we can close this.

[aendeavor]

@lukecyca have you already had time?:)

[lukecyca]

Sorry for the delay! How's this?

https://github.com/tomav/docker-mailserver/wiki/Full-text-search

[aendeavor]

Sorry for the delay! How's this?

https://github.com/tomav/docker-mailserver/wiki/Full-text-search

That's great! I'll close this since this should be resolved with this wiki entry.