Mailu: Admin: On UPDATE email field is never refreshed

[rnixx]

Hi,

we just installed mailu 1.5 via docker compose referring to the install docs.

We added the initial admin user with docker-compose run --rm admin python manage.py admin root example.net password.

Further we created a domain and some users and adopted the DNS records.

Sending to the created users from outside works, as does sending with one of those users from webmail.

When trying to send mail from admin interface (auto responder, global notification, welcome message), the process always tries to send with the initially created user, and postfix complains about a missing password.

in .env the RELAYNETS is set to the network reported by docker.

Any hints what we are missing?

Thank you!

[kaiyou]

Nope, but this has already been reported, I'll investigate since it seems to work on my side of things.

[jensens]

638 is probably related to this

[muhlemmer]

Root

This is reproducible when setting the admin user as [email protected] for the first admin. Root username is proposed by the documentation:

docker-compose run --rm admin python manage.py admin root example.net password

Trace from log

Announcement tries to send as [email protected], but fails as the user does not exist:

smtplib.SMTPRecipientsRefused: {'[email protected]': (550, b'5.1.0 <[email protected]>: Sender address rejected: User unknown in virtual mailbox table')}
188.27.138.94 - - [10/Oct/2018:13:00:45 +0000] "POST /ui/announcement HTTP/1.0" 500 291 "https://xxxxx.email/admin/ui/announcement" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"

Tested conditions

Tested and reproduced on 1.5 and feat-abstract-db.

• Setting the first admin account as "admin" and this bug does not expose
• Setting the first admin account to any other name, this bug exposes
• Creating the admin user on a later moment does not work around this bug; the interface still attempts to send authenticate as [email protected], while sender is set to [email protected].

Note: Is not related to #638, as it works properly in feat-abstract-db.

[jensens]

I can confirm that in our logs the [email protected] appears after the smtplib.SMTPRecipientsRefused.

(fyi, I set Mailu up together with @rnixx yesterday)

[muhlemmer]

As a workaround:

• You can try the query the /mailu/data/main.db and modify the name of root to admin, using sqlite. I never did this and don't know the layout of the database.
• If the server is not yet in production:

docker-compose down
rm -rfv /mailu
docker compose up -d
docker-compose run --rm admin python manage.py admin admin example.net password

Or wait until this is fixed.

[jensens]

If I cd into my docker volume and do a

sqlite3 main.db "update user set localpart='admin' where localpart='root';"

then there is no mention of root in not part of the database left.
Anyway, using the notification message system I stilll got

smtplib.SMTPRecipientsRefused: {'[email protected]': (553, b'5.7.1 <[email protected]>: Sender address rejected: not logged in')}

Looking into the code, the sender address localpart is the configured app.config['postmaster']:
https://github.com/Mailu/Mailu/blob/eb4cbc0f81db767f95be15d78cfcd4dfc6587cc6/core/admin/mailu/models.py#L210

This defaults to postmaster:
https://github.com/Mailu/Mailu/blob/eb4cbc0f81db767f95be15d78cfcd4dfc6587cc6/core/admin/mailu/__init__.py#L39

So, the root has to be in the os.environ:
https://github.com/Mailu/Mailu/blob/eb4cbc0f81db767f95be15d78cfcd4dfc6587cc6/core/admin/mailu/__init__.py#L70

But in my .env of docker-compose, there is the setting POSTMASTER=postmaster.
Where does it get the override from?

[jensens]

Btw., I just double-checked, no changes between 1.5 (which I run) and master (links above) on the above parts.

[muhlemmer]

In my installation, which works, POSTMASTER=admin, which is the same as the first account I created: admin.

I believe those values need to be the same. Can you test this? Note: docker-compose restart required when changing.

[jensens]

@muhlemmer well, I would really like to find the cause of this instead of poking in the dark.

Somewhere the initial user value seems to be stored and exposed to os.environ at some point. Its not in the sqlite database, which I checked.

In the code and configuration there is also no obvious hint, that the name admin or root is special in any way. In our case postmaster is the name we now want to have.

One last resort would be some magic in manage.py, but looking at it it is a very straight forward adding of up to two records in the database here:
https://github.com/Mailu/Mailu/blob/790e32cfb865f500c8a43a81c9fbd3a355bb48ed/core/admin/manage.py#L26-L41

All the code is very clean and readable. I have to call it a day now, but if nobody here has a better idea I try later or tomorrow a bit live debugging the flask app.

[jensens]

Anyway, I had to check: using a custom logger line in my container before
https://github.com/Mailu/Mailu/blob/eb4cbc0f81db767f95be15d78cfcd4dfc6587cc6/core/admin/mailu/models.py#L210
I really got the configured postmaster. So it must be something at a different place. Now it gets interesting.

[kaiyou]

I believe there are two issues, with two different error messages:

• if the sender address does not exist, Postfix complains that the sender does not exist
• if it does exist, Postfix complains the user is not authenticated

The weird part is that smtplib uses smtp as the default host (overwritable via env variable), on port 10025. The SMTP on that port is not supposed to enforce authentication or check that the sender address exists since it is only supposed to be accessed by authenticated users and internal services.

[jensens]

FTR, I set did a smtp.set_debuglevel(1) as first instruction in the with block. Here the protocol:

send: 'ehlo [192.168.96.8]\r\n'
reply: b'250-ourdomain.com\r\n'
reply: b'250-PIPELINING\r\n'
reply: b'250-SIZE 50000000\r\n'
reply: b'250-VRFY\r\n'
reply: b'250-ETRN\r\n'
reply: b'250-AUTH PLAIN LOGIN\r\n'
reply: b'250-ENHANCEDSTATUSCODES\r\n'
reply: b'250-8BITMIME\r\n'
reply: b'250 DSN\r\n'
reply: retcode (250); Msg: b'ourdomain.com\nPIPELINING\nSIZE 50000000\nVRFY\nETRN\nAUTH PLAIN LOGIN\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
send: 'mail FROM:<[email protected]> size=206\r\n'
reply: b'250 2.1.0 Ok\r\n'
reply: retcode (250); Msg: b'2.1.0 Ok'
send: 'rcpt TO:<[email protected]>\r\n'
reply: b'553 5.7.1 <[email protected]>: Sender address rejected: not logged in\r\n'
reply: retcode (553); Msg: b'5.7.1 <[email protected]>: Sender address rejected: not logged in'
send: 'rset\r\n'
reply: b'250 2.0.0 Ok\r\n'
reply: retcode (250); Msg: b'2.0.0 Ok'
send: 'QUIT\r\n'
reply: b'221 2.0.0 Bye\r\n'
reply: retcode (221); Msg: b'2.0.0 Bye'

First,
in my .env I set RELAYNETS=192.168.96.0/20 which matches the IP-range smtp is in, if this matters here (I am good at Python but my smtp is a bit rusty).

Second,
I am still baffeld that there is a [email protected] left somewhere! In the UI it is nowhere, I really double checked all screens. So I again opened the DB using sqlite3 command-line tool. After a SELECT * FROM user; I found it in a field as a whole email address for user with localpart admin. A PRAGMA table_info([user]); told me its the field email. I again opened the screen at https://ourdomain.com/admin/ui/user/edit/admin%40ourdomain.com - no email field visible.

So, to check if this is the problem, in sqlite3 I executed UPDATE user SET email="[email protected]" WHERE localpart="admin";

After this, the chat protocol with the smtp does not contain any traces of the [email protected] initial admin user any more.

So back to Python code looking where the email field is populated with data. It happens as a default value at object creation time here:
https://github.com/Mailu/Mailu/blob/790e32cfb865f500c8a43a81c9fbd3a355bb48ed/core/admin/mailu/models.py#L151-L157

But then, at the settings form it never got updated again:
https://github.com/Mailu/Mailu/blob/790e32cfb865f500c8a43a81c9fbd3a355bb48ed/core/admin/mailu/ui/views/users.py#L90-L101

This is bug number one.

Now, even when I do docker-compose restart (to avoid caching issues), I still can not send mail!

So, next bug at the next reply here. Stay tuned.

[jensens]

Manually applying https://github.com/Mailu/Mailu/commit/1b0b3a2b1e055200545143b697baae7b73b86aba in the running container and a postfix reload fixes the authentication issue.

So, we are using the release container 1.5 of https://hub.docker.com/r/mailu/postfix/
Now, we found there is a kind of 1.5.1 release which should contain the above mentioned fix, but it's not visible at Docker Hub.

Sorry, but the Release Management process is not very transparent. May you please explain how we can get the fix in?

[jensens]

Ok, wrong. The 1.5.1 ZIP file does not contain the fix. Just checked... only master has it.

[muhlemmer]

Sorry, but the Release Management process is not very transparent. May you please explain how we can get the fix in?

1.5.1 is a tag, which is a static thing on GitHub. We've continued with some bugfixes on the 1.5 branch (dynamic) after the tag. This has led to a lot of confusion, we know. It will be clarified in the FAQ (in progress) also.

[jensens]

More confusing than the tags on the 1.5 branch is, there is no 1.5.1 on docker hub. I would expect a 1.5.0 and a 1.5.1 tag there, and also a 1.5 tag always pointing to latest 1.5.x - this is how it works in 99% of the projects.

[jensens]

Well, but the fix is also not at the 1.5-branch.
What about using latest from Docker Hub at master? Is it a nightly build or how is it created?

[rnixx]

@muhlemmer how stable is master? is there a planned release date for 1.6 or whatever comes next?

[rnixx]

@muhlemmer we (@jensens and me) have some options now. 1.) Upgrade to latest, 2.) wait for a fix in 1.5 or 3.) create our own internal build pipeline. We'd prefer option one or two.

[muhlemmer]

@jensens VERSION=master in .env.
@rnixx Define your definition of stable. At this moment we some automated building and have a double review mechanism on pull requests. Review means live testing any changes made.

So in principle master is functional and not suppose to break at any moment. But since a review by 2 persons does not expose all use cases, something might go tits up.

As for the release date, we want to finish writing the setup utility #343. When that's done we can look into releasing.

[jensens]

I created a back porting Pull Request #658.

[kaiyou]

Backporting sounds sensible but we have to run some thorough tests before we change the postfix config. I hope tonight.

[kaiyou]

This should be fixed now

[jensens]

Have way, the issue described in https://github.com/Mailu/Mailu/issues/643#issuecomment-428912467 is still open.

[muhlemmer]

Maybe it's better to open a separate issue for that?

[jensens]

As you like. At first we were not aware that this are actually two issues.

[kaiyou]

@jensens there is indeed still one of the issues to be dealt with. I am working on it as we speak.

[kaiyou]

Okay, I was digging way too deep with this and came back to basics. The only plausible cause for your bug is simply that the postmaster from configuration does not match any existing address in your database. Could you try adding a root address to the database if your postmaster is set to root?

[jensens]

@kaiyou The problem is not related to the name root at all, as I hunted it down (see also my comment and analysis above).

I described it above, but since there is so much mixed up together, here the comprehension of the bug left:

In the DB-model there is an email field defined.
On INSERT it is filled automatically with a concatenation of localpart, @, and domain.
https://github.com/Mailu/Mailu/blob/790e32cfb865f500c8a43a81c9fbd3a355bb48ed/core/admin/mailu/models.py#L151-L157

OnUPDATE, email field is never refreshed. So, when the localpart or domain changes, the email field still contains the old, initial value. This is because at form.populate_obj(user), form has no email value and has no reason to touch the field, which already contains a value.
https://github.com/Mailu/Mailu/blob/790e32cfb865f500c8a43a81c9fbd3a355bb48ed/core/admin/mailu/models.py#L151-L157
Here either a form post-processing needs to be added or the SQLAlchemy model needs a modification to pick up the changes.

To check this, create a any user.
Change its localpart.
Login with this user.
Use one of the admin-features which sends out an email (auto-responder, notification).
Boom.

This is because postfix container looks at the email field of the DB.

[muhlemmer]

To check this, create a any user.
Change its localpart.
Login with this user.

I don't fully understand. Where is it you are updating / changing the user's email? In the UI the input is disabled:

Furthermore, changing a user's e-mail would also mean moving the mailbox, as mailboxes are arranged per username. Currently there is not such functionality implemented.

OnUPDATE, email field is never refreshed. So, when the localpart or domain changes, the email field still contains the old, initial value. This is because at form.populate_obj(user), form has no email value and has no reason to touch the field, which already contains a value.

So as far as I know, localpart or domain are never supposed to change. Please note that my earlier suggestion to rename "root" to "admin" was a troubleshooting effort, which is not a normal mode of operation.

[muhlemmer]

Voting to close, unless someone is able to explain what needs fixing or sends a PR.

[jensens]

Well, I think I described the problem well in https://github.com/Mailu/Mailu/issues/643#issuecomment-431786542 - and I would love to fix it, but I am not deep enough in the way of handling models and defaults to fix it w/o diving a lot deeper into the topic. So -1 for closing just because for the sake of closing from my side.

[jensens]

Well, you say it is not possible to edit the field. I quickly checked this and it is true. I am pretty sure I ran into the problem and just try to figure out how I managed to change the e-mail address.

[muhlemmer]

The way I understand in that comment: we should change/fix the python code and send a UPDATE to the DB. But that would disconnect the renamed user from the underlying mailbox, as that is mapped based on the email address. So he'll end up in a new and empty mailbox. In that case you might as well just create a new user with the intended name and delete the old one?

So I don't see how the current situation is a bug and therefore I don't see it necessary to keep it open. We can however label it as an feature request: "The ability to rename user email".

[kaiyou]

I really do not get the fact that the email address should be updated when the local part is updated. The local part should never be updated.

[jensens]

Well, maybe I run in some kind of edge condition. I think you can safely close the issue unless nobody else comes up with it.

[Nebukadneza]

hi,
@jensens , maybe you can explain the actual use-case for this. When the localpart is the user-defining part of a account, and can be mapped via aliases too — what is the exact end-user use-case for updating the localpart and taking all existing email (albeit with then-incorrect headers) with him.
If there’s a clear usecase, all the mailbox handling ~could~ be done, technically, I guess(?).

[jensens]

So, the field is not editable for me here. I tried to run into the same situation, but it is not possible. I have no idea how it was possible for me to change the value.

[kaiyou]

Maybe you did edit it in database? I am closing this issue, but feel free to reopen if you face this again.