Mailu: Catch-All Alias

[greshake]

How to create a catchall alias from the admin interface? '*' doesn't work, and neither does an empty string. Is this even a feature here? If not, why, and if it is, how do I set it up?

[kaiyou]

Catchall aliases or wildcard aliases are not supported yet.

Why? Mostly because we designed Freeposte.io with the strict minimum set of features in order to release early and we have been implementing missing features since.

This should be simple enough to implement. One way (option A) to go would be to leverage the power of SQL LIKE statements (https://www.sqlite.org/lang_expr.html#like). One could create an alias named % for catching all emails or admin-% for catching all emails sent to addresses that start with admin-.

The only issue is that % is a valid character in an email address. Also, _ (underscore) is wildly used in email addresses and would be interpreted as _any single character_.

With the proper documentation, it would be possible to replace those two with * and ? respectively, which are almost unused in local parts, would match your request exactly (an alias named * would indeed catch all emails) and would not bother users too much. We would then have to implement :

• an SQLAlchemy type adaptor for wildcard-enabled local parts (escape % and _, replace * and ? back and forth) ;
• LIKE statements in the Postfix configuration to handle aliases properly ;
• the proper Javascript snippet to warn users who are using * and ? in their statement that they are enabling the wildcard feature, maybe with a link to documentation.

Another way (option B) to go is to offer simple catchall aliases with a separate implementation and stored directly in the domain table. I personnally do not like this solution.

The last way (option C) that I can imagine would be to offer a specific name for catchall aliases and simply match it in Postfix (like a single * for catching all emails). But as this would still require both documentation and modificiations in the Postfix SQL queries, I prefer to implement the full power version once and for all.

What do you think? I do not really like option B, but options A and C both sound fine. You are welcome to pull request if you have some time to implement one of them (or any other way you think of that seems fit), otherwise I will proabably implement option A by the end of the week (I also find myself in the need for wildcard aliases!)

[kaiyou]

Also this is a standard enough feature to include in milestone 1.2 I believe, released as stable in a month or so.

[greshake]

SQL Like is possible, although Regexes would be more standard and elegant. I'd suggest to add a checkbox when entering account/alias names (disabled by default) to switch between SQL and regular interpretation. This way, users wanting to catch all admin_ adresses can just use standard, well documented SQL behaviour, having to escape _. By default, _ would just behave as expected as the normal character.

[greshake]

I just had a little look into the code & config; I don't understand how this stuff works so far. I tried to hack myself to a working catchall by replacing the = with LIKE in the postfix virtual alias map query, and manually inserting a matching alias. Two questions came up:

1. Who is sanitizing the input into the %s parameter? Postfix?
2. Where is this config (well) documented? I looked into the postfix docs, but couldn't make sense of it.

[kaiyou]

I like the idea of the checkbox to enable the wildcard behaviour. I still prefer standard SQL wildcards instead of regular expressions however. Regular expressions are nice but a lot more expansive and not standard in SQL to my knowledge, also simple behaviours like catch-all or catch-wildcard would be way more complicated to express by users.

Regarding the configuration itself, Postfix is responsible for escaping arguments, the Postfix documentation itself is very through but I agree it's a mess. The Postfix SQLite documentation references the MySQL documention, which states that %s is quoted (understand _escaped_) : http://www.postfix.org/mysql_table.5.html

The current Freeposte.io configuration is still poorly documented. Some of it is straightforward, but feel free to document any hack or tricky behaviour that you find yourself looking into.

The way I would implement it:

• modify the admin interface, add a field (wildcard ?) to the alias table and model, create the database migration script;
• modify the view to support modifying the wildcard attribute;
• add a UNION query in the alias virtual map.

[greshake]

The union query appears to me like an elegant solution to the implementation question. Although I think the whole idea of getting the aliases from an SQL query is very obscure and prone to security-relevant mistakes, be it implementation errors in postfix (maybe using old database drivers etc) or the query itself, it is easy to implement and synchronize with the admin interface. As far as I can read from the postfix docs, they also support explicit regex alias maps (which seem to me would be the more elegant and safe long-term solution) but this would require direct I/O from the python code to rewrite the config on each change. If I understand your plan correctly, it would result in the following virtual alias query:
SELECT destination FROM alias WHERE alias.email = '%s' UNION SELECT destination FROM alias WHERE alias.wildcard LIKE '%s' LIMIT 1
The limit would be my suggestion to avoid conflicting aliases, preferring the explicit aliases over wildcard ones.

[kaiyou]

Having two different fields email and wildcard both holding an email address to match sounds confusing. I would be more inclined to have a boolean field named wildcard and then localpart and email holding either a static address or a wildcard one.

Either a union-query or a more complex condition I am not sure which is the best way to go. Union query sounded simpler when it comes to managing priority between direct aliases or wildcard matching.

To avoid simple security issues, I would prefer matching only the localpart against a wildcard, never the whole email address. What other issues do you see (apart from the ones originating in Postfix implementation)?

[greshake]

You make a certainly valid point, but I was thinking about them as being mutually exclusive. Going the boolean way is certainly more elegant, and would result in a minimally different query:
SELECT destination FROM alias WHERE alias.email = '%s' AND alias.wildcard = False UNION SELECT destination FROM alias WHERE alias.wildcard LIKE '%s' AND alias.wildcard = True LIMIT 1
I'm not sure which security issues would be mitigated by allowing only the local part to be matched, what exactly do you mean?
I fear users will create aliases with overlapping matches; These will be randomly/obscurely resolved. This is not a primary issue though.

[kaiyou]

Sounds perfect. The confusion with overlapping matches is somewhat inevitable, unless we add some validations in the future. It should be properly documented for now.

The most direct security issue was per-domain administrators catching emails from other domains, even if the scenario seems implausible.

[kaiyou]

Did you start implementing this? If not I will have enough time tonight to give it a try.

[greshake]

I didn't start yet, and I don't know a lot about django flask, but I propably know enough to maybe get a good-looking patch up and running in the next few days. No promises though.

[kaiyou]

It is not based on Django but Flask instead. I will try and implement this one tonight but you are more than welcome to contribute when you have some time. The learning curve for Flask is fairly short.

[greshake]

Wait a sec. Already done, making a pullrequest right now... I already corrected myself with the django thing in the security thread... I knew it was flask but still wrote Django. Heck, they're not so different after all, Flask at least is not such a behemoth. I agree with your choice of Flask by the way

[kaiyou]

The issue is not completely resolved yet, there is still on guarantee that static aliases will override wildcards.

[greshake]

Even after the latest commits, mail to a different account is still landing in my catchall account. This may be due to how aliases are preferred over actual account adresses (not static aliases).
Example case:
I have an account called [email protected] and [email protected]
Emails to [email protected] land in [email protected]

[greshake]

Quick fix for latest-commit-users who read this: add an alias [email protected] with enabled Like syntax redirecting to [email protected]. This will also redirect mails for ful and fua and [email protected], so just a quick fix if anyone needs it.

[kaiyou]

Oh. Aliases prefered over users, I did not think about that one. So yes, the query should exclude wildcards if the email address matches a user. I can only think of a subquery right now, which sounds wrong. Any other idea?

[greshake]

I think a subquery would be okay. at least then all decisions about which account an email is going to be forawarded to are in one single place/query. I did you a favor and already extended the query:
SELECT destination FROM (SELECT destination, email, wildcard, localpart FROM alias UNION SELECT email AS destination, email, 0 as wildcard, localpart FROM user) WHERE ( wildcard = 0 AND email = '%s' ) OR ( wildcard = 1 AND '%s' LIKE email ) ORDER BY wildcard ASC, length(localpart) DESC LIMIT 1

Sorry for the missing formatting, still learning markdown. Here you go: http://pastebin.com/P2M5jD7R

[kaiyou]

Well it is definitely ugly but I do not see any other way and it does the job. I will add some comments and commit this tomorrow. Thank you.

[kaiyou]

It is working fine on my side. Could anyone else give it a try?

[greshake]

Working for me, too. However I did not test thouroughly enough yet. More confirmation would be nice.

[kaiyou]

I spent a couple hours trying to find error cases and could not. I fixed the migration script that did not specifically set the field to 0 for the newly added column. Except for this, that patch sounds good. Thank you!

[fritz-net]

I'm not quite sure if I should open a new issue.

I'm not sure how to define a catchall mail address so I added the following aliases:

[e-mail]
%[email protected]
*@example.com
[email protected]

It somehow works when I use a mail address from the same server (other domain). But when I use a other Mail server I get alias expanded in the delivery notification but the mail never reaches it's target.

[ThomasKoscheck]

Doesnt work for me too

[Nebukadneza]

Hi,

i just tried the following on current master:
main domain is test02.xxx, public hostnames is neb.xxx,test02.xxx, then i created the admin user [email protected], another user [email protected], and an alias [e-mail], destination [email protected], and made sure the checkbox [X] Use SQL LIKE Syntax (e.g. for catch-all aliases) is checked.

Then i sent emails to:

• [email protected] ⇒ arrived at admin
• [email protected] arrived at horst
• [email protected] arrived at horst
• [email protected] arrived at admin
• [email protected] arrived at admin
• [email protected] bounced

This is pretty much what i expected. How exactly do your aliases look? Since internally SQL LIKE is used, the correct syntax is %@domain, but you can also do foo-%_bar@domain for more fine-grained control.
Always make sure to check Use SQL LIKE Syntax

Hope this helps. Please report back if it doesn’t …

[ThomasKoscheck]

Hi @Nebukadneza

Always make sure to check Use SQL LIKE Syntax

Where do I find this? I have version 1.6, is this only a feature of the current master?
Thank you!

[Nebukadneza]

Hi,

it‘s in the create or edit dialog for aliases … Also been there in 1.6 ^_^. See here:

[ThomasKoscheck]

Oh, thanks!! I always searched for this feature in the create account section! :smile:

[arnydo]

I have followed the steps listed above on v1.7 but continue to get the following error:

smtp_1_19f06e350c25 | Nov 01 12:54:42 mail postfix/smtpd[192]: NOQUEUE: reject: RCPT from mail-eopbgr820082.outbound.protection.outlook.com[40.107.82.82]: 450 4.1.1 <[email protected]>: Recipient address rejected: unverified address: Address lookup failure; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<NAM01-SN1-obe.outbound.protection.outlook.com>

Any tips appreciated!

[kaiyou]

Could you check sending the email internally first? Also, could you check that you do not have any space around the percent sign?

[arnydo]

You disregard my comment. Further inspection revealed it was not the wildcard that was giving the issue. It was "reject_unverified_sender" in Postfix.

[SunMar]

Hi, I can confirm this issue. My catch-all wildcard used to work fine, and after updating today I also started having Recipient address rejected: unverified address: Address lookup failure;.

Running postconf showed the following line:

smtpd_client_restrictions = permit_mynetworks, check_sender_access ${podop}senderaccess, reject_non_fqdn_sender, reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_unverified_recipient, permit

I resolved it for now by adding a postfix.cf in the data/overrides directory with the line (just removing reject_unverified_recipient):

smtpd_client_restrictions = permit_mynetworks, check_sender_access socketmap:unix:/tmp/podop.socket:senderaccess, reject_non_fqdn_sender, reject_unknown_sender_domain, reject_unknown_recipient_domain, permit

Note that unfortunately ${podop} is not available from the overrides.

After removing reject_unverified_recipient my wildcard worked again.

[kaiyou]

Hey :) could you please open a separate issue about this?