Mail: Filtering of Incoming messages

[hitam4450]

for organizing message it would be very useful to incorporate a filtering mechanism, based on different conditions e.g. specific account, specific sender, Subject, etc., which will be applied to all incoming messages!

regards, hitam4450

---

There is a $800 open bounty on this issue. Add to the bounty at Bountysource.

---

There is a $800 open bounty on this issue. Add to the bounty at Bountysource.

[zeugmatis]

This is a nice idea but it is important to draw a clear distinction between the MUA (Mail User Agent) and MDA (Mail Delivery Agent - filtering). MDA solutions on the server side already exist e.g. procmail, maildrop, Dovecot LDA module and others - if you have SSH access to your server, you can set up your rules to do this today. 😄

Using an MDA, your filters are only maintained in one place and you have consistent behavior across _all_ of your devices / email clients. Otherwise you have to maintain the same rulesets across all your devices, and can have confusion / possible race conditions when checking on e.g. the cellphone, then check on webmail and suddenly emails moved around. Finally, when using an MDA rules will execute immediately when mail is delivered - so things like vacation autoresponders, forward-to based on From: etc. do not have to wait for you to login.

It is a good idea, but warrants it's own app I think? Since MDA's can have a lot of accidental complexity - it seems simple at first glance but can get so much more complex, see for example the number of options in the native clients like apple's Mail.app.

[zeugmatis]

BTW if this is to be pursued either in Mail app or a separate MDA app for NC, I would like to mention Sieve, which is based on RFC 5228 - already has existing support in many clients/servers/existing API's etc.

[laurentdinclaux]

Of course, a sieve UI is much needed.
A library that could be used: http://sieve-php.sourceforge.net/
Horde does it: https://www.horde.org/apps/ingo

[pierreozoux]

Also be able to filter read/unread on inbox.

[ediazcomellas]

A sieve management screen is pretty much a standard in webmail systems. I am willing to setup a bounty for this feature. Please contact me for details.

[jancborchardt]

@ediazcomellas you can donate to the bounty at https://www.bountysource.com/issues/37588385-filtering-of-incoming-messages :)

cc @hitam4450 @zeugmatis @laurentdinclaux @pierreozoux if you’d like this feature too, any donation or contribution is appreciated! :tada:

[ediazcomellas]

Is there anyone willing to make a try on this issue? I've expanded the bounty to $125. I hope it helps

[jancborchardt]

@ediazcomellas thanks a lot for the bounty! :)

@nextcloud/mail @ChristophWurst @Gomez @lcalaresu anyone up for solving this? :)

[ChristophWurst]

FYI: there's a pretty high bounty set for this ticket. Thus, a potential bounty hunter might be interested in implementing this feature. If so, please contact us first here in this ticket and let us know how you want to solve this problem. This will ensure that it's done right and you don't waste too much time on a solution that won't be integrated. Hope that makes sense. Thanks.

[ediazcomellas]

I've raised the bounty to $250... anyone willing to solve it? @denim2x @ChristophWurst @Gomez @lcalaresu @nextcloud/mail
Please contact @ChristophWurst to make sure it is integrated with the main tree. It wont be accepted as a solution if it is not merged into upstream.

[MG2R]

Added another 50 to the bounty. Would love this feature.

[ediazcomellas]

I don't know why it is not an attractive bounty. Besides, there is a reference implementation in horde that can be used as a guide.

[jancborchardt]

We tweeted about the issue to get people involved: https://twitter.com/Nextclouders/status/941701115195940865 – please retweet and spread the word to help make it happen. :smiley:

[nado]

https://github.com/Rainloop/rainloop-webmail has support for sieve filters, its another app, but it may be worth checking how to improve integration of rainloop within nextcloud or get a similar sieve ui from them.

[muppeth]

Rainloop from what i saw has a specific way of saving and formatting filters. It would be good to follow the same scheme so that future migrations are painless.

On 15 December 2017 17:13:39 CET, Corentin Pazdera [e-mail] wrote:

https://github.com/Rainloop/rainloop-webmail has support for sieve
filters, its another app, but it may be worth checking how to improve
integration of rainloop within nextcloud or get a similar sieve ui from
them.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/nextcloud/mail/issues/44#issuecomment-352045425

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[pierlon]

Hey @nextcloud/mail, I'd like to implement this.

Should this be under Settings as a section, or should it have its own page? I plan to use the pear/net_sieve package btw.

[ChristophWurst]

^ @jancborchardt

[jancborchardt]

@pierlon cool stuff! :) Should definitely be in the Settings as a section caled "Filters". That is, the account-specific settings which can be opened through the 3-dot-menu, right? :)

[pierlon]

@jancborchardt Right 👍

[loxK]

@pierlon have a look at rainloop's implementation. I find their filter UI great and capable.

I just forgot to save some new rules a couple of time making me loose them, but I guess it is better than a mistake leading to a "lost in the folders jungle" email because of that.

[pierlon]

@loxK OK thanks.

[kylepwarren]

Yeah this is pretty crucial. when I downloaded it, it was IMAP only. not sure if pop3 is supported now. either way, straight from host your own mail server - really needs filters. Probably hard to make. not sure $250 is enough. I'd do it for like $5000

[kylepwarren]

Would be cool if was built into cron or something, so this could be how you managed your IMAP mail (filters/sorting)

[kylepwarren]

To the person who said dovecot can do this already: Do you expect any email user to create a dovecot mail rules congif?

[loxK]

Nextcloud being about self-hosting I think most of us are using the mail app on self hosted mail servers.

Sieve server-side filtering rules aren't exclusively supported by Dovecot, see here. Also Sieve server side filters are what most webmails add support for when talking about filtering (roundcube, rainloop, sogo etc.)

[kylepwarren]

oh. In that case. sieve rules.

[kylepwarren]

Was thinking about this again: sieve... not everyone will have access to that. I have to think eventually pop3 being supported would be a plan. Also, pretty sure a cron task wouldn't scale well. Feels funny, but I think the decision making should be offset to client - I guess it is just like every other email client. I guess it makes sense

[kylepwarren]

is there a js sieve library?

[loxK]

POP3 support is out of that issue scope. Nextcloud Mail is an IMAP client for now.

[ediazcomellas]

IMHO sieve is the way to go. Server-side filters are extremely useful: email gets delivered to the correct folder automatically. It doesn't matter if you open the mail with thuderbird, netxtcloud or your email client in your mobile phone.

[kylepwarren]

yeah, but not just server side, just in case they cant change their config - what if that server doesn't support sieve, or pretty sure not all sieve [server] support includes remote management by regular user. I saw a thunderbird plugin that looks like it does a good job - in the screen shots. Haven't tried it. See, I'm not sure if I'd have to install extra stuff for dovecot to support the remote management of sieve configs. I bet some people would have to. So, it should have client side fall back, feedback, instructions on how to edit the config, etc.

[ediazcomellas]

Server side and client side are two different tasks. I will consider this task done, and concede the bounty, if the server side is done.

Being nextcloud about selfhosting, there is a good chance that the admin can add sieve in the server, and any commercial service that is worth the money will already have sieve in the imap server anyway.

[ediazcomellas]

Anyone willing to take this issue? I really want to give the bounty :)

[mezenok]

@ediazcomellas I made a quick look on the issue comments and I think I can help you with it. If this issue is still active and needs a solution - let me know.
BTW: Is it still related to this bounty - https://www.bountysource.com/issues/37588385-filtering-of-incoming-messages?

[loxK]

@mezenok if it was solved the status of this issue would have reflected it. Also, you'll find the related bounty referenced in the first post of this issue.

[TheLastTech]

Is this bounty still open?

[e-alfred]

@TheLastTech As there is no PR and this isn't closed, I would say yes.

[ghost]

Hey @TheLastTech @kylepwarren @pierlon @mezenok,

Any progress on this and are you interested in teaming up for completing this?

cc @ChristophWurst

[PanderMusubi]

You can add this line at the top of the issue description, it will update itself

![badge](https://api.bountysource.com/badge/issue?issue_id=37588385)

and it looks like

[ghost]

@PanderMusubi But it's a lie, the acutal bounty that can be cashed out is ~630$ because of the 10% bountysource fee.

[PanderMusubi]

🤔 Perhaps time for an API on top of this API that deducts the fee.

[ghost]

@ediazcomellas Why is this comment marked as outdated? :D https://gyazo.com/f9b55c46b4523bfefdc8c0b8904dd133

[ChristophWurst]

You can add this line at the top of the issue description, it will update itself

The issue description should update automatically. I notified the person who set this up to check if the integration has to be-enabled.

[ediazcomellas]

@LemonAndroid I have no idea... The bounty is still open and awaiting for someone to take it. Please note that only a solution that is accepted upstream will be awarded the bounty.

[andycowan]

Hi - if no-one else is working on this, I'd like to volunteer to have a crack at it.

[ChristophWurst]

@Krymson awesome. Please let us know which approach you plan on implementing before starting your actual work so that we can make sure it's actually something we'd eventually integrate. Also, please see #1045 because we're restructuring/rewriting parts of this app at the moment and this might conflict.

[andycowan]

It will be a settings page to manage sieve filters on the mail server via the managesieve protocol described in RFC5804.

Additionally there will be a page to set up a simple out of office notification using the same mechanism (but making it obvious as opposed to something you can do with filters).

Do you need more detail?

[ghost]

@ChristophWurst @Krymson Any progress on this?

[ChristophWurst]

It will be a settings page to manage sieve filters on the mail server via the managesieve protocol described in RFC5804.

You might want to wait for the completion of #1045 in this regard.

Other than that I'd like to see how you'd technically implement this. How will you build the UI? How will you configure sieve rules on the IMAP server? How do you handle servers that don't support sieve filtering?

Thanks.

[nado]

So basically, still waiting? Rainloop for nextcloud is not supported in nc14 and official mail app doesnt support filters.
I guess I’ll wait on nc13 for some time this year.

[ghost]

@nado why is rainloop not supported in nc14 and why is nextcloud/mail superior to rainloop?

[nado]

@LemonAndroid rainloop is not officially maintained for nc, the current port is incompatible with the drop of deprecated API in nc14.

Rainloop support sieve filters, thats the only feature I've considered up til now to compare rainloop and nc/mail, because I rely heavily on them.

[ghost]

@nado I don't really see the upside of re-writing apps for nextcloud, that already exist in oss.

It's the same for nextcloud/deck really, there is wekan already, which could be integrated into nextcloud.

[nado]

Im not sure to understand what you say. If I have to, I'll switch back to a "rainloop only" instance.
But my hope was to have it more integrated in my current nc install (login session, file picking for attachment, less checks needed to keep it all updated, etc).
I'd be happy to switch to nc/mail when the filters are developed because thats basically all Im waiting for.

[ghost]

@nado I'm saying rainloop could be integrated to share login etc. with nc install.

[nado]

@LemonAndroid yes it could. Problem is manpower. So I'm looking for alternatives since I dont have time for personally managing it right now.
See https://github.com/pierre-alain-b/rainloop-nextcloud/pull/64#issuecomment-403088925

[ghost]

@nado thanks for linking the rainloop-nextcloud repo, let's continue discussion over there: https://github.com/pierre-alain-b/rainloop-nextcloud/issues/72

[loxK]

I have the same rainloop dilemma. Just learning it won't be supported in NC14... 😫

Every webmail app has sieve filters support. Sogo, rainloop, roundcube etc. It is the de facto standard for IMAP mail filtering.

[ghost]

@loxK i was planning on putting up a bounty for the rainloop-nextcloud#45 issue for making it work on nc14. would you chime in? I thought 10$ each for now.

[plinss]

@LemonAndroid I'd be willing to put up $50 toward the bounty of getting rainloop working in NC14

[ghost]

@plinss I'll match that.

[ChristophWurst]

Please discuss these rainloop topics in the rainloop repo. Thanks :v:

[Sanookmakmak]

There are a lot of solutions already done, just install sieve:

http://sieve.info/clients

[andycowan]

@ChristophWurst - firstly, I don't plan on supporting servers that don't support managesieve. In the same way that Nextcloud doesn't support mail servers that don't support IMAP.

From a UI point of view, I'm expecting that I'll provide ready to use rules and the parameters require for them rather than access the sieve language - in my mind this is an end-user tool, not poweruser/admins. For this reason I'll short-circuit the most common functions, such as setting an out-of-office message.

As for how technically - I need to investigate how the frontend works in Nextcloud. This would be my first Nextcloud contribution, and I need to familiarise.

[ChristophWurst]

@ChristophWurst - firstly, I don't plan on supporting servers that don't support managesieve. In the same way that Nextcloud doesn't support mail servers that don't support IMAP.

We at least have to show a info that rules cannot be edited on these server and disable the rules dialogue.

[andycowan]

Sure, that's a given. I'll have to add some settings in the account settings dialog.

[ediazcomellas]

I am ok with a managesieve approach. Sieve is a sensible standard.
Please check how roundcube manages several sieve scripts, allowing to activate one or another.

[ghost]

I raised bug #1101, but it was closed as duplicate of this bug. However, from reading above comments it seems that server side filtering is preferred through sieve. This makes me wonder two things:

• Can sieve on the server side apply message filters across multiple accounts?
• My use case is using mail app to read many external accounts, such as Gmail, outlook, and a few others, including private ones. AFAIK most public mail services doesn't support sieve server side. In this respect I think my original #1101 request should remain open.

Also, I do not think that most users that self host also self host email. Just think about the press release about expanding nextcloud on consumer devices in Japan. Why limit such an important mail feature to advanced users.

[ytzelf]

@krymson are you still working on this?

[omocinteractive]

I installed Dovecot/Sieve and use Thunderbird to edit the sieve-filters for each account. I think it's a good way. It would be nice to have a editor in the nextcloud mailer, of course ....

[plinss]

I agree that filtering via Sieve on the mail server is the proper approach here. FWIW, I've been using RainLoop to edit sieve filters. Much better UX than Thunderbird, IMO.

Having a decent UI to edit Sieve filters via the managesieve protocol built in to the standard mail app would be great.

[omocinteractive]

This is also a reason i use Roundcube (was installed on my Plesk system and has a better UI for multiple accounts) ...

[ediazcomellas]

Hi,

No, Sieve only acts on message receival for a single account, so it wont work on external accounts or work across several local accounts.

Being mailserver-based, sieve is ideal if you access your email from several interfaces (which is the case of lots of users, using web, desktop and mobile MUAs).

I raised bug #1101, but it was closed as duplicate of this bug. However, from reading above comments it seems that server side filtering is preferred through sieve. This makes me wonder two things:

* Can sieve on the server side apply message filters across multiple accounts?

* My use case is using mail app to read many external accounts, such as Gmail, outlook, and a few others, including private ones. AFAIK most public mail services doesn't support sieve server side. In this respect I think my original #1101 request should remain open.

Also, I do not think that most users that self host also self host email. Just think about the press release about expanding nextcloud on consumer devices in Japan. Why limit such an important mail feature to advanced users.

[omocinteractive]

• Can sieve on the server side apply message filters across multiple accounts?

yes, but you have to add the filtersetting for each account (copy and paste directly on the host or export/import in your editor). i forward mails from gmx, gmail to a self-hosted account with sieve filter.

[ytzelf]

I'm currently using roundcube and there is a well made and functionnal managesieve plugin here https://github.com/roundcube/roundcubemail/tree/master/plugins/managesieve . To alleviate above concerns about sieve complexity, the way it works is with a graphical user-friendly filter editor translating comboboxes in sieve syntax.
I'd be really interested in this as I want to switch to nextcloud for contacts and calendar purpose but can't live without client-side edition and creation of server-side filters. I'll add 20€ to the bounty - I see that someone named daniilborovkov has started working on it some 20 days ago, has he pinged anyone from nextcloud?

[omocinteractive]

"I'd be really interested" - i too. I use a card-dav and web-dav in roundcube plugin to interact with nextcloud, but it would be better to do this in nextcloud mailer (if someday the signature feature available).

[ytzelf]

"I'd be really interested" - i too. I use a card-dav and web-dav in roundcube plugin to interact with nextcloud, but it would be better to do this in nextcloud mailer (if someday the signature feature available).

Then maybe add to the bounty? if it goes high enough someone will pick it up ^^

[omocinteractive]

... i still waiting for the confirmation mail (greylisting active).

[omocinteractive]

... done.

[sayir]

Hello. Is now someone except me working on this issue?

[TheLastTech]

Yeah, i don't do bounties after I did one and they refused to pay.

[ghost]

@TheLastTech @nextcloud/appstore refused to pay?

[ghost]

@daniilborovkov can i join you?

[ediazcomellas]

As one of the posters of the bounty, I feel a bit frustrated. After a loooong time, and with a quite high bounty after all, I don't see any progress on the issue. Is there anyone really working on it?

[ghost]

@ediazcomellas do you . plan to pull funding? is that possible?

[ghost]

I'd like to start working on this, but it's difficult alone.

[ghost]

At least we got the conversation sparking up again.

[ediazcomellas]

I really want to pay the bounty. I only put two conditions: work with sieve and produce a solution accepted upstream. I won't pull the bounty (it is not possible). I just feel a bit frustrated :)

[ghost]

I can understand.

[LorbusChris]

I added a little to the bounty. Somebody, please do this!

[ghost]

how about using ruby/truemail instead of sieve?

https://github.com/rubygarage/truemail

[loxK]

how about using ruby/truemail instead of sieve?

https://github.com/rubygarage/truemail

How is this related?

[ghost]

@loxK isn't it much like sieve?

[loxK]

It is totally unrelated it is an email validator for the Ruby programming language...

[ediazcomellas]

Using sieve is an obvious choice: let the email server do the work. Email will be stored in an IMAP server anyway, so why not use it for filtering too? Most providers support sieve (all the serious ones) and it is very easy to setup one if you self host your email. All it takes is a parser/editor of sieve scripts and a way to deploy them (sieve has a protocol for that). Even better: you can borrow all the necessary the code from Horde, roundcube and many other webmail programs out there.

[ghost]

@loxK how is it totally unrelated? sieve is also a dsl. ruby is a great choice for dsls. isn't truemail doing email validation as well?

[ghost]

btw, i started doing the bounty.

don't worry, i'll use sieve.

Does any1 want to help/join?

[ghost]

Great to see some discussions again. There are a few things sieve cant deal with. For example cross account filtering. Also I don't think that most major email providers like Gmail supports sieve.

[ghost]

http://eagle.botcompany.de progress can be seen here.

If you want access to the vm or just talk about the issue in a chat room, feel free to join my discord server: https://discord.gg/kwKzVX.

If you want to actually join the development, we'll find a sensible way to split the bounty.

[ediazcomellas]

Great to see some discussions again. There are a few things sieve cant deal with. For example cross account filtering. Also I don't think that most major email providers like Gmail supports sieve.

Most self-hosters (and Nextcloud is all about self hosting) can implement it. Gmail, like all the big guys, don't play well with others. Cross account filtering is something that requires explicit support in the client, and it sounds just too complicated for me. I can live with a simple sieve support (and, btw, it is one of my two requirements to release the bounty).

[ghost]

Great to see some discussions again. There are a few things sieve cant deal with. For example cross account filtering. Also I don't think that most major email providers like Gmail supports sieve.

Most self-hosters (and Nextcloud is all about self hosting) can implement it. Gmail, like all the big guys, don't play well with others. Cross account filtering is something that requires explicit support in the client, and it sounds just too complicated for me. I can live with a simple sieve support (and, btw, it is one of my two requirements to release the bounty).

Yes, you're right. Understandably difficult too. Could one way of seeing this be that Nextcloud is the client that uses IMAP to download from various accounts (like Gmail, Hotmail..) and applies the sieve locally? A sieve client would talk to NC and not the third party IMAP servers? It's just a thought. Thanks.

[morph027]

Right now it's just a client which does not download. One could "easily" setup a self hosted server including sieve which fetches from all remote accounts to apply cross account rules.

[ediazcomellas]

@morph027 is right. The current mail application in Nextcloud is just a webmail that talks IMAP to the server. What is needed here is a "settings" section that allows to edit/deploy sieve scripts to the email server. Let the server do the heavy work. Creating a new implementation of sieve in the client can be a nightmare, and I doubt it would be useful.

Filtering in the server (sieve) makes all views of your email (either using a desktop client, any webmail client or a mobile phone app) consistent and asynchronous (filtering happens on mail delivery, not when you open the mailbox with your email app). That is why I find it is the best place to do the work.

Besides, letting the server do the filtering makes this request much easier to accomplish.

[ChristophWurst]

@ediazcomellas yes, thanks for the summary. This is exactly what we should do there and also the only solution I'd allow integration into our code base :wink:

Reminder to whoever wants to work on this: let us know what and how you want to implement it. Thanks.

If nobody is interested I can have a look at this feature after summer vacation in a few weeks. As I'm a Nextcloud employee the Bounty would be put back onto other tickets as per usual policy.

I'm closing resolved discussion comments now to keep the thread clean for future readers.

[ghost]

@ChristophWurst I started working on the issue.

[ghost]

But now i'm getting a bit sceptical. A few comments ago I already mentioned I started working on this issue + another commenter raised concerns about Nextcloud paying out bounties + it seems the preferred way for bounties is to be "collected" by employees and put back into the pool.

You kinda completely defeated the purpose of bounties by this logic.

[ghost]

I also wish to not be a member anymore, I don't want to be associated with Nextcloud at this point.

[stefan-reich]

@ChristophWurst: @LemonAndroid posted here that he works on the issue and you ignored it. Is that the way you treat collaborators?

[kesselb]

Please calm down :sunglasses:

A few comments ago I already mentioned I started working on this issue + another commenter raised concerns about Nextcloud paying out bounties + it seems the preferred way for bounties is to be "collected" by employees and put back into the pool.

1) I have the impression that the person who wrote that statement never made a bounty for nextcloud. Not sure why he is even here. You may check his github profile.

2) I made a ticket with bounty once (https://github.com/nextcloud/mail/pull/1061) and got the bounty.

[morph027]

I guess not by purpose, probably by accident. Also being member of the nextcloud github organisation does not make you an employee. I'm pretty sure you'll get your bounty once it's done.

[ghost]

@morph027 @kesselb thanks for calming me down.

[loxK]

@LemonAndroid please read what @ChristophWurst said. He said to get in touch to anybody taking the work. If nobody does, he may do it after summer vacation. It is pretty clear and nothing is wrong with that.

If you want to do it, well just do it, this is how open source works.

Now please stop useless comments that notify everybody in the thread.

[ghost]

@kesselb I also did a small bounty like yours tho: https://github.com/nextcloud/deck/issues/259.

I'm sceptical about the payout of big bounties.

[ChristophWurst]

Sorry for my bad communication here. Of course I've read @LemonAndroid. Unfortunately so many people have tried to claim bounties before they even had a look at the code. You can search the issue tracker for bounty tickets and you'll find many where people said they were working on somehting, others held back and in the end there was not a single line of code.

My statement was only regarding the case that nobody would actually pick it up.

@LemonAndroid as soon as you have some code please open a PR so we can discuss the details there.

But now i'm getting a bit sceptical. A few comments ago I already mentioned I started working on this issue + another commenter raised concerns about Nextcloud paying out bounties + it seems the preferred way for bounties is to be "collected" by employees and put back into the pool.

That is not true. As @morph027 pointed out this is only done when an employee implements a feature which a bounty has been set for. Naturally this has happened many times because when I post bounties then I put them on tickets that make most sense and not some random edge case feature. Hence I, as the main developer, occasionally do stuff which others requested and offered their monetary support for.

I've only rejected paying bounties when a contributor provided an incomplete solution, usually claimed before it was even integrated. But those conflicts were resolved quickly and AFAIK all bounties have been paid out.

IMO the purpose of bounties was never to let them grow as much as it has here, but so be it. When I distribute money I do in batches of small amounts so it's not concentrated on a few tickets but allows earning a few bucks on as many tickets as possible: https://twitter.com/ChristophWurst/status/1120707552856215553.

:)

[ghost]

10 usd minus bountysource fees was never earning.

I still like your professionally-worded response.

[ChristophWurst]

I'm sorry to hear that. I was not aware of the fee as I never had a bounty paid out. But I think with the $800 here the $10 are negligible :see_no_evil: In the long run we should move away from Bountysource. We've had too many issues with it recently (both technically and similar to what you raised).

Now let's stop the off-topic discussion and focus on making Mail an awesome e-mail app :rocket:

[ghost]

just one last "off-topic": Bountysource charges a 10% fee on all withdrawals from your Bountysource account.

[pierlon]

@LemonAndroid started anything?

[ghost]

@pierlon I tried and yet I have issues setting up the mail app on the server. Some problem with npm/nodejs and probably the old version of Ubuntu, 16.04. Once I figure that out, I'll continue with a fork of the solution you started.

update:

now i'm having issues with the server, log files don't seem to have correct permissions. I wish I could see (detailed) errors in the web interface.

[mmittelb]

@LemonAndroid any progress?

[ghost]

Nope

On 29. Jul 2019, at 10:32, Merlin Mittelbach [e-mail] wrote:

@LemonAndroid https://github.com/LemonAndroid any progress?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/nextcloud/mail/issues/44?email_source=notifications&email_token=AAYPQIXZLA74ODKMZCJJMQ3QB2TJJA5CNFSM4COUONO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD277ZAY#issuecomment-515898499, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYPQIRU6OZA67OF3PPGEFLQB2TJJANCNFSM4COUONOQ.

[mmittelb]

If there is no progress on @ghost side, I'd like to take over the issue. I'm starting tomorrow.
Cheers

[kesselb]

@mmittelb do it :+1: The user @LemonAndroid deleted himself from github. To keep issues and comments such accounts are renamed (or merged) to @ghost.

[plinss]

@mmittelb great to see someone taking this on. I have a couple of requests:

1) please support STARTTLS when contacting managesieve, and possibly have an option to require STARTTLS before sending any passwords (or make it mandatory). Secure communication and protection against TLS downgrade attacks is important.

2) given that sieve scripts are just text files, I presume you'll have a nice UI for managing filters, similar to RainLoop and others (last I looked at Thunderbird it made the user edit the raw sieve scripts which is a non-starter for non-technical users). AFAICT there's no standard how filter UIs translate into sieve scripts and back again. I've used both RainLoop and RoundCube and they use different methods. Ideally you'll be compatible with one of these existing methods rather than inventing your own (RainLoop's would be my preference). It'd be nice to be able to use both RainLoop and this UI without breaking existing filters.

[mmittelb]

@kesselb oh good to know
@plinss
To 1: STARTTLS option is planned.
To 2: I only know the roundcube sieve client. But I'll have a look at rainloop before implementing the UI.

[eZii-jester-data]

Thanks for taking over this, I'm the @ghost btw 🤣

[loxK]

Roundcube does a pretty job as it gives access to all sieve rules. Rainloop only manages sieve rules in its own filters file.

Roundcube also has a nice vacation message form. But it is incompatible with Rainloop, it fails to parse rules conditions and doesn't keep the comments Rainloop needs.

[mmittelb]

Little update:
I set up the server with the mail app and started to experiment with the frontend. Since I'm new to developing webapps this may take some time.
Frontend:
I figured out how to add an option to the settings menu (AppSettingsMenu) and will start to work on the view. The idea is, to have a dropdown list to select the account. Once selected there will be a roundcube-like gui (I just like it better) to edit filters and an expandable settings menu to specify server, port and STARTTLS options plus testing button. Default should be the imap server, port 4190 and STARTTLS enabled with cert checking. As soon I gat something working I'll push it to git. Should I branch the current master or how are you using git?
Backend:
Didn't do much yet. But I will probably use the horde-managesieve library.

[loxK]

I had to have a deeper look in Sieve rules and webmails for another purpose and have some thoughts.

In terms of "Sieve", activating a filter set makes a symlink from sieve-script -> filtersetName.user.sieve.

Rainloop (and maybe other clients) maintains its rules in its own filter set file with some hash ids. What this implies is that any modification done from another client may make it unreadable by Rainloop afterwards.

So my idea would be that the GUI will present filter sets in a similar in what Roundcube does. But:

1. the filter sets GUI would support drag and drop sorting
2. one "main" filter set would be hidden, its purpose would be to handle other filter sets activation and sorting using Sieve's include

That way Rainloop (and other) rules can still be used along side Nextcloud's rules.

[ediazcomellas]

We use roundcube extensively here, so if what you implement supports it, It would be perfect (for us).

[eZii-jester-data]

Will the rules be saved to the NC database?

[plinss]

Hopefully the rules will only live in the sieve files, so they don't get out of sync if edited by another client.

Another ask would be that the sieve server settings (and possibly the filters?) be settable via occ (I'm not sure how much of that you get for free).

[loxK]

@ezii123 Sieve rules in the NC database is not a good idea nor needed. The rules are stored on the mail server.

@plinss occ support seems rather complicated to me. You'll need to specify the user's email account you want to set filters rules for, the rules set file etc. But you can do global sieve filters or default filters and so on on you mail server directly and script that easily.

[plinss]

@loxK at the very least we should be able to set default server parameters for managesieve as we do for the smtp and imap servers in config.php 'app.mail.accounts.default'. I don't feel strongly if that's in config.php or settable via occ. The general user won't/shouldn't know how to configure their sieve server.

[eZii-jester-data]

When will we see the first Pull Request?

[kesselb]

@ezii123 Have you read https://github.com/nextcloud/mail/issues/44#issuecomment-527294986?

@mmittelb There is a draft with some backend changes. Might be handy: https://github.com/nextcloud/mail/pull/1822

[mmittelb]

@ezii123 I'm not planing on saving the script on the server. I don't think there is any need for that. You will see the pull request as soon there is something to pull. It's taking some time setting up everything and getting to know the frameworks and concepts used. Please be patient.
@loxK Thanks for your ideas. I like both of them. But side-by-side use with Rainloop won't work.
@plinss Hmm I'm not sure about the occ feature. I don't know yet, how hard it will be to implement and I don't really see a scenario in that this is needed. Do you have one?
@kesselb Thx I'll have a look.

[plinss]

@mmittelb the requirement for this is two-fold:

1) we can already setup default parameters for smtp and imap servers, so that in an environment where all Nextcloud users use the same email server (and the mail server uses the same credentials), users can simply click the mail icon and see their email without doing any configuration on their end. We should be able to set the defaults for the sieve server so users can then simply use the filtering UI without any configuration as well.

2) most of Nextcloud can be setup automatically using tools like ansible by either specifying information in config.php or running occ to configure apps. The default sieve server information should be able to be setup automatically as well. I really don't care if it's done via config.phg (like the default mail server info already is), or via occ. Given that pretty much all apps can be configured via occ I'm guessing you'll get this for free so long as you use a standard way of storing app config information.

I run a business where we setup and maintain Nextcloud instances for customers, and everything is done via automation. If this has to be manually configured by each (non-technical) user or by using the admin web UI it would be a non-starter for us to adopt.

[kesselb]

we can already setup default parameters for smtp and imap servers

Sounds reasonable to me. Needs some getters and setters for the default values here (config.php to object) and here (default config to account config). Given that this ticket is heavy I would not demand it. @ChristophWurst @ediazcomellas opinions? (you're the top backers).

via occ I'm guessing you'll get this for free so long as you use a standard way of storing app config information.

Yes. Just use IConfig and you are good.

[loxK]

@loxK Thanks for your ideas. I like both of them. But side-by-side use with Rainloop won't work.

Roundcube may have issues too. No sure how it will show a file filed with includes.
In the current situation Rainloop and Roundube aren't interoperable either. Roundcube would break a saved Rainloop filters set on save and fails to parse conditions with brackets.

But, the more I think about it the more I like the idea of having a main file that has a list of user sorted and activated fiters set. The idea is so good to me that I think that, once implemented this way in Nextcloud as a proof of concept, pull requests to Rainloop and Roundcube to make them comptatible with it has good chances to make its way to their code base.

[mmittelb]

@plinss On scriptable server information I agree and that's easy to implement. I thought you mean the sieve script handling, which is in my opinion easier to do directly communicating with the sieve server.
I think it's best to do the script rendering in the browser to make the page more responsive. Any opinion on that?

[plinss]

I did mention possibly setting some initial filters, but that was just a random side thought as I was typing. I agree if there's a need for that we can always just set some initial filters directly on the sieve server. My main concern was the configuration of the server settings.

As far as doing the script rendering client side, I agree, that'll be a much more responsive experience for the user.

[ediazcomellas]

Initial filters are a job of the server. Don't waste efforts on that.

[pierlon]

Hey @mmittelb, made any progress? I'm updating #1822 with some more changes.

[mmittelb]

@pierlon I created a filter view with draggable lists and was about to start working on the script parser. I kind of overlooked how far you are.. Maybe I keep working on the parser and integrate it into your stuff? I pushed it to this repo https://github.com/mmittelb/mail

[pierlon]

Nice :100:. Will look at your changes later to see how we can get it merged.

[mmittelb]

@pierlon I'll fork your repo and merge.
Is the filters ressource fully working?
Is there a way to check sieve files by the mailserver? I think managesieve does have such functionality.
My view is integrated into the settings menu and I think it's better to leave it that way. For the server configuration, I like your solution better though.

[pierlon]

Yes there's a SieveController for fetching, updating and activating scripts

[mmittelb]

What is working:

• getting scripts from server
• multiple Accounts
• raw editing of scripts
• parsing and unparsing of scripts
• multiple active scripts (the discussed include method)
• dragable filters and filter sets to sort

What is missing:

• saving scripts to server
• more test options: currently supported are Subject, From, To, Header exists, Content (body)
• more action options: currently supported are Move, Copy
• disabling filters in account config

What other options are important? Please test for bugs and share your thoughts.
Cheers

[plinss]

Looking good so far.

One valuable action option will be to delete the message rather than move to trash. Commonly used for certain kinds of spam that there's no reason to keep in the trash.

A UI note, IMO the ability to edit the raw script is more of a power user feature that most users won't use (or understand), it might be better if that control was less prominent.

[ChristophWurst]

Is this jquery UI?

[jancborchardt]

Heya @mmittelb, thanks for your contribution! :)

So this seems to be very complex and complicated. With Nextcloud we aim to design stuff simply so regular people can use it. Editing the rules in plaintext format is not really accessible in that way.

What we need to do is start with the basics, like checking if the subject or sender matches or contains a specific text. Interface-wise this can be done very simply, similar to the Workflow redesign at https://github.com/nextcloud/server/pull/17506#issuecomment-542341901
And the "Filters" section can just be a section in the account settings.

From there on we can see what else is needed, but keep in mind all this needs to be maintained, so should be starting simple, with a minimum viable product.

What do you think? Also, it would be nice if you can open a pull request for this so we can use that to collaborate and discuss. :) Thanks!

[mmittelb]

Thanks for the input.

delete the message rather than move to trash

I think the command is discard. I will implement it.

raw script is more of a power user feature

Agreed. I implemented it as a fallback, because I don't want to add to many options to the parser. Imo too many options are confusing to the user. But I will think of a way to better disguise that button.

Is this jquery UI?

No the dragging is vue-draggable which is a Vue implementation of sortablejs (https://github.com/SortableJS/Vue.Draggable)

very complex and complicated

It's kind of a copy of the roundcube filter GUI. I always liked that one because it's straightforward for inexperienced user but at the same time powerful enough for the power users. I put the raw edit in the gif more to demonstrate the parser. The normal user shouldn't need to use that.

checking if the subject or sender matches or contains a specific text

That is implemented.

similar to the Workflow redesign at nextcloud/server#17506 (comment)

Could you please be more concrete? I already tried to hide away most of the Sieve possiblities. To make it easier to make filters withoutuch knowledge.

And the "Filters" section can just be a section in the account settings.

@pierlon suggested the same. Two reasons why I don't like it:

• Filters are just no server configuration option to me. I think it's cleaner to have its own section.
• The current UI would break the server configuration options design. But to implement the features we discussed in this thread I couldn't think of anything that would fit there. I didn't want to reinvent the wheel that's why I followed the roundcube way.

keep in mind all this needs to be maintained

I understand. I tried to code it modular (using Vue components) and separated the parser from the page. I should definitely implement some tests.

open a pull request

I used @pierlon work on the backend (https://github.com/nextcloud/mail/pull/1822) as a base. That's why I opened a pull request into his branch. Should I have my own pull request?

[jancborchardt]

@mmittelb ok, then we really have to check it. :)
Generally I'd say better look at Apple Mail or Google Mail rather than Roundcube for inspiration, as that is more widely used and tested.

Filters are just no server configuration option to me

That's why it's called "Account settings" and has Signature and Edit mode too. :) See https://github.com/nextcloud/mail/pull/2119
Filters belong there.

Regarding the last point on branching, @ChristophWurst how should it be done?

[ChristophWurst]

Regarding the last point on branching, @ChristophWurst how should it be done?

It should be pushed to some branch. I don't really mind which PR it is :)

[ediazcomellas]

I'm glad this issue is showing progress after all this time. I really appreciate it. Please continue and make sure @ChristophWurst accepts the solution so I can release the bounty. Following roundcube's example is OK for me (I'm already a roundcube user).

[Gunnar-GO]

anything new ?

[ChristophWurst]

Hey there,

I'd just like to let you know that due to the ongoing issue with Bountysource we're discussing ways to move to a different platform. As it looks right now, Bountysource might claim some of the money that was contributed by the community. We will try our best to prevent that. Historically we – the Nextcloud GmbH employees – simply put the claimed bounties back to other tickets until one was solved by a community member. So the money was always from the community and for the community. We'll therefore move to a better platform and currently don't post any new bounties until this situation is resolved. The current goal is to move/donate the money to Nextcloud include but details will follow in the forum post.

Hope this makes sense. If you have any questions please ask them on the forum. This comment will be copy-pasted to all open issues with bounties.

Christoph and the Mail team

[ediazcomellas]

Hey there,

I'd just like to let you know that due to the ongoing issue with Bountysource we're discussing ways to move to a different platform.

Please @ChristophWurst , claim the bounty and move it to somewhere useful. Bountysource has been a bad place from day one. I would rather see my money going to somewhere else in the nextcloud comunity.

[MG2R]

Please, claim the bounty, take whatever money you can, switch platforms and put the bounty up there. I posted my bounty end of 2017. No way that money is going to do anything useful if you don’t claim it. BS will drag this out until the deadline and then claim all the money.

[ChristophWurst]

The problem is that if not all 9 backers accept the bounty will remain on bountysource's side and we'll lose the money.

I've tried the quick claim on my own bounties like at https://github.com/nextcloud/mail/issues/317 but ironically the bountysource page is too broken to update the open/close state and I can't even claim the resolved issue at https://www.bountysource.com/issues/42324763-improve-caching-preloading-for-better-performance. It's really a mess.

[MG2R]

Their site is a mess since the announcement. People must be waltzing in like herds.

@ChristophWurst what do you suggest I/we do? The money I pledged should go to Nextcloud. Preferably towards this issue but definitely towards your organisation, that’s why I said claim it. If that’s not possible, what’s the best way of helping out?

[HLFH]

ownCloud/Nextcloud since 2010 is the leading project in protecting personal data in the cloud era.
It has built a great community.

What I suggest is to:

• instead of being manual in the process, count how much backed money is remaining in BS ;
• have a legal agreement with Bountysource Inc ;
• in this agreement, make them agree to move all the remaining backed money to Nextcloud Include ;
• wait for the transfer ;
• be transparent as always ;
• with Nextcloud Include, give backed money to contributors once they have pushed the required PRs.

1. Get all the backed money from BountySource and put it in Nextcloud Include.
2. Develop and internalize progressively a bounty solution for Nextcloud contributors.

And huge thanks to @ChristophWurst for the great rework of Nextcloud Mail.

[kesselb]

Besides forum post, see main GitHub discussion about BS here: #44 (comment).

@HLFH no. Any discussion should be at https://help.nextcloud.com/t/dont-use-bountysource-anymore/84943. The bountysource issue is completely unrelated to "Filtering of Incoming messages".

[dehnhardt]

To keep the hope alive, here is an illustrated intermediate result ;-) I curse Javascript and am still astonished what all this does...

Holger

[dumblob]

For those needing filtering/Sieve now, there is an opensource solution Sieve Editor.

I'd also guess Sieve/filetring implementation in NC could also leverage the mature code base of Sieve Editor. What do you think @thsmi ?

[alphashuro]

Is the bounty on this still open?

[ChristophWurst]

https://www.bountysource.com/issues/37588385-filtering-of-incoming-messages is still open but please don't add anything to it. We're trying to move away from the bountysource platform: https://help.nextcloud.com/t/dont-use-bountysource-anymore/84943

[jancborchardt]

@dehnhardt (and also maybe @ChristophWurst or @GretaD or @brueckner) do you want to sit together on figuring filtering out properly at some point?

We should start by collecting examples of how other systems do it, especially simple and easy to understand ones. Great examples are Sogo and Gmail:

• https://www.laptopmag.com/articles/filter-mail-gmail
• https://www.midgard.co.uk/how-tos/how-to-set-up-filters-for-email-in-sogo/

Additionally, Google does it nicely with different entry points to filtering like

• You can create filters from searches
• You can "Filter mails like this" to e.g. create a filter rule based on a sender, from a mail you are currently in

[plinss]

@jancborchardt Two other examples are RoundCube and RainLoop. RainLoop also has a Nextcloud app so that would be a MVP target to get people to switch from it to the Mail app. Many people, such as myself, use it over the built-in Mail app for the filter support (and its superior feature set, and better UI, but Mail is catching up).

To me, one of the fundamental problems with sieve filters is interoperability. When the filters are written to the server, they're script, and few tools (if any) are able to parse the script back into the filter primitives.

Most tools, like RainLoop and RoundCube, also store filter metadata as comments in the script. That way, when they read the filter later, they can populate the UI and regenerate the same script. Unfortunately, no two tools I've ever seen agree on a format for that metadata. So if you create filters with one, then edit with another, many of your filter rules are lost (or at best don't show in the UI).

A huge win here would be standardizing on a single metadata format, and what script that metadata generates (and have open documentation for that format). Then get as many existing tools as possible to use that same format (submitting patches as necessary). Ideally work with authors of tools that already support sieve filters on the metadata format.

[loxK]

RainLoop has a compatibility issue. It stores hashes (as comments) in its sieve file. As such, editing this file with another client prevents editing it back in RainLoop.

What would be awesome is that the Nextcloud's implementation uses a master file that includes a nextcloud.sieve file. If so, we could then ask the RainLoop developer to modify its code in order to look for the master sieve file and roll its own include.

[ChristophWurst]

While I support the idea of making things as interoperable as possible searching for the perfect solution as the initial implementation will just make things more complex and push it down the roadmap. In my opinion it would be better if we first have something that works with our app and in another step we make it as interoperable as possible. If the fully interoperable solution was so easy, other solutions would have it …

[ediazcomellas]

I also support the interoperability idea, but wont want to wait one more year to get it running. OTOH, starting a parallel work to create a definition of the standard would be just great. I guess this is something to discuss in another thread. I would gladly contribute to such a working group.

[dumblob]

Disclaimer: I don't know Sieve very well.

How about not having a real standard, but rather a "standardized" (i.e. detectable) convention for a "main" Sieve file which would by any means not contain anything else than includes of other Sieve files ordered lexicographically (pretty much how Linux distributions started managing /etc/ for different daemons during last 15 years).

The point would be, that if the included file can't be parsed by the given MUA, then that MUA has at least the opportunity to "get ahead" of other-MUA-filters and use the other-MUA-filters conditionally or even stop before other-MUA-filters.

This has the advantage, that if later there would actually be some standard, it'd stay compatible and useful as MVP for many use cases. On the other hand, despite being a small change this still requires all MUAs to make at least some minor changes to their behavior. But maybe Sieve projects like Pigeonhole could partially alleviate that by having defaults and recommendations to use such convention?

Is there any good place to discuss such potential standard/convention to not pollute this thread?

[plinss]

I accept that creating an open standard method of storing filter metadata takes time, and I agree that this feature shouldn't be gated on that. But at the least, I'd like to see some movement in that direction and a commitment from whoever implements this to help make it interoperable in the future. In fact, if an effort starts to create such a standard, you might find more people willing to help out.

Some suggestions:

1) Please do not simply create yet another non-interoperable format (unless you've studied all the existing formats and found them all wanting). This would just make the problem worse. At least start by following one existing format (ideally one that's easily detectable and extensible).

2) Ideally start with an exiting format that has open source code and refactor their metadata handling into a stand-alone reference library, that both they and this project can use. This could be done along with generating the documentation of the format and be used as a starting point for the standard. That way other tools can re-use the reference library to get interoperability instead of having to re-implement from scratch. (Without having looked closely at the actual metadata formats, I suggest starting with RainLoop simply because there are existing Nextcloud users.)

3) Given a common library, over time it can gain the ability to read other tools' proprietary metadata formats and convert into the standardized output format. Further improving interoperability without data loss.

4) Creating a standardized format doesn't have to be a formal standard (but it would be nice). Anyone can start such an effort by creating a GitHub repo and getting others to participate (I agree this is not the place for that effort). Ideally, invite authors of tools that have existing sieve filter handling and the original authors of sieve scripting. As it matures, you can search for a home for a more formal standard (I suggest IETF since that's where sieve lives).

I have some relevant skills and would be happy to participate in developing such a standard as time allows. Unfortunately I don't have the bandwidth to lead such an effort at this point in time, we just need one person to step-up and start it off...

[ChristophWurst]

Hey everyone :wave:

we (Nextcloud GmbH) would like to finish this feature as part of our current roadmap. @kesselb will help with that. Since this is a rather complex change we want to do it in two or three smaller chunks. First one would be the back-end support, acceptance criteria is passing unit and integration tests. In a second step we'll add a basic UI. Anything more advanced will be added in a third step.

Now I'd like to ask the original authors of the two open but stale PRs if it's ok if we build on their previous work. We won't take any bounty, as per usual, so the money will remain on the platform and I would like to ask those authors to later claim the remaining money when the PR for the second part is merged.

Is this ok for everone? I hope this way we can give credit where credit is due (or money) and from a bigger perspective we'll hopefully soon have our first release of Mail with this feature.

Please let me know if there are any objections or concerns.

Cheers

[ediazcomellas]

Hi!

It is ok for me. I would love to see this implemented.

Best regards