Mailu: Discussion about project management

[kaiyou]

Hi to everyone. First of I want to apologize for not being as reactive as I would like: my work and personal life have taken a turn where I cannot spend more than a couple hours per week on my projects, and I still have a few of them running.

Mailu remains one the most depending projects and I would like to give it a proper care, and that probably means giving the community more credit for its contributions. I can count at least 5 people providing more meaningful contributions than I have in the past month and that I would like to empower somehow in the project.

I have invited main contributors to the team, as team members, so they can review pull requests with automated merge coming along the way, and manage issues. I hope this will help Mailu live even when I cannot connect for multiple weeks. I will also make sure that I am privately available to main contributors and that we can properly discuss strategies.

Finally, we have discussed financing the development in a previous issue (#508). I have decided to setup a Patreon (https://www.patreon.com/kaiyou). The idea is that I might keep small amounts to keep the project going (pay for Github fees, maybe a demo server again as it expired) and will redistribute the rest to contributors.

Any comment is welcome :)

[benyanke]

Thanks for this!

As mentioned on the other thread, I think https://www.bountysource.com/ could be another great tool I'd love to see implemented. Not only does it provide a nice way of seeing the features people really want, but it provides incentives - at zero cost to the project - to get them done, either by the maintainers, or by 3rd parties via PRs. It integrates nicely within the github issue comments.

Finally - on a project administration note - have you been able to get all the project infrastructure (docker hub, github org, hosting, domain registrar) to at least one other trusted admin? I've heard of far too many promising FOSS projects get abandoned or have a rough time (Slackware, Solus, copperheados) get themselves into major trouble because of the "bus factor" of a single person being the only one with access to core project infrastructure. As someone who would absolutely want to see this project continue, I'd encourage you to look at this if you haven't already.

[kaiyou]

Github repos have a backup access and so do every piece I host. I guess Docker hub and now Travis are the missing pieces.

[curry684]

I can look into sponsoring a demo server and website infra via my company if you want.

I also don't mind taking 'notary' access to the Docker Hub, Travis and Github projects to help prevent them from ever getting abandoned completely.

[kaiyou]

Regarding Github, Docker Hub and Travis, I already have one trusted third party ready, we need to set some instructions about takeover in case I disappear in anyway but this should be handled pretty soon.

Regarding the hosting of the website, it is already running on my infra and is really costless. However, the demo server is a hassle and any help is welcome. I have to warn though: people will abuse it, maybe we should restrict the sending of outbound email to prevent spam and preserve ip reputation.

[benyanke]

I agree - demo server should dump to /dev/null for outbound. Or perhaps have two demos, that will only ever send to each other?

[HorayNarea]

Maybe we should have a "absolutly no outbound"-feature, that is set in the env-file and can't be overwritten anywhere else?

[mildred]

First, thank you for trusting me to become part of this project. Then, I have a few questions...

About project management, is there some existing guidelines about bug triage, reviews (what should and shouldn't be accepted, when a PR can be considered good to merge), workflow (is there an order to merge the tickets) ? There is now GitHub projects that provides Kanban boards, perhaps this can be useful in some way.

Also, I recently came across https://apenwarr.ca/log/?m=201712, a long article that talk about project management, and the best part I found was near the end when he talks about bug triage.

[kaiyou]

Hey Mildred, sorry for inviting you without prior notice, I probably was a bit rude on the matter.

Regarding bug triage we currently have no guidelines but any proposal is welcome and I will have a look at that article.

[curry684]

Maybe we should have a "absolutly no outbound"-feature, that is set in the env-file and can't be overwritten anywhere else?

Why bother, that's what firewalls are for.

It would likely even work to just disable the Postfix container, everything else should still be demoable.

[kaiyou]

I just created a whole bunch of labels and will start relabelling the issues and closing the stale ones.

[HorayNarea]

How does @kaiyou / do we decide when to create a new release or backport from master to a still supported release-branch?

What do you all think of production-ready rolling releases on master in addition to the release-branches?
(Maybe only after having automatic tests)

[kaiyou]

@HorayNarea agreed about marking master a bit more production ready once we have tests. In fact, I believe many are already using master as a production branch, which only suggests we should be more thorough with testing and careful not to break it too much.

As to releases, I believe we can mark issues for specific versions. I have not reviewed the state for these in a while, but I guess we should have a thread everytime a new release is available to discuss quickly which are the core points for the next one.

Currently, I would say 1.6 is waiting for :

• abstract database,
• configuration wizard.

Feel free to comment and discuss these :)

[muhlemmer]

Can I open the discussion again about alpine:edge? I see it as a risk using it. Yes, maybe more prompt updates. But also more risk of breaking, like I'm seeing now with the musl libc.

It seem that alpine:edge is not rebuild on a very regular basis. So a bug introduced in a build, will take time to be resolved.

In my opinion we have 3 real options:

1. Stay on alpine:edge and run a apk upgrade to get the latest bugfixes. I believe this is against best ocker practices.
2. Go to versioned Alpine versions
3. Use alpine:latest and always pull in the latest released version

[kaiyou]

I vote for option 2, starting with 1.8 in the master branch.

[curry684]

I've been operating under the assumption that Alpine is being deprecated over the past few months in favor of Debian Slim, and building all my containers based on that. Only slightly larger but no compatibility issues whatsoever.

[benyanke]

I've been operating under the assumption that Alpine is being deprecated over the past few months

This is news to me - more info?

[curry684]

Nothing official, I just saw it being updated badly over the past few years and then noticed myself that many of the bigger official containers started providing Debian Slim containers, while others removed Alpine varieties for compatibility reasons.

[HorayNarea]

I've only seen that projects which supported alpine & debian removed alpine-based images to reduce the amount of work they have to do

[curry684]

Like I said - it has been my assumption that I've been acting on, nothing official. But the move ofc makes sense that it's easier to build containers with a bare-metal version of the most common Linux server OS (Debian/Ubuntu) than to maintain 2 versions that have to be tested et al, and Slim is likely being maintained much better and most of all faster (I've had bad experiences waiting for Alpine to update even the not-quite-obscure packages within months).

[muhlemmer]

Alpine (which uses musl) is primarily focusing on size. As glibc based would be more performant. In my opinion it can be an advantage here to use something Debian based.

We can make up for the size difference by standardizing our images. Docker tries to reuse base images in a stacking matter. This means, that every step in a Dockerfile creates a new image layer. At this moment we are implementing this feature very poorly. We use the following base images:

• alpine:3.7: 3 times
• alpine (=latest/3.8): 1 time
• alpine:edge: 3 times
• python:alpine: 1 time
• php:5-apache: 1 time (based on debian-slim)
• php:7.2-apache: 1 time (based on debian-slim)

That's 3 different alpine base versions and one debian-slim. Meaning, if we move completely to debian-slim, the total footprint will be smaller.

We can optimize more. Some Dockerfiles are installing python and / or pip and some use a Python image. All of them are (or will) use a python script to start. So we can base most of the images on python:3.7-stretch-slim. Not for the webmails, as there primary resource is PHP/apache. (Can we use the same PHP version at least?)

[HorayNarea]

Can we use the same PHP version at least?

IMHO, we should always use the latest supported PHP Version, even if that means having 2 different base-images… but you are right, at the moment both webmails run fine on PHP 7.2

[curry684]

Also PHP5 is about to permanently die.

As for the PHP containers - I've been building most of my own containers based on stretch-slim directly and building Sury PHP FPM packages into them as it makes far smaller containers than the official ones that do custom compilation. Even apart from that we should switch to FPM anyway as the already included Nginx can serve from them right away, which would remove the unreasonably heavy and unneeded two Apache instances.

[muhlemmer]

FPM can also help in reusing resources like IMAP connections. (Also under discussion somewhere I remember).

On October 9, 2018 9:40:34 PM GMT+03:00, Niels Keurentjes [e-mail] wrote:

Also PHP5 is about to permanently
die.

As for the PHP containers - I've been building most of my own
containers based on stretch-slim directly and building Sury PHP FPM
packages into them as it makes far smaller containers than the official
ones that do custom compilation. Even apart from that we should switch
to FPM anyway as the already included Nginx can serve from them right
away, which would remove the unreasonably heavy and unneeded two Apache
instances.

--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/Mailu/Mailu/issues/593#issuecomment-428304183

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[kaiyou]

@curry684 I do not believe we can easily just use fpm without having an additional layer of either nginx or Apache to serve non php files though.

[curry684]

Correct, I usually build my static files into the Nginx container for that reason, with Gitlab CI it's pretty easy using artifacts. Don't really have any experience doing such a thing with Travis regrettably. We could consider using multi-stage Docker files to prepare the artifacts.

[kaiyou]

Well, I clearly would not want to insert artifacts for every supported webmail directly in the front container, that sounds both overkill for the performance gain and terribly difficult to maintain. Switching to fpm sounds ok as long as we have a proper way of handling these static files, e.g. an additional layer of nginx.

[curry684]

When set up properly it's actually extremely highly performant and easy to maintain, and less error prone than chaining webservers to eachother - I can attest from experience to how much you won't be having to debug which one is not properly forwarding or respecting X-Forwarded-Proto or X-Forwarded-For headers, thus accidentally causing backends to generate invalid URLs or, worse, ignoring IP whitelists and opening up sensitive data to the entire internet 😉

I agree for that exact reason it's not a priority to change this overly soon because of the (security) issues it might cause accidentally.

[kaiyou]

Well, it requires that the front container has a built in copy of assets for every possible back end application that runs php.

That not only make it a burden to build these asset bundles, but also creates specificities for php apps, which might not apply to other webmails.

If you have proper documentation on how this can be properly managed, maybe I can be persuaded. For now I am quite convinced this is a bad idea even if it brings a performance bonus.

[muhlemmer]

I agree with @kaiyou.

Also, we are developing a mail-server not a e-commerce website. The static assets are relatively small in our case. The biggest data the webmail will be dealing with will be attachments, which should be served from the web mail server dynamically. Not sure how the webmails are dealing with attachments in the background, but I guess it will be a hell to set it up correctly.

I'm not really familiar with ngnix, but you could just setup a proxy catch on the front for the static assets and be done with it.

As for getting rid of Apache, I agree. I was never a fan of it.

Base image

Meanwhile, back to the base image discussion. I'll send in some PRs moving to alpine:3.8 to fix #625 and #627.

On the long rung we are going to have to decide if we want to stick to alpine or not.

[kaiyou]

On sticking with Alpine or going with Debian slim, I have no strong feelings about one or the other. If the general trend is to switch to Debian slim, then it might be a good idea, but I see no immediate pressure and we still have a lot of operational issues to deal with before this becomes a priority :)

[curry684]

Agreed, I think in general the moves that have some sort of priority are:

1. Unify base images to 1 Alpine base and 1 Debian Slim base to reduce overall footprint. This is a big and easy save in disk and memory footprints, and install and update times.
2. Ensure all components use their latest supported platform (ie. PHP 7.2) and we are not at risk of running EOL components soon (ie. PHP5). This is just sane maintenance procedure of an active project.

Of less immediate importance:

1. Unify base images to a single platform, likely Debian Slim. Reducing 2 to 1 is a far more marginal improvement than bringing the current 6 down to 2.
2. Get rid of the unnecessary weight of Apache, switch to PHP-FPM and use a single Nginx instance for all internal routing. Due to inherent complications discussed before this is a complex operation and has no real priority as it is not 'broken' right now, just overweight (if this is started, highlight me in a dedicated issue and I'll share experiences on how to build and set up).

[muhlemmer]

Well, we just moved to PHP7.2 for both Rainloop and Rouncube. @HorayNarea submitted a new PR #649 to reduce even better (some testing required). Wrt alpine version, we are almost there.

We could use some help with testing/reviewing off outstanding pull requests. I feel that at this moment I spend too much time on this.

For the move to Debian, I think we can create a seperate issue for this, with low priority. Maybe putting it on the 2.0 milestone? I did some studying on the subject and I would like to volunteer for the task. But after we solve priority0 issues and release of 1.6.

[muhlemmer]

Regaring the issues tags. Would it be helpful if we specify something like field/php, field/python etc for feature requests and bugs? This way we can easily create an overview for people who want to start contributing, and have certain experience in only some fields.

[curry684]

Makes sense. I'm happy to help with PHP and Docker issues but my Python knowledge is kindergarten level.

[kaiyou]

Regarding branches and tags, I would very much like to:

• remove all my personal features and fix branches from this repo and move them to my personal fork;
• remove 1.5.1 and stop using tags altogether, no single version of this project is stuck in time.

What do you think?

[benyanke]

1) What type of fixes and features? Some might be good candidates for keeping here, if they're of general interest. I could see this cleanup being valuable too though.

2) For something being used in production, I think there's value to tagged releases, including two special tags: latest tag which just leapfrogs to the current tag, and nightly, which tracks every commit. Tagged releases allow better stability. It makes me a bit wary that every single commit to master instantly goes to production on people's mailservers. Tags could be fairly lightweight though, and frequently tagged on commits which have even been lightly hand-tested.

[muhlemmer]

@benyanke: latest and other version tags will still be in docker hub. This is about removing tags in GitHub. Releases then exist only in branches.

@kaiyou I agree removing the tags altogether. I think we are getting versioning questions a few times a week now.

Also, I agree remove personal branches. However, it should be possible in the future to create Mailu branches in case people have to collaborate on a major feature.

[benyanke]

In that case, full agreement @muhlemmer

[kaiyou]

Okay, I will start cleaning things up then. And @benyanke the reasoning behind latest not being the stable branch but the master instead for Mailu is the following: we introduce breaking changes in new branches, including breaking changes to the compose file and other deployment files, so there is no way we can provide a single latest tag that people will be able to upgrade forever.

This is why we encourage people seeking a really stable setup to use branch tags, which still benefit from backported patches, and regularly check for new branches and migrate when necessary (which does require manual maintenance).

[kaiyou]

@hoellen I see that you are reviewing a couple PRs, would you like an invitation to the team so that your approvals are counted?

[hoellen]

If you need any more help, I'll be happy to help.

[muhlemmer]

@hoellen Welcome to the team!

[curry684]

I fail to see completely how removing tags from repository and registry are in any way productive?

Common interpretation of SemVer and how it helps maintain system stability is that branches are minor versions and tags represent patch versions. So ie. if I run tag 3.2 I expect it to automatically update to 3.2.1, and 3.2.2 and so on. If I want more manual control over updates I'll run 3.2.1, and change it to 3.2.2 myself when ready.

In both cases the potentially dangerous update to 3.3 is manual (although many images allow you to do so by selecting image tag 3) and the by definition highly dangerous update to 4 is by definition manual.

This scheme works for just about every other product on Docker Hub, and is how people expect things to work. What good reason is there to consciously deviate from expected and well defined behavior?

[kaiyou]

@curry684 removing tags is simply meant to acknowledge that our way of tagging until now was definitely not consistent with semver, or any other standard for that matter. It was a very poor decision trying to display some version numbers in a non-consistent and non-useful manner. So we are definitely better off without them and without inconsistent branch names like 1.5.1, that mostly deceived people and led them to running unmaintained versions of Mailu.

As to the question: should we provide a more standard panel of Docker image tags, and follow standard naming, the answer is: yes, probably. But that will require some time and maturity. Also, it is not to be confused with Git tags, which are the ones I removed.

[curry684]

Ok, that seems to imply that we all agree it would be a good idea to re-embrace SemVer and common Docker versioning practices for the next major version. I can agree with that.

[muhlemmer]

Hey guys. For the last couple of weeks, I've been working together with @ionutfilip to get Mailu a bit up to speed for our business needs. Now I'm looking to hire another person with a sys-admin background. Partly I want to offload some of our internal jobs to him. Secondly, I'm thinking to use him as a testing engineer, doing reviews for Mailu.

I want to hear opinions if anybody sees this as a conflict of interest. In the worst case he'll be reviewing the commits of his employer.

Please know my plan is to follow the normal way: he'll be doing un-official reviews until the Mailu team considers the quality of the reviews sufficient to give him the official review permissions.

[muhlemmer]

@kaiyou, related to this, could you add @ionutfilip to this organization? This way I can assign some issues to his name. No need for write access, just regular membership.

[ofthesun9]

Hello,
Happy to contribute here from times to times !
What is the most efficient way to "test" the patches proposed by the other contributors ?
I would like to apply thoses patches to my own fork to build and test, but I don't know what is the most efficent way to do it ? (with git command line, if possible :-))

[muhlemmer]

@ofthesun9, that sounds great! It might be a bit complicated when you want to maintain / test against you own fork. But I'll write up the way I'm doing it on my testing server. You can change the branch names to whatever you want. Just make sure you never commit / merge anything to the local master of a development PC. That will mess up future syncs good.

Some definitions

origin: your own repository
upstream: Mailu repository
<sender>: placeholder you'll use for the repository where the PR originated from.

The base

First, clone your own repository to the server where you want to test. I prefer to always call my own repo "origin", as it matches my development station. It prevents mistakes. Secondly, we will add "upstream" as a remote, as we normally test against the Mailu:master.

git clone https://github.com/<you>/Mailu.git
cd Mailu
git remote add upstream https://github.com/Mailu/Mailu.git

Add the sender

I will use "kaiyou" as the <sender> of the PR. Also, later I will use an actual branch. You will need to adjust this to the needs of the review.

git remote add kaiyou https://github.com/kaiyou/Mailu.git

Merge conflicts

Before proceeding, check the PR page in the bottom. It should not indicate a merge conflict. If there are merge conflicts, you have 2 options:

1. Do a review "request changes" and ask the author to resolve the merge conflict.
2. Solve the merge conflict yourself on Github, using the web editor.

If it can't be done in the web editor, go for option 1. Unless you want to go through the trouble of importing the branch into your fork, do the merge and send a PR to the repository of the <sender>.

Merge a PR

When someone sends a PR, you need merge his PR into master locally. The following will put you in a "detached head" state and do the merge in that state. Any commits done in this state will be lost forever when you checkout a "normal" branch. This is exactly what we want, as we do not want to mess with our repositories. This is just a test run.

The following must be done on every PR or after a new commit to an existing PR.

1. Fetch the latest status of all the remotes.
2. List all local and remote available branches (this is not needed, but very helpful at times)
3. Checkout upstream/master
4. Merge upstream/master with <sender>/branch

git fetch --all
git checkout upstream/master
# ...You are in 'detached HEAD' state.... (bla bla bla)
git branch -a
# Hit `q` to exit the viewer, if it was opened. Uses arrows up/down for scrolling.
git merge kaiyou/fix-sender-checks

If git opens a editor for a commit message just save and exit as-is. If you have a merge conflict, see above and do the complete procedure from "fetch" onward again.

Build

Export the correct version. This should correspond with the the version you use in docker-compose.yml or .env for running. After that, you can run a build. You can suffix the build command with service names if you consider a partial build is enough for the test.

export VERSION="master"
docker-compose -f tests/build.yml [<service> ..]

Run

cd /path/to/compose-file
docker-compose up -d
docker-compose logs -f

Now, play around. See if (external) mails work. Check for whatever functionality the PR is trying to fix. When happy, you can approve the PR. When running into failures, mark the review as "request changes" and try to provide as much as possible details on the failure. (Logs, error codes form clients etc).

Notes:

1. Github marks positive reviews as obsolete when a new commit is added to a PR. This requires a new review from you side.
2. I wrote this from the top of my head. There might be typos in the git commands. When running into trouble, use git <command> --help to read the docs.
3. I was planning on writing this for a longer time. I will add this to the developer docs as well, later this week.

[ofthesun9]

Thanks for the detailed overview !

[muhlemmer]

Hi all, open for discussion: #722

[hoellen]

In summary, I think this idea is great. So small changes are approved faster (like #719).
But for big PRs, like #670 and #682, I would still request two approvals, because this changes are fundamental. So maybe we can introduce a new label, e.g. PR/need 2 which request 2 approvals.

[muhlemmer]

Separate from the #722 discussion.

682 seems big but does not address anything that affects functionality. The test suite and setup are both developed by us (usrpro) and this PR is just an continuation on that. Since it cannot break anything in production, I would say a single review would be enough. The reviewer would need only to check if there are no modified files in the core. If there are any actual errors or things to be improved, it would be more easy for people to contribute after it is merged. I will elaborate more in a comment on the PR.

On the discussion of #722: I could create a rule that protects the directories core, services, optionaland webmails from single review. And probably also the LICENSE.md file. But I would like to hear the opinion of others and @kaiyou first.

[muhlemmer]

The real "problem" I try to solver here: most people are doing this in their free time. In the case of a "trusted" PR, you currently need a total of 3 members to make the change happen. And it seems during the last month that that is hard to achieve. For me it sometimes feels I'm trying to push a dead horse. (I meant that to be funny, no offense)

[HorayNarea]

I'm with @muhlemmer and would approve PR #722 but I think this should be just @kaiyou's decision.

The suggestion of a "needs 2 Reviews" label seems like a good way to reduce the risk if we merge #722 but then we should also write a bit about "when you should add needs2reviews label to your PR" or we could end up with contributors adding the label always because they're not sure.

[kaiyou]

This sounds just right, especially in the light of my recent lack of involvement in the project. Regarding #722, I am just sad mergify does not have any smarter syntax, but I am perfectly fine with the spirit. Just checking the syntax and merge should happen in minutes.

[muhlemmer]

@HorayNarea I somebody of the team needs a second opinion, they can always select "comment" and write what they've already tested or why they need the support of a second team member for review. A comment neither approved or blocks a PR.

Nevertheless, I will implement the label now and we can always choose to drop it, if it does not work out in practice.

[muhlemmer]

Implemented review/need2 label in #726.

[muhlemmer]

Dear all,

I'm thrilled to announce you that @kaiyou has approached me this week to join him in managing this project. He has been to busy lately to be very involved in Mailu and the project really needs love and attention.

Our involvements

I will be mostly involved with the daily business of things around here. I will fall back to Kaiyou if I'm not sure or have questions. Kaiyou has granted me admin rights on GitHub and Docker hub, so I can manage all things necessary. I've also gotten log-in credentials to manage the mailu.io domain name. In the near future I will (re-)deploy the documentation web server, the demo server and the setup utility. Kaiyou whishes to remain involved in the Admin UI design, as he still has plenty of plans with that.

About me

My name is Tim Mohlmann. I'm from Holland and live in Romania for some time now. I'm running a small IT start-up.

Usrpro

Under the "usrpro" brand-name we aim to provide complete collaboration packages to small and medium sized companies. A "package" basically exists out of a Mail platform (Mailu), cloud package (Nextcloud) and web-hosting. We see that open source is out there and is capable of fully replacing Google suite or Microsoft Office 365, however most company owners don't really know how to get there. It is our business to implement, host and maintain open source packages for our clients, offering them full control of their own data.

Mailu

As mentioned, Mailu is one of our products. However, it is not commercially supported. This is how we became involved in this project. Now we're even considering offering commercial support for Mailu as a separate offering. The open license of Mailu is important. We will never keep code behind closed doors. Everything will always get submitted back to Mailu.

[muhlemmer]

On an operation note: I've disabled Docker Hub auto-builds and enabled image upload from Travis. This way we are sure that only tested images end up on the hub.

As a consequence, the following new images are now available for download:

1. Setup
2. Unbound resolver
3. Documentation

[curry684]

Heej medehollander, goed bezig zeg ik dan 😉

[muhlemmer]

Another request for comments: #761. Please reply in that thread if so.

[muhlemmer]

Apparently mergify can't work around or with branch protection. Neither is it capable of using it for protection by simply monitoring its status (yet), like Travis-CI. This is more to blame on the Github API I guess. More details: https://github.com/Mergifyio/mergify-engine/issues/312#issuecomment-447746672. As a consequence Merigify can't merge as long as we have the branch protection rule set to minimal 2 reviews.

We have quite a small group of people in the @Mailu/contributors group, which personally I trust enough to relax the branch protection rules a bit. I will drop the minimal review protection setting to 1. Mergify policy will still stay the same to merge after 2 reviews when the PR is from an external contributor. And 1 review for a PR from a trusted user.

This means, that in any case after a single review the merge button will turn green and could be used by anybody of the @Mailu/contributors team. However, I can expect that we are mature enough not to miss-use it.

I would like to add that anybody in the team is free to do a "relaxed" manual merge when the PR is trivial, like documentation and setup improvements. (until setup is kinda stable I guess...)

[muhlemmer]

Demo server is back up. For details see: #786

[kaiyou]

Closing this old thread of mine.