Marathon: Support mesos maintenance modes

[mindscratch]

With mesos 0.25.0, new maintenance primitives have been added. Based on the documentation it appears frameworks can be maintenance aware.

Marathon should support mesos maintenance modes.

[cmluciano]

👍🏽I'm super interested in this. Is there a public roadmap or developer list that I can post to for a status on this feature @aldipower ? My team and I would be interested in assisting with this code, but we don't want to replicate work if someone is already working on it.

[sledigabel]

+1
Super interested.

[wendigo]

+1 can help :)

[jpza]

Bumping this as well +1

[xperimental]

Any news on this feature?

[DavidXArnold]

+1

[vidhyaarvind]

+1

[Salaander]

+1

[travisgroth]

Any traction on this? Would be a much appreciated feature

[erSitzt]

+1

[richardmoe]

+1

[bateau84]

+1

[xperimental]

Can you please stop adding "+1" posts? Please use reactions instead.

[janisz]

This is blocked by #2382 for more details see MESOS-2063

[svshah]

Any updates on this?

[rhoml]

This is super important, would love to see this implemented

[janisz]

I think we should start with design doc on how we want it to work. There are more than one approach to handle maintenance mode and we need to decide what will works best and how to allow customization. It's we need to discuss on policy about handling reverse offers and respawning tasks on other nodes.
It's on maintainers side if they will merge community driven feature into upstream and if they approve this approach.
cc: @gkleiman @aquamatthias

Definitely making this feature will be reasonable to do after switch to HTTP API which is not super easy but doable. On the other hand I believe companies that really need maintenance mode have already implement it outside of Marathon. In fact supporting reverse offer is not required to have maintenance mode. It's worth to note this feature firstly appear in Aurora and then was implemented in Mesos. So I recommend you write how you want it to behave or how you implemented it. Compare how it's done in other open source framework and then do it.

[rhoml]

That makes sense.

This is important in order to ensure correct draining of tasks when we want to do maintenance on certain mesos-agent nodes. Otherwise, throw requests on the floor when let's say we want to change meta attributes in mesos-slave.

Yesterday, our marathon framework was stuck trying to deploy certain number of applications ~80 tasks, when we took one of our mesos nodes out of the fleet causing our loadbalancer to not be able to update HAproxy configuration because the http_callback never made it.

On a ~2k tasks node I don't know how marathon would behave trying to reconciliate tasks in the cluster

[janisz]

Because nobody is working on switching from libmesos to Mesos HTTP API I propose to handle maintenance without it by periodically checking /maintenance/schedule and taking actions according to data that are there.
I propose to handle it in following manner:

1. Change scheduler to ignore offers with Unavailability that occurs in next configurable time interval. For example we probably want to decline offer of a node that will be down in next quarter because we will need to reschedule that task. #4653
2. Create an agent that will be responsible of checking maintenance schedule and notifies when node is about to go down (with configurable advance).
3. When node is about to go down we will block all offers from that node (done in previous point) and spawn additional task on other node. We can kill task on maintained node but I think it's not necessary because when node will be down we will get TASK_LOST.
4. When Marathon starts using Mesos HTTP API we can replace maintenance monitoring agent with Inverse Offers handler and keep logic unchanged.

What do you think about this approach as first try. I can work on this ticket full time starting from Nov/Dec but before I want to discuss possible solution and choose best.

[rhoml]

Sounds like a good first approach

[jasongilanfarr]

Seems reasonable to me. We are working on mesos http support but it's a fair ways out.

[rushabh268]

Any update on this?

[janisz]

Today I returned from vacation and will be working on this. I'll follow above plan. I'm starting with #4653. BTW there is WIP branch to add mesos-http-adapter so maybe HTTP API will be working ealier than I expected.

[ottoyiu]

:+1:

[kangman]

has anyone found another work around while this is being developed?

[EvanKrall]

What we do in paasta is basically:

1. When we start draining a node, we create a reservation for all remaining space on that box.
2. When our bounce process runs, it counts how many of a Marathon app's tasks are on boxes with maintenance scheduled, and then scales up the app by that amount.
3. Once those new tasks are healthy, the bounce process will remove each of the tasks running on the at-risk hosts from the load balancer, and then kill-and-scale them.

This isn't great; it only happens as fast as we can poll (which keeps getting slower as we add more services), and we don't always manage to reserve all the resources as fast as they become available, so Marathon will occasionally start new tasks on the box we're trying to drain.

[erSitzt]

I found this gist some time ago, if i remember correctly it is redeploying tasks with an UNLIKE-constraint for the host you want to put in maintenance.
I would not use it as is, but it could be used as a base...

https://gist.github.com/pradeepchhetri/2eafdc75d89236018cbf9e6e57bc1885

[EvanKrall]

One problem with the UNLIKE approach is that it requires a full restart of your application.

[kangman]

@evanKrall how does your bounce process know which boxes are scheduled for maintenance, does it just look for where space are all booked up by the reservation Mesos api? Why use the reservation Mesos api rather than the maintenance primitives to mark a node for maintenance?

[EvanKrall]

We query /master/maintenance/status: https://mesos.apache.org/documentation/latest/endpoints/master/maintenance/status/

[meichstedt]

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-3216. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.