Marathon: Tasks in state TASK_LOST should expunged by marathon?

[kongbo1987]

I found when mesos slave is down , the task in that node become TASK_LOST in marathon, and marathon will rebuild the lost task, should marathon expunge the lost task?
otherwise when I delete the APP will cause like "Cannot kill task XXX because it is unknown; performing reconciliation" and at the meantime I can‘t create the APP with the same name.

[gabfssilva]

Just a question, which Mesos version are you using? I got the same problem when I upgraded Mesos to 1.0.0, then I downgraded Mesos back to 0.28.2 because of it.

[kongbo1987]

Mesos version 1.0.0, you mean Mesos 0.28.2 is good?

[gabfssilva]

Since Marathon didn't release any version after Mesos 1.0.0, I thought it was some kind of incompatibility. At least with Mesos 0.28.2 the problem didn't happen with me. I guess you should give a try. :)

[kongbo1987]

OK, I will have a try, thanks for your suggestion :)

[unterstein]

Hi @kongbo1987,

which version of marathon are you using? Can you post the marathon logs around the slave lost timestamp?

Thanks
Johannes

[miry]

I have same issue:

OS: Centos 7
Mesos: Package mesos-1.0.0
Marathon: Package marathon-1.1.2

Is it possible to install marathon 1.2 or 1.3 from the rpm similar to Mesos ?

yum -y install --enablerepo=mesosphere-unstable mesos

[JayH5]

We have the same issue:

OS: Ubuntu 14.04
Mesos: 0.28.2
Marathon: 1.1.2

If a worker goes down, the tasks on it get stuck in a Starting or Unknown state in the Marathon UI. Unable to remove those tasks.

I notice that there is mention of a fix for TASK_LOST behaviour in the 1.3.0 changelog here that will be backported? Any news about that?

Upgrading to 1.3.0 is quite difficult for us with the requirement that we shut down all Marathon instances and also do a major Mesos upgrade.

[JayH5]

Apologies, what we were seeing was the new TASK_LOST behaviour. After setting the task_lost_expunge parameters to smaller values, the tasks got cleaned up. These parameters don't seem to be documented in the official documentation website yet, but you can see them here.

[miry]

@JayH5 good catch. thanks

[jdef]

@meichstedt @gkleiman sounds like we need some docs for new features, probably should land in 1.3?

[r4um]

Seeing similar issues with mesos 0.28.2 with marathon 1.1.2, voting here for back porting fix to 1.1.2.
Its not possible for us to upgrade to mesos 1.x right now.

Additionally we have apps stuck in unknown / reconciliation for more than 24 hours.
Looks like the default gc is happening in 24 hours as expected, the TASK_LOST state apps are gone.

[r4um]

Looks like (version 1.1.2) expunge is only triggered on leader election and those new options take effect, reconciliation settings are default. expunge settings

 --task_lost_expunge_gc 180000 --task_lost_expunge_initial_delay 120000 

[r4um]

Needed to reduce --task_lost_expunge_interval also. Working now as expected.

--task_lost_expunge_gc 180000 --task_lost_expunge_initial_delay 120000 --task_lost_expunge_interval 300000

[feythin]

@r4um it works for me . marathon-1.1.2 mesos-1.0.1, and add --task_lost_expunge_gc 180000 --task_lost_expunge_initial_delay 120000 --task_lost_expunge_interval 300000. the TASK_LOST gone

[lowzj]

@r4um same issue. It's very useful for me, thanks.

[Azulinho]

I had this issue on a bunch of different marathon deployments, and the '--task_lost_expunge_*' settings fixed the problem in every one of them.
Probably worthy setting those properties as the default or making sure they're documented in the main marathon documentation, and then close this issue.

[jeschkies]

With 1.4 one can define expunge behaviour per App. That should solve this issue.