Marathon: Does marathon has a limit number of retry times ?

[xianlubird]

When I run a container on marathon.If this container is wrong , it can not launch correctly.Marathon will always try to start this app many times. Does marathon has a parameter that can control retry times ? I see a parameter that called maxLaunchDelaySeconds, but it seems not to be work.

[meichstedt]

Hey @xianlubird Marathon will try to get your app running. If tasks terminate, Marathon will apply a backoff and increase that backoff for further failures until it reaches a given maximum delay. The deployment of your application will display delayed in the GUI. With the combination of backoffSeconds, backoffFactor and maxLaunchDelaySeconds as described here, you can control the relaunch behaviour. The defaults start with a 1 second delay that is multiplied by a factor of 1.15 until it reaches a 1 hour delay or tasks turn out running.

If you think this is not working for you then please specify the problem you're experiencing and/or provide some logs. Thanks!

[xianlubird]

I set backoffSeconds=1 backoffFactor=2 maxLaunchDelaySeconds=6 and my task is a container that can not run corrently. Then the task should try first and fail and after 2 second it will try again and fail then after 4 second it will try and fail.This time, it has reached maxLaunchDelaySeconds and the task should not be try again. But I see the situation is that the task try again after retry 3 times and retry forever. The app's status is turning from running to waiting to delayed and cycle again and again. My marathon version is v0.11.0 mesos is v0.23.0. Thank you .@meichstedt

[meichstedt]

Hey @xianlubird Maybe the documentation is misleading on this. Once the backoff delay reaches maxLaunchDelaySeconds, it will not become _greater_ than that – but it will keep being delayed. As I wrote above, the default for maxLaunchDelaySeconds is 3600 – 1 hour. This means, if the tasks fails over and over again, it will at some point reach that maximum and only be retried after one hour. Setting maxLaunchDelaySeconds to 6 seconds effectively prevents the backoff mechanism.

[xianlubird]

Thank you @meichstedt . So I have not method to stop marathon retry ? If I want to control max retry times in marathon , how should I do ? Could you give me some suggestion? Thanks.

[xianlubird]

Are you still there ? @meichstedt

[meichstedt]

@xianlubird I don't think that a stop retrying functionality really makes sense. The goal of Marathon is to keep your applications running. Not retrying would be somewhat contrary to this goal. Can you please explain why you need such a functionality?

Please consider discussing this in the marathon google group

[xianlubird]

If I have a wrong container which will abort once it it running. Because this container's code or something is wrong, I was debug it. So if I do not notice this ,marathon will always retry this already wrong container, this will waste much time and resource, I have to write a code to receive marathon callback event, If container's state change for 5 times ,I have to call marathon API to destory this app.I want marathon have a setting env which can support this function, If this container has been tried three or five times, marathon will stop retrying this meanless behavior. Thank you @meichstedt

[Colstuwjx]

+1, stop retrying equals fast fail, it would be better if marathon support.

[meichstedt]

I think the general topic is around if/how deployments should fail, and what should happen if they do. Somehow related to #4046

@xianlubird Sorry this fell off the radar, @Colstuwjx thanks for bringing it up again.

How exactly is a failed deployment better than a deployment where Marathon retries over and over again? You'd still have to poll the API or subscribe to the event stream in order to find out that the deployment failed, right?

[Colstuwjx]

Thanks for your nice reply, I just need a retry_limit to avoid Marathon retries over and over again.

fast-fail can make other way works, such as let backup cluster up while active cluster down or just rollback while deployment failed, but dead-loop-retry will make things broken.

[meichstedt]

@zen-dog since you've restructured the deployment logic: can you add some thoughts on what it would mean to add the concept of a deployment that _fails_ after some time or a number of subsequent Condition.Failed?

Note that in this scenario (a broken container config) the deployment actor would only receive Condition.Failed (the equivalent InstanceChangedEvent depending on what it uses now), no Running for any newly launched instance.

[zen-dog]

Would be a bit of work I suppose. And if we're restructuring it, I'd rather simplify it (not complicate it further) and move the extra logic out either by listening to event bus events or elsewise. This may be a part of our v3 API discussion.

[chroto]

Hi. came across this issue because i was looking for this feature.

possible use case that brought me here was 'what if someone asks for resources that can never be allocated' i.e. i ask for 100 CPU for my application when the cluster only supports 50 at max.

I just tried this to see what happens and the application hangs in Waiting forever. I expected it to fail after a while which would prompt me to investigate logs to see why it could never run.

Just my two cents.

[ups216]

+1 I think it will be useful to allow the deployment to stop if it reach the retry_limit. If the deployment is wrong, there is no reason to try this again and again.

[janisz]

:-1: I'd prefer to keep Marathon simple. If deployments hangs due to lack of resources then monitoring should give you notification that your cluster is running out of resources. If deployments hangs because of wrong configuration (e.g., improper constraints) or failing task (e.g., wrong url) then somebody should be interested to check why deployment takes so long or why new version is not running yet.

This is related to https://github.com/mesosphere/marathon/issues/4046

[samart]

Whatever you guys have implemented is not working. If an app is failing repeatedly, why keep trying. It exhausts system resources (like docker thin pool, logs, service discovery add/removes) needlessly

A simple "retry_count" is all people are asking for. Amazing this does not exist. All we have are operational issues because of this. We now have to develop our own solution to detect this and scale the app down to 0 to stop it.

[jasongilanfarr]

We are planning in 1.5 to implement an automatic rollback after a certain
duration. Perhaps this will suffice your needs?
On Tue, Jan 17, 2017 at 12:25 PM Trevor Samaroo [e-mail]
wrote:

Whatever you guys have implemented is not working. If an app is failing
repeatedly, why keep trying. It exhausts system resources (like docker thin
pool, logs, service discovery add/removes) needlessly

A simple "retry_count" is all people are asking for. Amazing this does not
exist. All we have are operational issues because of this. We now have to
develop our own solution to detect this and scale the app down to 0 to stop
it.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mesosphere/marathon/issues/3035#issuecomment-273289061,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAweZp-tuXtRLWfkjYt_NQZUlwxKss50ks5rTSPHgaJpZM4HISI1
.

[samart]

It would nice to have the option simply stop. Very simple, no magic, no guesswork and operationally sound and explainable to our developers. You could have a dependency (db, etc) that is down and rollback is not what you might want. So - simple. Stop.

[samart]

You can imagine you upgraded your db schema or message format and the last thing you want is to rollback, as that would be catastrophic

[janisz]

As @samart pointed out there is no generic solution that will work for everyone. Rollback is dangerous, and simple Stop could lead to inconsistent state. What to do with applications that are partially deployed e.g. Fail to run on specific agents?
This feature could be done outside of Marathon with simple service listening on events and based on this taking proper actions. Probably similar to autosuspender mentioned here

[jasongilanfarr]

I concur. We currently have two supported mechanisms (force will be/is
deprecated as it is super dangerous): force - override an existing
deployment with a new one and rollback. In 1.5, we will optionally support
auto-rollback as well. There isn't really a viable "works for everyone"
solution in the current model, but we are hoping to support a much more
flexible model in the long term future.
On Tue, Jan 17, 2017 at 3:29 PM janisz [e-mail] wrote:

As @samart https://github.com/samart pointed out there is no generic
solution that will work for everyone. Rollback is dangerous, and simple
Stop could lead to inconsistent state. What to do with applications that
are partially deployed e.g. Fail to run on specific agents?
This feature could be done outside of Marathon with simple service
listening on events and based on this taking proper actions. Probably
similar to autosuspender mentioned here
https://github.com/mesosphere/marathon/issues/4942#issuecomment-272262441

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mesosphere/marathon/issues/3035#issuecomment-273334770,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAweZqSRElaLt4DiSuMzhHiB4FMnPR-dks5rTU7pgaJpZM4HISI1
.

[samart]

What about pause. Like a k8s deployment. So, literally, stop trying to schedule the task and leave it how it is. It's better than destroying your cluster! And it's then in delayed state.

Seriously, it's hosing our machines and it's because we don't have the knobs to control in marathon. Make it an option, I'm not asking this to be default. Then people can choose. Right now, there's not much that can be done other than, building something ourselves to detect and act. But it seems like a pretty simple knob to add. Anyways, thanks for considering

[samart]

Can i ask if i am understanding this correctly?

if i have
Backoff = 3 seconds, Backoffactor = 2, and maxlaunchdelay = 3600

That should give ~ 10 attempts, 3 seconds apart, then wait 3600 seconds. Rinse and repeat.?

[janisz]

Correct