Marathon: Marathon stops responding

[chris530]

mesos 1.0.1-2.0.93
marathon - 1.3.6-1.0.540
ubuntu 14.04 1.8.0_91

The last couple days, marathon will work for a couple hours then will stop working. Usually when it stops, it will go in the inactive framework section of the mesos gui. I tried a couple different other versions of marathon ( I have the newest one ) to see if it's version related issue with no luck. I run marathon on port 8080. When it stops working the port is not responsive but the process is running. Restarting marathon will fix it

[chris530]

I run 3 zookeepers ( exhibitor ) on 3 servers via docker, is it possible something in zookeeper is corrupted ? Running marathon with no HA to see if that works. It has been working for for a couple months prior

[unterstein]

Hey @chris530 ,

can you post the full log when the zookeeper issues comes up? Which zk version are you using? Did you update marathon or zookeeper?

Thanks
Johannes

[chris530]

http://pastebin.com/tHsW776y

Thanks for taking a look!. I am using exhibtor 1.0. which according to the Dockerfile it runs zookeeper-3.4.6. ( https://hub.docker.com/r/netflixoss/exhibitor/~/dockerfile/ )

I did destroy and recreate the zk cluster but still getting the same issue.

I upgraded marathon to 1.3.6-1.0.540 ( which it currently is on ) . I also tried 1.3.5 1.3.2-1.

A couple side notes...... this mesos cluster with marathon has been running for about 5 months without issue. I upgraded a couple weeks ago and it ran fine but this week it started with this issue.

Yesterday I turned off HA, marathon has not crashed for also 24 hours which may point to a HA problem maybe ? From the error it seems like it might be a zookeeper issue, maybe I should try and run it with a standard zookeeper setup and not exhibitor. Also mesos has been using the same leader.

[jasongilanfarr]

Looking at the logs, it appears that you're hitting connection timeouts with ZK. I only recently setup the leader connection to use the ZK timeout as the connection timeout (pretty sure you're seeing a warning where the connection timeout is less than the session timeout).

[chris530]

Looks like I can change that in --zk_timeout ? I'll try and adjust that settings to see if it makes a difference. Again, thanks a million !

[jasongilanfarr]

That is true on master, but its not true in 1.3.x. It'd be a super simple change to fix it.

[chris530]

Great!

[jasongilanfarr]

@chris530 Any luck?

[chris530]

Marathon ( 1.3.6-1.0.540 ) has been running for 6 days without going unresponsive. . It is currently NOT in a ha mode, just one host. I assume going back to 3 marathon hosts with HA will see the issue again. I will have to wait till after black Friday to test zookeeper and not exhibitor. Was a commit in the works for the zookeeper timeout issue ?

[jasongilanfarr]

We don't currently have any plans to release a 1.3.7 but this is fixed on master right now. I or someone else can quickly implement this on the release branch and spin a new build.

In CuratorElectionService.scala, under provideCuratorClient, there is a CuratorFrameworkFactory.builder() which should have .connectionTimeoutMs called with config.zooKeeperTimeout().toInt

[jasongilanfarr]

I put up the PR here: https://github.com/mesosphere/marathon/pull/4685 and here: https://phabricator.mesosphere.com/D207 - If we do spin a 1.3.7, I believe there is another change incoming before 1.3.7 can be released.

[chris530]

Thanks a million !

[eyalzek]

We're seeing the same issue with marathon 1.3.2 and zookeeper 3.4.5.
Did you mean that --zk_timeout currently has no effect? or that some setting coming from zookeeper is overriding it?

Also, the message I'm seeing is:
session timeout [10000] is less than connection timeout [15000] which is a bit confusing since we don't set --zk_session_timeout which is by default 30 mins. What timeouts exactly are referred to in this message?

[jasongilanfarr]

zk_timeout does not set the connection timeout, only the "amount of time a zk operation is allowed to be pending". In 1.3.7 (coming soon) and 1.4, it is also used for the connection timeout. This is why you're seeing the warning.

[eyalzek]

@jasongilanfarr is this issue closed with https://github.com/mesosphere/marathon/releases/tag/v1.3.7 ?

[unterstein]

@eyalzek Yes, #4685 is included in 1.3.7

[meichstedt]

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-2255. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.