Marathon: Upgrade to (pre-release) 1.4.0 results in all lost apps and all killed tasks

[solarkennedy]

I recently upgraded a test cluster from 1.3.0-1.0.506 to 1.4.0-0.1.20161021221011.snap19 at Yelp.

It is a 3 marathon node cluster, I upgraded the cluster one node at a time.

The last node was the "important one", because after upgrading it, a new leader was elected for the first time that was on 1.4.0.

After the last node, the marathon UI starts timing out. I give it a bit, and then restart all the marathon servers to get them to re-elect or whatever, but when the interface loads, there are no apps :(

Here is my timeline:

• 23:30: paasta1-uswest1adevc is upgraded and restarted
• 23:33: paasta1-uswest1cdevc is upgraded and restarted
• 23:37: paasta2-uswest1adevc is upgraded and restarted
• 23:38: UI timeouts all around
• 23:43-44: I restart marathons all around
• 23:43: paasta1-uswest1adevc is elected leader and the UI starts responding again
• 23:43:22+: paasta1-uswest1adevc expunges all apps and kills all tasks with messages like:

bla exists in InstanceTracker, but not store. The run spec was likely terminated. Will now expunge
...
Killing 1 tasks for reason: Orphaned (ids: instance [kibana.platform.git1

• 23:45: the paasta control plane starts launching new apps to get things back in sync (~300 apps)
• 23:49: paasta2-uswest1adevc becomes a new leader? Not sure why.

Of course 1.4.0 is not released yet, and looking at zk it looks to have a new "on disk" state layout. I bet this is the root reason for why this upgrade failed this way. I could have taken some zk backups, but honestly Marathon works so fast, a zk restore wouldn't bring my processes back :(

I have the full log in all its glory, I can easily share via google drive if requested.

Let me know if you would like me to try to reproduce, or if you would like more data.
All apps are "flat" on the "/" namespace. (no groups)

[unterstein]

Hi @solarkennedy,

thanks for reporting this. It would be great if you could try to reproduce this with dummy apps on your test cluster and share this logs. Please try RC3 as well, as the snapshots had known issues with migration.

Thanks

[solarkennedy]

Alright, but we only saw this bug on a real cluster. (we don't do itests for marathon upgrades).

And because RC3 has a hard requirement on mesos 1.1, it means we'll have to test and roll out that upgrade first. (starting with https://github.com/Yelp/paasta/pull/942)

And on real clusters, we install marathon via debs. Can you build a trusty RC3 deb for us to use?

[unterstein]

You can download RC3 here: https://s3.amazonaws.com/repos.mesosphere.com/ubuntu/pool/main/m/marathon/marathon_1.4.0-1.0.562.ubuntu1404_amd64.deb

[solarkennedy]

@unterstein I think in 1.4.0 the runnable-jar stuff broke.
For the itests we have now (using snap19) I had to hand-make the deb and had to separate out the jar from the bin file.
Do you know if anyone else has used this deb before? Does it even work?

Here is the output when I try to run this thing:

root@41e5c43359a3:/# marathon --zk zk://zookeeper:2181/marathon --master zk://zookeeper:2181/mesos-testcluster --no-logger --env_vars_prefix MARATHON_ --mesos_authentication_principal marathon --mesos_authentication_secret_file /etc/marathon_framework_secret --mesos_authentication true
Error: Could not find or load main class mesosphere.marathon.Main

Here is the docker image used in this:
https://github.com/Yelp/paasta/blob/4e0efaf073835057eb6d180b5569c268d93eae54/yelp_package/dockerfiles/itest/marathon/Dockerfile

I've looked at this for a couple hours before hand-building the snap19 deb. I couldn't figure out why the runnable jar technique didn't work. The jar zip headers looked like they were there and everything.

[jeschkies]

@solarkennedy the migration issue might be related to https://github.com/mesosphere/marathon/issues/4873. The broken deb package issue is tracked here https://github.com/mesosphere/marathon/issues/4854.

[meichstedt]

4873 affects tasks with Marathon-based HealthChecks and does not explain or fix anything re orphaned tasks.

[unterstein]

@solarkennedy,

we have issues with the linux packages, but addressed and fixed this today in https://github.com/mesosphere/marathon/pull/4890 and will include this changes to the next RC.

We also experienced this Could not find or load main class mesosphere.marathon.Main issue while trying to export the jar file from the binary manually. When downloading the jar file directly, e.g. from this package https://downloads.mesosphere.com/marathon/v1.4.0-RC3/marathon-1.4.0-RC3.tgz and extracting it afterwards, you should experience a starting marathon version with you arguments :)

Thanks for reporting and testing :)

[jasongilanfarr]

I believe we fixed the root cause of this issue (the migration being incorrect) in a later snapshot. I started digging to find where it was yesterday but couldn't remember where exactly it happened.

We hit an issue later where the zip file for the shell script required zip64 and I think that was ultimately fixed as well.

[aquamatthias]

I retested this with this setup:

• Start Marathon 1.3.6
• Start 1 simple cmd app with 10 tasks
• Start 1 complex app with health checks with 1 task
• Stop Marathon
• Start Marathon 1.4.0-RC4

I see this in the logs:

Starting migration to 1.3.6 
Finished 1.3.6 migration 
Migration for storage: Version(1, 3, 6, LEGACY) to current: Version(1, 4, 0, LEGACY): apply change for version: Version(1, 4, 0, PERSISTENCE_STORE)  
Migrated (tasks,11)(deployment plans,0)(task failures,0)(root + app versions,8)(framework-id,1)(event-subscribers,0) to new format 

No task was killed.
All apps are available.

[solarkennedy]

@unterstein to (finally) close the loop here, I can confirm that the 1.4.0-rc3 deb provided does work and I have a green build against it in https://github.com/Yelp/paasta/pull/943. Thanks!