Marathon: Show refused offers as part of deployment status

[cmaloney]

Frequently deploys, and just spinning up new instances doesn't get any acceptable offers (or sometimes any offers at all). Marathon should give some UI for this.

A "Time since last offer" + list of recently refused offers with reason for being refused would be really helpful for debugging why things aren't starting in marathon.

[ConnorDoyle]

Possible duplicate or extension of these issues:

• #126
• #652
• #951
• #968
• #1077

[sttts]

:+1:

[smarthall]

+1

[air]

Related to #951 and #1307, which are different cases of deployments going wrong.

[kolloch]

:+1:

[stuart-warren]

copied from #1728 (sorry for the duplicate, I did search honest ;)

I've noticed that trying to work out why an application is stuck in the queue is pretty much impossible to do without accessing the marathon log files. Though the information is at least present there.

eg:

Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: Conflicting constraints are: [field: "hostname"
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: operator: UNIQUE
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: ] (mesosphere.mesos.ResourceMatcher$:39)
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: [2015-06-30 13:44:18,497] INFO All
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: [2015-06-30 13:44:18,498] WARN App ports are not available in the offer. (mesosphere.mesos.ResourceMatcher$:50)
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: [2015-06-30 13:44:18,498] INFO No matching offer for /platform/services/discovery (need cpus=0.25, mem=64.0, disk=0.0, ports=List(8880)) : id {
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]:   value: "20150626-230939-3209314314-5050-1116-O259398"
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: }
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: framework_id {
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]:   value: "20150626-152352-3209314314-5050-1173-0000"
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: }
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: slave_id {
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]:   value: "20150626-152353-3226091530-5050-1189-S1"
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: }
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: hostname: "10.64.74.191"
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]: resources {
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]:   name: "cpus"
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]:   type: SCALAR
Jun 30 13:44:18 master-eeb65358-49f6-45e2-80ab-2018dc057a50 marathon[933]:   scalar {

Trying to parse syslog logs is no fun, can we expose these important logs as events or adding something to the object in the deployment queue?
Pretty much any logging occurring in src/main/scala/mesosphere/mesos

[Raffo]

Should I use this one or #1307 ? I think this is a major issue for deploying containers and I'd love to know what's the status of this issue to understand how to contribute. Also, could this https://issues.apache.org/jira/browse/MESOS-2020 be helpful?

[christianparpart]

:+1: we're currently opting into Mesos/Marathon for deploying all of our services and I ran into this issue numerous times. Checking logs for me is tolerable but not for the actual future daily-users. Do we have any intend to create an ETA for this story? -- Thanks for the great product so far ;-)

[Raffo]

+1 once again. We generally have to give the user a better way to understand why a deployment is stuck. Can we recap the status of this issue so that we can understand how to proceed?

[abraithwaite]

Relates to:

• #2869
• #2520

[jpza]

+1

[jpza]

Would be nice if this also distinguished between a lack of resources and a constraint

[air]

Good news everyone! We have a UI design for this (including offers and constraints) and we're in the process of getting the work scheduled. No hard delivery date but it's in the pipe.

[Raffo]

Cool! Do you mind sharing something?

[kamilchm]

@air cloud your share a screenshot?

[air]

Sure @Raffo @kamilchm @worstadmin - here's the current mockup https://mesosphere.invisionapp.com/share/PS5KLCF7Q#/screens/127360376_App_Detail_View_-Debug-_Waiting-

[kamilchm]

Looks great! Do you plan to make it for next release?
We have users that will be happy to see this and we can help to code this.

[tmmi]

@air Perhaps backend for this in such a way https://github.com/tmmi/marathon/tree/show_refused_offers_endpoint
and UI
https://github.com/mesosphere/marathon-ui/pull/767

[kamilchm]

@air your screen lacks one information. Some offers can be refused because app combine many requirements for resources and constraints, eg. I want to run app with 4 CPUs in two racks (["rack_id", "GROUP_BY", "2"]) and there's no CPUs left in one rack but there are plenty of it the second one. So as I understand your proposition it will show that there's no matches for rack constraint but CPUs are good? It wont tell me that as a developer I can decrease my CPU need for maybe 2 CPUs and then it will spawn successfully in both racks. @tmmi proposition with combined resources column can suggest that.
Do I read that correctly?

[kamilchm]

@air any thoughts?

[air]

Good feedback @kamilchm @tmmi, thanks for that. I'll review with the design team.

[kamilchm]

Backend PR in https://github.com/mesosphere/marathon/pull/4119

[jasongilanfarr]

Covered by rejected offers.

[bobrik]

@jasongilanfarr can you link to an issue / pr where I can look at screenshots? When will it be released?