Flux-core: [enhancement] command to remove a broker from the topology cleanly

[trws]

For right now, really just a way to not use it for work would be fine, but some way to cleanly remove a node (broker) would be very helpful.

[SteVwonder]

The "not use it for work" part is probably something that we could handle easily in sched. Either removing the node from the resrc or marking it with a very long "allocation" should keep work from being scheduled on it.

[garlick]

Allocate in rc1 after sched is loaded, free in rc3 before sched is unloaded maybe?

[dongahn]

Need an ability within sched to schedule by node name or rank, though... (a hack should be easy).

[trws]

Unfortunately my attempt at a dirty hack, setting a few nodes to have no cores in their hwloc data, causes sched to die... This is part of the larger resilience story, but even to manually remove nodes for thinks like their filesystem connection being down would be good.

[trws]

@SteVwonder, the very long allocation idea is a good one. Do you happen to know if there's a way to ask sched to allocate a specific node to something at the moment?

[dongahn]

In theory, one can add the name field in the resrc_request. Then, at resrc_match_resource within resrc.c, you can do string match on the name if the name field is presenr in the request..

[dongahn]

If you want to bypass the node managed by a certain rank, maybe just not reading the hwloc xml from that rank...

And yeah, we need to beef up how sched handles thr hwloc data. We can do whole lot by using fake or mutated hwloc xml data...

[SteVwonder]

Do you happen to know if there's a way to ask sched to allocate a specific node to something at the moment?

I do not. I was imagining some custom function, but what @dongahn suggested sounds really promising and very simple. Are you trying to do this at runtime on arbitrary nodes during the course of the allocation? Or do you have a list of "bad nodes" that you want to blacklist when first launching Flux?

[trws]

It's blacklisting in an existing instance we don't want to stop. I'm managing a 1727 node flux instance right now, and we found out (with jobs running) that four of the nodes had bad FS connections. It made me really wish for a way to blacklist them live...

[dongahn]

I don't think there is a way with the current version. Let me see if I can get to this today or tomorrow morning.

[dongahn]

@grondo: while I'm on this, I want to see if I can do a PR for the node_exclusive scheduling. Is there an option in submit requesting a node exclusively?

If not, I will just make "node exclusive" an option to sched load.

[trws]

There is not, but there should be.

On 2 Apr 2018, at 22:04, Dong H. Ahn wrote:

@grondo: while I'm on this, I want to see if I can do a PR for the
node_exclusive scheduling. Is there an option in submit requesting a
node exclusively?

If not, I will just make "node exclusive" an option to sched load.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/flux-framework/flux-core/issues/1415#issuecomment-378129334

[grondo]

No option for request of exclusive resources in the current submit,
though it is trivial to add it. Unfortunately, I'll be traveling with
family it for the next couple days or I could add it.

On Mon, Apr 2, 2018, 10:05 PM Dong H. Ahn [e-mail] wrote:

@grondo https://github.com/grondo: while I'm on this, I want to see if
I can do a PR for the node_exclusive scheduling. Is there an option in
submit requesting a node exclusively?

If not, I will just make "node exclusive" an option to sched load.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/flux-framework/flux-core/issues/1415#issuecomment-378129334,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtSUr-x56-xE2h8WM-H35nfI8XWuriDks5tkwLrgaJpZM4TEHwc
.

[dongahn]

@grondo: don't worry about it. I don't think this is urgent.

[dongahn]

@trws: I posted experimental PRs that allow you to exclude and include a node for scheduling

flux-core PR https://github.com/flux-framework/flux-core/pull/1418

flux-sched PR https://github.com/flux-framework/flux-sched/pull/305

Sorry for the experimental nature of this.

This was just how much I could afford this morning before today's trip. If you have issues, please send along. I won't be able to get to code throughout the day but I can respond via emails.

[dongahn]

Node exclusion/inclusion support is now available:

• https://github.com/flux-framework/flux-core/pull/1418
• https://github.com/flux-framework/flux-sched/pull/305.