Swarmkit: Task may hold onto network resources(IP address) forever

[mrjana]

When a node running a task fails, we may not deallocate network resources for the task for a long time or forever because the task state hasn't been updated by the agent to > TaskStateRunning (the condition the allocator looks for deallocating the task) and it might get removed from the store without ever reaching this condition. This will result in leaked resources.

[aluzzardi]

@mrjana Is this one still relevant? Do we need to move it to 1.13?

[aaronlehmann]

Yes, it's still relevant.

[mrjana]

yep we have to revisit this at some point in the future.

[aluzzardi]

Do you consider this a P1 though?

[mrjana]

May be not. It will only happen in a very rare case where the node has lost communication and we haven't purged the dead task.

[aluzzardi]

We also have to make sure we release IPs of desired=shutdown tasks (e.g. task history).

[aluzzardi]

Tentatively assigning to @mrjana for now who has the most context.

/cc @dongluochen @aaronlehmann

[stevvooe]

When deciding on the state of the task, you also need to take into account the state of the assigned node.

Typically, we disallow coercing status that is not from the agent, but it might be simpler if we relax this.

[stevvooe]

Adding an ORPHANED status might be okay...

[aaronlehmann]

Adding an ORPHANED status might be a good idea. If we do this, we should also add a state for nodes that have been down so long that we are treating their tasks as orphaned.

When the dispatcher sees that a node has been down long enough, it would put the node in this state and change all its tasks to the ORPHANED state.

The allocator would receive events for tasks moved to the ORPHANED state, and release resources associated with these.

[aluzzardi]

Sounds like the right thing to do.

I remember we talked about this in the past but decided against it. But then the overall architecture was much simpler so it makes a lot more sense now.

This is how the PoC used to work - when a node went down we'd orphan all his tasks and let the orchestrator re-schedule them.

[stevvooe]

@aluzzardi The biggest reason we did this is we didn't want to confuse the system of record for task status once the hand off to the node happened. For this case, I don't think we have that issue, since updates for the task won't be accepted if that node comes back after expiring the heartbeat.

For this to work well, ORPHANED MUST be greater than other existing task states. Setting a task as ORPHANED implies that no more task status updates will be accepted (which is true for other terminal states).