Swarmkit: TestManager hanging

[aaronlehmann]

When I run make test at 20f7542e48cc6727e6796b9be36e3ad4ac949c75, sometimes it hangs while testing the manager package.

I haven't been able to get stack trace so far.

I suspect this may be related to #668.

I don't think it's #693, becuase I think I saw it before updating to a version that included that.

cc @abronan @LK4D4

[aaronlehmann]

Also wonder if this could be related to #681.

[aaronlehmann]

Here's a trace: https://gist.github.com/aaronlehmann/864ac67180eb8dd5d56dc83f388bf3c7

[aaronlehmann]

Definitely blocked in WaitForCluster:

github.com/docker/swarm-v2/manager/state/raft.WaitForCluster(0x6a90288, 0xc82006a858, 0xc8201b0340, 0x0, 0x0, 0x0)
    /Users/aaronl/go/src/github.com/docker/swarm-v2/manager/state/raft/util.go:73 +0x530
github.com/docker/swarm-v2/manager.(*Manager).Run(0xc8201d4120, 0x6a90288, 0xc82006a858, 0x0, 0x0)
    /Users/aaronl/go/src/github.com/docker/swarm-v2/manager/manager.go:416 +0xd52
github.com/docker/swarm-v2/manager.TestManager.func1(0xc8204ee5a0, 0xc8201d4120, 0x6a90288, 0xc82006a858)
    /Users/aaronl/go/src/github.com/docker/swarm-v2/manager/manager_test.go:73 +0x43
created by github.com/docker/swarm-v2/manager.TestManager
    /Users/aaronl/go/src/github.com/docker/swarm-v2/manager/manager_test.go:74 +0xd27

[aaronlehmann]

Added some debugging, and the CreateCluster call is never reached.

Edit: never mind, my output was being suppressed.

[LK4D4]

Hmm, then probably whole leader code branch is skipped :/

[abronan]

@aaronlehmann Just reproduced the hang on master after running the tests 10 times, do you think there is a tight race on WaitForCluster that can trigger the deadlock? Other probable cause is that this is an issue with leadership and the node never reaches the state when an isLeader/isFollower event is fired. Thus the hang because we never create the Cluster Object.

[LK4D4]

I think it might be a race between SubscribeLeadership and writing to leadershipBroadcast, so manager won't became aware of its leadership. But it would be error before too :/

[aaronlehmann]

@LK4D4: This seems like the best theory so far.

What about moving the call to SubscribeLeadership outside of the goroutine, so it happens before raftNode.Run is called?

[LK4D4]

@aaronlehmann let me try it

[aaronlehmann]

What about moving the call to SubscribeLeadership outside of the goroutine, so it happens before raftNode.Run is called?

Tried this and I can't reproduce the issue after this change. Tried 5 times so far.

[aaronlehmann]

It seems to solve the problem.

[LK4D4]

Yup, works for me as well.