Ohara: [Phase2] Support dead cluster concept

[saivirtue]

This is phase 2 of #1438
For cluster, we should have more meaningful status beside "running" and "failed" ; We should

• [x] define the cluster status enumeration and rules (move to #1544)
• [x] save settings to configurator for cluster (zk, bk, wk) (move to #1544)
• [x] Show status and deadNodes in UI (#3444 )

[chia7712]

@saivirtue I'd say our @oharastream/frontend neglect this issue for a long time. please ping them for more discussion :)

[konekoya]

Seems like everything is done in the backend? But I'm not quite following about the issue 😅 @saivirtue or @chia7712 would you mind explaining a bit more to me here or we can have a quick meeting to discuss how we can go about it?

[chia7712]

@konekoya Please write the conclusion for our discussion :)

[konekoya]

Had a quick discussion on the topic with @chia7712 today.

In the cluster APIs (zookeepers, brokers, workers and streamApps) the added field __deadNodes__ will keep track of the dead nodes(nodes that are not running or failed) of the originally assigned nodes from the users. And the __nodeNames__ field will store the nodes even if a node is failed.

For example:

A worker cluster is started and assigned the following nodes

nodeA, nodeB and nodeC

In the API, the nodeNames field should look like (Please note that this is not the real implmetation of our API):

"nodeNames": ["nodeA", "nodeB", "nodeC"]
"deadNodes": []

Assuming that after the cluster running for a while, and for some reason, __nodeA__ failed. Our API would display something like:

"nodeNames":   ["nodeA", "nodeB", "nodeC"]
"deadNodes": ["nodeA"]

Notice that the __nodeNames__ still has all the nodes that were given by the user when initializing the cluster. But the __deadNodes__ value is updated.

For frontend. We will figure out where to display this info to the users and so they can use our logs API to see a dead node's logs and fix the issue.

Also, the __nodeNames__ field will be moved to __settings___

@chia7712 feel free to add more detail if I have missed something :)

[chia7712]

a nit

In the cluster APIs (zookeepers, brokers, and workers)

and streamapp

and for another thing. the nodeNames will be moved to "settings" (see https://github.com/oharastream/ohara/issues/2438)

[chia7712]

@konekoya @wu87988622 any suggestion about showing the dead node?

[konekoya]

@konekoya @wu87988622 any suggestion about showing the dead node?

I'm thinking about putting the info in the workspace, perhaps in the overview pages

[konekoya]

@vitojeng @chia7712 @eechih I think we should also include this in the new UI, right?

[eechih]

I think we should also include this in the new UI, right?

@konekoya That's right, it will be done in the new UI.

[vitojeng]

Great! Thanks for @oharastream/frontend

[konekoya]

@eechih Feel free to close or link to issue when it's done :)

[eechih]

Feel free to close or link to issue when it's done

@konekoya OK