Openwhisk: controller registered Redundant invokers

[axiqia]

Environment details:

• local deployment, native ubuntu:16.04
• version of docker: 17.03.0-ce

Provide the actual results and outputs:

Ansible inventory:

$ cat environments/local/hosts
; the first parameter in a host is the inventory_hostname

; used for local actions only
ansible ansible_connection=local

[edge]
172.17.0.1          ansible_host=172.17.0.1 ansible_connection=local

[controllers]
controller0         ansible_host=172.17.0.1 ansible_connection=local
;
[kafkas]
kafka0              ansible_host=172.17.0.1 ansible_connection=local

[zookeepers:children]
kafkas

[invokers]
invoker0            ansible_host=172.17.0.1 ansible_connection=local

; db group is only used if db.provider is CouchDB
[db]
172.17.0.1          ansible_host=172.17.0.1 ansible_connection=local

[redis]
172.17.0.1          ansible_host=172.17.0.1 ansible_connection=local

[apigateway]
172.17.0.1          ansible_host=172.17.0.1 ansible_connection=local

Deploy cmd:

ansible-playbook -i environments/local openwhisk.yml -e skip_pull_runtimes=True -e  invoker_user_memory="4096m" -e limit_invocations_per_minute=99999 -e limit_invocations_concurrent=99999

Part of Controller logs:

[2019-05-16T05:41:41.996Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] registered a new invoker: invoker0
[2019-05-16T05:41:42.011Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker0 is unhealthy [marker:loadbalancer_invokerState.unhealthy_counter:17425]
[2019-05-16T05:41:42.026Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker0' with activation id '35ac202c481b4eb9ac202c481bbeb957' [marker:controll#
[2019-05-16T05:41:42.034Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Unhealthy
[2019-05-16T05:41:42.037Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 1 blackboxInvokers = 1
[2019-05-16T05:41:42.125Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posted to invoker0[0][0] [marker:controller_kafka_finish:17539:99]
[2019-05-16T05:41:49.415Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] received completion ack for health action on invoker0/0
[2019-05-16T05:41:49.416Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker0' with activation id '633e2a97acbe479abe2a97acbe479a4e' [marker:controll#
[2019-05-16T05:41:49.418Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker0 is up
[2019-05-16T05:41:49.419Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Healthy
[2019-05-16T05:41:49.419Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 1 blackboxInvokers = 1
[2019-05-16T05:41:49.422Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posted to invoker0[0][1] [marker:controller_kafka_finish:24835:5]
[2019-05-16T05:41:50.027Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] received completion ack for health action on invoker0/0
[2019-05-16T05:42:40.823Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker0 is down [marker:loadbalancer_invokerState.down_counter:76236]
[2019-05-16T05:42:40.858Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Offline
[2019-05-16T05:42:40.859Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 1 blackboxInvokers = 1
[2019-05-16T05:42:44.142Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker0 is unhealthy [marker:loadbalancer_invokerState.unhealthy_counter:79555]
[2019-05-16T05:42:44.143Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker0' with activation id '5918fca3da944d7e98fca3da94ed7e8a' [marker:controll#
[2019-05-16T05:42:44.146Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Unhealthy
[2019-05-16T05:42:44.147Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 1 blackboxInvokers = 1
[2019-05-16T05:42:44.149Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posted to invoker0[0][2] [marker:controller_kafka_finish:79563:6]
[2019-05-16T05:43:31.812Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] registered a new invoker: invoker3
[2019-05-16T05:43:31.832Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker3 is unhealthy [marker:loadbalancer_invokerState.unhealthy_counter:127228]
[2019-05-16T05:43:31.832Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker3' with activation id 'bc6c300096584701ac30009658b7019f' [marker:controll#
[2019-05-16T05:43:31.832Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Unhealthy, 1 -> Offline, 2 -> Offline, 3 -> Unhealthy
[2019-05-16T05:43:31.832Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 4 blackboxInvokers = 1
[2019-05-16T05:43:40.841Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] received completion ack for health action on invoker0/0
[2019-05-16T05:43:40.864Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker0 is up
[2019-05-16T05:43:40.865Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker0' with activation id '3a19cc9bad4e4d3199cc9bad4efd3151' [marker:controll#
[2019-05-16T05:43:40.865Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 4 blackboxInvokers = 1
[2019-05-16T05:43:40.865Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Healthy, 1 -> Offline, 2 -> Offline, 3 -> Unhealthy
[2019-05-16T05:43:40.865Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posted to invoker0[0][3] [marker:controller_kafka_finish:136262:6]
[2019-05-16T05:43:42.190Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] received completion ack for health action on invoker0/0
[2019-05-16T05:43:42.495Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] registered a new invoker: invoker1
[2019-05-16T05:43:42.496Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker1 is unhealthy [marker:loadbalancer_invokerState.unhealthy_counter:137910]
[2019-05-16T05:43:42.497Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker1' with activation id '93b5181c50904932b5181c509049328e' [marker:controll#
[2019-05-16T05:43:42.499Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Healthy, 1 -> Unhealthy, 2 -> Offline, 3 -> Unhealthy

[style95]

Is there any chance that the controller receives any ping message via Kafka from other than your invokers?

This is a similar case that I faced once.
If controllers receive a ping message from the invokerN, it will populate all statuses for invoker0 ~ invoker(N-1) even if there is no message from them. Surely their statuses are offline.

Last time, when I faced this I only had 10 invokers but statuses for invoker10 ~ 14 were also being maintained.
And I found one debris invoker whose name is invoker14. And it kept sending Ping message to the Kafka.

If you make sure there is no other invoker configured in the host file and you chose the correct target environment, I suspect you may be under the same situation.

[axiqia]

Thanks for your reply @style95 .

Yes, it seems a similar case. I found three debris invokers,invoker1, invoker2, invoker3 running by docker ps. I stop them all and redeploy controller, then all is well.