Emqx: Broker restarts for unknown reasons

Created on 25 Nov 2019  路  28Comments  路  Source: emqx/emqx

Environment

  • OS: GKE OS
  • Erlang/OTP: 21.2
  • EMQ: v3.2.5

Description

We are running an emqx cluster on kubernetes. Lately, as traffic grew, we noticed that the brokers try to connect to an ip which is not in the cluster anymore. This happened after a cluster upgrade.
Logs show some kind of peer connection attempt to an unknown node:

emqtt-0 emqtt 2019-11-25 17:24:25.951 [error] event=connect_to_remote_server peer="[email protected]" result=failure reason="ehostunreach"
emqtt-0 emqtt 2019-11-25 17:24:25.951 [error]   crasher:
emqtt-0 emqtt     initial call: gen_rpc_client:init/1
emqtt-0 emqtt     pid: <0.4619.16>
emqtt-0 emqtt     registered_name: []
emqtt-0 emqtt     exception exit: {badrpc,ehostunreach}
emqtt-0 emqtt       in function  `gen_server:init_it/6` (gen_server.erl, line 358)
emqtt-0 emqtt     ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]
emqtt-0 emqtt     message_queue_len: 0
emqtt-0 emqtt     messages: []
emqtt-0 emqtt     links: [<0.1353.0>]
emqtt-0 emqtt     dictionary: []
emqtt-0 emqtt     trap_exit: true
emqtt-0 emqtt     status: running
emqtt-0 emqtt     heap_size: 1598
emqtt-0 emqtt     stack_size: 27
emqtt-0 emqtt     reductions: 4779
emqtt-0 emqtt   neighbours:
emqtt-0 emqtt

```bash
emqx_ctl cluster status
Cluster status: [{running_nodes,['[email protected]','[email protected]',
'[email protected]']}]

This ip seems cached somehow and appears also after restart of all nodes. This was noticed on v3.1.1 also, however after upgrade to v3.2.5 didn't appear in logs anymore for a while. Today it appeared again.
Also, another symptom which might or might not be related to this, is that, ocasionally (maybe once every two days) the broker is restarted by kubernetes because of liveness probe failure. Liveness config is:

```yml
readinessProbe:
  tcpSocket:
    port: 1883
  initialDelaySeconds: 60
  periodSeconds: 10
livenessProbe:
  tcpSocket:
    port: 1883
  initialDelaySeconds: 60
  periodSeconds: 3
picture mariusstaicu
👍1

Most helpful comment

downgraded to v3.1.1 and restart problems disappeared. Broker uptime is now 6d20h.

picture mariusstaicu on 27 Dec 2019
👍2

All 28 comments

I have logs just before crash:

"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4783\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:01.227 [warning] <<\"client-3681687\">>@10.154.15.202:64797 [Protocol] Client client-3681687 (Username: 'client-3681687') login failed for not_authorized\n"
"2019-11-26 06:13:01.981 [warning] <<\"client-6862262\">>@10.154.15.203:55066 [Protocol] Client client-6862262 (Username: 'client-6862262') login failed for not_authorized\n"
"2019-11-26 06:13:02.237 [warning] <<\"client-615694\">>@10.154.15.200:62732 [Protocol] Client client-615694 (Username: 'client-615694') login failed for not_authorized\n"
"2019-11-26 06:13:02.753 [warning] <<\"client-8867908\">>@10.154.15.203:25347 [Protocol] Client client-8867908 (Username: 'client-8867908') login failed for not_authorized\n"
"2019-11-26 06:13:03.428 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:03.428 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.13992.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4783\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:05.558 [warning] <<\"client-8213660\">>@10.56.11.1:52988 [Protocol] Client client-8213660 (Username: 'client-8213660') login failed for not_authorized\n"
"2019-11-26 06:13:05.564 [warning] <<\"client-6863850\">>@10.154.15.202:60956 [Protocol] Client client-6863850 (Username: 'client-6863850') login failed for not_authorized\n"
"2019-11-26 06:13:06.527 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:06.527 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14444.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4783\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:06.565 [warning] <<\"client-6199265\">>@10.154.15.205:60255 [Protocol] Client client-6199265 (Username: 'client-6199265') login failed for not_authorized\n"
"2019-11-26 06:13:06.802 [warning] <<\"client-4164875\">>@10.56.11.1:61084 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-26 06:13:09.599 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:09.599 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14938.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4783\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:12.099 [warning] <<\"client-8818144\">>@10.154.15.200:25400 [Protocol] Client client-8818144 (Username: 'client-8818144') login failed for not_authorized\n"
"2019-11-26 06:13:12.552 [warning] <<\"client-6863594\">>@10.154.15.203:64536 [Protocol] Client client-6863594 (Username: 'client-6863594') login failed for not_authorized\n"
"2019-11-26 06:13:12.672 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:12.672 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14944.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4781\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:13.042 [warning] <<\"client-8828141\">>@10.154.15.201:60496 [Protocol] Client client-8828141 (Username: 'client-8828141') login failed for not_authorized\n"
"2019-11-26 06:13:14.112 [warning] <<\"client-8803045\">>@10.154.15.200:61628 [Protocol] Client client-8803045 (Username: 'client-8803045') login failed for not_authorized\n"
"2019-11-26 06:13:15.743 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:15.744 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14957.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4782\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:16.271 [warning] <<\"client-6175221\">>@10.154.15.205:56065 [Protocol] Client client-6175221 (Username: 'client-6175221') login failed for not_authorized\n"
"2019-11-26 06:13:17.739 [warning] <<\"client-6185135\">>@10.154.15.202:50869 [Protocol] Client client-6185135 (Username: 'client-6185135') login failed for not_authorized\n"
"2019-11-26 06:13:18.815 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:18.815 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14965.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4781\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:19.483 [warning] <<\"client-8840958\">>@10.154.15.205:52930 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-26 06:13:19.601 [warning] <<\"client-8817634\">>@10.154.15.203:63471 [Protocol] Client client-8817634 (Username: 'client-8817634') login failed for not_authorized\n"
"2019-11-26 06:13:21.887 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:21.887 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14978.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4781\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:24.034 [warning] <<\"client-8828033\">>@10.154.15.205:52528 [Protocol] Client client-8828033 (Username: 'client-8828033') login failed for not_authorized\n"
"2019-11-26 06:13:24.959 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:24.959 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14986.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4781\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:26.767 [warning] <<\"client-8199230\">>@10.154.15.203:56099 [Protocol] Client client-8199230 (Username: 'client-8199230') login failed for not_authorized\n"
"2019-11-26 06:13:26.975 [warning] <<\"client-611173\">>@10.154.15.201:62454 [Protocol] Client client-611173 (Username: 'client-611173') login failed for not_authorized\n"
"2019-11-26 06:13:27.591 [warning] <<\"client-8867908\">>@10.154.15.200:25392 [Protocol] Client client-8867908 (Username: 'client-8867908') login failed for not_authorized\n"
"2019-11-26 06:13:28.031 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:28.031 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.14995.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4783\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:28.086 [warning] <<\"client-8852409\">>@10.154.15.203:59396 [Protocol] Client client-8852409 (Username: 'client-8852409') login failed for not_authorized\n"
"2019-11-26 06:13:28.635 [warning] <<\"client-11381203\">>@10.154.15.200:55736 [Protocol] Client client-11381203 (Username: 'client-11381203') login failed for not_authorized\n"
"2019-11-26 06:13:31.103 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"2019-11-26 06:13:31.103 [error]   crasher:\n"
"    initial call: gen_rpc_client:init/1\n"
"    pid: <0.15005.58>\n"
"    registered_name: []\n"
"    exception exit: {badrpc,ehostunreach}\n"
"      in function  gen_server:init_it/6 (gen_server.erl, line 358)\n"
"    ancestors: [gen_rpc_client_sup,gen_rpc_sup,<0.1348.0>]\n"
"    message_queue_len: 0\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4781\n"
"  neighbours:\n"
"\n"
"2019-11-26 06:13:31.840 [warning] <<\"client-3680677\">>@10.154.15.200:50374 [Channel] Discarded by client-3680677:<0.15023.58>\n"
"2019-11-26 06:13:32.052 [warning] <<\"client-8769599\">>@10.154.15.201:25341 [Protocol] Client client-8769599 (Username: 'client-8769599') login failed for not_authorized\n"
"2019-11-26 06:13:32.962 [warning] <<\"client-4185523\">>@10.154.15.200:57488 [Protocol] Client client-4185523 (Username: 'client-4185523') login failed for not_authorized\n"
"2019-11-26 06:13:34.175 [error] event=connect_to_remote_server peer=\"[email protected]\" result=failure reason=\"ehostunreach\"\n"
"    messages: []\n"
"    links: [<0.1353.0>]\n"
"    dictionary: []\n"
"    trap_exit: true\n"
"    status: running\n"
"    heap_size: 1598\n"
"    stack_size: 27\n"
"    reductions: 4781\n"
"  neighbours:\n"
"\n"
"sh: invalid number '306687emqx'\n"
"2019/11/26 06:13:37 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/26 06:13:37 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/26 06:13:37 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/26 06:13:37 Client closed local connection on 127.0.0.1:5432\n"
"cluster.k8s.address_type=ip\n"
"cluster.k8s.address_type=ip\n"
"auth.pgsql.username=device-service\n"
"node.max_ports=1048576\n"
"listener.tcp.external.acceptors=64\n"
"auth.pgsql.auth_query=select password from device_credentials where username = '%u' limit 1\n"
"allow_anonymous=false\n"
"auth.pgsql.encoding=utf8\n"
"listener.ssl.external.acceptors=32\n"
"node.process_limit=2097152\n"
"node.max_ets_tables=2097152\n"
"auth.pgsql.password_hash=plain\n"
"cluster.k8s.service_name=emqtt\n"
"cluster.k8s.service_name=emqtt\n"
"auth.pgsql.pool=4\n"
"auth.pgsql.password=device-service\n"
"auth.pgsql.super_query=select is_superuser from device_credentials where username = '%u' limit 1\n"
"auth.pgsql.server=127.0.0.1:5432\n"
"cluster.discovery=k8s\n"
"cluster.discovery=k8s\n"
"listener.ws.external.acceptors=16\n"
"auth.pgsql.acl_query=select allow, ipaddr, username, clientid, access, topic from device_acl where username = '%u'\n"
"auth.pgsql.ssl=false\n"
"[email protected]\n"
"cluster.k8s.app_name=emqtt\n"
"cluster.k8s.app_name=emqtt\n"
"cluster.k8s.apiserver=https://kubernetes.default.svc.cluster.local:443\n"
"cluster.k8s.apiserver=https://kubernetes.default.svc.cluster.local:443\n"
"cluster.k8s.namespace=infrastructure\n"
"cluster.k8s.namespace=infrastructure\n"
"auth.pgsql.database=device-service\n"
"acl_nomatch=deny\n"
"EMQX_LOADED_PLUGINS=emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_auth_pgsql\n"
"2019/11/26 06:14:00 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/26 06:14:00 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/26 06:14:00 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/26 06:14:00 New connection for \"myproject:europe-west2:production\"\n"
"emqx v3.2.5 is started successfully!\n"
"2019/11/26 06:14:03 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/26 06:14:03 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/26 06:14:03 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/26 06:14:03 Client closed local connection on 127.0.0.1:5432\n"
"2019-11-26 06:14:03.046 [critical] [EMQ X] emqx shutdown for join\n"
"['2019-11-26T06:14:04Z']:waiting emqx\n"
"2019/11/26 06:14:06 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/26 06:14:06 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/26 06:14:06 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/26 06:14:06 New connection for \"myproject:europe-west2:production\"\n"
"['2019-11-26T06:14:06Z']:waiting emqx\n"
"['2019-11-26T06:14:13Z']:emqx start\n"
"2019-11-26 06:14:45.582 [warning] <<\"client-8213660\">>@10.154.15.200:56498 [Protocol] Client client-8213660 (Username: 'client-8213660') login failed for not_authorized\n"
"2019-11-26 06:14:46.825 [warning] <<\"client-4164875\">>@10.154.15.202:60492 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-26 06:14:47.057 [warning] <<\"client-8769599\">>@10.154.15.203:25336 [Protocol] Client client-8769599 (Username: 'client-8769599') login failed for not_authorized\n"
"2019-11-26 06:14:50.331 [warning] <<\"client-10081520\">>@10.154.15.200:49936 [Protocol] Client client-10081520 (Username: 'client-10081520') login failed for not_authorized\n"
"2019-11-26 06:14:53.070 [warning] <<\"client-8828141\">>@10.154.15.202:60500 [Protocol] Client client-8828141 (Username: 'client-8828141') login failed for not_authorized\n"
"2019-11-26 06:14:54.153 [warning] <<\"client-8803045\">>@10.154.15.202:61632 [Protocol] Client client-8803045 (Username: 'client-8803045') login failed for not_authorized\n"
"2019-11-26 06:14:54.931 [warning] <<\"client-8851949\">>@10.154.15.203:51774 [Protocol] Client client-8851949 (Username: 'client-8851949') login failed for not_authorized\n"
"2019-11-26 06:14:58.615 [warning] <<\"client-8213772\">>@10.154.15.202:49864 [Protocol] Client client-8213772 (Username: 'client-8213772') login failed for not_authorized\n"
"2019-11-26 06:14:59.226 [warning] <<\"client-8777559\">>@10.154.15.201:60328 [Protocol] Client client-8777559 (Username: 'client-8777559') login failed for not_authorized\n"
mariusstaicu picture mariusstaicu on 26 Nov 2019
👍1

Another restart, this time with some strange numbers appearing just before restart. Is it possible that this is related to fix that is fixed in v3.2.6 ?

Fix emqx crash caused by the crash of auth plugin
Github PR: emqx/emqx#3048

If so, when will there be a v3.2.6 docker image released to docker hub ?

Also, broker is fine tuned according to specs here: https://docs.emqx.io/broker/v3/en/tune.html.
An identical setup on staging was load tested using emqx_bech with 60k connections and 2.5k messages / sec (512k with qos 1) and holds with no problem.

"2019-11-27 06:57:13.924 [warning] <<\"client-4176755\">>@10.56.1.1:50990 [Protocol] Client client-4176755 (Username: 'client-4176755') login failed for not_authorized\n"
"2019-11-27 06:57:14.420 [warning] <<\"client-6175221\">>@10.154.15.203:59626 [Protocol] Client client-6175221 (Username: 'client-6175221') login failed for not_authorized\n"
"2019-11-27 06:57:15.277 [warning] <<\"client-10081461\">>@10.154.15.204:50611 [Protocol] Client client-10081461 (Username: 'client-10081461') login failed for not_authorized\n"
"2019-11-27 06:57:15.710 [warning] <<\"client-4185337\">>@10.154.15.203:49600 [Protocol] Client client-4185337 (Username: 'client-4185337') login failed for not_authorized\n"
"2019-11-27 06:57:16.261 [warning] <<\"client-4207830\">>@10.154.15.204:55113 [Protocol] Client client-4207830 (Username: 'client-4207830') login failed for not_authorized\n"
"2019-11-27 06:57:16.760 [warning] <<\"client-10081391\">>@10.154.15.202:54658 [Protocol] Client client-10081391 (Username: 'client-10081391') login failed for not_authorized\n"
"2019-11-27 06:57:17.255 [warning] <<\"client-8851621\">>@10.154.15.201:64647 [Protocol] Client client-8851621 (Username: 'client-8851621') login failed for not_authorized\n"
"2019-11-27 06:57:17.857 [warning] <<\"client-8811549\">>@10.154.15.204:56246 [Protocol] Client client-8811549 (Username: 'client-8811549') login failed for not_authorized\n"
"2019-11-27 06:57:18.221 [warning] <<\"client-8213772\">>@10.154.15.200:55681 [Protocol] Client client-8213772 (Username: 'client-8213772') login failed for not_authorized\n"
"2019-11-27 06:57:18.684 [warning] <<\"client-8840958\">>@10.154.15.202:56491 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 06:57:22.790 [warning] <<\"client-4185274\">>@10.154.15.204:5976 [Protocol] Client client-4185274 (Username: 'client-4185274') login failed for not_authorized\n"
"2019-11-27 06:57:22.993 [warning] <<\"client-8817634\">>@10.154.15.200:58941 [Protocol] Client client-8817634 (Username: 'client-8817634') login failed for not_authorized\n"
"2019-11-27 06:57:23.997 [warning] <<\"client-8828033\">>@10.154.15.200:56089 [Protocol] Client client-8828033 (Username: 'client-8828033') login failed for not_authorized\n"
"2019-11-27 06:57:28.461 [warning] <<\"client-3682404\">>@10.154.15.202:32240 [Protocol] Client client-3682404 (Username: 'client-3682404') login failed for not_authorized\n"
"2019-11-27 06:57:28.983 [warning] <<\"client-8867908\">>@10.154.15.201:25333 [Protocol] Client client-8867908 (Username: 'client-8867908') login failed for not_authorized\n"
"2019-11-27 06:57:29.851 [warning] <<\"client-8820347\">>@10.154.15.201:63930 [Protocol] Client client-8820347 (Username: 'client-8820347') login failed for not_authorized\n"
"2019-11-27 06:57:30.273 [warning] <<\"client-4185523\">>@10.56.1.1:62765 [Protocol] Client client-4185523 (Username: 'client-4185523') login failed for not_authorized\n"
"2019-11-27 06:57:31.517 [warning] <<\"client-4161719\">>@10.154.15.204:51444 [Protocol] Client client-4161719 (Username: 'client-4161719') login failed for not_authorized\n"
"2019-11-27 06:57:31.753 [warning] <<\"client-8769599\">>@10.154.15.203:25336 [Protocol] Client client-8769599 (Username: 'client-8769599') login failed for not_authorized\n"
"2019-11-27 06:57:33.120 [warning] <<\"client-11381203\">>@10.154.15.200:51622 [Protocol] Client client-11381203 (Username: 'client-11381203') login failed for not_authorized\n"
"2019-11-27 06:57:37.936 [warning] <<\"client-8852409\">>@10.154.15.203:61287 [Protocol] Client client-8852409 (Username: 'client-8852409') login failed for not_authorized\n"
"2019-11-27 06:57:38.646 [warning] <<\"client-6185135\">>@10.154.15.204:54923 [Protocol] Client client-6185135 (Username: 'client-6185135') login failed for not_authorized\n"
"2019-11-27 06:57:39.517 [warning] <<\"client-6175221\">>@10.154.15.203:59627 [Protocol] Client client-6175221 (Username: 'client-6175221') login failed for not_authorized\n"
"2019-11-27 06:57:40.360 [warning] <<\"client-10081461\">>@10.154.15.202:50612 [Protocol] Client client-10081461 (Username: 'client-10081461') login failed for not_authorized\n"
"2019-11-27 06:57:42.227 [warning] <<\"client-8776666\">>@10.154.15.202:64311 [Protocol] Client client-8776666 (Username: 'client-8776666') login failed for not_authorized\n"
"2019-11-27 06:57:42.233 [warning] <<\"client-8851621\">>@10.154.15.203:2643 [Protocol] Client client-8851621 (Username: 'client-8851621') login failed for not_authorized\n"
"2019-11-27 06:57:43.645 [warning] <<\"client-8840958\">>@10.154.15.201:56492 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 06:57:44.100 [warning] <<\"client-4185317\">>@10.56.1.1:57032 [Protocol] Client client-4185317 (Username: 'client-4185317') login failed for not_authorized\n"
"2019-11-27 06:57:46.231 [warning] <<\"client-3681687\">>@10.154.15.204:50294 [Protocol] Client client-3681687 (Username: 'client-3681687') login failed for not_authorized\n"
"2019-11-27 06:57:47.997 [warning] <<\"client-8817634\">>@10.154.15.203:52966 [Protocol] Client client-8817634 (Username: 'client-8817634') login failed for not_authorized\n"
"2019-11-27 06:57:48.993 [warning] <<\"client-8828033\">>@10.154.15.201:56090 [Protocol] Client client-8828033 (Username: 'client-8828033') login failed for not_authorized\n"
"2019-11-27 06:57:53.450 [warning] <<\"client-3682404\">>@10.154.15.201:32242 [Protocol] Client client-3682404 (Username: 'client-3682404') login failed for not_authorized\n"
"2019-11-27 06:57:53.824 [warning] <<\"client-8199230\">>@10.154.15.202:50086 [Protocol] Client client-8199230 (Username: 'client-8199230') login failed for not_authorized\n"
"2019-11-27 06:57:54.021 [warning] <<\"client-8867908\">>@10.154.15.201:25328 [Protocol] Client client-8867908 (Username: 'client-8867908') login failed for not_authorized\n"
"2019-11-27 06:57:54.333 [warning] <<\"client-8767983\">>@10.154.15.200:51189 [Protocol] Client client-8767983 (Username: 'client-8767983') login failed for not_authorized\n"
"2019-11-27 06:57:55.366 [warning] <<\"client-8213660\">>@10.154.15.200:59030 [Protocol] Client client-8213660 (Username: 'client-8213660') login failed for not_authorized\n"
"2019-11-27 06:57:56.531 [warning] <<\"client-4161719\">>@10.154.15.201:55961 [Protocol] Client client-4161719 (Username: 'client-4161719') login failed for not_authorized\n"
"2019-11-27 06:57:58.992 [warning] <<\"client-10081520\">>@10.154.15.201:58485 [Protocol] Client client-10081520 (Username: 'client-10081520') login failed for not_authorized\n"
"2019-11-27 06:58:03.763 [warning] <<\"client-8803045\">>@10.154.15.204:65191 [Protocol] Client client-8803045 (Username: 'client-8803045') login failed for not_authorized\n"
"2019-11-27 06:58:03.823 [warning] <<\"client-8851949\">>@10.154.15.200:51096 [Protocol] Client client-8851949 (Username: 'client-8851949') login failed for not_authorized\n"
"2019-11-27 06:58:05.718 [warning] <<\"client-4185337\">>@10.56.1.1:62029 [Protocol] Client client-4185337 (Username: 'client-4185337') login failed for not_authorized\n"
"2019-11-27 06:58:08.261 [warning] <<\"client-8213772\">>@10.56.1.1:57554 [Protocol] Client client-8213772 (Username: 'client-8213772') login failed for not_authorized\n"
"2019-11-27 06:58:08.925 [warning] <<\"client-8777559\">>@10.154.15.201:63887 [Protocol] Client client-8777559 (Username: 'client-8777559') login failed for not_authorized\n"
"2019-11-27 06:58:10.044 [warning] <<\"client-641085\">>@10.154.15.202:54189 [Protocol] Client client-641085 (Username: 'client-641085') login failed for not_authorized\n"
"2019-11-27 06:58:12.763 [warning] <<\"client-4185274\">>@10.56.1.1:57881 [Protocol] Client client-4185274 (Username: 'client-4185274') login failed for not_authorized\n"
"2019-11-27 06:58:13.635 [warning] <<\"client-8830005\">>@10.154.15.202:57649 [Protocol] Client client-8830005 (Username: 'client-8830005') login failed for not_authorized\n"
"2019-11-27 06:58:13.919 [warning] <<\"client-7348624\">>@10.154.15.203:64008 [Protocol] Client client-7348624 (Username: 'client-7348624') login failed for not_authorized\n"
"2019-11-27 06:58:14.756 [warning] <<\"client-8851495\">>@10.56.1.1:51687 [Protocol] Client client-8851495 (Username: 'client-8851495') login failed for not_authorized\n"
"2019-11-27 06:58:15.285 [warning] <<\"client-6188966\">>@10.154.15.203:56152 [Protocol] Client client-6188966 (Username: 'client-6188966') login failed for not_authorized\n"
"2019-11-27 06:58:15.865 [warning] <<\"client-8826347\">>@10.56.1.1:25339 [Protocol] Client client-8826347 (Username: 'client-8826347') login failed for not_authorized\n"
"2019-11-27 06:58:18.934 [warning] <<\"client-4164875\">>@10.154.15.204:56113 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-27 06:58:19.324 [warning] <<\"client-10891083\">>@10.154.15.201:55749 [Protocol] Client client-10891083 (Username: 'client-10891083') login failed for not_authorized\n"
"2019-11-27 06:58:19.329 [warning] <<\"client-8767983\">>@10.154.15.203:60398 [Protocol] Client client-8767983 (Username: 'client-8767983') login failed for not_authorized\n"
"2019-11-27 06:58:22.967 [warning] <<\"client-4204002\">>@10.154.15.203:62117 [Protocol] Client client-4204002 (Username: 'client-4204002') login failed for not_authorized\n"
"2019-11-27 06:58:28.034 [warning] <<\"client-4176755\">>@10.154.15.204:63496 [Protocol] Client client-4176755 (Username: 'client-4176755') login failed for not_authorized\n"
"2019-11-27 06:58:28.516 [warning] <<\"client-6185135\">>@10.154.15.201:64783 [Protocol] Client client-6185135 (Username: 'client-6185135') login failed for not_authorized\n"
"2019-11-27 06:58:29.424 [warning] <<\"client-6175221\">>@10.56.1.1:59629 [Protocol] Client client-6175221 (Username: 'client-6175221') login failed for not_authorized\n"
"2019-11-27 06:58:30.331 [warning] <<\"client-10081461\">>@10.154.15.203:50614 [Protocol] Client client-10081461 (Username: 'client-10081461') login failed for not_authorized\n"
"2019-11-27 06:58:31.221 [warning] <<\"client-4207830\">>@10.154.15.202:55116 [Protocol] Client client-4207830 (Username: 'client-4207830') login failed for not_authorized\n"
"2019-11-27 06:58:31.769 [warning] <<\"client-10081391\">>@10.154.15.203:21291 [Protocol] Client client-10081391 (Username: 'client-10081391') login failed for not_authorized\n"
"2019-11-27 06:58:33.621 [warning] <<\"client-8840958\">>@10.154.15.203:56494 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 06:58:34.151 [warning] <<\"client-4185317\">>@10.154.15.200:51707 [Protocol] Client client-4185317 (Username: 'client-4185317') login failed for not_authorized\n"
"2019-11-27 06:58:35.461 [warning] <<\"client-611173\">>@10.154.15.202:58843 [Protocol] Client client-611173 (Username: 'client-611173') login failed for not_authorized\n"
"2019-11-27 06:58:37.239 [warning] <<\"client-8803127\">>@10.154.15.204:54360 [Protocol] Client client-8803127 (Username: 'client-8803127') login failed for not_authorized\n"
"2019-11-27 06:58:38.248 [warning] <<\"client-4166199\">>@10.154.15.203:49639 [Protocol] Client client-4166199 (Username: 'client-4166199') login failed for not_authorized\n"
"2019-11-27 06:58:38.622 [warning] <<\"client-8830005\">>@10.154.15.204:57650 [Protocol] Client client-8830005 (Username: 'client-8830005') login failed for not_authorized\n"
"2019-11-27 06:58:38.833 [warning] <<\"client-7348624\">>@10.154.15.201:56161 [Protocol] Client client-7348624 (Username: 'client-7348624') login failed for not_authorized\n"
"2019-11-27 06:58:39.798 [warning] <<\"client-8851495\">>@10.154.15.202:54833 [Protocol] Client client-8851495 (Username: 'client-8851495') login failed for not_authorized\n"
"2019-11-27 06:58:43.908 [warning] <<\"client-8199230\">>@10.154.15.204:60632 [Protocol] Client client-8199230 (Username: 'client-8199230') login failed for not_authorized\n"
"2019-11-27 06:58:43.930 [warning] <<\"client-4164875\">>@10.154.15.203:62605 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-27 06:58:43.988 [warning] <<\"client-8867908\">>@10.154.15.203:25336 [Protocol] Client client-8867908 (Username: 'client-8867908') login failed for not_authorized\n"
"2019-11-27 06:58:44.333 [warning] <<\"client-8767983\">>@10.154.15.204:64158 [Protocol] Client client-8767983 (Username: 'client-8767983') login failed for not_authorized\n"
"2019-11-27 06:58:44.338 [warning] <<\"client-10891083\">>@10.154.15.201:53947 [Protocol] Client client-10891083 (Username: 'client-10891083') login failed for not_authorized\n"
"2019-11-27 06:58:47.146 [warning] <<\"client-8818144\">>@10.56.1.1:25334 [Protocol] Client client-8818144 (Username: 'client-8818144') login failed for not_authorized\n"
"2019-11-27 06:58:47.974 [warning] <<\"client-4204002\">>@10.154.15.204:62118 [Protocol] Client client-4204002 (Username: 'client-4204002') login failed for not_authorized\n"
"2019-11-27 06:58:48.055 [warning] <<\"client-11381203\">>@10.154.15.204:61432 [Protocol] Client client-11381203 (Username: 'client-11381203') login failed for not_authorized\n"
"2019-11-27 06:58:50.001 [warning] <<\"client-4161315\">>@10.154.15.200:58833 [Protocol] Client client-4161315 (Username: 'client-4161315') login failed for not_authorized\n"
"2019-11-27 06:58:54.263 [warning] <<\"client-8803045\">>@10.154.15.203:65193 [Protocol] Client client-8803045 (Username: 'client-8803045') login failed for not_authorized\n"
"2019-11-27 06:58:54.730 [warning] <<\"client-6185675\">>@10.154.15.203:51272 [Protocol] Client client-6185675 (Username: 'client-6185675') login failed for not_authorized\n"
"2019-11-27 06:58:56.400 [warning] <<\"client-4207830\">>@10.154.15.200:55117 [Protocol] Client client-4207830 (Username: 'client-4207830') login failed for not_authorized\n"
"2019-11-27 06:58:56.774 [warning] <<\"client-10081391\">>@10.154.15.203:21291 [Protocol] Client client-10081391 (Username: 'client-10081391') login failed for not_authorized\n"
"2019-11-27 06:58:57.255 [warning] <<\"client-8851621\">>@10.154.15.203:50343 [Protocol] Client client-8851621 (Username: 'client-8851621') login failed for not_authorized\n"
"2019-11-27 06:58:58.590 [warning] <<\"client-8840958\">>@10.56.1.1:56495 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 06:58:58.688 [warning] <<\"client-8829135\">>@10.154.15.202:56124 [Protocol] Client client-8829135 (Username: 'client-8829135') login failed for not_authorized\n"
"2019-11-27 06:58:58.933 [warning] <<\"client-8777559\">>@10.154.15.200:63889 [Protocol] Client client-8777559 (Username: 'client-8777559') login failed for not_authorized\n"
"2019-11-27 06:59:00.061 [warning] <<\"client-641085\">>@10.56.1.1:49912 [Protocol] Client client-641085 (Username: 'client-641085') login failed for not_authorized\n"
"2019-11-27 06:59:00.085 [warning] <<\"client-3033236\">>@10.154.15.201:56585 [Protocol] Client client-3033236 (Username: 'client-3033236') login failed for not_authorized\n"
"2019-11-27 06:59:00.543 [warning] <<\"client-611173\">>@10.56.1.1:65193 [Protocol] Client client-611173 (Username: 'client-611173') login failed for not_authorized\n"
"2019-11-27 06:59:04.001 [warning] <<\"client-8828033\">>@10.154.15.200:56093 [Protocol] Client client-8828033 (Username: 'client-8828033') login failed for not_authorized\n"
"2019-11-27 06:59:05.863 [warning] <<\"client-8826347\">>@10.56.1.1:25339 [Protocol] Client client-8826347 (Username: 'client-8826347') login failed for not_authorized\n"
"2019-11-27 06:59:08.939 [warning] <<\"client-4164875\">>@10.154.15.200:54818 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-27 06:59:09.858 [warning] <<\"client-8820347\">>@10.154.15.201:63934 [Protocol] Client client-8820347 (Username: 'client-8820347') login failed for not_authorized\n"
"2019-11-27 06:59:11.774 [warning] <<\"client-8769599\">>@10.154.15.203:25336 [Protocol] Client client-8769599 (Username: 'client-8769599') login failed for not_authorized\n"
"2019-11-27 06:59:14.436 [warning] <<\"client-4161315\">>@10.154.15.202:64859 [Protocol] Client client-4161315 (Username: 'client-4161315') login failed for not_authorized\n"
"2019-11-27 06:59:18.853 [warning] <<\"client-8803045\">>@10.154.15.202:65194 [Protocol] Client client-8803045 (Username: 'client-8803045') login failed for not_authorized\n"
"2019-11-27 06:59:18.870 [warning] <<\"client-8851949\">>@10.154.15.204:56325 [Protocol] Client client-8851949 (Username: 'client-8851949') login failed for not_authorized\n"
"2019-11-27 06:59:20.402 [warning] <<\"client-10081461\">>@10.154.15.200:50616 [Protocol] Client client-10081461 (Username: 'client-10081461') login failed for not_authorized\n"
"2019-11-27 06:59:25.056 [warning] <<\"client-641085\">>@10.154.15.204:62921 [Protocol] Client client-641085 (Username: 'client-641085') login failed for not_authorized\n"
"2019-11-27 06:59:28.856 [warning] <<\"client-7348624\">>@10.154.15.203:54714 [Protocol] Client client-7348624 (Username: 'client-7348624') login failed for not_authorized\n"
"2019-11-27 06:59:30.300 [warning] <<\"client-6188966\">>@10.154.15.203:56155 [Protocol] Client client-6188966 (Username: 'client-6188966') login failed for not_authorized\n"
"2019-11-27 06:59:33.941 [warning] <<\"client-4164875\">>@10.56.1.1:60830 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-27 06:59:35.516 [warning] <<\"client-8213660\">>@10.154.15.204:62738 [Protocol] Client client-8213660 (Username: 'client-8213660') login failed for not_authorized\n"
"2019-11-27 06:59:37.162 [warning] <<\"client-8818144\">>@10.154.15.201:25332 [Protocol] Client client-8818144 (Username: 'client-8818144') login failed for not_authorized\n"
"2019-11-27 06:59:39.004 [warning] <<\"client-10081520\">>@10.154.15.201:52357 [Protocol] Client client-10081520 (Username: 'client-10081520') login failed for not_authorized\n"
"2019-11-27 06:59:43.022 [warning] <<\"client-8852409\">>@10.154.15.203:52406 [Protocol] Client client-8852409 (Username: 'client-8852409') login failed for not_authorized\n"
"2019-11-27 06:59:44.564 [warning] <<\"client-6175221\">>@10.154.15.204:59632 [Protocol] Client client-6175221 (Username: 'client-6175221') login failed for not_authorized\n"
"2019-11-27 06:59:44.763 [warning] <<\"client-6185675\">>@10.154.15.203:59741 [Protocol] Client client-6185675 (Username: 'client-6185675') login failed for not_authorized\n"
"2019-11-27 06:59:46.297 [warning] <<\"client-4185361\">>@10.56.1.1:50396 [Protocol] Client client-4185361 (Username: 'client-4185361') login failed for not_authorized\n"
"2019-11-27 06:59:46.779 [warning] <<\"client-10081391\">>@10.154.15.200:50633 [Protocol] Client client-10081391 (Username: 'client-10081391') login failed for not_authorized\n"
"2019-11-27 06:59:48.685 [warning] <<\"client-8829135\">>@10.154.15.202:56126 [Protocol] Client client-8829135 (Username: 'client-8829135') login failed for not_authorized\n"
"2019-11-27 06:59:49.097 [warning] <<\"client-8803045\">>@10.154.15.201:65195 [Protocol] Client client-8803045 (Username: 'client-8803045') login failed for not_authorized\n"
"2019-11-27 06:59:53.019 [warning] <<\"client-8817634\">>@10.56.1.1:55097 [Protocol] Client client-8817634 (Username: 'client-8817634') login failed for not_authorized\n"
"2019-11-27 06:59:53.260 [warning] <<\"client-4166199\">>@10.154.15.204:59146 [Protocol] Client client-4166199 (Username: 'client-4166199') login failed for not_authorized\n"
"2019-11-27 06:59:54.013 [warning] <<\"client-8828033\">>@10.154.15.200:56095 [Protocol] Client client-8828033 (Username: 'client-8828033') login failed for not_authorized\n"
"2019-11-27 06:59:54.751 [warning] <<\"client-8851495\">>@10.154.15.203:62386 [Protocol] Client client-8851495 (Username: 'client-8851495') login failed for not_authorized\n"
"2019-11-27 06:59:59.873 [warning] <<\"client-8820347\">>@10.154.15.202:63936 [Protocol] Client client-8820347 (Username: 'client-8820347') login failed for not_authorized\n"
"2019-11-27 07:00:01.543 [warning] <<\"client-4161719\">>@10.154.15.202:54556 [Protocol] Client client-4161719 (Username: 'client-4161719') login failed for not_authorized\n"
"2019-11-27 07:00:01.813 [warning] <<\"client-8769599\">>@10.154.15.201:25328 [Protocol] Client client-8769599 (Username: 'client-8769599') login failed for not_authorized\n"
"2019-11-27 07:00:02.997 [warning] <<\"client-4204002\">>@10.154.15.202:62121 [Protocol] Client client-4204002 (Username: 'client-4204002') login failed for not_authorized\n"
"2019-11-27 07:00:03.077 [warning] <<\"client-11381203\">>@10.154.15.204:50861 [Protocol] Client client-11381203 (Username: 'client-11381203') login failed for not_authorized\n"
"2019-11-27 07:00:04.549 [warning] <<\"client-4161315\">>@10.154.15.200:56022 [Protocol] Client client-4161315 (Username: 'client-4161315') login failed for not_authorized\n"
"2019-11-27 07:00:07.975 [warning] <<\"client-4176755\">>@10.56.1.1:54465 [Protocol] Client client-4176755 (Username: 'client-4176755') login failed for not_authorized\n"
"2019-11-27 07:00:09.831 [warning] <<\"client-6185675\">>@10.154.15.203:56408 [Protocol] Client client-6185675 (Username: 'client-6185675') login failed for not_authorized\n"
"2019-11-27 07:00:11.818 [warning] <<\"client-10081391\">>@10.154.15.202:61660 [Protocol] Client client-10081391 (Username: 'client-10081391') login failed for not_authorized\n"
"2019-11-27 07:00:12.059 [warning] <<\"client-3033236\">>@10.154.15.201:50643 [Protocol] Client client-3033236 (Username: 'client-3033236') login failed for not_authorized\n"
"2019-11-27 07:00:12.245 [warning] <<\"client-8851621\">>@10.154.15.200:51768 [Protocol] Client client-8851621 (Username: 'client-8851621') login failed for not_authorized\n"
"2019-11-27 07:00:12.882 [warning] <<\"client-8811549\">>@10.154.15.200:50369 [Protocol] Client client-8811549 (Username: 'client-8811549') login failed for not_authorized\n"
"2019-11-27 07:00:16.242 [warning] <<\"client-641971\">>@10.154.15.204:60535 [Channel] Discarded by client-641971:<0.7671.164>\n"
"2019-11-27 07:00:17.234 [warning] <<\"client-8803127\">>@10.154.15.200:54364 [Protocol] Client client-8803127 (Username: 'client-8803127') login failed for not_authorized\n"
"2019-11-27 07:00:19.858 [warning] <<\"client-8851495\">>@10.154.15.203:50974 [Protocol] Client client-8851495 (Username: 'client-8851495') login failed for not_authorized\n"
"2019-11-27 07:00:20.322 [warning] <<\"client-6188966\">>@10.154.15.204:56157 [Protocol] Client client-6188966 (Username: 'client-6188966') login failed for not_authorized\n"
"2019-11-27 07:00:23.973 [warning] <<\"client-4164875\">>@10.154.15.201:2883 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-27 07:00:24.002 [warning] <<\"client-8199230\">>@10.154.15.200:55582 [Protocol] Client client-8199230 (Username: 'client-8199230') login failed for not_authorized\n"
"2019-11-27 07:00:24.348 [warning] <<\"client-8767983\">>@10.154.15.204:53837 [Protocol] Client client-8767983 (Username: 'client-8767983') login failed for not_authorized\n"
"2019-11-27 07:00:24.918 [warning] <<\"client-8820347\">>@10.154.15.201:63937 [Protocol] Client client-8820347 (Username: 'client-8820347') login failed for not_authorized\n"
"2019-11-27 07:00:25.419 [warning] <<\"client-4185523\">>@10.154.15.200:52736 [Protocol] Client client-4185523 (Username: 'client-4185523') login failed for not_authorized\n"
"2019-11-27 07:00:27.163 [warning] <<\"client-8818144\">>@10.154.15.202:25340 [Protocol] Client client-8818144 (Username: 'client-8818144') login failed for not_authorized\n"
"2019-11-27 07:00:28.003 [warning] <<\"client-4204002\">>@10.154.15.202:62122 [Protocol] Client client-4204002 (Username: 'client-4204002') login failed for not_authorized\n"
"2019-11-27 07:00:28.057 [warning] <<\"client-11381203\">>@10.56.1.1:51035 [Protocol] Client client-11381203 (Username: 'client-11381203') login failed for not_authorized\n"
"2019-11-27 07:00:33.452 [warning] <<\"client-6185135\">>@10.154.15.201:57686 [Protocol] Client client-6185135 (Username: 'client-6185135') login failed for not_authorized\n"
"2019-11-27 07:00:36.258 [warning] <<\"client-4207830\">>@10.154.15.200:55121 [Protocol] Client client-4207830 (Username: 'client-4207830') login failed for not_authorized\n"
"2019-11-27 07:00:37.240 [warning] <<\"client-8851621\">>@10.56.1.1:51848 [Protocol] Client client-8851621 (Username: 'client-8851621') login failed for not_authorized\n"
"2019-11-27 07:00:37.923 [warning] <<\"client-8811549\">>@10.56.1.1:49645 [Protocol] Client client-8811549 (Username: 'client-8811549') login failed for not_authorized\n"
"2019-11-27 07:00:38.318 [warning] <<\"client-8213772\">>@10.56.1.1:61967 [Protocol] Client client-8213772 (Username: 'client-8213772') login failed for not_authorized\n"
"2019-11-27 07:00:38.635 [warning] <<\"client-8840958\">>@10.154.15.204:56499 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 07:00:40.059 [warning] <<\"client-641085\">>@10.56.1.1:65262 [Protocol] Client client-641085 (Username: 'client-641085') login failed for not_authorized\n"
"2019-11-27 07:00:41.329 [warning] <<\"client-3681687\">>@10.154.15.200:3396 [Protocol] Client client-3681687 (Username: 'client-3681687') login failed for not_authorized\n"
"2019-11-27 07:00:42.220 [warning] <<\"client-8803127\">>@10.154.15.204:54365 [Protocol] Client client-8803127 (Username: 'client-8803127') login failed for not_authorized\n"
"2019-11-27 07:00:42.798 [warning] <<\"client-4185274\">>@10.56.1.1:54029 [Protocol] Client client-4185274 (Username: 'client-4185274') login failed for not_authorized\n"
"2019-11-27 07:00:43.027 [warning] <<\"client-8817634\">>@10.154.15.204:53241 [Protocol] Client client-8817634 (Username: 'client-8817634') login failed for not_authorized\n"
"2019-11-27 07:00:43.267 [warning] <<\"client-4166199\">>@10.154.15.202:57355 [Protocol] Client client-4166199 (Username: 'client-4166199') login failed for not_authorized\n"
"2019-11-27 07:00:45.322 [warning] <<\"client-6188966\">>@10.56.1.1:56158 [Protocol] Client client-6188966 (Username: 'client-6188966') login failed for not_authorized\n"
"2019-11-27 07:00:48.946 [warning] <<\"client-4164875\">>@10.154.15.202:54989 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-27 07:00:49.003 [warning] <<\"client-8867908\">>@10.56.1.1:25339 [Protocol] Client client-8867908 (Username: 'client-8867908') login failed for not_authorized\n"
"2019-11-27 07:00:52.218 [warning] <<\"client-8818144\">>@10.154.15.203:25343 [Protocol] Client client-8818144 (Username: 'client-8818144') login failed for not_authorized\n"
"2019-11-27 07:00:52.994 [warning] <<\"client-4204002\">>@10.154.15.204:62123 [Protocol] Client client-4204002 (Username: 'client-4204002') login failed for not_authorized\n"
"2019-11-27 07:00:54.009 [warning] <<\"client-10081520\">>@10.56.1.1:63374 [Protocol] Client client-10081520 (Username: 'client-10081520') login failed for not_authorized\n"
"2019-11-27 07:00:57.463 [warning] <<\"client-8828141\">>@10.154.15.204:64066 [Protocol] Client client-8828141 (Username: 'client-8828141') login failed for not_authorized\n"
"2019-11-27 07:00:58.533 [warning] <<\"client-6185135\">>@10.154.15.200:54062 [Protocol] Client client-6185135 (Username: 'client-6185135') login failed for not_authorized\n"
"2019-11-27 07:01:00.745 [warning] <<\"client-4185337\">>@10.154.15.204:50272 [Protocol] Client client-4185337 (Username: 'client-4185337') login failed for not_authorized\n"
"2019-11-27 07:01:01.308 [warning] <<\"client-4185361\">>@10.154.15.203:49755 [Protocol] Client client-4185361 (Username: 'client-4185361') login failed for not_authorized\n"
"2019-11-27 07:01:02.149 [warning] <<\"client-3033236\">>@10.154.15.202:58472 [Protocol] Client client-3033236 (Username: 'client-3033236') login failed for not_authorized\n"
"2019-11-27 07:01:03.273 [warning] <<\"client-8213772\">>@10.154.15.204:50395 [Protocol] Client client-8213772 (Username: 'client-8213772') login failed for not_authorized\n"
"2019-11-27 07:01:03.772 [warning] <<\"client-8829135\">>@10.154.15.202:56129 [Protocol] Client client-8829135 (Username: 'client-8829135') login failed for not_authorized\n"
"2019-11-27 07:01:05.136 [warning] <<\"client-641085\">>@10.154.15.201:56397 [Protocol] Client client-641085 (Username: 'client-641085') login failed for not_authorized\n"
"2019-11-27 07:01:05.588 [warning] <<\"client-8777559\">>@10.154.15.202:63894 [Protocol] Client client-8777559 (Username: 'client-8777559') login failed for not_authorized\n"
"2019-11-27 07:01:07.260 [warning] <<\"client-8803127\">>@10.154.15.202:54366 [Protocol] Client client-8803127 (Username: 'client-8803127') login failed for not_authorized\n"
"2019-11-27 07:01:09.801 [warning] <<\"client-8851495\">>@10.154.15.203:50525 [Protocol] Client client-8851495 (Username: 'client-8851495') login failed for not_authorized\n"
"2019-11-27 07:01:10.287 [warning] <<\"client-6188966\">>@10.56.1.1:56159 [Protocol] Client client-6188966 (Username: 'client-6188966') login failed for not_authorized\n"
"2019-11-27 07:01:13.950 [warning] <<\"client-8199230\">>@10.154.15.204:5978 [Protocol] Client client-8199230 (Username: 'client-8199230') login failed for not_authorized\n"
"2019-11-27 07:01:13.950 [warning] <<\"client-4164875\">>@10.154.15.202:56876 [Protocol] Client client-4164875 (Username: 'client-4164875') login failed for not_authorized\n"
"2019-11-27 07:01:14.871 [warning] <<\"client-8820347\">>@10.56.1.1:63939 [Protocol] Client client-8820347 (Username: 'client-8820347') login failed for not_authorized\n"
"2019-11-27 07:01:15.344 [warning] <<\"client-4185523\">>@10.154.15.201:51204 [Protocol] Client client-4185523 (Username: 'client-4185523') login failed for not_authorized\n"
"2019-11-27 07:01:18.064 [warning] <<\"client-11381203\">>@10.154.15.204:64724 [Protocol] Client client-11381203 (Username: 'client-11381203') login failed for not_authorized\n"
"2019-11-27 07:01:22.473 [warning] <<\"client-8828141\">>@10.154.15.202:64067 [Protocol] Client client-8828141 (Username: 'client-8828141') login failed for not_authorized\n"
"2019-11-27 07:01:23.522 [warning] <<\"client-6185135\">>@10.56.1.1:55321 [Protocol] Client client-6185135 (Username: 'client-6185135') login failed for not_authorized\n"
"2019-11-27 07:01:24.583 [warning] <<\"client-6175221\">>@10.154.15.203:59636 [Protocol] Client client-6175221 (Username: 'client-6175221') login failed for not_authorized\n"
"2019-11-27 07:01:24.842 [warning] <<\"client-6185675\">>@10.154.15.204:64384 [Protocol] Client client-6185675 (Username: 'client-6185675') login failed for not_authorized\n"
"2019-11-27 07:01:25.818 [warning] <<\"client-4185337\">>@10.154.15.200:56153 [Protocol] Client client-4185337 (Username: 'client-4185337') login failed for not_authorized\n"
"2019-11-27 07:01:26.320 [warning] <<\"client-4185361\">>@10.154.15.204:5978 [Protocol] Client client-4185361 (Username: 'client-4185361') login failed for not_authorized\n"
"2019-11-27 07:01:26.323 [warning] <<\"client-4207830\">>@10.154.15.200:55123 [Protocol] Client client-4207830 (Username: 'client-4207830') login failed for not_authorized\n"
"2019-11-27 07:01:27.251 [warning] <<\"client-8851621\">>@10.56.1.1:53069 [Protocol] Client client-8851621 (Username: 'client-8851621') login failed for not_authorized\n"
"2019-11-27 07:01:27.321 [warning] <<\"client-8776666\">>@10.154.15.204:56489 [Protocol] Client client-8776666 (Username: 'client-8776666') login failed for not_authorized\n"
"2019-11-27 07:01:28.606 [warning] <<\"client-8840958\">>@10.154.15.203:56501 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 07:01:29.184 [warning] <<\"client-4185317\">>@10.154.15.203:55588 [Protocol] Client client-4185317 (Username: 'client-4185317') login failed for not_authorized\n"
"2019-11-27 07:01:30.072 [warning] <<\"client-641085\">>@10.154.15.202:62282 [Protocol] Client client-641085 (Username: 'client-641085') login failed for not_authorized\n"
"2019-11-27 07:01:34.127 [warning] <<\"client-8828033\">>@10.154.15.202:56099 [Protocol] Client client-8828033 (Username: 'client-8828033') login failed for not_authorized\n"
"2019-11-27 07:01:35.889 [warning] <<\"client-8826347\">>@10.154.15.201:25325 [Protocol] Client client-8826347 (Username: 'client-8826347') login failed for not_authorized\n"
"2019-11-27 07:01:36.761 [warning] <<\"client-633226\">>@10.154.15.200:62456 [Protocol] Client client-633226 (Username: 'client-633226') login failed for not_authorized\n"
"2019-11-27 07:01:39.419 [warning] <<\"client-8767983\">>@10.154.15.204:53433 [Protocol] Client client-8767983 (Username: 'client-8767983') login failed for not_authorized\n"
"2019-11-27 07:01:43.070 [warning] <<\"client-11381203\">>@10.154.15.204:64465 [Protocol] Client client-11381203 (Username: 'client-11381203') login failed for not_authorized\n"
"2019-11-27 07:01:44.454 [warning] <<\"client-4161315\">>@10.154.15.202:62026 [Protocol] Client client-4161315 (Username: 'client-4161315') login failed for not_authorized\n"
"2019-11-27 07:01:50.428 [warning] <<\"client-10081461\">>@10.154.15.200:50622 [Protocol] Client client-10081461 (Username: 'client-10081461') login failed for not_authorized\n"
"2019-11-27 07:01:52.965 [warning] <<\"client-8811549\">>@10.154.15.200:59658 [Protocol] Client client-8811549 (Username: 'client-8811549') login failed for not_authorized\n"
"2019-11-27 07:01:53.622 [warning] <<\"client-8840958\">>@10.56.1.1:56502 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 07:01:55.506 [warning] <<\"client-611173\">>@10.154.15.201:64555 [Protocol] Client client-611173 (Username: 'client-611173') login failed for not_authorized\n"
"2019-11-27 07:01:55.913 [warning] <<\"client-615694\">>@10.154.15.202:63656 [Protocol] Client client-615694 (Username: 'client-615694') login failed for not_authorized\n"
"2019-11-27 07:01:56.259 [warning] <<\"client-3681687\">>@10.154.15.203:52736 [Protocol] Client client-3681687 (Username: 'client-3681687') login failed for not_authorized\n"
"2019-11-27 07:01:58.267 [warning] <<\"client-4166199\">>@10.56.1.1:59019 [Protocol] Client client-4166199 (Username: 'client-4166199') login failed for not_authorized\n"
"2019-11-27 07:01:59.038 [warning] <<\"client-8828033\">>@10.154.15.204:56100 [Protocol] Client client-8828033 (Username: 'client-8828033') login failed for not_authorized\n"
"2019-11-27 07:02:00.925 [warning] <<\"client-8826347\">>@10.154.15.202:25342 [Protocol] Client client-8826347 (Username: 'client-8826347') login failed for not_authorized\n"
"2019-11-27 07:02:03.477 [warning] <<\"client-3682404\">>@10.154.15.202:32281 [Protocol] Client client-3682404 (Username: 'client-3682404') login failed for not_authorized\n"
"2019-11-27 07:02:04.386 [warning] <<\"client-8767983\">>@10.154.15.203:65426 [Protocol] Client client-8767983 (Username: 'client-8767983') login failed for not_authorized\n"
"2019-11-27 07:02:05.382 [warning] <<\"client-4185523\">>@10.154.15.202:5504 [Protocol] Client client-4185523 (Username: 'client-4185523') login failed for not_authorized\n"
"2019-11-27 07:02:06.823 [warning] <<\"client-8769599\">>@10.154.15.201:25325 [Protocol] Client client-8769599 (Username: 'client-8769599') login failed for not_authorized\n"
"2019-11-27 07:02:08.019 [warning] <<\"client-4204002\">>@10.154.15.200:62126 [Protocol] Client client-4204002 (Username: 'client-4204002') login failed for not_authorized\n"
"2019-11-27 07:02:14.634 [warning] <<\"client-6175221\">>@10.154.15.202:59638 [Protocol] Client client-6175221 (Username: 'client-6175221') login failed for not_authorized\n"
"2019-11-27 07:02:15.336 [warning] <<\"client-10081461\">>@10.154.15.203:50623 [Protocol] Client client-10081461 (Username: 'client-10081461') login failed for not_authorized\n"
"2019-11-27 07:02:17.163 [warning] <<\"client-3033236\">>@10.154.15.204:54328 [Protocol] Client client-3033236 (Username: 'client-3033236') login failed for not_authorized\n"
"2019-11-27 07:02:18.686 [warning] <<\"client-8840958\">>@10.154.15.203:56503 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"             102,45,108,105,103,104,116,115,47,51,54,56,49,50,56,50,47,99,111,\n"
"             109,109,97,110,100,45,114,101,115,112,111,110,115,101,88,57,47,55,\n"
"             78,43,105,72,72,51,75,109,66,113,47,103,122,90,118,79,115,50,84,\n"
"             52,47,79,114,53,49,50,87,69,122,75,52,118,68,99,78,112,120,81,105,\n"
"             114,112,99,71,70,114,115,112,52,53,73,86,90,122,106,115,43,78,78,\n"
"             100,119,117,108,108,100,87,71,76,47,78,99,87,74,57,119,122,68,72,\n"
"             76,70,73,116,57,53,76,81,113,73,56,116,76,86,50,97,52,100,73,116,\n"
"             53,57,72,49,98,69,61>>\n"
"2019-11-27 07:02:20.547 [warning] <<\"client-611173\">>@10.154.15.201:56855 [Protocol] Client client-611173 (Username: 'client-611173') login failed for not_authorized\n"
"2019-11-27 07:02:21.255 [warning] <<\"client-3681687\">>@10.154.15.203:58592 [Protocol] Client client-3681687 (Username: 'client-3681687') login failed for not_authorized\n"
"sh: invalid number '681541emqx'\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"
"auth.pgsql.username=device-service\n"
"cluster.k8s.address_type=ip\n"
"cluster.k8s.address_type=ip\n"
"node.max_ports=1048576\n"
"listener.tcp.external.acceptors=64\n"
"auth.pgsql.auth_query=select password from device_credentials where username = '%u' limit 1\n"
"allow_anonymous=false\n"
"auth.pgsql.encoding=utf8\n"
"listener.ssl.external.acceptors=32\n"
"node.process_limit=2097152\n"
"node.max_ets_tables=2097152\n"
"auth.pgsql.password_hash=plain\n"
"cluster.k8s.service_name=emqtt\n"
"cluster.k8s.service_name=emqtt\n"
"auth.pgsql.pool=4\n"
"auth.pgsql.super_query=select is_superuser from device_credentials where username = '%u' limit 1\n"
"auth.pgsql.password=device-service\n"
"auth.pgsql.server=127.0.0.1:5432\n"
"cluster.discovery=k8s\n"
"cluster.discovery=k8s\n"
"listener.ws.external.acceptors=16\n"
"auth.pgsql.acl_query=select allow, ipaddr, username, clientid, access, topic from device_acl where username = '%u'\n"
"auth.pgsql.ssl=false\n"
"[email protected]\n"
"cluster.k8s.app_name=emqtt\n"
"cluster.k8s.app_name=emqtt\n"
"cluster.k8s.apiserver=https://kubernetes.default.svc.cluster.local:443\n"
"cluster.k8s.apiserver=https://kubernetes.default.svc.cluster.local:443\n"
"cluster.k8s.namespace=infrastructure\n"
"cluster.k8s.namespace=infrastructure\n"
"auth.pgsql.database=device-service\n"
"acl_nomatch=deny\n"
"EMQX_LOADED_PLUGINS=emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_auth_pgsql\n"
"2019/11/27 07:02:47 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/27 07:02:47 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/27 07:02:47 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/27 07:02:47 New connection for \"myproject:europe-west2:production\"\n"
"emqx v3.2.5 is started successfully!\n"
"2019/11/27 07:02:49 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:49 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:49 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:49 Client closed local connection on 127.0.0.1:5432\n"
"2019-11-27 07:02:49.465 [critical] [EMQ X] emqx shutdown for join\n"
"2019/11/27 07:02:50 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/27 07:02:50 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/27 07:02:50 New connection for \"myproject:europe-west2:production\"\n"
"2019/11/27 07:02:50 New connection for \"myproject:europe-west2:production\"\n"
"['2019-11-27T07:02:53Z']:emqx start\n"
"2019-11-27 07:03:27.993 [warning] <<\"client-8852409\">>@10.154.15.202:49160 [Protocol] Client client-8852409 (Username: 'client-8852409') login failed for not_authorized\n"
"2019-11-27 07:03:30.804 [warning] <<\"client-4185337\">>@10.154.15.202:62986 [Protocol] Client client-4185337 (Username: 'client-4185337') login failed for not_authorized\n"
"2019-11-27 07:03:32.936 [warning] <<\"client-8811549\">>@10.154.15.202:58142 [Protocol] Client client-8811549 (Username: 'client-8811549') login failed for not_authorized\n"
"2019-11-27 07:03:33.646 [warning] <<\"client-8840958\">>@10.154.15.201:56506 [Protocol] Client client-8840958 (Username: 'client-8840958') login failed for not_authorized\n"
"2019-11-27 07:03:35.916 [warning] <<\"client-615694\">>@10.154.15.204:57731 [Protocol] Client client-615694 (Username: 'client-615694') login failed for not_authorized\n"
"2019-11-27 07:03:38.077 [warning] <<\"client-8817634\">>@10.154.15.204:52003 [Protocol] Client client-8817634 (Username: 'client-8817634') login failed for not_authorized\n"
"2019-11-27 07:03:38.673 [warning] <<\"client-8830005\">>@10.154.15.203:57662 [Protocol] Client client-8830005 (Username: 'client-8830005') login failed for not_authorized\n"
"2019-11-27 07:03:40.359 [warning] <<\"client-6188966\">>@10.154.15.202:56165 [Protocol] Client client-6188966 (Username: 'client-6188966') login failed for not_authorized\n"
"2019-11-27 07:03:40.903 [warning] <<\"client-8826347\">>@10.154.15.202:25340 [Protocol] Client client-8826347 (Username: 'client-8826347') login failed for not_authorized\n"
"2019-11-27 07:03:43.955 [warning] <<\"client-8199230\">>@10.56.1.1:59205 [Protocol] Client client-8199230 (Username: 'client-8199230') login failed for not_authorized\n"
"2019-11-27 07:03:44.041 [warning] <<\"client-8867908\">>@10.154.15.202:25330 [Protocol] Client client-8867908 (Username: 'client-8867908') login failed for not_authorized\n"
"2019-11-27 07:03:44.372 [warning] <<\"client-10891083\">>@10.56.1.1:52658 [Protocol] Client client-10891083 (Username: 'client-10891083') login failed for not_authorized\n"
"2019-11-27 07:03:44.417 [warning] <<\"client-8767983\">>@10.154.15.203:58754 [Protocol] Client client-8767983 (Username: 'client-8767983') login failed for not_authorized\n"
"2019-11-27 07:03:45.448 [warning] <<\"client-8213660\">>@10.154.15.201:64965 [Protocol] Client client-8213660 (Username: 'client-8213660') login failed for not_authorized\n"
"2019-11-27 07:03:46.591 [warning] <<\"client-4161719\">>@10.154.15.200:61217 [Protocol] Client client-4161719 (Username: 'client-4161719') login failed for not_authorized\n"
"2019-11-27 07:03:48.024 [warning] <<\"client-4204002\">>@10.154.15.202:62130 [Protocol] Client client-4204002 (Username: 'client-4204002') login failed for not_authorized\n"
"2019-11-27 07:03:53.265 [warning] <<\"client-4176755\">>@10.56.1.1:65423 [Protocol] Client client-4176755 (Username: 'client-4176755') login failed for not_authorized\n"
"2019-11-27 07:03:53.537 [warning] <<\"client-6185135\">>@10.154.15.204:59518 [Protocol] Client client-6185135 (Username: 'client-6185135') login failed for not_authorized\n"
"2019-11-27 07:03:55.769 [warning] <<\"client-4185337\">>@10.56.1.1:55810 [Protocol] Client client-4185337 (Username: 'client-4185337') login failed for not_authorized\n"
"2019-11-27 07:03:56.823 [warning] <<\"client-10081391\">>@10.154.15.200:60931 [Protocol] Client client-10081391 (Username: 'client-10081391') login failed for not_authorized\n"
"2019-11-27 07:03:57.289 [warning] <<\"client-8851621\">>@10.154.15.202:62100 [Protocol] Client client-8851621 (Username: 'client-8851621') login failed for not_authorized\n"
"2019-11-27 07:04:00.497 [warning] <<\"client-611173\">>@10.154.15.200:61461 [Protocol] Client client-611173 (Username: 'client-611173') login failed for not_authorized\n"
"2019-11-27 07:04:01.270 [warning] <<\"client-3681687\">>@10.56.1.1:58055 [Protocol] Client client-3681687 (Username: 'client-3681687') login failed for not_authorized\n"
"2019-11-27 07:04:02.357 [warning] <<\"client-8803127\">>@10.154.15.202:54373 [Protocol] Client client-8803127 (Username: 'client-8803127') login failed for not_authorized\n"
"2019-11-27 07:04:03.328 [warning] <<\"client-4166199\">>@10.154.15.204:50602 [Protocol] Client client-4166199 (Username: 'client-4166199') login failed for not_authorized\n"
"2019-11-27 07:04:05.904 [warning] <<\"client-8826347\">>@10.56.1.1:25339 [Protocol] Client client-8826347 (Username: 'client-8826347') login failed for not_authorized\n"
"2019-11-27 07:04:06.711 [warning] <<\"client-633226\">>@10.154.15.203:50984 [Protocol] Client client-633226 (Username: 'client-633226') login failed for not_authorized\n"
mariusstaicu picture mariusstaicu on 27 Nov 2019

@mariusstaicu Did you load any auth plugin?

tigercl picture tigercl on 28 Nov 2019

yes, emqx_auth_pgsql

mariusstaicu picture mariusstaicu on 28 Nov 2019

From the latest crashes I can see that the emqx broker crashes with exit code 1. Only thing contained in logs now is auth failed warnings from auth plugin. No other error whatsoever. Memory and cpu does not spike, it doesn't run out of memory.

mariusstaicu picture mariusstaicu on 30 Nov 2019

Hi, I think something wrong with your pgsql service caused emqx to stop. Please check your pqsql service. In emqx v3.2.6 we fixed the emqx crash caused by the auth plugin. Now you can get emqx/emqx:v3.2.6 image on the docker hub

zhanghongtong picture zhanghongtong on 2 Dec 2019
👍1

Thanks, I will try the new version.
Also, can I see logs from psql plugin somewhere ? is there a way to find out if psql plugin is unable to acquire new connection ?

mariusstaicu picture mariusstaicu on 2 Dec 2019

Error logs should be stored in log/emqx.log. *

zhanghongtong picture zhanghongtong on 2 Dec 2019
👍2

Yes, however there are no error logs anywhere, just warnings of authorization failures. Broker just shuts down with error code 1 without any error in log.

Log level is:
emqx_ctl log primary-level
warning

mariusstaicu picture mariusstaicu on 2 Dec 2019

upgrading to 3.2.6 does not fix the issue, broker still restarts

mariusstaicu picture mariusstaicu on 4 Dec 2019 
"sh: invalid number '681541emqx'\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"
"2019/11/27 07:02:21 Client closed local connection on 127.0.0.1:5432\n"

I wonder how the above error in your log was generated?

zhanghongtong picture zhanghongtong on 4 Dec 2019

Client closed local connection is a log of cloudsql proxy container in the same pod. It is used to connect to cloudsql db in google cloud. It appears when broker restarts, when the db connection is closed.
sh: invalid number I suspect it is output for when emqx container exits with error code: 1.
emqx broker logs contain nothing except warnings for authorization failed and logs of broker starting up normally after.

I could change logging level to see more detailed logging, however, no errors, except this, which appears once in a while.

[error] <<"clientid-4215080">>@10.56.0.1:56013 [Postgres] Auth from pgsql failed: not_authorized
mariusstaicu picture mariusstaicu on 4 Dec 2019

Please tell me the hardware configuration of your host? And what resources are allocated to the pod?

zhanghongtong picture zhanghongtong on 5 Dec 2019

GCP machine type: n1-standard-4 4 CPUs, 15 GB memory
we have 6-7 hosts like this

Emqx is deployed in a cluster of 3 nodes, each having
1 cpu dedicated and memory of 1G.

Normal CPU usage per broker is 50-70%, memory usage is 40-65% depending on how many clients are connected to that broker. So from resources point of view, the broker never exceeds the resources allocated. I suspect it has something to do with unauthorized devices that try to connect every second to the broker. The connection pool allocated to each broker for postgres auth plugin was 4 connections / broker. I increased it to more for one broker and I am watching if it will solve the problem or if the restarts will be less often.

Currently, each broker restarts approximately once every 48h at 5,5k connections and ~150 unauthorized clients trying to connect each every 30 seconds.

mariusstaicu picture mariusstaicu on 5 Dec 2019

I guess the reason for the emqx restart may have something to do with the lack of resources allocated by pods. Check out the events related to k8s pods

zhanghongtong picture zhanghongtong on 6 Dec 2019

I can allocate more resources, however there is no case when k8s kills the pods, there is no moment when emqx consumes more than allocated.

How many resources should I allocate to the pods?

mariusstaicu picture mariusstaicu on 6 Dec 2019

It seems the problem goes away when I increase the connection pool of the psql auth plugin. However, I wonder if it's normal that the psql plugin doesn't log anything about not being able to obtain connections, not even a warning. Is there a way to enable finer logging only for a specific plugin ?

mariusstaicu picture mariusstaicu on 8 Dec 2019

Actually, increasing pg connection pool has no effect. It seems all the brokers that have many unauthorized clients trying to connect will eventually restart. I have one broker with a single unauthorized client trying to connect and that one has no restart in over 1 week.

So the problem is directly connected to the number of unauthorized clients that keep trying to connect to the broker. This problem seems pretty big since it allows someone with bad intentions to make a broker with public ip restart.

mariusstaicu picture mariusstaicu on 11 Dec 2019

cc @turtleDeng

zhanghongtong picture zhanghongtong on 11 Dec 2019

@turtleDeng Please note that the node that had problems with the resources was the node that had the most of the connections and it never got killed by the K8S. so there's no problem with the resource, not in this case

vborcea picture vborcea on 17 Dec 2019

Currently, our emqx pods have QoS Guaranteed (GA) so they can never be killed by k8s.
CPU usage is now at < 30% on each node, memory usage is <35% on each node and still the broker restarts every ~48h.
emqx

The broker restarts itself and is not killed by k8s because I can see it from the termination logs and from k8s status history.

mariusstaicu picture mariusstaicu on 17 Dec 2019
👍1

downgraded to v3.1.1 and restart problems disappeared. Broker uptime is now 6d20h.

mariusstaicu picture mariusstaicu on 27 Dec 2019
👍2

@turtleDeng @zhanghongtong do you need more info on this?

vborcea picture vborcea on 8 Jan 2020

@mariusstaicu Hi, Can you provide the full emqx.log file? Logs are stored in log/emqx.log.*. , you can mount the log directory from the container to the host to retrieve the log files after the container crashes

zhanghongtong picture zhanghongtong on 8 Jan 2020

I can't, I don't have the logs anymore. We've downgraded our production env to v3.1.1 and the problem went away. I have another env with much less traffic with v4.0.0. In that one, it only happened once, however I'm not sure if it's the same problem. I can send you the logs for v4.0.0 if it helps.

mariusstaicu picture mariusstaicu on 27 Jan 2020

Upgraded to v4.0.1, the issue remains. Restarts on all brokers within one day.
I managed to get the logs from two hours before the crash from k8s.
emqx-before-crash.log
emqx.log

There is nothing out of the usual.

mariusstaicu picture mariusstaicu on 10 Feb 2020

If it helps, I can get all logs from the container before restart.

mariusstaicu picture mariusstaicu on 11 Feb 2020

decreased reconnection rate of clients, thus the number of unauthorized connection attempts. Also upgraded to 4.0.7 and it seems to be stable.

mariusstaicu picture mariusstaicu on 5 Jun 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jemc picture jemc  路  5Comments
jbrzozoski picture jbrzozoski  路  4Comments
stefano055415 picture stefano055415  路  4Comments
rootqa picture rootqa  路  3Comments
mikybone picture mikybone  路  3Comments