Netdata: Streaming Issues post Netdata Upgrade(build from source vs prebuilt) + Base image OS changeover

[jeremyjpj0916]

Question summary

I run Netdata on an OpenShift project with a multi slave(2 right now) -> master

So I was running on Debian Jessie, compiling/building from source for Netdata 1.10.0, never really had flakey connections or issues w streaming. But Now I am running on Alpine Linux and Netdata version 1.18.1 already compiled run(netdata-v1.18.1.gz.run) I am seeing a lot more problems.

Example taken from stdout of my slave node I see having an issue:

2019-11-15 17:52:54: netdata ERROR : STATSD : STREAM kong-170-r2wqk [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
--
  | 2019-11-15 17:52:55: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: connecting...
  | 2019-11-15 17:52:55: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-15 17:52:55: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-15 17:52:55: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-15 17:52:55: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send]: discarding 1100218 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2019-11-15 17:52:55: netdata INFO  : STATSD : STREAM kong-170-r2wqk [send]: sending metrics...
  | 2019-11-15 17:52:55: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: too many data pending - buffer is 1100460 bytes long, 1029162 unsent - we have sent 117328439736 bytes in total, 74094 on this connection. Closing connection to flush the data.
  | 2019-11-15 17:52:55: netdata ERROR : STATSD : STREAM kong-170-r2wqk [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2019-11-15 17:52:56: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: connecting...
  | 2019-11-15 17:52:56: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-15 17:52:56: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-15 17:52:56: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-15 17:52:56: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send]: discarding 1101805 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2019-11-15 17:52:56: netdata INFO  : STATSD : STREAM kong-170-r2wqk [send]: sending metrics...
  | 2019-11-15 17:52:56: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: too many data pending - buffer is 1081141 bytes long, 917575 unsent - we have sent 117328603544 bytes in total, 163808 on this connection. Closing connection to flush the data.
  | 2019-11-15 17:52:56: netdata ERROR : STATSD : STREAM kong-170-r2wqk [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2019-11-15 17:52:57: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: connecting...
  | 2019-11-15 17:52:57: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-15 17:52:57: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-15 17:52:57: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-15 17:52:57: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send]: discarding 1082773 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2019-11-15 17:52:57: netdata INFO  : STATSD : STREAM kong-170-r2wqk [send]: sending metrics...
  | 2019-11-15 17:52:57: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: too many data pending - buffer is 1125062 bytes long, 1091510 unsent - we have sent 117328637338 bytes in total, 33794 on this connection. Closing connection to flush the data.
  | 2019-11-15 17:52:57: netdata ERROR : STATSD : STREAM kong-170-r2wqk [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2019-11-15 17:52:58: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: connecting...
  | 2019-11-15 17:52:58: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-15 17:52:58: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-15 17:52:58: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-15 17:52:58: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send]: discarding 1126968 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2019-11-15 17:52:58: netdata INFO  : STATSD : STREAM kong-170-r2wqk [send]: sending metrics...
  | 2019-11-15 17:52:58: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: too many data pending - buffer is 1107527 bytes long, 1034831 unsent - we have sent 117328711794 bytes in total, 74456 on this connection. Closing connection to flush the data.
  | 2019-11-15 17:52:58: netdata ERROR : STATSD : STREAM kong-170-r2wqk [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2019-11-15 17:52:59: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: connecting...
  | 2019-11-15 17:52:59: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-15 17:52:59: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-15 17:52:59: netdata INFO  : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-15 17:52:59: netdata ERROR : STREAM_SENDER[kong-170-r2wqk] : STREAM kong-170-r2wqk [send]: discarding 1109003 bytes of metrics already in the buffer. (errno 22, Invalid argument)

And from the master node when I tab into the slave node to look at its charts, see how there are breaks in the chart, some charts much more broken up than others(not included in screenshot):

But the master nodes dashboard itself loads healthy and fine. No pod restarts on slave or master.

I see some logs like this in the master netdata node, but they seem harmless:

2019-11-18 05:52:12: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:43350] : RRDSET: chart name 'statsd_counter_kong.xxxx_xxx_xxx_service_v3_test15.user.xxxxx.request.status.200' on host 'kong-546-jbh6q' already exists.

And here is the full startup logs of the master netdata node:

2019-11-18 06:03:43: netdata INFO  : MAIN : SIGNAL: Enabling reaper
  | 2019-11-18 06:03:43: netdata INFO  : MAIN : process tracking enabled.
  | 2019-11-18 06:03:43: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
  | 2019-11-18 06:03:43: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value -998
  | 2019-11-18 06:03:43: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
  | 2019-11-18 06:03:43: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
  | 2019-11-18 06:03:43: netdata ERROR : MAIN : Cannot switch to user's netdata group (gid: 1000). (errno 1, Operation not permitted)
  | 2019-11-18 06:03:43: netdata ERROR : MAIN : Cannot become user 'netdata'. Continuing as we are.
  | 2019-11-18 06:03:43: netdata INFO  : MAIN : netdata started on pid 1.
  | 2019-11-18 06:03:43: netdata INFO  : MAIN : Executing /opt/netdata/usr/libexec/netdata/plugins.d/system-info.sh
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_NAME="Alpine Linux"
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_ID=alpine
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_ID_LIKE=unknown
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_VERSION=unknown
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_VERSION_ID=3.10.2
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_DETECTION=/etc/os-release
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-957.27.2.el7.x86_64
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=hypervisor
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=/proc/cpuinfo
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=docker
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=dockerenv
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : Found 0 files in path /opt/netdata/var/cache/netdata/dbengine
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : Data files not found, creating in path "/opt/netdata/var/cache/netdata/dbengine".
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : Creating new data and journal files in path /opt/netdata/var/cache/netdata/dbengine
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : Created data file "/opt/netdata/var/cache/netdata/dbengine/datafile-1-0000000001.ndf".
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : Created journal file "/opt/netdata/var/cache/netdata/dbengine/journalfile-1-0000000001.njf".
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : Host 'netdata-107-9rmkv' (at registry as 'netdata-107-9rmkv') with guid '2f35e312-09c9-11ea-9096-0a580a804e77' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.18.1', update every 1, memory mode dbengine, history entries 924, streaming disabled (to '' with api key ''), health disabled, cache_dir '/opt/netdata/var/cache/netdata', varlib_dir '/opt/netdata/var/lib/netdata', health_log '/opt/netdata/var/lib/netdata/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2019-11-18 06:03:47: netdata INFO  : PLUGIN[proc] : thread created with task id 17
  | 2019-11-18 06:03:47: netdata INFO  : STATSD : thread created with task id 18
  | 2019-11-18 06:03:47: netdata INFO  : STATSD : set name of thread 18 to STATSD
  | 2019-11-18 06:03:47: netdata INFO  : BACKENDS : thread created with task id 19
  | 2019-11-18 06:03:47: netdata INFO  : BACKENDS : set name of thread 19 to BACKENDS
  | 2019-11-18 06:03:47: netdata INFO  : PLUGINSD : thread created with task id 21
  | 2019-11-18 06:03:47: netdata INFO  : PLUGINSD : set name of thread 21 to PLUGINSD
  | 2019-11-18 06:03:47: netdata INFO  : PLUGIN[proc] : set name of thread 17 to PLUGIN[proc]
  | 2019-11-18 06:03:47: netdata INFO  : MAIN : netdata initialization completed. Enjoy real-time performance monitoring!
  | 2019-11-18 06:03:47: netdata INFO  : HEALTH : thread created with task id 22
  | 2019-11-18 06:03:47: netdata INFO  : HEALTH : set name of thread 22 to HEALTH
  | 2019-11-18 06:03:47: netdata INFO  : BACKENDS : cleaning up...
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : thread created with task id 20
  | 2019-11-18 06:03:47: netdata INFO  : BACKENDS : thread with task id 19 finished
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : set name of thread 20 to WEB_SERVER[stat
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : To use encryption it is necessary to set "ssl certificate" and "ssl key" in [web] !
  |  
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : starting worker 2
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : starting worker 3
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static2] : thread created with task id 23
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : starting worker 4
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static2] : set name of thread 23 to WEB_SERVER[stat
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : starting worker 5
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static4] : thread created with task id 25
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static4] : set name of thread 25 to WEB_SERVER[stat
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static5] : thread created with task id 26
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static5] : set name of thread 26 to WEB_SERVER[stat
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static3] : thread created with task id 24
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static3] : set name of thread 24 to WEB_SERVER[stat
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : starting worker 6
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static6] : thread created with task id 27
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static6] : set name of thread 27 to WEB_SERVER[stat
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-18 06:03:47: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-18 06:03:47: netdata INFO  : STATSD : cleaning up...
  | 2019-11-18 06:03:47: netdata INFO  : STATSD : STATSD: closing sockets...
  | 2019-11-18 06:03:47: netdata INFO  : STATSD : STATSD: cleanup completed.
  | 2019-11-18 06:03:47: netdata INFO  : STATSD : thread with task id 18 finished
  | 2019-11-18 06:03:48: netdata INFO  : WEB_SERVER[static6] : clients wants to STREAM metrics.
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : thread created with task id 28
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : set name of thread 28 to STREAM_RECEIVER
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : STREAM kong-546-jbh6q [10.131.78.1]:47376: receive thread created (task id 28)
  | 2019-11-18 06:03:48: netdata INFO  : WEB_SERVER[static5] : clients wants to STREAM metrics.
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : thread created with task id 29
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : set name of thread 29 to STREAM_RECEIVER
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : STREAM kong-546-r5j8t [10.130.72.1]:39122: receive thread created (task id 29)
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : HEALTH [kong-546-jbh6q]: cannot open health file: /opt/netdata/var/lib/netdata/2ffc6e18-0908-11ea-931c-0a580a834e44/health/health-log.db.old (errno 2, No such file or directory)
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : HEALTH [kong-546-jbh6q]: cannot open health file: /opt/netdata/var/lib/netdata/2ffc6e18-0908-11ea-931c-0a580a834e44/health/health-log.db (errno 2, No such file or directory)
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : Host 'kong-546-jbh6q' (at registry as 'kong-546-jbh6q') with guid '2ffc6e18-0908-11ea-931c-0a580a834e44' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.18.1', update every 1, memory mode ram, history entries 3996, streaming disabled (to '' with api key ''), health enabled, cache_dir '/opt/netdata/var/cache/netdata/2ffc6e18-0908-11ea-931c-0a580a834e44', varlib_dir '/opt/netdata/var/lib/netdata/2ffc6e18-0908-11ea-931c-0a580a834e44', health_log '/opt/netdata/var/lib/netdata/2ffc6e18-0908-11ea-931c-0a580a834e44/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : STREAM kong-546-jbh6q [receive from [10.131.78.1]:47376]: initializing communication...
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : Postponing health checks for 60 seconds, on host 'kong-546-jbh6q', because it was just connected.
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : STREAM kong-546-jbh6q [receive from [10.131.78.1]:47376]: receiving metrics...
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : HEALTH [kong-546-r5j8t]: cannot open health file: /opt/netdata/var/lib/netdata/cba9d56c-0908-11ea-9f17-0a580a82482a/health/health-log.db.old (errno 2, No such file or directory)
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : HEALTH [kong-546-r5j8t]: cannot open health file: /opt/netdata/var/lib/netdata/cba9d56c-0908-11ea-9f17-0a580a82482a/health/health-log.db (errno 2, No such file or directory)
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : Host 'kong-546-r5j8t' (at registry as 'kong-546-r5j8t') with guid 'cba9d56c-0908-11ea-9f17-0a580a82482a' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.18.1', update every 1, memory mode ram, history entries 3996, streaming disabled (to '' with api key ''), health enabled, cache_dir '/opt/netdata/var/cache/netdata/cba9d56c-0908-11ea-9f17-0a580a82482a', varlib_dir '/opt/netdata/var/lib/netdata/cba9d56c-0908-11ea-9f17-0a580a82482a', health_log '/opt/netdata/var/lib/netdata/cba9d56c-0908-11ea-9f17-0a580a82482a/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : STREAM kong-546-r5j8t [receive from [10.130.72.1]:39122]: initializing communication...
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : Postponing health checks for 60 seconds, on host 'kong-546-r5j8t', because it was just connected.
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : STREAM kong-546-r5j8t [receive from [10.130.72.1]:39122]: receiving metrics...
  | 2019-11-18 06:03:48: netdata INFO  : PLUGIN[proc] : Using now_boottime_usec() for uptime (dt is 1 ms)
  | 2019-11-18 06:03:48: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/net/sctp/snmp' (errno 2, No such file or directory)
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : read failed: end of file
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : STREAM kong-546-jbh6q [receive from [10.131.78.1]:47376]: disconnected (completed 110 updates). (errno 22, Invalid argument)
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : STREAM kong-546-jbh6q [receive from [10.131.78.1]:47376]: receive thread ended (task id 28)
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47376] : thread with task id 28 finished
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : requested a CHART, without a type.id, on host 'kong-546-r5j8t'. Disabling it.
  | 2019-11-18 06:03:48: netdata ERROR : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : STREAM kong-546-r5j8t [receive from [10.130.72.1]:39122]: disconnected (completed 34 updates). (errno 22, Invalid argument)
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : STREAM kong-546-r5j8t [receive from [10.130.72.1]:39122]: receive thread ended (task id 29)
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39122] : thread with task id 29 finished
  | 2019-11-18 06:03:48: netdata INFO  : WEB_SERVER[static3] : clients wants to STREAM metrics.
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : thread created with task id 30
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : set name of thread 30 to STREAM_RECEIVER
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : STREAM kong-546-jbh6q [10.131.78.1]:47382: receive thread created (task id 30)
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : STREAM kong-546-jbh6q [receive from [10.131.78.1]:47382]: initializing communication...
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : Postponing health checks for 60 seconds, on host 'kong-546-jbh6q', because it was just connected.
  | 2019-11-18 06:03:48: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : STREAM kong-546-jbh6q [receive from [10.131.78.1]:47382]: receiving metrics...
  | 2019-11-18 06:03:49: netdata INFO  : WEB_SERVER[static4] : clients wants to STREAM metrics.
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39124] : thread created with task id 31
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39124] : set name of thread 31 to STREAM_RECEIVER
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39124] : STREAM kong-546-r5j8t [10.130.72.1]:39124: receive thread created (task id 31)
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39124] : STREAM kong-546-r5j8t [receive from [10.130.72.1]:39124]: initializing communication...
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39124] : Postponing health checks for 60 seconds, on host 'kong-546-r5j8t', because it was just connected.
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-r5j8t,[10.130.72.1]:39124] : STREAM kong-546-r5j8t [receive from [10.130.72.1]:39124]: receiving metrics...
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : RRDSET: chart name 'statsd_counter_kong.oauth2perf.user.phyconapi.request.count' on host 'kong-546-jbh6q' already exists.
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : RRDSET: chart name 'statsd_counter_kong.phyconapi_processmacros_uat.user.phyconapi.request.count' on host 'kong-546-jbh6q' already exists.
  | 2019-11-18 06:03:49: netdata INFO  : STREAM_RECEIVER[kong-546-jbh6q,[10.131.78.1]:47382] : RRDSET: chart name 'statsd_counter_kong.phyconapi_processmacros_uat.request.count' on host 'kong-546-jbh6q' already exists.

I checked this thread because this error has posted before it seems:
https://github.com/netdata/netdata/issues/6307

This one above ^ seems to be the the one that hits home most to me. I may revert to building from source(as much as I hate having to do that) to fix the flakey connection error issues.

I also checked these thread too because errors looked similar, but my guid is confirmed unique(so not this, master logs above show the uuids are unique too, also unique hostnames too):
https://github.com/netdata/netdata/issues/4049
https://github.com/netdata/netdata/issues/5014

Relevant Config files for slave nodes:

slave-netdata.conf.txt

slave-netdata-streams.conf.txt

Relevant Config files for master:

master-netdata-conf.txt

master-netdata-stream-conf.txt

Did a netstat on the SLAVE node by port for time_wait wondering if there was anything there with ephemeral port exhaustion playing a role in tcp streaming issues and re-connections but seems fine to me:

/ $ netstat -nt | sed -r -n 's/^tcp +[0-9]+ +[0-9]+ [0-9\.]+(:[0-9]+).+TIME_WAIT/\1/p' | sort | uniq -c | sort -n
      1 :41392
      1 :52224
      1 :52232
      1 :52238
      1 :52240
      1 :52252
      1 :52260
      1 :52266
      1 :52272
      1 :52276
      1 :52290
      1 :52296
      1 :52298
      1 :52304
      1 :52310
      1 :52324
      1 :52332
      1 :52342
      1 :52356
      1 :52364
      1 :52366
      1 :52378
      1 :52380
      1 :52392
      1 :52404
      1 :52410
      1 :52416
      1 :52418
      1 :52446
      1 :52458
      1 :52470
      1 :52484
      1 :52498
      1 :52504
      1 :52514
      1 :52516
      1 :52522
      1 :52526
      1 :52534
      1 :52544
      1 :52550
      1 :52562
      1 :52568
      1 :52578
      1 :52594
      1 :52602
      1 :52610
      1 :52616
      1 :52624
      1 :52632
      1 :52642
      1 :52646
      1 :52652
      1 :52668
      1 :52678
      1 :52692
      1 :52708
      1 :52726
      1 :52738
      1 :52752
      1 :59332
      1 :59358
      1 :59360

Master node seems fine too:

/ $ netstat -nt | sed -r -n 's/^tcp +[0-9]+ +[0-9]+ [0-9\.]+(:[0-9]+).+TIME_WAIT/\1/p' | sort | uniq -c | sort -n
      6 :19999

Lastly, here is how I build my final docker image based on that prebuilt binary(for the slave nodes for example):

FROM docker.company.com/repo/alpine:3.10

USER root

ENV NETDATA_VERSION=1.18.1

# install required packages
RUN apk add alpine-sdk bash curl zlib-dev util-linux-dev libmnl-dev gcc make git autoconf automake pkgconfig python logrotate

ADD . /gitsources
#Fix /opt/netdata path and logrotate
RUN chmod -R 777 /gitsources \
    && touch /etc/logrotate.d/netdata \
    && chmod 777 /etc/logrotate.d/netdata \
    && mkdir /opt/netdata \
    && chmod -R 777 /opt/netdata \
    && chmod 777 /gitsources/kickstart-static64.sh \
    && chmod 777 /gitsources/netdata-v$NETDATA_VERSION.gz.run

RUN bash /gitsources/kickstart-static64.sh --dont-wait --dont-start-it --no-updates --local-files /gitsources/netdata-v$NETDATA_VERSION.gz.run /gitsources/sha256sums.txt

COPY ./system/netdata.conf /opt/netdata/etc/netdata/netdata.conf
COPY ./conf.d/stream.conf /opt/netdata/etc/netdata/stream.conf
COPY ./conf.d/python.d.conf /opt/netdata/etc/netdata/python.d.conf
COPY ./conf.d/python.d/nginx.conf /opt/netdata/etc/netdata/python.d/nginx.conf
RUN touch /opt/netdata/etc/netdata/.opt-out-from-anonymous-statistics

RUN chmod -R 777 /opt/netdata

WORKDIR /
ENV NETDATA_PORT 19999
EXPOSE 19999 19999/udp 8125 8125/udp

CMD /opt/netdata/bin/netdata -D -p 19999

OS / Environment

Apline Linux / Docker containers(OpenShift/Kubernetes project).

Component Name

Netdata slave->master stream functionality.

Expected results

Stream is healthy with no connectivity and dataflow issues.

[jeremyjpj0916]

Originally I was writing this up as a question, but it should probs get the bug label.

[jeremyjpj0916]

Lastly, my netdata slave node startup logs:

2019-11-17 07:06:35: netdata INFO  : MAIN : SIGNAL: Enabling reaper
  | 2019-11-17 07:06:35: netdata INFO  : MAIN : process tracking enabled.
  | 2019-11-17 07:06:35: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
  | 2019-11-17 07:06:35: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value -998
  | 2019-11-17 07:06:35: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
  | 2019-11-17 07:06:35: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
  | 2019-11-17 07:06:35: netdata ERROR : MAIN : Cannot switch to user's netdata group (gid: 1000). (errno 1, Operation not permitted)
  | 2019-11-17 07:06:35: netdata ERROR : MAIN : Cannot become user 'netdata'. Continuing as we are.
  | 2019-11-17 07:06:35: netdata INFO  : MAIN : netdata started on pid 1.
  | 2019-11-17 07:06:35: netdata INFO  : MAIN : Executing /opt/netdata/usr/libexec/netdata/plugins.d/system-info.sh
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_NAME="Alpine Linux"
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_ID=alpine
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_ID_LIKE=unknown
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_VERSION=unknown
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_VERSION_ID=3.10.2
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_OS_DETECTION=/etc/os-release
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-957.27.2.el7.x86_64
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=hypervisor
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=/proc/cpuinfo
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=docker
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=dockerenv
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : Host 'kong-546-r5j8t' (at registry as 'kong-546-r5j8t') with guid 'cba9d56c-0908-11ea-9f17-0a580a82482a' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.18.1', update every 1, memory mode none, history entries 924, streaming enabled (to 'tcp:netdata:19999' with api key '11111111-2222-3333-4444-555555555555'), health disabled, cache_dir '/opt/netdata/var/cache/netdata', varlib_dir '/opt/netdata/var/lib/netdata', health_log '/opt/netdata/var/lib/netdata/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2019-11-17 07:06:36: netdata INFO  : PLUGIN[proc] : thread created with task id 16
  | 2019-11-17 07:06:36: netdata INFO  : PLUGIN[proc] : set name of thread 16 to PLUGIN[proc]
  | 2019-11-17 07:06:36: netdata INFO  : PLUGINSD : thread created with task id 20
  | 2019-11-17 07:06:36: netdata INFO  : HEALTH : thread created with task id 21
  | 2019-11-17 07:06:36: netdata INFO  : PLUGINSD : set name of thread 20 to PLUGINSD
  | 2019-11-17 07:06:36: netdata INFO  : HEALTH : set name of thread 21 to HEALTH
  | 2019-11-17 07:06:36: netdata INFO  : BACKENDS : thread created with task id 18
  | 2019-11-17 07:06:36: netdata INFO  : BACKENDS : set name of thread 18 to BACKENDS
  | 2019-11-17 07:06:36: netdata INFO  : BACKENDS : cleaning up...
  | 2019-11-17 07:06:36: netdata INFO  : BACKENDS : thread with task id 18 finished
  | 2019-11-17 07:06:36: netdata INFO  : PLUGINSD[python.d] : thread created with task id 22
  | 2019-11-17 07:06:36: netdata INFO  : MAIN : netdata initialization completed. Enjoy real-time performance monitoring!
  | 2019-11-17 07:06:36: netdata INFO  : PLUGINSD[python.d] : set name of thread 22 to PLUGINSD[python
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : thread created with task id 19
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : set name of thread 19 to WEB_SERVER[stat
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : To use encryption it is necessary to set "ssl certificate" and "ssl key" in [web] !
  |  
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : starting worker 2
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : starting worker 3
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : starting worker 4
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static3] : thread created with task id 24
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : starting worker 5
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static3] : set name of thread 24 to WEB_SERVER[stat
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static4] : thread created with task id 25
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static5] : thread created with task id 26
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static4] : set name of thread 25 to WEB_SERVER[stat
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : starting worker 6
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static5] : set name of thread 26 to WEB_SERVER[stat
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static6] : thread created with task id 27
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static6] : set name of thread 27 to WEB_SERVER[stat
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static2] : thread created with task id 23
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static2] : set name of thread 23 to WEB_SERVER[stat
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2019-11-17 07:06:36: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2019-11-17 07:06:36: netdata INFO  : STATSD : thread created with task id 17
  | 2019-11-17 07:06:36: netdata INFO  : STATSD : set name of thread 17 to STATSD
  | 2019-11-17 07:06:36: netdata INFO  : STATSD_COLLECTOR[1] : thread created with task id 28
  | 2019-11-17 07:06:36: netdata INFO  : STATSD_COLLECTOR[1] : set name of thread 28 to STATSD_COLLECTO
  | 2019-11-17 07:06:36: netdata INFO  : STATSD_COLLECTOR[1] : STATSD collector thread started with taskid 28
  | 2019-11-17 07:06:36: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'udp:0.0.0.0:8125'
  | 2019-11-17 07:06:36: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'udp:[::]:8125'
  | 2019-11-17 07:06:36: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:8125'
  | 2019-11-17 07:06:36: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'tcp:[::]:8125'
  | 2019-11-17 07:06:37: netdata ERROR : STATSD : STREAM kong-546-r5j8t [send]: not ready - discarding collected metrics. (errno 2, No such file or directory)
  | 2019-11-17 07:06:37: netdata INFO  : STREAM_SENDER[kong-546-r5j8t] : thread created with task id 29
  | 2019-11-17 07:06:37: netdata INFO  : STREAM_SENDER[kong-546-r5j8t] : set name of thread 29 to STREAM_SENDER[k
  | 2019-11-17 07:06:37: netdata INFO  : STREAM_SENDER[kong-546-r5j8t] : STREAM kong-546-r5j8t [send]: thread created (task id 29)
  | 2019-11-17 07:06:37: netdata INFO  : PLUGIN[proc] : Using now_boottime_usec() for uptime (dt is 1 ms)
  | 2019-11-17 07:06:37: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/net/sctp/snmp' (errno 2, No such file or directory)
  | 2019-11-17 07:06:38: netdata INFO  : PLUGINSD[python.d] : connected to '/opt/netdata/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 30
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : using python v2
  | 2019-11-17 07:06:38: python.d WARNING: plugin[main] : 'pythond-jobs-statuses.json' was not found
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [adaptec_raid] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [apache] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [beanstalk] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [bind_rndc] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [boinc] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [ceph] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [chrony] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [couchdb] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [dns_query_time] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [dnsdist] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [dockerd] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [dovecot] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [elasticsearch] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:38: python.d INFO: plugin[main] : [energid] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [example] built 1 job(s) configs
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [exim] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [fail2ban] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [freeradius] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [gearman] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [go_expvar] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [haproxy] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [hddtemp] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [httpcheck] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [icecast] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [ipfs] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [isc_dhcpd] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [litespeed] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [logind] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [megacli] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [memcached] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [mongodb] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [monit] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [mysql] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [nginx] built 1 job(s) configs
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [nginx_plus] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [nsd] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [ntpd] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [nvidia_smi] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [openldap] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [oracledb] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [ovpn_status_log] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [phpfpm] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [portcheck] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [postfix] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [postgres] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [powerdns] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [proxysql] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [puppet] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [rabbitmq] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [redis] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [rethinkdbs] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [retroshare] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [riakkv] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [samba] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [sensors] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [smartd_log] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [spigotmc] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [springboot] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [squid] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [tomcat] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [tor] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [traefik] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [unbound] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [uwsgi] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [varnish] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [w1sensor] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : [web_log] is disabled in the configuration file, skipping it
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : example[example] : check success
  | 2019-11-17 07:06:39: python.d INFO: plugin[main] : nginx[localhost] : check success
  | 2019-11-17 07:06:42: netdata INFO  : STREAM_SENDER[kong-546-r5j8t] : STREAM kong-546-r5j8t [send to tcp:netdata:19999]: connecting...
  | 2019-11-17 07:06:42: netdata INFO  : STREAM_SENDER[kong-546-r5j8t] : STREAM kong-546-r5j8t [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-17 07:06:42: netdata INFO  : STREAM_SENDER[kong-546-r5j8t] : STREAM kong-546-r5j8t [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-17 07:06:42: netdata INFO  : STREAM_SENDER[kong-546-r5j8t] : STREAM kong-546-r5j8t [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-17 07:06:43: netdata INFO  : STATSD : STREAM kong-546-r5j8t [send]: sending metrics...
  | 2019-11-18 01:26:58: netdata ERROR : STREAM_SENDER[kong-546-r5j8t] : STREAM kong-546-r5j8t [send to tcp:netdata:19999]: too many data pending - buffer is 1048618 bytes long, 1048618 unsent - we have sent 13905909532 bytes in total, 13905909532 on this connection. Closing connection to flush the data.

[thiagoftsm]

Hi @jeremyjpj0916 ,

Between 1.10 and 1.18.1 we had many changes, but initially I think that you are probably having problems with your database configuration, I will do few questions for you for we confirm and we fix your problem.

1 - On your netdata master your, configuration does not say nothing about the memory mode and I am assuming that you are using dbengine, please, instead to copy the netdata.conf from the directory, can you access http://localhost:19999/netdata.conf and give us the output? Please, do the same with slave.
2 - Are both master and slave running the latest version?
3 - Only to confirm, in your slave destination you are using destination = tcp:netdata:19999, but are you using the master ip address instead netdata word inside the original stream.conf?

Best regards!

[thiagoftsm]

@knatsakis and @oxplot , please, can you check the docker compilation process to confirm that we do not have a problem there? I do not have condition to do this.

[jeremyjpj0916]

@thiagoftsm appreciate the prompt response!

1. Master netdata conf from the endpoint:

# netdata configuration
#
# You can download the latest version of this file, using:
#
#  wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
# or
#  curl -o /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
#
# You can uncomment and change any of the options below.
# The value shown in the commented settings, is the default value.
#

# global netdata configuration

[global]
    run as user = netdata
    history = 924
    access log = none
    error log = /dev/stderr
    debug log = none
    cleanup obsolete charts after seconds = 300
    cleanup orphan hosts after seconds = 300
    delete obsolete charts files = yes
    delete orphan hosts files = yes
    # glibc malloc arena max for plugins = 1
    # hostname = netdata-108-f5trj
    # update every = 1
    # config directory = /opt/netdata/etc/netdata
    # stock config directory = /opt/netdata/usr/lib/netdata/conf.d
    # log directory = /opt/netdata/var/log/netdata
    # web files directory = /opt/netdata/usr/share/netdata/web
    # cache directory = /opt/netdata/var/cache/netdata
    # lib directory = /opt/netdata/var/lib/netdata
    # home directory = /opt/netdata/var/cache/netdata
    # plugins directory = "/opt/netdata/usr/libexec/netdata/plugins.d" "/opt/netdata/etc/netdata/custom-plugins.d"
    # memory mode = dbengine
    # page cache size = 32
    # dbengine disk space = 256
    # host access prefix = 
    # memory deduplication (ksm) = yes
    # TZ environment variable = :/etc/localtime
    # timezone = UTC
    # debug flags = 0x0000000000000000
    # facility log = daemon
    # errors flood protection period = 1200
    # errors to trigger flood protection = 200
    # OOM score = -998
    # process scheduling policy = idle
    # process nice level = 19
    # pthread stack size = 81920
    # gap when lost iterations above = 1
    # enable zero metrics = no

[plugins]
    proc = yes
    diskspace = no
    cgroups = no
    tc = no
    idlejitter = no
    enable running new plugins = no
    charts.d = no
    fping = no
    node.d = no
    python.d = no
    apps = no
    # PATH environment variable = /opt/netdata/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
    # PYTHONPATH environment variable = 
    # check for new plugins every = 60
    # slabinfo = no
    # go.d = no
    # ioping = no
    # perf = no

[web]
    # default port = 19999
    web files owner = netdata
    web files group = netdata
    allow streaming from = *

    # option 'allow connections by dns' is not used.
    allow connections by dns = no

    # option 'allow dashboard by dns' is not used.
    allow dashboard by dns = no

    # option 'allow badges by dns' is not used.
    allow badges by dns = no

    # option 'allow registry by dns' is not used.
    allow registry by dns = no

    # option 'allow streaming by dns' is not used.
    allow streaming by dns = no

    # option 'allow netdata.conf by dns' is not used.
    allow netdata.conf by dns = no

    # option 'allow management by dns' is not used.
    allow management by dns = no
    # ssl key = /opt/netdata/etc/netdata/ssl/key.pem
    # ssl certificate = /opt/netdata/etc/netdata/ssl/cert.pem
    # ses max window = 15
    # des max window = 15
    # mode = static-threaded
    # listen backlog = 4096
    # bind to = *
    # disconnect idle clients after seconds = 60
    # timeout for first request = 60
    # accept a streaming request every seconds = 0
    # respect do not track policy = no
    # x-frame-options response header = 
    # allow connections from = localhost *
    # allow dashboard from = localhost *
    # allow badges from = *
    # allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.*
    # allow management from = localhost
    # enable gzip compression = yes
    # gzip compression strategy = default
    # gzip compression level = 3
    # web server threads = 6
    # web server max sockets = 262144
    # custom dashboard_info.js = 

1. Slave conf from the endpoint:

#  wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
# or
#  curl -o /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
#
# You can uncomment and change any of the options below.
# The value shown in the commented settings, is the default value.
#

# global netdata configuration

[global]
        memory mode = none
        history = 924
        run as user = netdata
        access log = none
        error log = /dev/stderr
        debug log = none
        cleanup obsolete charts after seconds = 300
        delete obsolete charts files = yes
        # glibc malloc arena max for plugins = 1
        # hostname = kong-89-rrtc8
        # update every = 1
        # config directory = /opt/netdata/etc/netdata
        # stock config directory = /opt/netdata/usr/lib/netdata/conf.d
        # log directory = /opt/netdata/var/log/netdata
        # web files directory = /opt/netdata/usr/share/netdata/web
        # cache directory = /opt/netdata/var/cache/netdata
        # lib directory = /opt/netdata/var/lib/netdata
        # home directory = /opt/netdata/var/cache/netdata
        # plugins directory = "/opt/netdata/usr/libexec/netdata/plugins.d" "/opt/netdata/etc/netdata/custom-plugins.d"
        # page cache size = 32
        # dbengine disk space = 256
        # host access prefix =
        # memory deduplication (ksm) = yes
        # TZ environment variable = :/etc/localtime
        # timezone = UTC
        # debug flags = 0x0000000000000000
        # facility log = daemon
        # errors flood protection period = 1200
        # errors to trigger flood protection = 200
        # OOM score = -998
        # process scheduling policy = idle
        # process nice level = 19
        # pthread stack size = 81920
        # gap when lost iterations above = 1
        # cleanup orphan hosts after seconds = 3600
        # delete orphan hosts files = yes
        # enable zero metrics = no

[plugins]
        proc = yes
        diskspace = no
        cgroups = no
        tc = no
        idlejitter = no
        enable running new plugins = no
        charts.d = no
        fping = no
        node.d = no
        python.d = yes
        apps = no
        # PATH environment variable = /opt/netdata/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
        # PYTHONPATH environment variable =
        # check for new plugins every = 60
        # slabinfo = no
        # go.d = no
        # ioping = no
        # perf = no

[web]
        # default port = 19999
        web files owner = netdata
        web files group = netdata

        # option 'allow connections by dns' is not used.
        allow connections by dns = no

        # option 'allow dashboard by dns' is not used.
        allow dashboard by dns = no

        # option 'allow badges by dns' is not used.
        allow badges by dns = no

        # option 'allow registry by dns' is not used.
        allow registry by dns = no

        # option 'allow streaming by dns' is not used.
        allow streaming by dns = no

        # option 'allow netdata.conf by dns' is not used.
        allow netdata.conf by dns = no

        # option 'allow management by dns' is not used.
        allow management by dns = no
        # ssl key = /opt/netdata/etc/netdata/ssl/key.pem
        # ssl certificate = /opt/netdata/etc/netdata/ssl/cert.pem
        # ses max window = 15
        # des max window = 15
        # mode = static-threaded
        # listen backlog = 4096
        # bind to = *
        # disconnect idle clients after seconds = 60
        # timeout for first request = 60
        # accept a streaming request every seconds = 0
        # respect do not track policy = no
        # x-frame-options response header =
        # allow connections from = localhost *
        # allow dashboard from = localhost *
        # allow badges from = *
        # allow streaming from = *
        # allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172
        # allow management from = localhost
        # enable gzip compression = yes
        # gzip compression strategy = default
        # gzip compression level = 3
        # web server threads = 6
        # web server max sockets = 262144

1. Yes, its funny in dev I actually was running a new slave node version and old master briefly during all this upgrade testing process by accident and saw OOM crashes happening on master, so there was certainly a compatibility issue early on in my testing before I got them both on the same version. That problem went away when I got them on same version.

netdata
v1.18.1

1. No I am not using an actual static IP address because the master node is not on a predictable IP as we rotate application pods on a weekly basis. Its in openshift/kubernetes so its resolving by its service name in my openshift project(which has never been a problem before). If it was having resolution issues I would see that in the slave logs I believe but you can see it connecting up above to the local ip in the original logs I posted.

[jeremyjpj0916]

Checking my prod environment Netdata node(still on working 1.10.0) I see this in the conf's:

1. Master netdata node:

# netdata configuration
#
# You can download the latest version of this file, using:
#
#  wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
# or
#  curl -o /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
#
# You can uncomment and change any of the options below.
# The value shown in the commented settings, is the default value.
#

# global netdata configuration

[global]
    run as user = netdata
    history = 924
    access log = none
    debug log = none
    cleanup obsolete charts after seconds = 300
    cleanup orphan hosts after seconds = 300
    delete obsolete charts files = yes
    delete orphan hosts files = yes
    # glibc malloc arena max for plugins = 1
    # glibc malloc arena max for netdata = 1
    # hostname = netdata-96-lphfc
    # update every = 1
    # config directory = /etc/netdata
    # log directory = /var/log/netdata
    # web files directory = /usr/share/netdata/web
    # cache directory = /var/cache/netdata
    # lib directory = /var/lib/netdata
    # home directory = /var/cache/netdata
    # plugins directory = "/usr/libexec/netdata/plugins.d" "/etc/netdata/custom-plugins.d"
    # memory mode = save
    # host access prefix = 
    # memory deduplication (ksm) = yes
    # TZ environment variable = :/etc/localtime
    # timezone = Etc/UTC
    # debug flags = 0x0000000000000000
    # error log = /var/log/netdata/error.log
    # errors flood protection period = 1200
    # errors to trigger flood protection = 200
    # OOM score = -998
    # process scheduling policy = idle
    # pthread stack size = 8388608
    # gap when lost iterations above = 1

[plugins]
    proc = yes
    diskspace = no
    cgroups = no
    tc = no
    idlejitter = no
    enable running new plugins = no
    charts.d = no
    fping = no
    node.d = no
    python.d = no
    apps = no
    # PATH environment variable = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
    # PYTHONPATH environment variable = 
    # check for new plugins every = 60

[web]
    # default port = 19999
    web files owner = netdata
    web files group = netdata
    allow streaming from = *
    # mode = static-threaded
    # listen backlog = 4096
    # bind to = *
    # disconnect idle clients after seconds = 60
    # timeout for first request = 60
    # respect do not track policy = no
    # x-frame-options response header = 
    # allow connections from = localhost *
    # allow dashboard from = localhost *
    # allow badges from = *
    # allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.*
    # enable gzip compression = yes
    # gzip compression strategy = default
    # gzip compression level = 3
    # web server threads = 6
    # web server max sockets = 524288
    # custom dashboard_info.js = 

1. Slave netdata node:

# netdata configuration
#
# You can download the latest version of this file, using:
#
#  wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
# or
#  curl -o /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
#
# You can uncomment and change any of the options below.
# The value shown in the commented settings, is the default value.
#

# global netdata configuration

[global]
        memory mode = none
        history = 924
        run as user = netdata
        access log = none
        debug log = none
        cleanup obsolete charts after seconds = 300
        delete obsolete charts files = yes
        # glibc malloc arena max for plugins = 1
        # glibc malloc arena max for netdata = 1
        # hostname = kong-112-n9pjn
        # update every = 1
        # config directory = /etc/netdata
        # log directory = /var/log/netdata
        # web files directory = /usr/share/netdata/web
        # cache directory = /var/cache/netdata
        # lib directory = /var/lib/netdata
        # home directory = /var/cache/netdata

[thiagoftsm]

Hi @jeremyjpj0916 ,

I apologize due the delay to answer you.
In normal situation we always expect that old Netdata can continue communicating with new Netdata, the opposite I need to confess for you that I never tried. I will test during this week the stream between 1.10 and 1.18.1 to understand the problems that you reported here. Did you also tried to connect a 1.10 with an 1.18.1?

Best regards!

[jeremyjpj0916]

I am not running old netdata and new netdata in an environment together regarding this github issue. Master and slave are on 1.18.1 in dev/stage where the problem is. I just gave my older configs of 1.10.0 as a reference point that we have in prod that faces no issues(and is building from src not prebuild binary).

Ill be testing building from src on 1.18.1 to see if it fixes the issue and report back too in a week or so. I expect it will based on what I have read in other git issues.

[stelfrag]

Ill be testing building from src on 1.18.1 to see if it fixes the issue and report back too in a week or so. I expect it will based on what I have read in other git issues.

@jeremyjpj0916 thank you for this. If building from source fixes things, then we will know for sure and
focus further on the prebuild (trying to recreate this in anycase)

The part of the log that you provided

2019-11-17 07:06:43: netdata INFO : STATSD : STREAM kong-546-r5j8t [send]: sending metrics...
2019-11-18 01:26:58: netdata ERROR : STREAM_SENDER[kong-546-r5j8t] : STREAM kong-546-r5j8t [send to tcp:netdata:19999]: too many data pending - buffer is 1048618 bytes long, 1048618 unsent - we have sent 13905909532 bytes in total, 13905909532 on this connection. Closing connection to flush the data.

indicates that the connection with the master was established but then (unclear so far why) stalled.

[jeremyjpj0916]

@stelfrag sure thing, will get back here when I get results. Your conclusion is similar to my thoughts, seems like a tcp stream that constantly breaks and reconnects and breaks again. Odd thing is when I first start up the netdata master/slaves they both log stable for a little while before things start behaving like that. Slave will connect and stay streaming for a bit before the flaky connection logs start showing up.

[thiagoftsm]

H @jeremyjpj0916 ,

It is possible we need to adjust your dbengine variables to keep your connection stable.

best regards!

[jeremyjpj0916]

@thiagoftsm , in the slave on 1.18.1 I run memory mode = none which should hold the same value as it did in 1.10.0 right? Where as a stream everything is in RAM real time as it streams all data quickly to master.

So you are suggesting adjusting my Master nodes config. Is there a configuration option for master that would take it back to the behavior of the 1.10.0 default? If so happy to do so. Maybe it needs

memory mode = save declared ? Open to suggestions, maybe there is value using dbengine now but idk. I use netdata as an exposed API for Prometheus to scrape APM/StatsD metrics. As well as browse its pretty chart dashboard when I think server errrors are going on too :)

[thiagoftsm]

@thiagoftsm , in the slave on 1.18.1 I run memory mode = none which should hold the same value as it did in 1.10.0 right? Where as a stream everything is in RAM real time as it streams all data quickly to master.

You are completely correct here. You could also set a memory mode on slave to have a kind of backup, but it is not necessary.

So you are suggesting adjusting my Master nodes config. Is there a configuration option for master that would take it back to the behavior of the 1.10.0 default? If so happy to do so. Maybe it needs

Case you set the memory mode = save, this would return to the configuration of 1.10.0.

memory mode = save declared ? Open to suggestions, maybe there is value using dbengine now but idk. I use netdata as an exposed API for Prometheus to scrape APM/StatsD metrics. As well as browse its pretty chart dashboard when I think server errrors are going on too :)

Yes, it is this mode. I quoted and began to comment, so I only see this now. :)

Please, do these change and give us a feed back, for we move in front. The dbengine has variables to adjust, but we cannot discard the possibility to adjust the Operate system too as you can see in this link https://docs.netdata.cloud/database/engine/#file-descriptor-requirements . Personally I prefer the dbengine, because it is capable to keep more data and with this we can have a better vision of the host.

[jeremyjpj0916]

@thiagoftsm makes sense, will try the older memory mode as a quick fix and will report back before i dig into how to build from source. Appreciate all the feedback and collaboration here 💯 .

[jeremyjpj0916]

Gave a test of memory mode = save on Master node 1.18.1 , did not improve the stream view of the slave node from master:

Still broken chunky charting indicating connections issues. Next step will be to build from src when I have time, likely over the weekend.

[thiagoftsm]

All right @jeremyjpj0916 !

I got the previous netdata that you were running and today I will test the stream between it and the newest version to confirm that we do not have broken compatibility. I will write the results here late today.

[jeremyjpj0916]

How does testing a stream connection between older and newer netdata versions relate to the bug raised in the original comment?

[thiagoftsm]

Between 1.10 and 1.18 we had changes in the stream, for example, now we can have a TLS channel between slave and master, I decided to have this environment to be sure that the changes that was brought in this period of time did not break any communication between the versions.

I will continue running during few hours, but in the first minutes of the tests I did not have any gap in the charts. In my tests I compiled both master(1.18.1) and slave(1.10.0) using the source.

[jeremyjpj0916]

Right, but my problem is not streaming between different versions of netdata. I only mention 1.10.0 as a baseline for what used to work for me(and the fact I used to build 1.10.0 from source and on 1.18.1 I now do not is a critical piece of the puzzle in my mind right now). I maintain version parity between slave/master nodes. Do a stream test between 1.18.1 master/slave using the pre-built run package if your looking for apples to apples. and let it run for a full 1-2 days and you should probably see the issue present itself, send some statsD data too if you want apples to apples for my use case 😀 .

[thiagoftsm]

You are completely correct!

I was only confirming that the stream was ok to be sure that we did not have more problems.

Returning for the main problem, I also tested the prebuild on Slackware current and I did not have problem with both during 2 hours, but users already reported for us problems like yours on Red Hat, so probably there are something wrong with our compilation process.

[amoss]

There is not enough information in the error.log to track what is happening inside rrdpush_sender_thread. It is important to discover why the connection is intermittently failing and causing the buffer flush. If the system is being recompiled from source can we enable the debug log with D_STREAM debug and see a log from a period of time when the connection fails to try to understand what is going on. Please be aware that debug should be switched back off after capturing the problem as it consumes a lot of disk-space.

[stelfrag]

I could not reproduce on a (as close as) possible configuration using a recent prebuild (v.1.18.1-155-g5b83a5a1), also alpine:3.10 --- the original docker.company.com/repo/alpine:3.10 won't work

No change on the default config files, except from the stream.conf-- to get it to work on my test machines

FROM alpine:3.10

USER root

ENV NETDATA_VERSION=1.18.1-155-g5b83a5a1

# install required packages
RUN apk add alpine-sdk bash curl zlib-dev util-linux-dev libmnl-dev gcc make git autoconf automake pkgconfig python logrotate

ADD . /gitsources
#Fix /opt/netdata path and logrotate
RUN chmod -R 777 /gitsources \
    && touch /etc/logrotate.d/netdata \
    && chmod 777 /etc/logrotate.d/netdata \
    && mkdir /opt/netdata \
    && chmod -R 777 /opt/netdata \
    && chmod 777 /gitsources/kickstart-static64.sh

RUN bash /gitsources/kickstart-static64.sh --dont-wait --dont-start-it --no-updates --local-files /gitsources/netdata-v$NETDATA_VERSION.gz.run /gitsources/sha256sums.txt

COPY ./system/netdata.conf /opt/netdata/etc/netdata/netdata.conf
COPY ./conf.d/stream.conf /opt/netdata/etc/netdata/stream.conf
COPY ./conf.d/python.d.conf /opt/netdata/etc/netdata/python.d.conf
COPY ./conf.d/python.d/nginx.conf /opt/netdata/etc/netdata/python.d/nginx.conf
RUN touch /opt/netdata/etc/netdata/.opt-out-from-anonymous-statistics

RUN chmod -R 777 /opt/netdata

WORKDIR /
ENV NETDATA_PORT 19999
EXPOSE 19999 19999/udp 8125 8125/udp

CMD /opt/netdata/bin/netdata -D -p 19999

[jeremyjpj0916]

I am going to try to build from source today to see if that helps. I believe our underlying VM's that make up an OpenShift cluster do run RHEL if the does make a difference, I know someone mentioned it earlier but our containers run alpine 3.10 or debian buster.

To give a view into a grafana chart that is produced from netdata slave -> master -> prom scraped -> grafana it looked like this:

Looks mostly okay from about 12-20 hrs then dropped off like a rock.

Will report back on if building frm src helps when confirmed.

[jeremyjpj0916]

Aight folks I am now building my master/slaves from source with this docker file format essentially:

FROM docker.company.com/kongadmin/debian:buster

USER root

ENV NETDATA_VERSION=1.18.1

# install required packages
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y zlib1g-dev uuid-dev libuv1-dev liblz4-dev libjudy-dev libssl-dev libmnl-dev gcc make vim htop bash git autoconf autoconf-archive autogen automake pkg-config curl python

ADD . /gitsources
#Fix /opt/netdata path and logrotate
RUN chmod -R 777 /gitsources \
    && touch /etc/logrotate.d/netdata \
    && chmod 777 /etc/logrotate.d/netdata \
    && mkdir /opt/netdata \
    && chmod -R 777 /opt/netdata \
    && chmod 777 /gitsources/kickstart-static64.sh \
    && chmod 777 /gitsources/netdata-v$NETDATA_VERSION.gz.run

#Install from pre-built binary
#RUN bash /gitsources/kickstart-static64.sh --dont-wait --dont-start-it --no-updates --local-files /gitsources/netdata-v$NETDATA_VERSION.gz.run /gitsources/sha256sums.txt

#Install from source
# download it - the directory 'netdata' will be created
RUN git clone -b v$NETDATA_VERSION https://github.com/netdata/netdata.git --depth=100 \
    && chmod -R 777 netdata \
    && cd netdata \
    && rm netdata-installer.sh \
    && mv /gitsources/netdata-installer.sh /netdata/netdata-installer.sh \
    && chmod 777 /netdata/netdata-installer.sh \
    && ./netdata-installer.sh --install /opt --dont-wait --dont-start-it --disable-go

WORKDIR /

COPY ./system/netdata.conf /opt/netdata/etc/netdata/netdata.conf
COPY ./conf.d/stream.conf /opt/netdata/etc/netdata/stream.conf
COPY ./conf.d/python.d.conf /opt/netdata/etc/netdata/python.d.conf
COPY ./conf.d/python.d/nginx.conf /opt/netdata/etc/netdata/python.d/nginx.conf
RUN touch /opt/netdata/etc/netdata/.opt-out-from-anonymous-statistics

RUN chmod -R 777 /opt/netdata

ENV NETDATA_PORT 19999
EXPOSE 19999 19999/udp 8125 8125/udp

#Static prebuild run
#CMD /opt/netdata/bin/netdata -D -p 19999
#Run from source installation
CMD /opt/netdata/usr/sbin/netdata -D -p 19999

Will report back shortly but in sandbox I did see this alert if it means anything too:

And did see this one instance in dev of logs too as some preliminary data for when it may have messed up once:

  | 2019-11-23 04:42:43: python.d INFO: plugin[main] : nginx[localhost] : check success
  | 2019-11-23 04:42:47: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: connecting...
  | 2019-11-23 04:42:47: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-23 04:42:47: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-23 04:42:47: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-23 04:42:48: netdata INFO  : STATSD : STREAM kong-93-wc4nt [send]: sending metrics...
  | 2019-11-23 05:33:13: netdata ERROR : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 13170761 bytes on this connection. (errno 113, No route to host)
  | 2019-11-23 05:33:13: netdata ERROR : STATSD : STREAM kong-93-wc4nt [send]: not ready - discarding collected metrics.
  | 2019-11-23 05:33:13: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: connecting...
  | 2019-11-23 05:33:13: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: initializing communication...
  | 2019-11-23 05:33:13: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2019-11-23 05:33:13: netdata INFO  : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send to tcp:netdata:19999]: established communication - ready to send metrics...
  | 2019-11-23 05:33:13: netdata ERROR : STREAM_SENDER[kong-93-wc4nt] : STREAM kong-93-wc4nt [send]: discarding 540 bytes of metrics already in the buffer.
  | 2019-11-23 05:33:14: netdata INFO  : STATSD : STREAM kong-93-wc4nt [send]: sending metrics...

If you have some debug in mind to add to the above docker file to help better capture the problem, maybe around D_STREAM thing one of yall were posting about earlier I will add it and report those logs if I start seeing the issue pop back up in stage. Just got it deployed out in dev/stage so I will wait out to see if we get similar results to the pre-built.

Edit - Stage ENV is also showing this mismatch alarms too.

[thiagoftsm]

Hi @jeremyjpj0916,

I do not have a great experience with docker image, so I will ask our devops team. @knatsakis and @ncmans, please, can you verify the docker file from the previous comment?

[jeremyjpj0916]

Following up here this weekend, I have built from src in now active running on my pods from friday for 24-48+ hours with no issues its looking like. So indeed something with running from the pre-built package vs not seems to be the culprit. Too bad using pre-built means no adding the debug aforementioned here. I will be happy to help run debug in a pre-built run file if you end up adding the tag and drop me the run file to drop in as a test to give you the logging you need.

[thiagoftsm]

Hi @jeremyjpj0916 ,

Thank you for your report, we will work to fix the problem reported here.

Best regards!

[ncmans]

I cannot reproduce the issue. Tested netdata-v1.18.1.gz.run in a master+slave setup with the configuration provided by the OP on both Ubuntu Bionic and Alpine 3.10.3.

@jeremyjpj0916 Let's try to isolate the issue a little. Have you tried running the master+slave setup without any intermediate network (e.g. two docker containers on localhost) with the the compiled binaries (netdata-v1.18.1.gz.run)? This is to make sure your network isn't the issue.

[jeremyjpj0916]

Yeah OS does not make a difference, I tried both Alpine and Debian and the problem persisted when running the containers using the compiled binaries. Only running from src seems to resolve the issue similarly reported in #6307 . Also more so evidenced because I also ran 1.10.0 from the get go from src and never saw this issues and only thought it would be more optimal to revisit that and use pre-built when looking into 1.18.1 .

If it was the network then my large enterprises Kubernetes cluster is misconfigured but we have over 600+ app teams leveraging it in production so I really can't go down a goose chase on that one. I would however be happy to spend some time running a .run file in our lower logical environments for yall if you want to enable any debug flags or extra print statements in the master/slave stream logic that may help you get an RCA here to both mine and the other issue that seems to hit a similar issue with what was the same fix in my case.

[amoss]

It looks as if we will ultimately end up closing this bug report as non-reproducible. Before we do that, it is very gracious of Jeremy to offer to run a container with debugging enabled to try to find more information about what causes this problem. We should try to take full advantage of this offer as this does look very similar to #6307. Let's have a conversation internally (after today's stand-up?) to work out how we can maximize the benefit of this test.

• Before we run any build with debugging we need to make sure that there is enough information in the debug to show us exactly what is happening in this case.
• We need to check that we have attempted to replicate the bug in the same conditions -> how are we simulating high-latency on the slave-client link while attempting to replicate?

[ilyam8]

how are we simulating high-latency on the slave-client link while attempting to replicate?

tc-netem

[amoss]

What is the tc command-line being used for replication? We can try to work out if it is a good model of what is happening on the real system.

[ilyam8]

See the link

http://man7.org/linux/man-pages/man8/tc-netem.8.html

[stelfrag]

Looks like we are running into a dead end here. I would, in summary, be inclined to say that we are dealing with some sort of network problem that stalls the connection.

I will close this, unfortunately, as "cannot reproduce" in a couple of days, unless someone has an idea on how to further investigate.

[jeremyjpj0916]

I would conclude that's highly unlikely as I can switch from pre-built binary to source and its night and day with the streaming problem occurring or not.

What I can't wrap my head around as a newer problem is even after fixing the "stream" issue by building from source, now at this stage my netdata nodes are hitting OOM and crashing after a longer window(2-3 days). I just upgraded to 1.19.0 to see if that helped any but it did not. I am looking to move statsD slave data traffic away from netdata slave->master and using statsD exporter sidecar w netdata on the pod, will be watching to see if stopping the statsD streaming from slave-> master and just collecting APM data w netdata fixes these OOM crashes I see now(since i run netdata in an openshift cluster the pod spins back up with a new instance right after it OOM's and keeps trucking).

Likely may end up rolling back to 1.10.0 where I used to not face issues.

[amoss]

Reassigning this to myself as it relates to a bug that I’m looking at in the streaming code. I will take another look at replicating this next week.

[jeremyjpj0916]

glad something might come of this yet. I am sticking to a really old version of netdata where I don't have this problem and I build from source for now. I don't use netdata for much fancy stuff so am happy to sit on ancient version 😆 .

[amoss]

Hi @jeremyjpj0916, we made some major changes to the streaming code that may affect the issue that you were seeing. Could you try a new version and let us know if you still see the problem?

[jeremyjpj0916]

I will try to pull down a new build over the next few weeks and deploy it out in dev/stage and see how it goes. Will reach back out here with results @amoss .

[MaikStohn]

I want to report that I had similar issues with one of my (multiple) streaming nodes, which was always loosing connection, resulting in gaps and alerts.

I tried increasing buffers, changing network connections, always used latest version of netdata... but nothing helped.

Finally, last week I upgraded my stream receiver server (now much faster CPU + HDD) and since then I never had a gap or alert for the problematic streaming node.

So I was looking at the wrong side all the time. The stream receiver was just not powerful enough.

Maybe this issue is also related: #6679

[amoss]

We are seeing similar problems on other issues (for example https://github.com/netdata/netdata/issues/9821) where we are trying to track down if the system is being starved or there is something in the code that is not servicing the path from network to disk quickly enough.

The underlying problem is that our current streaming code is best-effort only. So if points are missed, or the connection drops because it is not being serviced then data is lost. We are in the process of replacing it with code that replicates the data from the sender to the receiver (and backtracks to request missing data). This will solve some problems immediately, and it will make the underlying causes of other problems more visible so that we can tackle them.

The first attempt at replication was abandoned because of integration issues with the existing code. The second attempt is currently one test away from passing the checks that I need so it should be merged real-soon-now (TM).

[jeremyjpj0916]

@amoss Finally getting a chance to test this, going to re-iterate what my DOCKER file kinda looks like atm for prepping these images between the slave stream and master node:

Note: company is internal for us, masking our TLD.

FROM docker.repo1.company.com/kongadmin/alpine:3.11

USER root

ENV NETDATA_VERSION=1.24.0

# install required packages
RUN apk add --update --no-cache alpine-sdk bash curl libuv-dev zlib-dev util-linux-dev libmnl-dev gcc make git autoconf automake pkgconfig python logrotate

ADD . /gitsources
#Fix /opt/netdata path and logrotate
RUN chmod -R 777 /gitsources \
    && touch /etc/logrotate.d/netdata \
    && chmod 777 /etc/logrotate.d/netdata \
    && mkdir /opt/netdata \
    && chmod -R 777 /opt/netdata \
    && chmod 777 /gitsources/kickstart-static64.sh \
    && chmod 777 /gitsources/netdata-v$NETDATA_VERSION.gz.run

#From prebuilt binary
RUN bash /gitsources/kickstart-static64.sh --dont-wait --dont-start-it --disable-telemetry --no-updates --local-files /gitsources/netdata-v$NETDATA_VERSION.gz.run /gitsources/sha256sums.txt

COPY ./system/netdata.conf /opt/netdata/etc/netdata/netdata.conf
COPY ./conf.d/stream.conf /opt/netdata/etc/netdata/stream.conf
#RUN touch /opt/netdata/etc/netdata/.opt-out-from-anonymous-statistics No need, new kickstart has a disable telemetry arg.

RUN chmod -R 777 /opt/netdata

WORKDIR /
ENV NETDATA_PORT 19999
EXPOSE 19999 19999/udp 8125 8125/udp 9125 9125/udp 9102

CMD /opt/netdata/bin/netdata -D -p 19999

Will be deploying to dev tomorrow and hopefully stage shortly after, I remember I tried alpine for awhile last time and it seemed to have issues so I went back to using debian w netdata as base image. Trying alpine again now because I like it for its small size and because alpine is what we use for our other containerized apps(besides netdata so far).

Will report back with my findings. I essentially just need netdata to help stream around statsD metrics for prom to scrape for me from the master node. Some of its monitoring of the node and UI dashboard is just icing on the cake though so glad to have it. Hopefully one day this gets fixed because now I get containers crashing all the time and restarting and because of that the metrics straight up "blip" sometimes and give me crazy data through grafana for what prom scraped off that netdata master node heh.

[jeremyjpj0916]

So far put it on dev, no issues detected just yet. but dev is ghost town, wait till I get it on stage where there is lots of statsd data flowing and the streaming probably goes much harder. btw like the newer looking theme on 1.24.0 compared to what it used to look like, more modern web look 👍 .

[jeremyjpj0916]

Put it in stage now, copied the startup of one of the master nodes recieving the streams. Output looks like this just as an FYI:

2020-09-23 17:08:12: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/opt/netdata/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
  | 2020-09-23 17:08:12: netdata INFO  : MAIN : Found 0 legacy dbengines, setting multidb diskspace to 256MB
  | 2020-09-23 17:08:12: netdata INFO  : MAIN : Created file '/opt/netdata/var/lib/netdata/dbengine_multihost_size' to store the computed value
  | 2020-09-23 17:08:13: netdata INFO  : MAIN : SIGNAL: Enabling reaper
  | 2020-09-23 17:08:13: netdata INFO  : MAIN : process tracking enabled.
  | 2020-09-23 17:08:13: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
  | 2020-09-23 17:08:13: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value -998
  | 2020-09-23 17:08:13: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
  | 2020-09-23 17:08:13: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
  | 2020-09-23 17:08:13: netdata ERROR : MAIN : Cannot chown directory '/opt/netdata/var/lib/netdata/lock' to 1000:1000 (errno 1, Operation not permitted)
  | 2020-09-23 17:08:13: netdata ERROR : MAIN : Cannot switch to user's netdata group (gid: 1000). (errno 1, Operation not permitted)
  | 2020-09-23 17:08:13: netdata ERROR : MAIN : Cannot become user 'netdata'. Continuing as we are.
  | 2020-09-23 17:08:13: netdata INFO  : MAIN : netdata started on pid 1.
  | 2020-09-23 17:08:13: netdata INFO  : MAIN : Initializing spawn client.
  | 2020-09-23 17:08:13: netdata INFO  : MAIN : Executing /opt/netdata/usr/libexec/netdata/plugins.d/system-info.sh
  | Spawn server is up.
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_NAME=Alpine Linux
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID=alpine
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=3.11.6
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_DETECTION=/etc/os-release
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_HOST_OS_NAME=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_HOST_OS_ID=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_HOST_OS_ID_LIKE=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION_ID=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_HOST_OS_DETECTION=unknown
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-1062.18.1.el7.x86_64
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=none
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=docker
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=dockerenv
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=56
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_FREQ=3300000
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_DETECTION=nproc procfs sysfs
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_RAM=270177816576
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=30726156238848
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Configuring locking mechanism for global GUID map
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Host 'netdata-149-nj4ls' (at registry as 'netdata-149-nj4ls') with guid '61991730-fdbf-11ea-90e3-0a580a8205a9' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.24.0', update every 1, memory mode save, history entries 924, streaming disabled (to '' with api key ''), health disabled, cache_dir '/opt/netdata/var/cache/netdata', varlib_dir '/opt/netdata/var/lib/netdata', health_log '/opt/netdata/var/lib/netdata/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Found 0 files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Data files not found, creating in path "/opt/netdata/var/cache/netdata/dbengine".
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Creating new data and journal files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Created data file "/opt/netdata/var/cache/netdata/dbengine/datafile-1-0000000001.ndf".
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Created journal file "/opt/netdata/var/cache/netdata/dbengine/journalfile-1-0000000001.njf".
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Found 2 files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Scanning file "/opt/netdata/var/cache/netdata/dbengine/datafile-1-0000000001.ndf"
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Scanning file "/opt/netdata/var/cache/netdata/dbengine/journalfile-1-0000000001.njf"
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Metadata log files not found, creating in path "/opt/netdata/var/cache/netdata/dbengine".
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Creating new metadata log file in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Created metadata log file "/opt/netdata/var/cache/netdata/dbengine/metadatalog-00000-00001.mlf".
  | 2020-09-23 17:08:20: netdata INFO  : MAIN : Unable to load '/opt/netdata/var/lib/netdata/cloud.d/claimed_id', setting state to AGENT_UNCLAIMED
  | 2020-09-23 17:08:23: netdata INFO  : PLUGIN[proc] : thread created with task id 1144
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Main : thread created with task id 1146
  | 2020-09-23 17:08:23: netdata INFO  : STATSD : thread created with task id 1145
  | 2020-09-23 17:08:23: netdata INFO  : BACKENDS : thread created with task id 1147
  | 2020-09-23 17:08:23: netdata INFO  : PLUGIN[proc] : set name of thread 1144 to PLUGIN[proc]
  | 2020-09-23 17:08:23: netdata INFO  : EXPORTING : thread created with task id 1148
  | 2020-09-23 17:08:23: netdata INFO  : EXPORTING : set name of thread 1148 to EXPORTING
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Main : set name of thread 1146 to ACLK_Main
  | 2020-09-23 17:08:23: netdata INFO  : STATSD : set name of thread 1145 to STATSD
  | 2020-09-23 17:08:23: netdata INFO  : EXPORTING : CONFIG: cannot load user exporting config '/opt/netdata/etc/netdata/exporting.conf'. Will try the stock version.
  | 2020-09-23 17:08:23: netdata INFO  : BACKENDS : set name of thread 1147 to BACKENDS
  | 2020-09-23 17:08:23: netdata INFO  : MAIN : Initializing command server.
  | 2020-09-23 17:08:23: netdata INFO  : BACKENDS : cleaning up...
  | 2020-09-23 17:08:23: netdata INFO  : BACKENDS : thread with task id 1147 finished
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Main : Waiting for netdata to be ready
  | 2020-09-23 17:08:23: netdata INFO  : PLUGINSD : thread created with task id 1150
  | 2020-09-23 17:08:23: netdata INFO  : HEALTH : thread created with task id 1151
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : thread created with task id 1149
  | 2020-09-23 17:08:23: netdata INFO  : PLUGINSD : set name of thread 1150 to PLUGINSD
  | 2020-09-23 17:08:23: netdata INFO  : HEALTH : set name of thread 1151 to HEALTH
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : set name of thread 1149 to WEB_SERVER[stat
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : To use encryption it is necessary to set "ssl certificate" and "ssl key" in [web] !
  |  
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : starting worker 2
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : starting worker 3
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static2] : thread created with task id 1153
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static2] : set name of thread 1153 to WEB_SERVER[stat
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static3] : thread created with task id 1154
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : starting worker 4
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static3] : set name of thread 1154 to WEB_SERVER[stat
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static4] : thread created with task id 1155
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : starting worker 5
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static4] : set name of thread 1155 to WEB_SERVER[stat
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static5] : thread created with task id 1156
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : starting worker 6
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static5] : set name of thread 1156 to WEB_SERVER[stat
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static6] : thread created with task id 1157
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static6] : set name of thread 1157 to WEB_SERVER[stat
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 17:08:23: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 17:08:23: netdata INFO  : EXPORTING : No connector instances to activate
  | 2020-09-23 17:08:23: netdata INFO  : EXPORTING : EXPORTING: no exporting connectors configured
  | 2020-09-23 17:08:23: netdata INFO  : EXPORTING : cleaning up...
  | 2020-09-23 17:08:23: netdata INFO  : EXPORTING : thread with task id 1148 finished
  | 2020-09-23 17:08:23: netdata INFO  : STATSD : cleaning up...
  | 2020-09-23 17:08:23: netdata INFO  : STATSD : STATSD: closing sockets...
  | 2020-09-23 17:08:23: netdata INFO  : STATSD : STATSD: cleanup completed.
  | 2020-09-23 17:08:23: netdata INFO  : STATSD : thread with task id 1145 finished
  | 2020-09-23 17:08:23: netdata INFO  : MAIN : netdata initialization completed. Enjoy real-time performance monitoring!
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Main : Waiting for Cloud to be enabled
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Main : Waiting for netdata to be claimed
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Stats : thread created with task id 1158
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Stats : set name of thread 1158 to ACLK_Stats
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_status/main.db.
  | 2020-09-23 17:08:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_status/online.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/guest_nice.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/guest.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/steal.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/softirq.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/irq.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/user.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/system.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/nice.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/iowait.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/idle.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_per_second/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_per_second/added.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_per_second/dispatched.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ctxt/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ctxt/switches.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_write_q/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_write_q/added.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_write_q/consumed.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.forks/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.forks/started.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_read_q/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_read_q/added.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_read_q/consumed.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static3] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread3_cpu/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static3] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread3_cpu/user.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static3] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread3_cpu/system.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static4] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread4_cpu/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static4] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread4_cpu/user.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static4] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread4_cpu/system.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static2] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread2_cpu/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static2] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread2_cpu/user.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static2] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread2_cpu/system.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.processes/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.processes/running.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.processes/blocked.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static1] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread1_cpu/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static1] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread1_cpu/user.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static1] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread1_cpu/system.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static5] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread5_cpu/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static5] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread5_cpu/user.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static5] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread5_cpu/system.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static6] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread6_cpu/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static6] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread6_cpu/user.db.
  | 2020-09-23 17:08:24: netdata INFO  : WEB_SERVER[static6] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread6_cpu/system.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.uptime/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.uptime/uptime.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Using now_boottime_usec() for uptime (dt is 1 ms)
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_cloud_req/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_cloud_req/received.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_cloud_req/malformed.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/load1.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/load5.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/load15.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_threads/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_threads/Query_0.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_threads/Query_1.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.active_processes/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.active_processes/active.db.
  | 2020-09-23 17:08:24: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/cpu' (errno 2, No such file or directory)
  | 2020-09-23 17:08:24: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/cpu.
  | 2020-09-23 17:08:24: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/memory' (errno 2, No such file or directory)
  | 2020-09-23 17:08:24: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/memory.
  | 2020-09-23 17:08:24: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/io' (errno 2, No such file or directory)
  | 2020-09-23 17:08:24: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/io.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/avg.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/max.db.
  | 2020-09-23 17:08:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/total.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.pgpgio/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.pgpgio/in.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.pgpgio/out.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.pgfaults/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.pgfaults/minor.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.pgfaults/major.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/free.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/used.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/cached.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/buffers.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.available/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.available/MemAvailable.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.committed/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.committed/Committed_AS.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/Dirty.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/Writeback.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/FuseWriteback.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/NfsWriteback.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/Bounce.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/Slab.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/KernelStack.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/PageTables.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/VmallocUsed.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.slab/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.slab/reclaimable.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.slab/unreclaimable.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/numa_hit.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/numa_miss.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/local_node.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/numa_foreign.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/interleave_hit.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/other_node.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/numa_hit.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/numa_miss.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/local_node.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/numa_foreign.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/interleave_hit.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/other_node.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/net.eth0/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/net.eth0/received.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/net.eth0/sent.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/net_packets.eth0/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/net_packets.eth0/received.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/net_packets.eth0/sent.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/net_packets.eth0/multicast.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_sockets/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_sockets/used.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_tcp_sockets/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_tcp_sockets/alloc.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_tcp_sockets/orphan.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_tcp_sockets/inuse.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_tcp_sockets/timewait.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_tcp_mem/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_tcp_mem/mem.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_udp_mem/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/ipv4.sockstat_udp_mem/mem.db.
  | 2020-09-23 17:08:24: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/net/sctp/snmp' (errno 2, No such file or directory)
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.softnet_stat/main.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.softnet_stat/processed.db.
  | 2020-09-23 17:08:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.softnet_stat/dropped.db.
  | 2020-09-23 17:08:24: netdata LOG FLOOD PROTECTION too many logs (201 logs in 4 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1196 seconds.

Let me know if anything seems to amiss there, nodes are up and running, time to see if they do that OOM crashing cycle I usually see them do a few 100 times a week.

[jeremyjpj0916]

So far no crashes but will need time to tell, looking at UI from master browsing the slaves from using the prebuild run file I see little windows of gaps in the charts I think that building from src for us doesn't cause again:

Will keep watching it though.

[jeremyjpj0916]

Watching the slave nodes stream I see some connection reset logs, these generally never occured when I used to build from src on older versions in the logs I saw:

2020-09-23 17:29:23: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
--
  | 2020-09-23 17:29:23: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 89707 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:23: netdata ERROR : STATSD : STREAM kong-662-47296 [send]: not ready - discarding collected metrics.
  | 2020-09-23 17:29:23: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 89707 bytes transmitted.
  | 2020-09-23 17:29:23: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:23: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:23: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:23: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:23: netdata INFO  : PLUGINSD[python.d] : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:24: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 17:29:24: netdata ERROR : STATSD : STREAM kong-662-47296 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 17:29:24: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 93743 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:24: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 93743 bytes transmitted.
  | 2020-09-23 17:29:24: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:24: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:24: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:24: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:25: netdata INFO  : ACLK_Stats : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:25: netdata INFO  : PLUGIN[proc] : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:25: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 17:29:25: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 89090 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:25: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 89090 bytes transmitted.
  | 2020-09-23 17:29:25: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:25: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:25: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:25: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:26: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 17:29:26: netdata ERROR : STATSD : STREAM kong-662-47296 [send]: not ready - discarding collected metrics.
  | 2020-09-23 17:29:26: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 90768 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:26: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 90768 bytes transmitted.
  | 2020-09-23 17:29:26: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:26: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:26: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:26: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:27: netdata INFO  : ACLK_Stats : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:27: netdata INFO  : STATSD : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:27: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connection closed by far end. Restarting connection (errno 22, Invalid argument)
  | 2020-09-23 17:29:27: netdata ERROR : STATSD : STREAM kong-662-47296 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 17:29:27: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:27: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:27: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:27: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:28: netdata INFO  : ACLK_Stats : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:28: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 17:29:28: netdata ERROR : STATSD : STREAM kong-662-47296 [send]: not ready - discarding collected metrics.
  | 2020-09-23 17:29:28: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 88309 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:28: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 88309 bytes transmitted.
  | 2020-09-23 17:29:28: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:28: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:28: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:28: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:28: netdata INFO  : PLUGINSD[python.d] : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:29: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 17:29:29: netdata ERROR : STATSD : STREAM kong-662-47296 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 17:29:29: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 91609 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:29: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 91609 bytes transmitted.
  | 2020-09-23 17:29:29: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:29: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:29: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:29: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:30: netdata INFO  : STATSD : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:30: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 17:29:30: netdata ERROR : STATSD : STREAM kong-662-47296 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 17:29:30: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 90096 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:30: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 90096 bytes transmitted.
  | 2020-09-23 17:29:30: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:30: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:30: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:30: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:31: netdata INFO  : ACLK_Stats : STREAM kong-662-47296 [send]: sending metrics...
  | 2020-09-23 17:29:31: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 17:29:31: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 92651 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 17:29:31: netdata ERROR : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 92651 bytes transmitted.
  | 2020-09-23 17:29:31: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 17:29:31: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 17:29:31: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 17:29:31: netdata INFO  : STREAM_SENDER[kong-662-47296] : STREAM kong-662-47296 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 17:29:32: netdata LOG FLOOD PROTECTION too many logs (201 logs in 24 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1176 seconds.

Maybe that explains some of the little gaps in the stream data and its visual presentation. Why I get connection resets between the two nodes streaming I am not sure but I think it likely has something to do with the differences between build from src vs static run file or the versions of netdata.

The (errno 9, Bad file descriptor) seems to likely be playing a role there:

https://stackoverflow.com/questions/7686275/what-can-lead-to-ioerror-errno-9-bad-file-descriptor-during-os-system ??

[mfundul]

Can you install the latest stable (v1.25)? It is mostly a bugfix release.

[jeremyjpj0916]

@mfundul Yes I can, didn't realize it was out there. if it has any stream adjustments or bug fixes there then maybe that will help. trying now will report back!

[jeremyjpj0916]

Early reporting, the actual charts and visuals don't have the glitchy gaps and such in them. The logs from the streaming node however do have some error logs being printed occasionally.

Example slave stream node log:

2020-09-23 21:44:58: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/opt/netdata/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : Found 0 legacy dbengines, setting multidb diskspace to 256MB
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : Created file '/opt/netdata/var/lib/netdata/dbengine_multihost_size' to store the computed value
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : SIGNAL: Enabling reaper
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : process tracking enabled.
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value -998
  | 2020-09-23 21:44:58: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
  | 2020-09-23 21:44:58: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
  | 2020-09-23 21:44:58: netdata ERROR : MAIN : Cannot chown directory '/opt/netdata/var/lib/netdata/lock' to 1000:1000 (errno 1, Operation not permitted)
  | 2020-09-23 21:44:58: netdata ERROR : MAIN : Cannot switch to user's netdata group (gid: 1000). (errno 1, Operation not permitted)
  | 2020-09-23 21:44:58: netdata ERROR : MAIN : Cannot become user 'netdata'. Continuing as we are.
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : netdata started on pid 1.
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : Initializing spawn client.
  | 2020-09-23 21:44:58: netdata INFO  : MAIN : Executing /opt/netdata/usr/libexec/netdata/plugins.d/system-info.sh
  | Spawn server is up.
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_NAME=Alpine Linux
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID=alpine
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=3.11.6
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_DETECTION=/etc/os-release
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_HOST_OS_NAME=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_HOST_OS_ID=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_HOST_OS_ID_LIKE=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION_ID=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_HOST_OS_DETECTION=unknown
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-1062.18.1.el7.x86_64
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=none
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=docker
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=dockerenv
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=56
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_FREQ=3300000
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_DETECTION=nproc procfs sysfs
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_RAM=270177816576
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=30726156238848
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Configuring locking mechanism for global GUID map
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Host 'kong-462-pcfc4' (at registry as 'kong-462-pcfc4') with guid '08078770-fde6-11ea-873f-0a580a8307fd' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.25.0', update every 1, memory mode none, history entries 924, streaming enabled (to 'tcp:netdata:19999' with api key '11111111-2222-3333-4444-555555555555'), health disabled, cache_dir '/opt/netdata/var/cache/netdata', varlib_dir '/opt/netdata/var/lib/netdata', health_log '/opt/netdata/var/lib/netdata/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Found 0 files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Data files not found, creating in path "/opt/netdata/var/cache/netdata/dbengine".
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Creating new data and journal files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Created data file "/opt/netdata/var/cache/netdata/dbengine/datafile-1-0000000001.ndf".
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Created journal file "/opt/netdata/var/cache/netdata/dbengine/journalfile-1-0000000001.njf".
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Found 2 files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Scanning file "/opt/netdata/var/cache/netdata/dbengine/datafile-1-0000000001.ndf"
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Scanning file "/opt/netdata/var/cache/netdata/dbengine/journalfile-1-0000000001.njf"
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Metadata log files not found, creating in path "/opt/netdata/var/cache/netdata/dbengine".
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Creating new metadata log file in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Created metadata log file "/opt/netdata/var/cache/netdata/dbengine/metadatalog-00000-00001.mlf".
  | 2020-09-23 21:45:00: netdata INFO  : MAIN : Unable to load '/opt/netdata/var/lib/netdata/cloud.d/claimed_id', setting state to AGENT_UNCLAIMED
  | 2020-09-23 21:45:01: netdata INFO  : PLUGIN[proc] : thread created with task id 700
  | 2020-09-23 21:45:01: netdata INFO  : PLUGIN[proc] : set name of thread 700 to PLUGIN[proc]
  | 2020-09-23 21:45:01: netdata INFO  : STATSD : thread created with task id 701
  | 2020-09-23 21:45:01: netdata INFO  : ACLK_Main : thread created with task id 702
  | 2020-09-23 21:45:01: netdata INFO  : STATSD : set name of thread 701 to STATSD
  | 2020-09-23 21:45:01: netdata INFO  : BACKENDS : thread created with task id 703
  | 2020-09-23 21:45:01: netdata INFO  : PLUGINSD : thread created with task id 706
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : thread created with task id 705
  | 2020-09-23 21:45:01: netdata INFO  : HEALTH : thread created with task id 707
  | 2020-09-23 21:45:01: netdata INFO  : MAIN : Initializing command server.
  | 2020-09-23 21:45:01: netdata INFO  : BACKENDS : set name of thread 703 to BACKENDS
  | 2020-09-23 21:45:01: netdata INFO  : PLUGINSD : set name of thread 706 to PLUGINSD
  | 2020-09-23 21:45:01: netdata INFO  : EXPORTING : thread created with task id 704
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : set name of thread 705 to WEB_SERVER[stat
  | 2020-09-23 21:45:01: netdata INFO  : HEALTH : set name of thread 707 to HEALTH
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : To use encryption it is necessary to set "ssl certificate" and "ssl key" in [web] !
  |  
  | 2020-09-23 21:45:01: netdata INFO  : EXPORTING : set name of thread 704 to EXPORTING
  | 2020-09-23 21:45:01: netdata INFO  : ACLK_Main : set name of thread 702 to ACLK_Main
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : starting worker 2
  | 2020-09-23 21:45:01: netdata INFO  : ACLK_Main : Waiting for netdata to be ready
  | 2020-09-23 21:45:01: netdata INFO  : EXPORTING : CONFIG: cannot load user exporting config '/opt/netdata/etc/netdata/exporting.conf'. Will try the stock version.
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : starting worker 3
  | 2020-09-23 21:45:01: netdata INFO  : PLUGINSD[python.d] : thread created with task id 710
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static2] : thread created with task id 709
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : starting worker 4
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static2] : set name of thread 709 to WEB_SERVER[stat
  | 2020-09-23 21:45:01: netdata INFO  : PLUGINSD[python.d] : set name of thread 710 to PLUGINSD[python
  | 2020-09-23 21:45:01: netdata INFO  : BACKENDS : cleaning up...
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : starting worker 5
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static3] : thread created with task id 711
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static4] : thread created with task id 712
  | 2020-09-23 21:45:01: netdata INFO  : BACKENDS : thread with task id 703 finished
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static3] : set name of thread 711 to WEB_SERVER[stat
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : starting worker 6
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static4] : set name of thread 712 to WEB_SERVER[stat
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static5] : thread created with task id 713
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static6] : thread created with task id 714
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static5] : set name of thread 713 to WEB_SERVER[stat
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static6] : set name of thread 714 to WEB_SERVER[stat
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:45:01: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:45:01: netdata INFO  : EXPORTING : No connector instances to activate
  | 2020-09-23 21:45:01: netdata INFO  : EXPORTING : EXPORTING: no exporting connectors configured
  | 2020-09-23 21:45:01: netdata INFO  : EXPORTING : cleaning up...
  | 2020-09-23 21:45:01: netdata INFO  : EXPORTING : thread with task id 704 finished
  | 2020-09-23 21:45:01: netdata INFO  : MAIN : netdata initialization completed. Enjoy real-time performance monitoring!
  | 2020-09-23 21:45:01: netdata INFO  : STATSD_COLLECTOR[1] : thread created with task id 715
  | 2020-09-23 21:45:01: netdata INFO  : STATSD_COLLECTOR[1] : set name of thread 715 to STATSD_COLLECTO
  | 2020-09-23 21:45:01: netdata INFO  : STATSD_COLLECTOR[1] : STATSD collector thread started with taskid 715
  | 2020-09-23 21:45:01: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'udp:0.0.0.0:8125'
  | 2020-09-23 21:45:01: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'udp:[::]:8125'
  | 2020-09-23 21:45:01: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:8125'
  | 2020-09-23 21:45:01: netdata INFO  : STATSD_COLLECTOR[1] : POLLFD: LISTENER: listening on 'tcp:[::]:8125'
  | 2020-09-23 21:45:01: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 2, No such file or directory)
  | 2020-09-23 21:45:01: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : thread created with task id 716
  | 2020-09-23 21:45:01: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : set name of thread 716 to STREAM_SENDER[k
  | 2020-09-23 21:45:01: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send]: thread created (task id 716)
  | 2020-09-23 21:45:01: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:01: netdata INFO  : PLUGIN[proc] : Using now_boottime_usec() for uptime (dt is 1 ms)
  | 2020-09-23 21:45:01: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/cpu' (errno 2, No such file or directory)
  | 2020-09-23 21:45:01: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/cpu.
  | 2020-09-23 21:45:01: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/memory' (errno 2, No such file or directory)
  | 2020-09-23 21:45:01: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/memory.
  | 2020-09-23 21:45:01: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/io' (errno 2, No such file or directory)
  | 2020-09-23 21:45:01: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/io.
  | 2020-09-23 21:45:01: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:01: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:01: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/net/sctp/snmp' (errno 2, No such file or directory)
  | 2020-09-23 21:45:01: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:02: netdata INFO  : ACLK_Main : Waiting for Cloud to be enabled
  | 2020-09-23 21:45:02: netdata INFO  : ACLK_Main : Waiting for netdata to be claimed
  | 2020-09-23 21:45:02: netdata INFO  : ACLK_Stats : thread created with task id 717
  | 2020-09-23 21:45:02: netdata INFO  : ACLK_Stats : set name of thread 717 to ACLK_Stats
  | 2020-09-23 21:45:02: netdata INFO  : PLUGINSD[python.d] : connected to '/opt/netdata/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 718
  | 2020-09-23 21:45:02: netdata INFO  : ACLK_Stats : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : using python v2
  | 2020-09-23 21:45:03: python.d WARNING: plugin[main] : 'pythond-jobs-statuses.json' was not found
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [adaptec_raid] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [am2320] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [apache] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [beanstalk] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [bind_rndc] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [boinc] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [ceph] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [chrony] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [couchdb] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [dns_query_time] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [dnsdist] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [dockerd] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [dovecot] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [elasticsearch] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [energid] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [example] built 1 job(s) configs
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [exim] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [fail2ban] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [freeradius] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [gearman] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [go_expvar] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [haproxy] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [hddtemp] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [hpssa] is disabled by default, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [httpcheck] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [icecast] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [ipfs] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [isc_dhcpd] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [litespeed] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [logind] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [megacli] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [memcached] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [mongodb] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [monit] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [mysql] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [nginx] built 1 job(s) configs
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [nginx_plus] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [nsd] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [ntpd] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [nvidia_smi] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [openldap] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [oracledb] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [ovpn_status_log] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [phpfpm] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [portcheck] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [postfix] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [postgres] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [powerdns] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [proxysql] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [puppet] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [rabbitmq] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [redis] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [rethinkdbs] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [retroshare] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [riakkv] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [samba] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [sensors] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [smartd_log] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [spigotmc] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [springboot] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [squid] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [tomcat] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [tor] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [traefik] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [uwsgi] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [varnish] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [w1sensor] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : [web_log] is disabled in the configuration file, skipping it
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : example[example] : check success
  | 2020-09-23 21:45:03: python.d INFO: plugin[main] : nginx[localhost] : check success
  | 2020-09-23 21:45:45: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connection closed by far end. Restarting connection (errno 22, Invalid argument)
  | 2020-09-23 21:45:45: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:45: netdata ERROR : PLUGIN[proc] : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 2, No such file or directory)
  | 2020-09-23 21:45:45: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:45: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:45: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:45: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:46: netdata INFO  : ACLK_Stats : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:46: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:46: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:46: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 11913 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:46: netdata ERROR : PLUGIN[proc] : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics.
  | 2020-09-23 21:45:46: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 11913 bytes transmitted.
  | 2020-09-23 21:45:46: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:46: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:46: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:46: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:46: netdata INFO  : PLUGINSD[python.d] : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:47: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:47: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 26926 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:47: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:47: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 26926 bytes transmitted.
  | 2020-09-23 21:45:47: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:47: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:47: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:47: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:48: netdata INFO  : STATSD : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:48: netdata INFO  : ACLK_Stats : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:48: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:48: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 14754 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:48: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:48: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 14754 bytes transmitted.
  | 2020-09-23 21:45:48: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:48: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:48: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:48: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:49: netdata INFO  : STATSD : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:49: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:49: netdata ERROR : PLUGIN[proc] : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:49: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 14094 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:49: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 14094 bytes transmitted.
  | 2020-09-23 21:45:49: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:49: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:49: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:49: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:49: netdata INFO  : PLUGINSD[python.d] : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:50: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:50: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 20967 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:50: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:50: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 20967 bytes transmitted.
  | 2020-09-23 21:45:50: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:50: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:50: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:50: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:51: netdata INFO  : STATSD : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:51: netdata INFO  : ACLK_Stats : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:51: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:51: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 18948 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:51: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:51: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 18948 bytes transmitted.
  | 2020-09-23 21:45:51: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:51: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:51: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:51: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:51: netdata INFO  : PLUGINSD[python.d] : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:52: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connection closed by far end. Restarting connection (errno 22, Invalid argument)
  | 2020-09-23 21:45:52: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:52: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics.
  | 2020-09-23 21:45:52: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:52: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:52: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:52: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send]: discarding 882 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2020-09-23 21:45:53: netdata INFO  : STATSD : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:53: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 15852 bytes on this connection. (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:53: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:53: netdata ERROR : PLUGIN[proc] : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics.
  | 2020-09-23 21:45:53: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:53: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:53: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:53: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:53: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send]: discarding 810 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2020-09-23 21:45:54: netdata INFO  : ACLK_Stats : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:54: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
  | 2020-09-23 21:45:54: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 22603 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:54: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:54: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 22603 bytes transmitted.
  | 2020-09-23 21:45:54: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:54: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:54: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:54: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:55: netdata INFO  : ACLK_Stats : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:55: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connection closed by far end. Restarting connection (errno 22, Invalid argument)
  | 2020-09-23 21:45:55: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics.
  | 2020-09-23 21:45:55: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: failed to send metrics - closing connection - we have sent 14215 bytes on this connection. (errno 9, Bad file descriptor)
  | 2020-09-23 21:45:55: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:55: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:55: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:55: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:56: netdata INFO  : PLUGINSD[python.d] : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:56: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connection closed by far end. Restarting connection (errno 22, Invalid argument)
  | 2020-09-23 21:45:56: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:56: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:56: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:56: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:56: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:56: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send]: discarding 770 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2020-09-23 21:45:56: netdata INFO  : PLUGINSD[python.d] : STREAM kong-462-pcfc4 [send]: sending metrics...
  | 2020-09-23 21:45:57: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: restart stream because socket reports errors (POLLERR) - 30526 bytes transmitted.
  | 2020-09-23 21:45:57: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: connecting...
  | 2020-09-23 21:45:57: netdata ERROR : STATSD : STREAM kong-462-pcfc4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-09-23 21:45:57: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-09-23 21:45:57: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-09-23 21:45:57: netdata INFO  : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 2 - ready to send metrics...
  | 2020-09-23 21:45:57: netdata ERROR : STREAM_SENDER[kong-462-pcfc4] : STREAM kong-462-pcfc4 [send]: discarding 786 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2020-09-23 21:45:58: netdata LOG FLOOD PROTECTION too many logs (201 logs in 59 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1141 seconds.

Master node logs:

2020-09-23 21:46:12: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/opt/netdata/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
--
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : Found 0 legacy dbengines, setting multidb diskspace to 256MB
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : Created file '/opt/netdata/var/lib/netdata/dbengine_multihost_size' to store the computed value
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : SIGNAL: Enabling reaper
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : process tracking enabled.
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value -998
  | 2020-09-23 21:46:12: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
  | 2020-09-23 21:46:12: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
  | 2020-09-23 21:46:12: netdata ERROR : MAIN : Cannot chown directory '/opt/netdata/var/lib/netdata/lock' to 1000:1000 (errno 1, Operation not permitted)
  | 2020-09-23 21:46:12: netdata ERROR : MAIN : Cannot switch to user's netdata group (gid: 1000). (errno 1, Operation not permitted)
  | 2020-09-23 21:46:12: netdata ERROR : MAIN : Cannot become user 'netdata'. Continuing as we are.
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : netdata started on pid 1.
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : Initializing spawn client.
  | 2020-09-23 21:46:12: netdata INFO  : MAIN : Executing /opt/netdata/usr/libexec/netdata/plugins.d/system-info.sh
  | Spawn server is up.
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_NAME=Alpine Linux
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID=alpine
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=3.11.6
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_DETECTION=/etc/os-release
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_HOST_OS_NAME=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_HOST_OS_ID=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_HOST_OS_ID_LIKE=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION_ID=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_HOST_OS_DETECTION=unknown
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-1062.18.1.el7.x86_64
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=none
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=docker
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=dockerenv
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=56
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_FREQ=3300000
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_DETECTION=nproc procfs sysfs
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_RAM=270177816576
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=30726156238848
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Configuring locking mechanism for global GUID map
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Host 'netdata-133-mlp8c' (at registry as 'netdata-133-mlp8c') with guid '373bcc04-fde6-11ea-99a1-0a580a8006c7' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.25.0', update every 1, memory mode save, history entries 924, streaming disabled (to '' with api key ''), health disabled, cache_dir '/opt/netdata/var/cache/netdata', varlib_dir '/opt/netdata/var/lib/netdata', health_log '/opt/netdata/var/lib/netdata/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Found 0 files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Data files not found, creating in path "/opt/netdata/var/cache/netdata/dbengine".
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Creating new data and journal files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Created data file "/opt/netdata/var/cache/netdata/dbengine/datafile-1-0000000001.ndf".
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Created journal file "/opt/netdata/var/cache/netdata/dbengine/journalfile-1-0000000001.njf".
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Found 2 files in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Scanning file "/opt/netdata/var/cache/netdata/dbengine/datafile-1-0000000001.ndf"
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Scanning file "/opt/netdata/var/cache/netdata/dbengine/journalfile-1-0000000001.njf"
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Metadata log files not found, creating in path "/opt/netdata/var/cache/netdata/dbengine".
  | 2020-09-23 21:46:19: netdata INFO  : MAIN : Creating new metadata log file in path /opt/netdata/var/cache/netdata/dbengine
  | 2020-09-23 21:46:20: netdata INFO  : MAIN : Created metadata log file "/opt/netdata/var/cache/netdata/dbengine/metadatalog-00000-00001.mlf".
  | 2020-09-23 21:46:20: netdata INFO  : MAIN : Unable to load '/opt/netdata/var/lib/netdata/cloud.d/claimed_id', setting state to AGENT_UNCLAIMED
  | 2020-09-23 21:46:22: netdata INFO  : PLUGIN[proc] : thread created with task id 956
  | 2020-09-23 21:46:22: netdata INFO  : STATSD : thread created with task id 957
  | 2020-09-23 21:46:22: netdata INFO  : PLUGIN[proc] : set name of thread 956 to PLUGIN[proc]
  | 2020-09-23 21:46:22: netdata INFO  : BACKENDS : thread created with task id 959
  | 2020-09-23 21:46:22: netdata INFO  : STATSD : set name of thread 957 to STATSD
  | 2020-09-23 21:46:22: netdata INFO  : ACLK_Main : thread created with task id 958
  | 2020-09-23 21:46:22: netdata INFO  : EXPORTING : thread created with task id 960
  | 2020-09-23 21:46:22: netdata INFO  : EXPORTING : set name of thread 960 to EXPORTING
  | 2020-09-23 21:46:22: netdata INFO  : PLUGINSD : thread created with task id 962
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : thread created with task id 961
  | 2020-09-23 21:46:22: netdata INFO  : ACLK_Main : set name of thread 958 to ACLK_Main
  | 2020-09-23 21:46:22: netdata INFO  : BACKENDS : set name of thread 959 to BACKENDS
  | 2020-09-23 21:46:22: netdata INFO  : PLUGINSD : set name of thread 962 to PLUGINSD
  | 2020-09-23 21:46:22: netdata INFO  : MAIN : Initializing command server.
  | 2020-09-23 21:46:22: netdata INFO  : EXPORTING : CONFIG: cannot load user exporting config '/opt/netdata/etc/netdata/exporting.conf'. Will try the stock version.
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : set name of thread 961 to WEB_SERVER[stat
  | 2020-09-23 21:46:22: netdata INFO  : ACLK_Main : Waiting for netdata to be ready
  | 2020-09-23 21:46:22: netdata INFO  : HEALTH : thread created with task id 963
  | 2020-09-23 21:46:22: netdata INFO  : HEALTH : set name of thread 963 to HEALTH
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : To use encryption it is necessary to set "ssl certificate" and "ssl key" in [web] !
  |  
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : starting worker 2
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : starting worker 3
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static2] : thread created with task id 965
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static2] : set name of thread 965 to WEB_SERVER[stat
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static3] : thread created with task id 966
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : starting worker 4
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static3] : set name of thread 966 to WEB_SERVER[stat
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static2] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static4] : thread created with task id 967
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static3] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static4] : set name of thread 967 to WEB_SERVER[stat
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : starting worker 5
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static4] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : starting worker 6
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static5] : thread created with task id 968
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static5] : set name of thread 968 to WEB_SERVER[stat
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static2] : clients wants to STREAM metrics.
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static1] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static3] : clients wants to STREAM metrics.
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static6] : thread created with task id 969
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static6] : set name of thread 969 to WEB_SERVER[stat
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
  | 2020-09-23 21:46:22: netdata INFO  : WEB_SERVER[static6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : thread created with task id 970
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : set name of thread 970 to STREAM_RECEIVER
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : thread created with task id 971
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : set name of thread 971 to STREAM_RECEIVER
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : STREAM kong-462-pdw5x [10.131.6.1]:41980: receive thread created (task id 971)
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : STREAM kong-462-pcfc4 [10.131.6.1]:42778: receive thread created (task id 970)
  | 2020-09-23 21:46:22: netdata INFO  : BACKENDS : cleaning up...
  | 2020-09-23 21:46:22: netdata INFO  : BACKENDS : thread with task id 959 finished
  | 2020-09-23 21:46:22: netdata INFO  : EXPORTING : No connector instances to activate
  | 2020-09-23 21:46:22: netdata INFO  : EXPORTING : EXPORTING: no exporting connectors configured
  | 2020-09-23 21:46:22: netdata INFO  : EXPORTING : cleaning up...
  | 2020-09-23 21:46:22: netdata INFO  : EXPORTING : thread with task id 960 finished
  | 2020-09-23 21:46:22: netdata INFO  : MAIN : netdata initialization completed. Enjoy real-time performance monitoring!
  | 2020-09-23 21:46:22: netdata INFO  : STATSD : cleaning up...
  | 2020-09-23 21:46:22: netdata INFO  : STATSD : STATSD: closing sockets...
  | 2020-09-23 21:46:22: netdata INFO  : STATSD : STATSD: cleanup completed.
  | 2020-09-23 21:46:22: netdata INFO  : STATSD : thread with task id 957 finished
  | 2020-09-23 21:46:22: netdata ERROR : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : HEALTH [kong-462-pdw5x]: cannot open health file: /opt/netdata/var/lib/netdata/fe33c2fe-fde5-11ea-b575-0a580a8307fb/health/health-log.db.old (errno 2, No such file or directory)
  | 2020-09-23 21:46:22: netdata ERROR : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : HEALTH [kong-462-pdw5x]: cannot open health file: /opt/netdata/var/lib/netdata/fe33c2fe-fde5-11ea-b575-0a580a8307fb/health/health-log.db (errno 2, No such file or directory)
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : Host 'kong-462-pdw5x' (at registry as 'kong-462-pdw5x') with guid 'fe33c2fe-fde5-11ea-b575-0a580a8307fb' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.25.0', update every 1, memory mode ram, history entries 3996, streaming disabled (to '' with api key ''), health enabled, cache_dir '/opt/netdata/var/cache/netdata/fe33c2fe-fde5-11ea-b575-0a580a8307fb', varlib_dir '/opt/netdata/var/lib/netdata/fe33c2fe-fde5-11ea-b575-0a580a8307fb', health_log '/opt/netdata/var/lib/netdata/fe33c2fe-fde5-11ea-b575-0a580a8307fb/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : STREAM kong-462-pdw5x [receive from [10.131.6.1]:41980]: initializing communication...
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : STREAM kong-462-pdw5x [receive from [10.131.6.1]:41980]: Netdata is using the stream version 3.
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : Postponing health checks for 60 seconds, on host 'kong-462-pdw5x', because it was just connected.
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pdw5x,[10.131.6.1]:41980] : STREAM kong-462-pdw5x [receive from [10.131.6.1]:41980]: receiving metrics...
  | 2020-09-23 21:46:22: netdata ERROR : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : HEALTH [kong-462-pcfc4]: cannot open health file: /opt/netdata/var/lib/netdata/08078770-fde6-11ea-873f-0a580a8307fd/health/health-log.db.old (errno 2, No such file or directory)
  | 2020-09-23 21:46:22: netdata ERROR : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : HEALTH [kong-462-pcfc4]: cannot open health file: /opt/netdata/var/lib/netdata/08078770-fde6-11ea-873f-0a580a8307fd/health/health-log.db (errno 2, No such file or directory)
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : Host 'kong-462-pcfc4' (at registry as 'kong-462-pcfc4') with guid '08078770-fde6-11ea-873f-0a580a8307fd' initialized, os 'linux', timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.25.0', update every 1, memory mode ram, history entries 3996, streaming disabled (to '' with api key ''), health enabled, cache_dir '/opt/netdata/var/cache/netdata/08078770-fde6-11ea-873f-0a580a8307fd', varlib_dir '/opt/netdata/var/lib/netdata/08078770-fde6-11ea-873f-0a580a8307fd', health_log '/opt/netdata/var/lib/netdata/08078770-fde6-11ea-873f-0a580a8307fd/health/health-log.db', alarms default handler '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : STREAM kong-462-pcfc4 [receive from [10.131.6.1]:42778]: initializing communication...
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : STREAM kong-462-pcfc4 [receive from [10.131.6.1]:42778]: Netdata is using the stream version 3.
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : Postponing health checks for 60 seconds, on host 'kong-462-pcfc4', because it was just connected.
  | 2020-09-23 21:46:22: netdata INFO  : STREAM_RECEIVER[kong-462-pcfc4,[10.131.6.1]:42778] : STREAM kong-462-pcfc4 [receive from [10.131.6.1]:42778]: receiving metrics...
  | 2020-09-23 21:46:22: netdata INFO  : ACLK_Main : Waiting for Cloud to be enabled
  | 2020-09-23 21:46:22: netdata INFO  : ACLK_Main : Waiting for netdata to be claimed
  | 2020-09-23 21:46:22: netdata INFO  : ACLK_Stats : thread created with task id 972
  | 2020-09-23 21:46:22: netdata INFO  : ACLK_Stats : set name of thread 972 to ACLK_Stats
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/guest_nice.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/guest.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/steal.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/softirq.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/irq.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/user.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/system.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/nice.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/iowait.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.cpu/idle.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_status/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_status/online.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ctxt/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ctxt/switches.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_per_second/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_per_second/added.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_per_second/dispatched.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.forks/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.forks/started.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.processes/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.processes/running.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.processes/blocked.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_write_q/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_write_q/added.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_write_q/consumed.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.uptime/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.uptime/uptime.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Using now_boottime_usec() for uptime (dt is 1 ms)
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static3] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread3_cpu/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static3] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread3_cpu/user.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static3] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread3_cpu/system.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_read_q/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_read_q/added.db.
  | 2020-09-23 21:46:23: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_read_q/consumed.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static2] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread2_cpu/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static1] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread1_cpu/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static2] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread2_cpu/user.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static2] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread2_cpu/system.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static1] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread1_cpu/user.db.
  | 2020-09-23 21:46:23: netdata INFO  : WEB_SERVER[static1] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread1_cpu/system.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/main.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/load1.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/load5.db.
  | 2020-09-23 21:46:23: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.load/load15.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static6] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread6_cpu/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static6] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread6_cpu/user.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static6] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread6_cpu/system.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static4] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread4_cpu/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static4] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread4_cpu/user.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static4] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread4_cpu/system.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_cloud_req/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_cloud_req/received.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_cloud_req/malformed.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.active_processes/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.active_processes/active.db.
  | 2020-09-23 21:46:24: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/cpu' (errno 2, No such file or directory)
  | 2020-09-23 21:46:24: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/cpu.
  | 2020-09-23 21:46:24: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/memory' (errno 2, No such file or directory)
  | 2020-09-23 21:46:24: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/memory.
  | 2020-09-23 21:46:24: netdata ERROR : PLUGIN[proc] : PROCFILE: Cannot open file '/proc/pressure/io' (errno 2, No such file or directory)
  | 2020-09-23 21:46:24: netdata ERROR : PLUGIN[proc] : Cannot read pressure information from /proc/pressure/io.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static5] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread5_cpu/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_threads/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_threads/Query_0.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_threads/Query_1.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static5] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread5_cpu/user.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.pgpgio/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.pgpgio/in.db.
  | 2020-09-23 21:46:24: netdata INFO  : WEB_SERVER[static5] : Initializing file /opt/netdata/var/cache/netdata/netdata.web_thread5_cpu/system.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.pgpgio/out.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/avg.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/max.db.
  | 2020-09-23 21:46:24: netdata INFO  : ACLK_Stats : Initializing file /opt/netdata/var/cache/netdata/netdata.aclk_query_time/total.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.pgfaults/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.pgfaults/minor.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.pgfaults/major.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/free.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/used.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/cached.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/system.ram/buffers.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.available/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.available/MemAvailable.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.committed/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.committed/Committed_AS.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/Dirty.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/Writeback.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/FuseWriteback.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/NfsWriteback.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.writeback/Bounce.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/Slab.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/KernelStack.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/PageTables.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.kernel/VmallocUsed.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.slab/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.slab/reclaimable.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.slab/unreclaimable.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/numa_hit.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/numa_miss.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/local_node.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/numa_foreign.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/interleave_hit.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node1/other_node.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/main.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/numa_hit.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/numa_miss.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/local_node.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/numa_foreign.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/interleave_hit.db.
  | 2020-09-23 21:46:24: netdata INFO  : PLUGIN[proc] : Initializing file /opt/netdata/var/cache/netdata/mem.node0/other_node.db.
  | 2020-09-23 21:46:24: netdata LOG FLOOD PROTECTION too many logs (201 logs in 4 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1196 seconds.
  | 2020-09-23 21:47:22: alarm-notify.sh: WARNING: Cannot find file '/opt/netdata/etc/netdata/health_alarm_notify.conf'.
  | 2020-09-23 21:47:22: alarm-notify.sh: WARNING: Cannot find file '/opt/netdata/etc/netdata/health_alarm_notify.conf'.

[jeremyjpj0916]

@mfundul

Anyone have some knowledge on the error 22 invalid argument stuff? Is the stream.conf files changed too from 1.10.x to these newer iterations 1.25.0 ? I remember it was just a secret key that set the auth before and I didn't touch my stream.conf when I upgraded.

[jeremyjpj0916]

Also curious if this seems normal for a master node as far as data mem growth goes... almost up to 2GB in mem now:

Master netdata node conf:

# NetData Configuration
#
# To see defaults, grab one from your instance:
# http://localhost:19999/netdata.conf

#history = seconds for Graph/Chart data TTL, 900 = 15 minutes
# global netdata configuration

[global]
        memory mode = save
    run as user = netdata
    history = 900
#   host access prefix = /
    access log = none
    error log = /dev/stderr
    debug log = none
        cleanup obsolete charts after seconds = 300
    cleanup orphan hosts after seconds = 300
    delete obsolete charts files = yes
    delete orphan hosts files = yes

[plugins]
    proc = yes
    diskspace = no
    cgroups = no
    tc = no
    idlejitter = no
    enable running new plugins = no
    charts.d = no
    fping = no
    node.d = no
    python.d = no
    apps = no

[web]
    web files owner = netdata
    web files group = netdata
        allow streaming from = *
        allow connections by dns = no
        allow dashboard by dns = no
        allow badges by dns = no
        allow registry by dns = no
        allow streaming by dns = no
        allow netdata.conf by dns = no                                                                                                                                                           
        allow management by dns = no

[health]
    enabled = no

[statsd]
    enabled = no

# per plugin configuration
[plugin:proc]
    # netdata server resources = yes
    # /proc/stat = yes
    # /proc/uptime = yes
    # /proc/loadavg = yes
    /proc/sys/kernel/random/entropy_avail = no
    /proc/interrupts = no
    /proc/softirqs = no
    # /proc/vmstat = yes
    # /proc/meminfo = yes
    # /sys/kernel/mm/ksm = yes
    /sys/devices/system/edac/mc = no
    # /sys/devices/system/node = yes
    # /proc/net/dev = yes
    # /proc/net/sockstat = yes
    /proc/net/sockstat6 = no
    # /proc/net/netstat = yes
    /proc/net/snmp = no
        /proc/net/snmp6 = no
    # /proc/net/softnet_stat = yes
    /proc/net/ip_vs/stats = no
    # /proc/net/stat/conntrack = yes
    /proc/net/stat/synproxy = no
    /proc/diskstats = no
    /proc/net/rpc/nfsd = no
    /proc/net/rpc/nfs = no
    /proc/spl/kstat/zfs/arcstats = no
    /sys/fs/btrfs = no
    ipc = no

[plugin:proc:/proc/interrupts]
    interrupts per core = no

[plugin:proc:/proc/net/softnet_stat]
    softnet_stat per core = no

[plugin:proc:/proc/stat]
    per cpu core utilization = no
    cpu interrupts = no

[plugin:proc:/proc/net/stat/nf_conntrack]
    netfilter new connections = no
    netfilter connection changes = no
    netfilter connection expectations = no
    netfilter connection searches = no

[plugin:proc:/proc/meminfo]
    hugepages = no
    transparent hugepages = no

[plugin:proc:/proc/vmstat]
    system-wide numa metric summary = no

[users.cpu]
    enabled = no

[users.mem]
    enabled = no

[users.threads]
    enabled = no

[users.processes]
    enabled = no

[users.cpu_user]
    enabled = no

[users.cpu_system]
    enabled = no

[users.major_faults]
    enabled = no

[users.minor_faults]
    enabled = no

[users.lreads]
    enabled = no

[users.lwrites]
    enabled = no

[users.preads]
    enabled = no

[users.pwrites]
    enabled = no

[users.files]
    enabled = no

[users.sockets]
    enabled = no

[users.pipes]
    enabled = no

[groups.cpu]
    enabled = no

[groups.mem]
    enabled = no

[groups.threads]
    enabled = no

[groups.processes]
    enabled = no

[groups.cpu_user]
    enabled = no

[groups.cpu_system]
    enabled = no

[groups.major_faults]
    enabled = no

[groups.minor_faults]
    enabled = no

[groups.lreads]
    enabled = no

[groups.lwrites]
    enabled = no

[groups.preads]
    enabled = no

[groups.pwrites]
    enabled = no

[groups.files]
    enabled = no

[groups.sockets]
    enabled = no

[groups.pipes]
    enabled = no

[groups.vmem]
    enabled = no

[netdata.plugin_tc_cpu]
    enabled = no

[netdata.plugin_tc_time]
    enabled = no

[netdata.compression_ratio]
    enabled = no

[netdata.plugin_cgroups_cpu]
    enabled = no

[netdata.apps_cpu]
    enabled = no

[netdata.apps_files]
    enabled = no

[netdata.plugin_proc_cpu]
    enabled = no

[netdata.server_cpu]
    enabled = no

[netdata.clients]
    enabled = no

[netdata.net]
    enabled = no

[netdata.plugin_proc_modules]
    enabled = no

[netdata.web_thread1_cpu]
    enabled = no

[netdata.web_thread2_cpu]
    enabled = no

[netdata.web_thread3_cpu]
    enabled = no

[netdata.web_thread4_cpu]
    enabled = no

[netdata.web_thread5_cpu]
    enabled = no

[netdata.web_thread6_cpu]
    enabled = no

Maybe 15 mins is too significant? I would not think so because if mem growth was based on retention and considering my data flow is pretty typical for metrics with some load during day and slower at night then I would expect it to peak at a 15 min interval during day and go down at night, yet netdata pods seem to have consistent growth. Prometheus and graphana scrape my statsD metrics like every 10 seconds I think so as long as I retained more than 10 seconds I am golden anyways eh xD ? Or honestly prom scrapes an instant in time right so like 1 second retention is all thats really needed eh xD ? Might be incorrect in my thinking there but yeah. Kinda wanna minimize all dashboards and logged data besides my statsd and some really important high level stuffs like cpu/ram/tcp issues and such....

[jeremyjpj0916]

And looks like my prod netdata pod is pretty stable in this kubernetes(openshift) project:

[jeremyjpj0916]

Ehh am actually seeing the chunky chart data now on 1.25.0 using the prebuilt run file thing:

Had high hopes hah.

If any netdata dev's ever wanna see it in action I am happy to host a zoom call and demo the behaviors.

[mfundul]

I can see from the parent logs that the child node storage has memory mode = ram and history = 3996. Those settings are in etc/netdata/stream.conf in the parent node and apply for each child node separately (see here). First of all, this means that there is no persistent history (due to ram). Secondly, this means there are 3996 x 4 = 16KB of RAM per metric, which can be a lot depending on the number of metrics per node.

I would suggest trying to use memory mode = dbengine in both etc/netdata/netdata.conf and etc/netdata/stream.conf (see here). This will limit your memory usage and give you persistent history of metrics.

Moreover, we will merge a Pull Request that implements replication instead of simple streaming soon. This will be compatible with memory mode = dbengine, and fills the gaps by design, as long as the gaps don't exist at the child node side.

Edit: a note on replication, memory mode = none in the child side is not going to be compatible with database replication. For this to work, you should change etc/netdata/netdata.conf in the child nodes to use memory mode = ram with very small history, e.g. history = 100.

[jeremyjpj0916]

@mfundul The architecure for me is like so.

I could have 1...n pods of a api gateway application(kong) and it also runs a small netdata(child node) paired with each whose single job is to really stream basic metrics about the pods they are sidecar on in the pod as well as stream statsD metrics they receive from the Kong application over localhost to the parent netdata node which will serve as the 5-15 min of real-time data exposer for things like prometheus to scrape.

One annoyance is the logs I get since the applications statsD metrics are same keys between the 1...n pods sometimes so I see stuff like this in the logs:

2020-10-02 13:13:37: netdata INFO  : STREAM_RECEIVER[kong-449-hdvj4,[10.130.8.1]:40318] : RRDSET: chart name 'statsd_counter_kong.xyz_test_orders_dmz_stage_v3.request.status.502' on host 'kong-449-hdvj4' already exists.

I remember seeing posts once that netdata devs thought its problematic to have similar statsD metrics coming out of similar pods -> streaming to parent netdata node. Not sure if that is still the case, it should not be the case anyways imo, master should be able to aggregate the stream results it receives for statsD data.

In an ideal world I run these netdata child pods as stream only and minimal resources since they should just be real time streaming with maybe some retry in there if it ever fails to stream given metrics a few times...

Then the parent aggregates all these metrics for us and exposes it on the netdata prom endpoint for scraping to handle visualizations and alerts in grafana for us.

My Parent netdata node has this for /conf.d/stream.conf :

# netdata configuration for aggregating data from remote hosts
#
# API keys authorize a pair of sending-receiving netdata servers.
# Once their communication is authorized, they can exchange metrics for any
# number of hosts.
#
# You can generate API keys, with the linux command: uuidgen


# -----------------------------------------------------------------------------
# 1. ON SLAVE NETDATA - THE ONE THAT WILL BE SENDING METRICS

[stream]
    # Enable this on slaves, to have them send metrics.
    enabled = no

    # Where is the receiving netdata?
    # A space separated list of:
    #
    #      [PROTOCOL:]HOST[%INTERFACE][:PORT]
    #
    # If many are given, the first available will get the metrics.
    #
    # PROTOCOL  = tcp, udp, or unix (only tcp and unix are supported by masters)
    # HOST      = an IPv4, IPv6 IP, or a hostname, or a unix domain socket path.
    #             IPv6 IPs should be given with brackets [ip:address]
    # INTERFACE = the network interface to use (only for IPv6)
    # PORT      = the port number or service name (/etc/services)
    #
    # This communication is not HTTP (it cannot be proxied by web proxies).
    destination =

    # The API_KEY to use (as the sender)
    api key =

    # The timeout to connect and send metrics
    timeout seconds = 60

    # If the destination line above does not specify a port, use this
    default port = 19999

    # The buffer to use for sending metrics.
    # 1MB is good for 10-20 seconds of data, so increase this
    # if you expect latencies.
    buffer size bytes = 1048576

    # If the connection fails, or it disconnects,
    # retry after that many seconds.
    reconnect delay seconds = 5

    # Attempt to sync the clock the of the master with the clock of the
    # slave for that many iterations, when starting.
    initial clock resync iterations = 60


# -----------------------------------------------------------------------------
# 2. ON MASTER NETDATA - THE ONE THAT WILL BE RECEIVING METRICS

#    You can have one API key per slave,
#         or the same API key for all slaves.
#
#    netdata searches for options in this order:
#
#    a) master netdata settings (netdata.conf)
#    b) [API_KEY] section       (below, settings for the API key)
#    c) [MACHINE_GUID] section  (below, settings for each machine)
#
#    You can combine the above (the more specific setting will be used).

# API key authentication
# If the key is not listed here, it will not be able to push metrics.

# [API_KEY] is [YOUR-API-KEY], i.e [11111111-2222-3333-4444-555555555555]
[XXXXXXX-XXXXX-XXXX-XXXX-XXXXXXXXXXXX]
    # Default settings for this API key

    # You can disable the API key, by setting this to: no
    # The default (for unknown API keys) is: no
    enabled = yes

    # A list of simple patterns matching the IPs of the servers that
    # will be pushing metrics using this API key.
    # The metrics are received via the API port, so the same IPs
    # should also be matched at netdata.conf [web].allow connections from
    allow from = *

    # The default history in entries, for all hosts using this API key.
    # You can also set it per host below.
    # If you don't set it here, the history size of the central netdata
    # will be used.
    default history = 3600

    # The default memory mode to be used for all hosts using this API key.
    # You can also set it per host below.
    # If you don't set it here, the memory mode of netdata.conf will be used.
    # Valid modes:
    #    save    save on exit, load on start
    #    map     like swap (continuously syncing to disks)
    #    ram     keep it in RAM, don't touch the disk
    #    none    no database at all (use this on headless proxies)
    default memory mode = ram

    # Shall we enable health monitoring for the hosts using this API key?
    # 3 possible values:
    #    yes     enable alarms
    #    no      do not enable alarms
    #    auto    enable alarms, only when the sending netdata is connected
    # You can also set it per host, below.
    # The default is the same as to netdata.conf
    health enabled by default = auto

    # postpone alarms for a short period after the sender is connected
    default postpone alarms on connect seconds = 60

    # allow or deny multiple connections for the same host?
    # If you are sure all your netdata have their own machine GUID,
    # set this to 'allow', since it allows faster reconnects.
    # When set to 'deny', new connections for a host will not be
    # accepted until an existing connection is cleared.
    multiple connections = allow

    # need to route metrics differently? set these.
    # the defaults are the ones at the [stream] section
    #default proxy enabled = yes | no
    #default proxy destination = IP:PORT IP:PORT ...
    #default proxy api key = API_KEY


# -----------------------------------------------------------------------------
# 3. PER SENDING HOST SETTINGS, ON MASTER NETDATA
#    THIS IS OPTIONAL - YOU DON'T NEED IT

# This section exists to give you finer control of the master settings for each
# slave host, when the same API key is used by many netdata slaves / proxies.
#
# Each netdata has a unique GUID - generated the first time netdata starts.
# You can find it at /var/lib/netdata/registry/netdata.public.unique.id
# (at the slave).
#
# The host sending data will have one. If the host is not ephemeral,
# you can give settings for each sending host here.

[MACHINE_GUID]
    # enable this host: yes | no
    # When disabled, the master will not receive metrics for this host.
    # THIS IS NOT A SECURITY MECHANISM - AN ATTACKER CAN SET ANY OTHER GUID.
    # Use only the API key for security.
    enabled = no

    # A list of simple patterns matching the IPs of the servers that
    # will be pushing metrics using this MACHINE GUID.
    # The metrics are received via the API port, so the same IPs
    # should also be matched at netdata.conf [web].allow connections from
    # and at stream.conf [API_KEY].allow from
    allow from = *

    # The number of entries in the database
    history = 3600

    # The memory mode of the database: save | map | ram | none
    memory mode = save

    # Health / alarms control: yes | no | auto
    health enabled = yes

    # postpone alarms when the sender connects
    postpone alarms on connect seconds = 60

    # allow or deny multiple connections for the same host?
    # If you are sure all your netdata have their own machine GUID,
    # set this to 'allow', since it allows faster reconnects.
    # When set to 'deny', new connections for a host will not be
    # accepted until an existing connection is cleared.
    multiple connections = allow

    # need to route metrics differently?
    #proxy enabled = yes | no
    #proxy destination = IP:PORT IP:PORT ...
    #proxy api key = API_KEY

My parent node system/netdata.conf looks like so as well for reference, need to get a better way to remove more charts I don't like haha related to random stats but I tried to kill most the noise here so i can focus on core stats and statsD, I don't wanna need to much cpu/mem in my setup:

# NetData Configuration
#
# To see defaults, grab one from your instance:
# http://localhost:19999/netdata.conf

#history = seconds for Graph/Chart data TTL, 900 = 15 minutes
# global netdata configuration

[global]
        memory mode = save
    run as user = netdata
    history = 900
#   host access prefix = /
    access log = none
    error log = /dev/stderr
    debug log = none
        cleanup obsolete charts after seconds = 300
    cleanup orphan hosts after seconds = 300
    delete obsolete charts files = yes
    delete orphan hosts files = yes

[plugins]
    proc = yes
    diskspace = no
    cgroups = no
    tc = no
    idlejitter = no
    enable running new plugins = no
    charts.d = no
    fping = no
    node.d = no
    python.d = no
    apps = no

[web]
    web files owner = netdata
    web files group = netdata
        allow streaming from = *
        allow connections by dns = no
        allow dashboard by dns = no
        allow badges by dns = no
        allow registry by dns = no
        allow streaming by dns = no
        allow netdata.conf by dns = no                                                                                                                                                           
        allow management by dns = no

[health]
    enabled = no

[statsd]
    enabled = no

# per plugin configuration
[plugin:proc]
    # netdata server resources = yes
    # /proc/stat = yes
    # /proc/uptime = yes
    # /proc/loadavg = yes
    /proc/sys/kernel/random/entropy_avail = no
    /proc/interrupts = no
    /proc/softirqs = no
    # /proc/vmstat = yes
    # /proc/meminfo = yes
    # /sys/kernel/mm/ksm = yes
    /sys/devices/system/edac/mc = no
    # /sys/devices/system/node = yes
    # /proc/net/dev = yes
    # /proc/net/sockstat = yes
    /proc/net/sockstat6 = no
    # /proc/net/netstat = yes
    /proc/net/snmp = no
        /proc/net/snmp6 = no
    # /proc/net/softnet_stat = yes
    /proc/net/ip_vs/stats = no
    # /proc/net/stat/conntrack = yes
    /proc/net/stat/synproxy = no
    /proc/diskstats = no
    /proc/net/rpc/nfsd = no
    /proc/net/rpc/nfs = no
    /proc/spl/kstat/zfs/arcstats = no
    /sys/fs/btrfs = no
    ipc = no

[plugin:proc:/proc/interrupts]
    interrupts per core = no

[plugin:proc:/proc/net/softnet_stat]
    softnet_stat per core = no

[plugin:proc:/proc/stat]
    per cpu core utilization = no
    cpu interrupts = no

[plugin:proc:/proc/net/stat/nf_conntrack]
    netfilter new connections = no
    netfilter connection changes = no
    netfilter connection expectations = no
    netfilter connection searches = no

[plugin:proc:/proc/meminfo]
    hugepages = no
    transparent hugepages = no

[plugin:proc:/proc/vmstat]
    system-wide numa metric summary = no

[users.cpu]
    enabled = no

[users.mem]
    enabled = no

[users.threads]
    enabled = no

[users.processes]
    enabled = no

[users.cpu_user]
    enabled = no

[users.cpu_system]
    enabled = no

[users.major_faults]
    enabled = no

[users.minor_faults]
    enabled = no

[users.lreads]
    enabled = no

[users.lwrites]
    enabled = no

[users.preads]
    enabled = no

[users.pwrites]
    enabled = no

[users.files]
    enabled = no

[users.sockets]
    enabled = no

[users.pipes]
    enabled = no

[groups.cpu]
    enabled = no

[groups.mem]
    enabled = no

[groups.threads]
    enabled = no

[groups.processes]
    enabled = no

[groups.cpu_user]
    enabled = no

[groups.cpu_system]
    enabled = no

[groups.major_faults]
    enabled = no

[groups.minor_faults]
    enabled = no

[groups.lreads]
    enabled = no

[groups.lwrites]
    enabled = no

[groups.preads]
    enabled = no

[groups.pwrites]
    enabled = no

[groups.files]
    enabled = no

[groups.sockets]
    enabled = no

[groups.pipes]
    enabled = no

[groups.vmem]
    enabled = no

[netdata.plugin_tc_cpu]
    enabled = no

[netdata.plugin_tc_time]
    enabled = no

[netdata.compression_ratio]
    enabled = no

[netdata.plugin_cgroups_cpu]
    enabled = no

[netdata.apps_cpu]
    enabled = no

[netdata.apps_files]
    enabled = no

[netdata.plugin_proc_cpu]
    enabled = no

[netdata.server_cpu]
    enabled = no

[netdata.clients]
    enabled = no

[netdata.net]
    enabled = no

[netdata.plugin_proc_modules]
    enabled = no

[netdata.web_thread1_cpu]
    enabled = no

[netdata.web_thread2_cpu]
    enabled = no

[netdata.web_thread3_cpu]
    enabled = no

[netdata.web_thread4_cpu]
    enabled = no

[netdata.web_thread5_cpu]
    enabled = no

[netdata.web_thread6_cpu]
    enabled = no

Right now for my setup I run:

Parent node:
CPU: 200 millicores to 200 millicores
Memory: 2 GB to 2 GB

Children nodes(2 of them right now):

CPU: 500 millicores to 500 millicores
Memory: 1 GB to 1 GB

As for physical disk as well, not sure how much i get on a random openshift/kubernetes "pod".

I suppose it makes more sense for the parent to probs have more CPU than it does. But what actual settings would you suggest to achieve optimal metrics making it to my prometheus and graphana instance?

My prom configuration for scraping looks like:

scrape_configs:

• job_name: prometheus_kong
  honor_labels: true
  honor_timestamps: true
  params:
  format:
  
  
  
  • prometheus_all_hosts
    
    scrape_interval: 10s
    
    scrape_timeout: 10s
  
  

http://parent-netdata:80/api/v1/allmetrics

Maybe it makes sense to take down the master node persisting 1m or less of data?

[mfundul]

The debug prints shouldn't be a problem.

As far as the parent netdata CPU is concerned, I wasn't aware that it had so little resources. In my mind, a parent node should have at least 2 hardware threads to allow for netdata thread concurrency. Seeing gaps at the parent node side is to be expected if the CPU cannot keep up.

You should experiment with giving more CPU resources to the parent node and see if the gaps issue is resolved.

[jeremyjpj0916]

@mfundul would you recommend the parent node have say 1 CPU or does it need as much as 2 CPU then?

[mfundul]

@mfundul would you recommend the parent node have say 1 CPU or does it need as much as 2 CPU then?

I would say 2 for a parent node.

[jeremyjpj0916]

Editing netdata parent node from 200 millicores to 2 cores and 2 GB to 4 GB of RAM to see if behavior changes in stage with respect to chart gappings and some seen container restarts.

[jeremyjpj0916]

Even after deploying with more resources the result is still the same from gaps in the chart from the netdata child nodes streaming to master, didn't think resource limits were the issue, the master nodes are hardly getting a workout at .01 cores being utilized right now processing 1700 Kib's/sec of data:

Worth noting the UI of the parent nodes data itself is perfectly fine, no line breaks or anything in the charting. Feels like a loss of streaming data to me still. Flakey data getting dropped somehow. Netdata probably needs to implement some kinda retry or some kinda confirmation handoff of data payloads being passed around.

[mfundul]

The replication PR mentioned above should fix the gaps, but it's still strange to see them in this use case. Do you still see those messages such as:

too many data pending - buffer is 1100460 bytes long, 1029162 unsent - we have sent 117328439736 bytes in total, 74094 on this connection. Closing connection to flush the data.

[jeremyjpj0916]

@mfundul Seeing things like this in the children nodes, but not the log you are asking about at the moment above:

2020-10-15 04:56:42: netdata INFO  : STATSD : STREAM kong-847-gkhk4 [send]: sending metrics...
--
  | 2020-10-15 04:56:42: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 118268 bytes. Restarting connection
  | 2020-10-15 04:56:42: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-10-15 04:56:42: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:42: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:42: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:42: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:43: netdata INFO  : PLUGIN[proc] : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:43: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 183017 bytes. Restarting connection
  | 2020-10-15 04:56:43: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics.
  | 2020-10-15 04:56:43: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:43: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:43: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:43: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:44: netdata INFO  : STATSD : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:44: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 221720 bytes. Restarting connection
  | 2020-10-15 04:56:44: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-10-15 04:56:44: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:44: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:44: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:44: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:45: netdata INFO  : ACLK_Stats : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:45: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 132248 bytes. Restarting connection
  | 2020-10-15 04:56:45: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics.
  | 2020-10-15 04:56:45: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:45: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:45: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:45: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:46: netdata INFO  : PLUGIN[proc] : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:46: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 119803 bytes. Restarting connection
  | 2020-10-15 04:56:46: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:46: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-10-15 04:56:46: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:46: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:46: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:46: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send]: discarding 822 bytes of metrics already in the buffer. (errno 22, Invalid argument)
  | 2020-10-15 04:56:47: netdata INFO  : PLUGIN[proc] : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:47: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 129893 bytes. Restarting connection
  | 2020-10-15 04:56:47: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics.
  | 2020-10-15 04:56:47: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:47: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:47: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:47: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:48: netdata INFO  : PLUGIN[proc] : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:48: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 111719 bytes. Restarting connection
  | 2020-10-15 04:56:48: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-10-15 04:56:48: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:48: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:48: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:48: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:49: netdata INFO  : STATSD : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:49: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 192362 bytes. Restarting connection
  | 2020-10-15 04:56:49: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics. (errno 22, Invalid argument)
  | 2020-10-15 04:56:49: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:49: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:49: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:49: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: established communication with a parent using protocol version 3 - ready to send metrics...
  | 2020-10-15 04:56:50: netdata INFO  : ACLK_Stats : STREAM kong-847-gkhk4 [send]: sending metrics...
  | 2020-10-15 04:56:50: netdata ERROR : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: buffer full (1048576-bytes) after 218031 bytes. Restarting connection
  | 2020-10-15 04:56:50: netdata ERROR : STATSD : STREAM kong-847-gkhk4 [send]: not ready - discarding collected metrics.
  | 2020-10-15 04:56:50: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: connecting...
  | 2020-10-15 04:56:50: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: initializing communication...
  | 2020-10-15 04:56:50: netdata INFO  : STREAM_SENDER[kong-847-gkhk4] : STREAM kong-847-gkhk4 [send to tcp:netdata:19999]: waiting response from remote netdata...
  | 2020-10-15 04:56:50: netdata LOG FLOOD PROTECTION too many logs (201 logs in 27 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1173 seconds.

Parent node not showing too much in logs atm besides the standard statsD chart type already exists warning thing, would be nice if I could silence that since I expect many apps have a multi child to parent relationship sending data over the same chart type name because its the same type of service intended for statsD aggregation of results. I do see the parent node restarted once in the last week due to what looks like OOM, can tell by the metrics, see how it grew and peaked and then crashed back down and growing again slowly?

Guessing some kinda application mem leak never found w respect to statsD or steaming but unsure.

May eventually have to just give up on Netdata in our stack, been too unstable. Would have loved it to be reliable and a good way to move statsD data around though in our cloud platform. I may revert my code and go back to building from source(this usually fixes the metrics charting gaps and such) and maybe tweak the mem and storage parameters on the childen/parent nodes as you say and see if that helps any. Seems maybe the debian as base image does better for netdata than alpine as well in the past but can't draw any conclusion there, just a hunch.

[mfundul]

It looks like you're running out of buffer space at the child nodes, take a look here.

You should update etc/netdata/stream.conf in the child nodes to use a larger buffer size, e.g.

[stream]
    buffer size bytes = 10485760

[jeremyjpj0916]

But why is the buffer full every second, and why do get weird connection invalid arg erros (errno 22, Invalid argument) . In a stream w mostly my statsD data the buffer should relatively fluctuate w the traffic coming through the gateway app the child is sidecar with. But this error seems an ongoing thing that happens night(when utilization is low) or day. Will try playing w that param though.