Node_exporter: Consider implementing timeout for collectors
I've just noticed prometheus fail to get metrics from node_exporter in last 2 days (my workstation uptime is 3 days, so node_exporter actually worked only about a day after last reboot). Log of node_exporter complains about "too many open files", and it's really hit 1024 open file limit:
# ls /proc/$(pidof node_exporter)/fd | wc -l
1023
But most of them are sockets:
# ls -l /proc/$(pidof node_exporter)/fd | grep socket | wc -l
1018
And I believe all of them except one is leaked because there are no related connections:
# ss -anp | grep node_exporter
tcp LISTEN 0 128 127.0.0.1:9100 *:* users:(("node_exporter",pid=9801,fd=4))
Also I noticed node_exporter uses too many memory, not sure is this related to this issue, but it shouldn't be second process in memory usage top:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
1638 powerman 20 0 2514M 1476M 77384 S 0.5 18.6 1h44:02 /usr/bin/firefox
9801 root 20 0 12.5G 432M 2764 S 0.0 5.4 10:53.37 /home/powerman/gocode/bin/node_exporter
I'm running node_exporter as a service under runit using this run file:
#!/bin/sh
exec 2>&1
exec /home/powerman/gocode/bin/node_exporter -web.listen-address 127.0.0.1:9100 \
-collectors.enabled "conntrack,diskstats,entropy,filefd,filesystem,loadavg,mdadm,meminfo,netdev,netstat,sockstat,stat,textfile,time,uname,version,vmstat,runit,tcpstat"
I'm now using node_exporter d890b63fb5ebf6144766cd43bf29f0a0e6192491 (about month old) and at a glance next commits doesn't mention anything related to this issue so I suppose it's still actual. I will try to avoid restarting it in next couple of days in case you'll need more details from live process, but if it continues to eat RAM I may have to restart it.
powerman
👍2
All 50 comments
Thanks for the report! Please paste the output of lsof -p $(pidof node_exporter), this should give some first clues.
By the way, we have standard process metrics in several client libraries including golang. There is a process_open_fds metric, so you should be able to plot open fds across all your nodes. max(max_over_time(process_open_fds{job="node}[1d])) returns 37 open fds across all our nodes for example.
grobie
on 24 May 2016
I wonder if these are all half-closed connections. Can you report the netstat output matching the node_exporter port? How does your Prometheus server connect to the node_exporters? How many servers scrape the exporter? Which Prometheus version do you use? Does anything else scrape the exporters as well?
grobie
on 24 May 2016
I wonder if these are all half-closed connections. Can you report the netstat output matching the node_exporter port?
No - I've included ss output above (ss is replacement for netstat by iproute2).
How does your Prometheus server connect to the node_exporters?
It's default config, here is relevant part:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'node_exporter'
target_groups:
- targets: ['localhost:9100']
How many servers scrape the exporter?
Just one.
Which Prometheus version do you use?
0.18.0
Does anything else scrape the exporters as well?
No.
Please paste the output of lsof -p $(pidof node_exporter), this should give some first clues.
lsof: WARNING: can't stat() fuse file system /mnt/webdav.4shared
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node_expo 9801 root cwd DIR 8,5 4096 6300864 /service/node_exporter
node_expo 9801 root rtd DIR 8,5 4096 2 /
node_expo 9801 root txt REG 8,5 14541808 7606846 /home/powerman/gocode/bin/node_exporter
node_expo 9801 root mem REG 8,5 1734128 7740629 /lib64/libc-2.22.so
node_expo 9801 root mem REG 8,5 141440 7740632 /lib64/libpthread-2.22.so
node_expo 9801 root mem REG 8,5 148712 7740625 /lib64/ld-2.22.so
node_expo 9801 root 0w CHR 5,1 0t0 1035 /dev/console
node_expo 9801 root 1w FIFO 0,8 0t0 708 pipe
node_expo 9801 root 2w FIFO 0,8 0t0 708 pipe
node_expo 9801 root 3u FIFO 8,5 0t0 8650993 /var/log/all/.log
node_expo 9801 root 4u IPv4 1499975 0t0 TCP localhost:jetdirect (LISTEN)
node_expo 9801 root 5u a_inode 0,9 0 7 [eventpoll]
node_expo 9801 root 6u sock 0,7 0t0 1499979 protocol: TCP
node_expo 9801 root 7u sock 0,7 0t0 1487824 protocol: TCP
node_expo 9801 root 8u sock 0,7 0t0 1500243 protocol: TCP
node_expo 9801 root 9u sock 0,7 0t0 1490844 protocol: TCP
node_expo 9801 root 10u sock 0,7 0t0 1487869 protocol: TCP
node_expo 9801 root 11u sock 0,7 0t0 1500085 protocol: TCP
node_expo 9801 root 12u sock 0,7 0t0 1490873 protocol: TCP
node_expo 9801 root 13u sock 0,7 0t0 1500135 protocol: TCP
node_expo 9801 root 14u sock 0,7 0t0 1488561 protocol: TCP
node_expo 9801 root 15u sock 0,7 0t0 1488582 protocol: TCP
node_expo 9801 root 16u sock 0,7 0t0 1503335 protocol: TCP
node_expo 9801 root 17u sock 0,7 0t0 1503403 protocol: TCP
node_expo 9801 root 18u sock 0,7 0t0 1504804 protocol: TCP
node_expo 9801 root 19u sock 0,7 0t0 1493757 protocol: TCP
node_expo 9801 root 20u sock 0,7 0t0 1501462 protocol: TCP
node_expo 9801 root 21u sock 0,7 0t0 1505447 protocol: TCP
node_expo 9801 root 22u sock 0,7 0t0 1501482 protocol: TCP
node_expo 9801 root 23u sock 0,7 0t0 1503750 protocol: TCP
node_expo 9801 root 24u sock 0,7 0t0 1503759 protocol: TCP
node_expo 9801 root 25u sock 0,7 0t0 1500824 protocol: TCP
node_expo 9801 root 26u sock 0,7 0t0 1503822 protocol: TCP
node_expo 9801 root 27u sock 0,7 0t0 1488727 protocol: TCP
node_expo 9801 root 28u sock 0,7 0t0 1501607 protocol: TCP
node_expo 9801 root 29u sock 0,7 0t0 1488737 protocol: TCP
node_expo 9801 root 30u sock 0,7 0t0 1505601 protocol: TCP
node_expo 9801 root 31u sock 0,7 0t0 1504975 protocol: TCP
node_expo 9801 root 32u sock 0,7 0t0 1501665 protocol: TCP
node_expo 9801 root 33u sock 0,7 0t0 1488785 protocol: TCP
node_expo 9801 root 34u sock 0,7 0t0 1502590 protocol: TCP
node_expo 9801 root 35u sock 0,7 0t0 1501735 protocol: TCP
node_expo 9801 root 36u sock 0,7 0t0 1504054 protocol: TCP
node_expo 9801 root 37u sock 0,7 0t0 1504077 protocol: TCP
node_expo 9801 root 38u sock 0,7 0t0 1501121 protocol: TCP
node_expo 9801 root 39u sock 0,7 0t0 1504160 protocol: TCP
node_expo 9801 root 40u sock 0,7 0t0 1488871 protocol: TCP
node_expo 9801 root 41u sock 0,7 0t0 1505723 protocol: TCP
node_expo 9801 root 42u sock 0,7 0t0 1508371 protocol: TCP
node_expo 9801 root 43u sock 0,7 0t0 1505829 protocol: TCP
node_expo 9801 root 44u sock 0,7 0t0 1505162 protocol: TCP
node_expo 9801 root 45u sock 0,7 0t0 1505853 protocol: TCP
node_expo 9801 root 46u sock 0,7 0t0 1508411 protocol: TCP
node_expo 9801 root 47u sock 0,7 0t0 1502879 protocol: TCP
node_expo 9801 root 48u sock 0,7 0t0 1502058 protocol: TCP
node_expo 9801 root 49u sock 0,7 0t0 1507560 protocol: TCP
node_expo 9801 root 50u sock 0,7 0t0 1502911 protocol: TCP
node_expo 9801 root 51u sock 0,7 0t0 1505911 protocol: TCP
node_expo 9801 root 52u sock 0,7 0t0 1502125 protocol: TCP
node_expo 9801 root 53u sock 0,7 0t0 1505948 protocol: TCP
node_expo 9801 root 54u sock 0,7 0t0 1502942 protocol: TCP
node_expo 9801 root 55u sock 0,7 0t0 1508623 protocol: TCP
node_expo 9801 root 56u sock 0,7 0t0 1502190 protocol: TCP
node_expo 9801 root 57u sock 0,7 0t0 1510407 protocol: TCP
node_expo 9801 root 58u sock 0,7 0t0 1506008 protocol: TCP
node_expo 9801 root 59u sock 0,7 0t0 1508680 protocol: TCP
node_expo 9801 root 60u sock 0,7 0t0 1506738 protocol: TCP
node_expo 9801 root 61u sock 0,7 0t0 1503069 protocol: TCP
node_expo 9801 root 62u sock 0,7 0t0 1506809 protocol: TCP
node_expo 9801 root 63u sock 0,7 0t0 1503106 protocol: TCP
node_expo 9801 root 64u sock 0,7 0t0 1506108 protocol: TCP
node_expo 9801 root 65u sock 0,7 0t0 1507971 protocol: TCP
node_expo 9801 root 66u sock 0,7 0t0 1506135 protocol: TCP
node_expo 9801 root 67u sock 0,7 0t0 1503154 protocol: TCP
node_expo 9801 root 68u sock 0,7 0t0 1508842 protocol: TCP
node_expo 9801 root 69u sock 0,7 0t0 1506916 protocol: TCP
node_expo 9801 root 70u sock 0,7 0t0 1510636 protocol: TCP
node_expo 9801 root 71u sock 0,7 0t0 1506933 protocol: TCP
node_expo 9801 root 72u sock 0,7 0t0 1508862 protocol: TCP
node_expo 9801 root 73u sock 0,7 0t0 1506958 protocol: TCP
node_expo 9801 root 74u sock 0,7 0t0 1513473 protocol: TCP
node_expo 9801 root 75u sock 0,7 0t0 1511744 protocol: TCP
node_expo 9801 root 76u sock 0,7 0t0 1513505 protocol: TCP
node_expo 9801 root 77u sock 0,7 0t0 1512556 protocol: TCP
node_expo 9801 root 78u sock 0,7 0t0 1507242 protocol: TCP
node_expo 9801 root 79u sock 0,7 0t0 1512594 protocol: TCP
node_expo 9801 root 80u sock 0,7 0t0 1511828 protocol: TCP
node_expo 9801 root 81u sock 0,7 0t0 1508957 protocol: TCP
node_expo 9801 root 82u sock 0,7 0t0 1513538 protocol: TCP
node_expo 9801 root 83u sock 0,7 0t0 1510784 protocol: TCP
node_expo 9801 root 84u sock 0,7 0t0 1509026 protocol: TCP
node_expo 9801 root 85u sock 0,7 0t0 1512030 protocol: TCP
node_expo 9801 root 86u sock 0,7 0t0 1515626 protocol: TCP
node_expo 9801 root 87u sock 0,7 0t0 1513579 protocol: TCP
node_expo 9801 root 88u sock 0,7 0t0 1512101 protocol: TCP
node_expo 9801 root 89u sock 0,7 0t0 1513582 protocol: TCP
node_expo 9801 root 90u sock 0,7 0t0 1512832 protocol: TCP
node_expo 9801 root 91u sock 0,7 0t0 1512869 protocol: TCP
node_expo 9801 root 92u sock 0,7 0t0 1512884 protocol: TCP
node_expo 9801 root 93u sock 0,7 0t0 1512935 protocol: TCP
node_expo 9801 root 94u sock 0,7 0t0 1509190 protocol: TCP
node_expo 9801 root 95u sock 0,7 0t0 1513643 protocol: TCP
node_expo 9801 root 96u sock 0,7 0t0 1511117 protocol: TCP
node_expo 9801 root 97u sock 0,7 0t0 1509256 protocol: TCP
node_expo 9801 root 98u sock 0,7 0t0 1512341 protocol: TCP
node_expo 9801 root 99u sock 0,7 0t0 1512396 protocol: TCP
node_expo 9801 root 100u sock 0,7 0t0 1513754 protocol: TCP
node_expo 9801 root 101u sock 0,7 0t0 1515994 protocol: TCP
node_expo 9801 root 102u sock 0,7 0t0 1513764 protocol: TCP
node_expo 9801 root 103u sock 0,7 0t0 1513096 protocol: TCP
node_expo 9801 root 104u sock 0,7 0t0 1516060 protocol: TCP
node_expo 9801 root 105u sock 0,7 0t0 1513109 protocol: TCP
node_expo 9801 root 106u sock 0,7 0t0 1515063 protocol: TCP
node_expo 9801 root 107u sock 0,7 0t0 1516777 protocol: TCP
node_expo 9801 root 108u sock 0,7 0t0 1513811 protocol: TCP
node_expo 9801 root 109u sock 0,7 0t0 1511200 protocol: TCP
node_expo 9801 root 110u sock 0,7 0t0 1516785 protocol: TCP
node_expo 9801 root 111u sock 0,7 0t0 1515239 protocol: TCP
node_expo 9801 root 112u sock 0,7 0t0 1513245 protocol: TCP
node_expo 9801 root 113u sock 0,7 0t0 1514063 protocol: TCP
node_expo 9801 root 114u sock 0,7 0t0 1516352 protocol: TCP
node_expo 9801 root 115u sock 0,7 0t0 1514136 protocol: TCP
node_expo 9801 root 116u sock 0,7 0t0 1513332 protocol: TCP
node_expo 9801 root 117u sock 0,7 0t0 1516412 protocol: TCP
node_expo 9801 root 118u sock 0,7 0t0 1513347 protocol: TCP
node_expo 9801 root 119u sock 0,7 0t0 1516464 protocol: TCP
node_expo 9801 root 120u sock 0,7 0t0 1516938 protocol: TCP
node_expo 9801 root 121u sock 0,7 0t0 1514266 protocol: TCP
node_expo 9801 root 122u sock 0,7 0t0 1513435 protocol: TCP
node_expo 9801 root 123u sock 0,7 0t0 1516994 protocol: TCP
node_expo 9801 root 124u sock 0,7 0t0 1518782 protocol: TCP
node_expo 9801 root 125u sock 0,7 0t0 1520680 protocol: TCP
node_expo 9801 root 126u sock 0,7 0t0 1519693 protocol: TCP
node_expo 9801 root 127u sock 0,7 0t0 1519722 protocol: TCP
node_expo 9801 root 128u sock 0,7 0t0 1514396 protocol: TCP
node_expo 9801 root 129u sock 0,7 0t0 1518868 protocol: TCP
node_expo 9801 root 130u sock 0,7 0t0 1519758 protocol: TCP
node_expo 9801 root 131u sock 0,7 0t0 1520822 protocol: TCP
node_expo 9801 root 132u sock 0,7 0t0 1519812 protocol: TCP
node_expo 9801 root 133u sock 0,7 0t0 1517139 protocol: TCP
node_expo 9801 root 134u sock 0,7 0t0 1514455 protocol: TCP
node_expo 9801 root 135u sock 0,7 0t0 1521727 protocol: TCP
node_expo 9801 root 136u sock 0,7 0t0 1518997 protocol: TCP
node_expo 9801 root 137u sock 0,7 0t0 1518258 protocol: TCP
node_expo 9801 root 138u sock 0,7 0t0 1519040 protocol: TCP
node_expo 9801 root 139u sock 0,7 0t0 1522695 protocol: TCP
node_expo 9801 root 140u sock 0,7 0t0 1522702 protocol: TCP
node_expo 9801 root 141u sock 0,7 0t0 1522706 protocol: TCP
node_expo 9801 root 142u sock 0,7 0t0 1519209 protocol: TCP
node_expo 9801 root 143u sock 0,7 0t0 1519980 protocol: TCP
node_expo 9801 root 144u sock 0,7 0t0 1521042 protocol: TCP
node_expo 9801 root 145u sock 0,7 0t0 1520002 protocol: TCP
node_expo 9801 root 146u sock 0,7 0t0 1517344 protocol: TCP
node_expo 9801 root 147u sock 0,7 0t0 1522779 protocol: TCP
node_expo 9801 root 148u sock 0,7 0t0 1521142 protocol: TCP
node_expo 9801 root 149u sock 0,7 0t0 1521148 protocol: TCP
node_expo 9801 root 150u sock 0,7 0t0 1521162 protocol: TCP
node_expo 9801 root 151u sock 0,7 0t0 1519368 protocol: TCP
node_expo 9801 root 152u sock 0,7 0t0 1522841 protocol: TCP
node_expo 9801 root 153u sock 0,7 0t0 1522844 protocol: TCP
node_expo 9801 root 154u sock 0,7 0t0 1522851 protocol: TCP
node_expo 9801 root 155u sock 0,7 0t0 1518541 protocol: TCP
node_expo 9801 root 156u sock 0,7 0t0 1520213 protocol: TCP
node_expo 9801 root 157u sock 0,7 0t0 1518571 protocol: TCP
node_expo 9801 root 158u sock 0,7 0t0 1518592 protocol: TCP
node_expo 9801 root 159u sock 0,7 0t0 1522939 protocol: TCP
node_expo 9801 root 160u sock 0,7 0t0 1522944 protocol: TCP
node_expo 9801 root 161u sock 0,7 0t0 1521402 protocol: TCP
node_expo 9801 root 162u sock 0,7 0t0 1524827 protocol: TCP
node_expo 9801 root 163u sock 0,7 0t0 1521440 protocol: TCP
node_expo 9801 root 164u sock 0,7 0t0 1517542 protocol: TCP
node_expo 9801 root 165u sock 0,7 0t0 1523010 protocol: TCP
node_expo 9801 root 166u sock 0,7 0t0 1523040 protocol: TCP
node_expo 9801 root 167u sock 0,7 0t0 1526789 protocol: TCP
node_expo 9801 root 168u sock 0,7 0t0 1525869 protocol: TCP
node_expo 9801 root 169u sock 0,7 0t0 1524971 protocol: TCP
node_expo 9801 root 170u sock 0,7 0t0 1525005 protocol: TCP
node_expo 9801 root 171u sock 0,7 0t0 1521558 protocol: TCP
node_expo 9801 root 172u sock 0,7 0t0 1523307 protocol: TCP
node_expo 9801 root 173u sock 0,7 0t0 1521585 protocol: TCP
node_expo 9801 root 174u sock 0,7 0t0 1526871 protocol: TCP
node_expo 9801 root 175u sock 0,7 0t0 1521663 protocol: TCP
node_expo 9801 root 176u sock 0,7 0t0 1526029 protocol: TCP
node_expo 9801 root 177u sock 0,7 0t0 1525132 protocol: TCP
node_expo 9801 root 178u sock 0,7 0t0 1523369 protocol: TCP
node_expo 9801 root 179u sock 0,7 0t0 1523437 protocol: TCP
node_expo 9801 root 180u sock 0,7 0t0 1526965 protocol: TCP
node_expo 9801 root 181u sock 0,7 0t0 1526170 protocol: TCP
node_expo 9801 root 182u sock 0,7 0t0 1528036 protocol: TCP
node_expo 9801 root 183u sock 0,7 0t0 1523489 protocol: TCP
node_expo 9801 root 184u sock 0,7 0t0 1526232 protocol: TCP
node_expo 9801 root 185u sock 0,7 0t0 1523491 protocol: TCP
node_expo 9801 root 186u sock 0,7 0t0 1529005 protocol: TCP
node_expo 9801 root 187u sock 0,7 0t0 1527102 protocol: TCP
node_expo 9801 root 188u sock 0,7 0t0 1526301 protocol: TCP
node_expo 9801 root 189u sock 0,7 0t0 1528119 protocol: TCP
node_expo 9801 root 190u sock 0,7 0t0 1525389 protocol: TCP
node_expo 9801 root 191u sock 0,7 0t0 1523548 protocol: TCP
node_expo 9801 root 192u sock 0,7 0t0 1523550 protocol: TCP
node_expo 9801 root 193u sock 0,7 0t0 1527128 protocol: TCP
node_expo 9801 root 194u sock 0,7 0t0 1526507 protocol: TCP
node_expo 9801 root 195u sock 0,7 0t0 1528223 protocol: TCP
node_expo 9801 root 196u sock 0,7 0t0 1523564 protocol: TCP
node_expo 9801 root 197u sock 0,7 0t0 1526627 protocol: TCP
node_expo 9801 root 198u sock 0,7 0t0 1529151 protocol: TCP
node_expo 9801 root 199u sock 0,7 0t0 1529183 protocol: TCP
node_expo 9801 root 200u sock 0,7 0t0 1527206 protocol: TCP
node_expo 9801 root 201u sock 0,7 0t0 1526694 protocol: TCP
node_expo 9801 root 202u sock 0,7 0t0 1528362 protocol: TCP
node_expo 9801 root 203u sock 0,7 0t0 1529205 protocol: TCP
node_expo 9801 root 204u sock 0,7 0t0 1528433 protocol: TCP
node_expo 9801 root 205u sock 0,7 0t0 1523687 protocol: TCP
node_expo 9801 root 206u sock 0,7 0t0 1525620 protocol: TCP
node_expo 9801 root 207u sock 0,7 0t0 1530982 protocol: TCP
node_expo 9801 root 208u sock 0,7 0t0 1528484 protocol: TCP
node_expo 9801 root 209u sock 0,7 0t0 1528489 protocol: TCP
node_expo 9801 root 210u sock 0,7 0t0 1524409 protocol: TCP
node_expo 9801 root 211u sock 0,7 0t0 1524425 protocol: TCP
node_expo 9801 root 212u sock 0,7 0t0 1531037 protocol: TCP
node_expo 9801 root 213u sock 0,7 0t0 1527289 protocol: TCP
node_expo 9801 root 214u sock 0,7 0t0 1531065 protocol: TCP
node_expo 9801 root 215u sock 0,7 0t0 1528611 protocol: TCP
node_expo 9801 root 216u sock 0,7 0t0 1529389 protocol: TCP
node_expo 9801 root 217u sock 0,7 0t0 1528635 protocol: TCP
node_expo 9801 root 218u sock 0,7 0t0 1531946 protocol: TCP
node_expo 9801 root 219u sock 0,7 0t0 1527336 protocol: TCP
node_expo 9801 root 220u sock 0,7 0t0 1531160 protocol: TCP
node_expo 9801 root 221u sock 0,7 0t0 1528702 protocol: TCP
node_expo 9801 root 222u sock 0,7 0t0 1528713 protocol: TCP
node_expo 9801 root 223u sock 0,7 0t0 1524497 protocol: TCP
node_expo 9801 root 224u sock 0,7 0t0 1531232 protocol: TCP
node_expo 9801 root 225u sock 0,7 0t0 1534985 protocol: TCP
node_expo 9801 root 226u sock 0,7 0t0 1527374 protocol: TCP
node_expo 9801 root 227u sock 0,7 0t0 1534069 protocol: TCP
node_expo 9801 root 228u sock 0,7 0t0 1535029 protocol: TCP
node_expo 9801 root 229u sock 0,7 0t0 1529546 protocol: TCP
node_expo 9801 root 230u sock 0,7 0t0 1527481 protocol: TCP
node_expo 9801 root 231u sock 0,7 0t0 1532041 protocol: TCP
node_expo 9801 root 232u sock 0,7 0t0 1527527 protocol: TCP
node_expo 9801 root 233u sock 0,7 0t0 1531354 protocol: TCP
node_expo 9801 root 234u sock 0,7 0t0 1534154 protocol: TCP
node_expo 9801 root 235u sock 0,7 0t0 1534178 protocol: TCP
node_expo 9801 root 236u sock 0,7 0t0 1524575 protocol: TCP
node_expo 9801 root 237u sock 0,7 0t0 1535191 protocol: TCP
node_expo 9801 root 238u sock 0,7 0t0 1535197 protocol: TCP
node_expo 9801 root 239u sock 0,7 0t0 1529704 protocol: TCP
node_expo 9801 root 240u sock 0,7 0t0 1534294 protocol: TCP
node_expo 9801 root 241u sock 0,7 0t0 1531551 protocol: TCP
node_expo 9801 root 242u sock 0,7 0t0 1529766 protocol: TCP
node_expo 9801 root 243u sock 0,7 0t0 1535266 protocol: TCP
node_expo 9801 root 244u sock 0,7 0t0 1532156 protocol: TCP
node_expo 9801 root 245u sock 0,7 0t0 1531616 protocol: TCP
node_expo 9801 root 246u sock 0,7 0t0 1533324 protocol: TCP
node_expo 9801 root 247u sock 0,7 0t0 1537107 protocol: TCP
node_expo 9801 root 248u sock 0,7 0t0 1531682 protocol: TCP
node_expo 9801 root 249u sock 0,7 0t0 1536123 protocol: TCP
node_expo 9801 root 250u sock 0,7 0t0 1531738 protocol: TCP
node_expo 9801 root 251u sock 0,7 0t0 1534444 protocol: TCP
node_expo 9801 root 252u sock 0,7 0t0 1531807 protocol: TCP
node_expo 9801 root 253u sock 0,7 0t0 1534465 protocol: TCP
node_expo 9801 root 254u sock 0,7 0t0 1537251 protocol: TCP
node_expo 9801 root 255u sock 0,7 0t0 1535475 protocol: TCP
node_expo 9801 root 256u sock 0,7 0t0 1535495 protocol: TCP
node_expo 9801 root 257u sock 0,7 0t0 1532238 protocol: TCP
node_expo 9801 root 258u sock 0,7 0t0 1537311 protocol: TCP
node_expo 9801 root 259u sock 0,7 0t0 1533498 protocol: TCP
node_expo 9801 root 260u sock 0,7 0t0 1537342 protocol: TCP
node_expo 9801 root 261u sock 0,7 0t0 1535565 protocol: TCP
node_expo 9801 root 262u sock 0,7 0t0 1536250 protocol: TCP
node_expo 9801 root 263u sock 0,7 0t0 1535596 protocol: TCP
node_expo 9801 root 264u sock 0,7 0t0 1534623 protocol: TCP
node_expo 9801 root 265u sock 0,7 0t0 1539196 protocol: TCP
node_expo 9801 root 266u sock 0,7 0t0 1537409 protocol: TCP
node_expo 9801 root 267u sock 0,7 0t0 1537439 protocol: TCP
node_expo 9801 root 268u sock 0,7 0t0 1537467 protocol: TCP
node_expo 9801 root 269u sock 0,7 0t0 1534734 protocol: TCP
node_expo 9801 root 270u sock 0,7 0t0 1534740 protocol: TCP
node_expo 9801 root 271u sock 0,7 0t0 1536369 protocol: TCP
node_expo 9801 root 272u sock 0,7 0t0 1533737 protocol: TCP
node_expo 9801 root 273u sock 0,7 0t0 1537566 protocol: TCP
node_expo 9801 root 274u sock 0,7 0t0 1537576 protocol: TCP
node_expo 9801 root 275u sock 0,7 0t0 1536409 protocol: TCP
node_expo 9801 root 276u sock 0,7 0t0 1535764 protocol: TCP
node_expo 9801 root 277u sock 0,7 0t0 1536455 protocol: TCP
node_expo 9801 root 278u sock 0,7 0t0 1537686 protocol: TCP
node_expo 9801 root 279u sock 0,7 0t0 1537732 protocol: TCP
node_expo 9801 root 280u sock 0,7 0t0 1537739 protocol: TCP
node_expo 9801 root 281u sock 0,7 0t0 1537751 protocol: TCP
node_expo 9801 root 282u sock 0,7 0t0 1537757 protocol: TCP
node_expo 9801 root 283u sock 0,7 0t0 1537804 protocol: TCP
node_expo 9801 root 284u sock 0,7 0t0 1536719 protocol: TCP
node_expo 9801 root 285u sock 0,7 0t0 1533813 protocol: TCP
node_expo 9801 root 286u sock 0,7 0t0 1537823 protocol: TCP
node_expo 9801 root 287u sock 0,7 0t0 1537836 protocol: TCP
node_expo 9801 root 288u sock 0,7 0t0 1536832 protocol: TCP
node_expo 9801 root 289u sock 0,7 0t0 1535929 protocol: TCP
node_expo 9801 root 290u sock 0,7 0t0 1536878 protocol: TCP
node_expo 9801 root 291u sock 0,7 0t0 1537895 protocol: TCP
node_expo 9801 root 292u sock 0,7 0t0 1537918 protocol: TCP
node_expo 9801 root 293u sock 0,7 0t0 1538319 protocol: TCP
node_expo 9801 root 294u sock 0,7 0t0 1537991 protocol: TCP
node_expo 9801 root 295u sock 0,7 0t0 1541131 protocol: TCP
node_expo 9801 root 296u sock 0,7 0t0 1538042 protocol: TCP
node_expo 9801 root 297u sock 0,7 0t0 1539843 protocol: TCP
node_expo 9801 root 298u sock 0,7 0t0 1533896 protocol: TCP
node_expo 9801 root 299u sock 0,7 0t0 1539917 protocol: TCP
node_expo 9801 root 300u sock 0,7 0t0 1539958 protocol: TCP
node_expo 9801 root 301u sock 0,7 0t0 1543176 protocol: TCP
node_expo 9801 root 302u sock 0,7 0t0 1540491 protocol: TCP
node_expo 9801 root 303u sock 0,7 0t0 1543190 protocol: TCP
node_expo 9801 root 304u sock 0,7 0t0 1540048 protocol: TCP
node_expo 9801 root 305u sock 0,7 0t0 1543204 protocol: TCP
node_expo 9801 root 306u sock 0,7 0t0 1544349 protocol: TCP
node_expo 9801 root 307u sock 0,7 0t0 1542322 protocol: TCP
node_expo 9801 root 308u sock 0,7 0t0 1544428 protocol: TCP
node_expo 9801 root 309u sock 0,7 0t0 1542336 protocol: TCP
node_expo 9801 root 310u sock 0,7 0t0 1545251 protocol: TCP
node_expo 9801 root 311u sock 0,7 0t0 1542356 protocol: TCP
node_expo 9801 root 312u sock 0,7 0t0 1545294 protocol: TCP
node_expo 9801 root 313u sock 0,7 0t0 1545300 protocol: TCP
node_expo 9801 root 314u sock 0,7 0t0 1540680 protocol: TCP
node_expo 9801 root 315u sock 0,7 0t0 1540719 protocol: TCP
node_expo 9801 root 316u sock 0,7 0t0 1543273 protocol: TCP
node_expo 9801 root 317u sock 0,7 0t0 1545390 protocol: TCP
node_expo 9801 root 318u sock 0,7 0t0 1543275 protocol: TCP
node_expo 9801 root 319u sock 0,7 0t0 1542442 protocol: TCP
node_expo 9801 root 320u sock 0,7 0t0 1542450 protocol: TCP
node_expo 9801 root 321u sock 0,7 0t0 1544636 protocol: TCP
node_expo 9801 root 322u sock 0,7 0t0 1542481 protocol: TCP
node_expo 9801 root 323u sock 0,7 0t0 1545545 protocol: TCP
node_expo 9801 root 324u sock 0,7 0t0 1532740 protocol: TCP
node_expo 9801 root 325u sock 0,7 0t0 1542517 protocol: TCP
node_expo 9801 root 326u sock 0,7 0t0 1542525 protocol: TCP
node_expo 9801 root 327u sock 0,7 0t0 1540898 protocol: TCP
node_expo 9801 root 328u sock 0,7 0t0 1542554 protocol: TCP
node_expo 9801 root 329u sock 0,7 0t0 1543377 protocol: TCP
node_expo 9801 root 330u sock 0,7 0t0 1545662 protocol: TCP
node_expo 9801 root 331u sock 0,7 0t0 1543387 protocol: TCP
node_expo 9801 root 332u sock 0,7 0t0 1541558 protocol: TCP
node_expo 9801 root 333u sock 0,7 0t0 1545711 protocol: TCP
node_expo 9801 root 334u sock 0,7 0t0 1544799 protocol: TCP
node_expo 9801 root 335u sock 0,7 0t0 1542765 protocol: TCP
node_expo 9801 root 336u sock 0,7 0t0 1541102 protocol: TCP
node_expo 9801 root 337u sock 0,7 0t0 1541646 protocol: TCP
node_expo 9801 root 338u sock 0,7 0t0 1545793 protocol: TCP
node_expo 9801 root 339u sock 0,7 0t0 1545830 protocol: TCP
node_expo 9801 root 340u sock 0,7 0t0 1546288 protocol: TCP
node_expo 9801 root 341u sock 0,7 0t0 1541688 protocol: TCP
node_expo 9801 root 342u sock 0,7 0t0 1543468 protocol: TCP
node_expo 9801 root 343u sock 0,7 0t0 1545900 protocol: TCP
node_expo 9801 root 344u sock 0,7 0t0 1543494 protocol: TCP
node_expo 9801 root 345u sock 0,7 0t0 1546384 protocol: TCP
node_expo 9801 root 346u sock 0,7 0t0 1545936 protocol: TCP
node_expo 9801 root 347u sock 0,7 0t0 1543013 protocol: TCP
node_expo 9801 root 348u sock 0,7 0t0 1543041 protocol: TCP
node_expo 9801 root 349u sock 0,7 0t0 1546441 protocol: TCP
node_expo 9801 root 350u sock 0,7 0t0 1546446 protocol: TCP
node_expo 9801 root 351u sock 0,7 0t0 1541820 protocol: TCP
node_expo 9801 root 352u sock 0,7 0t0 1546114 protocol: TCP
node_expo 9801 root 353u sock 0,7 0t0 1546476 protocol: TCP
node_expo 9801 root 354u sock 0,7 0t0 1546134 protocol: TCP
node_expo 9801 root 355u sock 0,7 0t0 1543590 protocol: TCP
node_expo 9801 root 356u sock 0,7 0t0 1546160 protocol: TCP
node_expo 9801 root 357u sock 0,7 0t0 1543610 protocol: TCP
node_expo 9801 root 358u sock 0,7 0t0 1543612 protocol: TCP
node_expo 9801 root 359u sock 0,7 0t0 1550366 protocol: TCP
node_expo 9801 root 360u sock 0,7 0t0 1550376 protocol: TCP
node_expo 9801 root 361u sock 0,7 0t0 1541919 protocol: TCP
node_expo 9801 root 362u sock 0,7 0t0 1545090 protocol: TCP
node_expo 9801 root 363u sock 0,7 0t0 1549423 protocol: TCP
node_expo 9801 root 364u sock 0,7 0t0 1550438 protocol: TCP
node_expo 9801 root 365u sock 0,7 0t0 1545116 protocol: TCP
node_expo 9801 root 366u sock 0,7 0t0 1542001 protocol: TCP
node_expo 9801 root 367u sock 0,7 0t0 1549508 protocol: TCP
node_expo 9801 root 368u sock 0,7 0t0 1543689 protocol: TCP
node_expo 9801 root 369u sock 0,7 0t0 1547567 protocol: TCP
node_expo 9801 root 370u sock 0,7 0t0 1543699 protocol: TCP
node_expo 9801 root 371u sock 0,7 0t0 1543727 protocol: TCP
node_expo 9801 root 372u sock 0,7 0t0 1542074 protocol: TCP
node_expo 9801 root 373u sock 0,7 0t0 1542080 protocol: TCP
node_expo 9801 root 374u sock 0,7 0t0 1550625 protocol: TCP
node_expo 9801 root 375u sock 0,7 0t0 1551413 protocol: TCP
node_expo 9801 root 376u sock 0,7 0t0 1549648 protocol: TCP
node_expo 9801 root 377u sock 0,7 0t0 1550689 protocol: TCP
node_expo 9801 root 378u sock 0,7 0t0 1551426 protocol: TCP
node_expo 9801 root 379u sock 0,7 0t0 1542106 protocol: TCP
node_expo 9801 root 380u sock 0,7 0t0 1549719 protocol: TCP
node_expo 9801 root 381u sock 0,7 0t0 1543816 protocol: TCP
node_expo 9801 root 382u sock 0,7 0t0 1547744 protocol: TCP
node_expo 9801 root 383u sock 0,7 0t0 1547756 protocol: TCP
node_expo 9801 root 384u sock 0,7 0t0 1543861 protocol: TCP
node_expo 9801 root 385u sock 0,7 0t0 1547056 protocol: TCP
node_expo 9801 root 386u sock 0,7 0t0 1547099 protocol: TCP
node_expo 9801 root 387u sock 0,7 0t0 1550884 protocol: TCP
node_expo 9801 root 388u sock 0,7 0t0 1549891 protocol: TCP
node_expo 9801 root 389u sock 0,7 0t0 1549899 protocol: TCP
node_expo 9801 root 390u sock 0,7 0t0 1543903 protocol: TCP
node_expo 9801 root 391u sock 0,7 0t0 1551656 protocol: TCP
node_expo 9801 root 392u sock 0,7 0t0 1543922 protocol: TCP
node_expo 9801 root 393u sock 0,7 0t0 1551019 protocol: TCP
node_expo 9801 root 394u sock 0,7 0t0 1551027 protocol: TCP
node_expo 9801 root 395u sock 0,7 0t0 1547858 protocol: TCP
node_expo 9801 root 396u sock 0,7 0t0 1547872 protocol: TCP
node_expo 9801 root 397u sock 0,7 0t0 1543999 protocol: TCP
node_expo 9801 root 398u sock 0,7 0t0 1547240 protocol: TCP
node_expo 9801 root 399u sock 0,7 0t0 1551167 protocol: TCP
node_expo 9801 root 400u sock 0,7 0t0 1551175 protocol: TCP
node_expo 9801 root 401u sock 0,7 0t0 1551720 protocol: TCP
node_expo 9801 root 402u sock 0,7 0t0 1550111 protocol: TCP
node_expo 9801 root 403u sock 0,7 0t0 1544039 protocol: TCP
node_expo 9801 root 404u sock 0,7 0t0 1551737 protocol: TCP
node_expo 9801 root 405u sock 0,7 0t0 1544043 protocol: TCP
node_expo 9801 root 406u sock 0,7 0t0 1552844 protocol: TCP
node_expo 9801 root 407u sock 0,7 0t0 1550218 protocol: TCP
node_expo 9801 root 408u sock 0,7 0t0 1547954 protocol: TCP
node_expo 9801 root 409u sock 0,7 0t0 1547963 protocol: TCP
node_expo 9801 root 410u sock 0,7 0t0 1553645 protocol: TCP
node_expo 9801 root 411u sock 0,7 0t0 1553685 protocol: TCP
node_expo 9801 root 412u sock 0,7 0t0 1553691 protocol: TCP
node_expo 9801 root 413u sock 0,7 0t0 1552951 protocol: TCP
node_expo 9801 root 414u sock 0,7 0t0 1551809 protocol: TCP
node_expo 9801 root 415u sock 0,7 0t0 1556539 protocol: TCP
node_expo 9801 root 416u sock 0,7 0t0 1557601 protocol: TCP
node_expo 9801 root 417u sock 0,7 0t0 1551822 protocol: TCP
node_expo 9801 root 418u sock 0,7 0t0 1557605 protocol: TCP
node_expo 9801 root 419u sock 0,7 0t0 1556584 protocol: TCP
node_expo 9801 root 420u sock 0,7 0t0 1556601 protocol: TCP
node_expo 9801 root 421u sock 0,7 0t0 1555757 protocol: TCP
node_expo 9801 root 422u sock 0,7 0t0 1555786 protocol: TCP
node_expo 9801 root 423u sock 0,7 0t0 1553897 protocol: TCP
node_expo 9801 root 424u sock 0,7 0t0 1556658 protocol: TCP
node_expo 9801 root 425u sock 0,7 0t0 1557691 protocol: TCP
node_expo 9801 root 426u sock 0,7 0t0 1553126 protocol: TCP
node_expo 9801 root 427u sock 0,7 0t0 1551908 protocol: TCP
node_expo 9801 root 428u sock 0,7 0t0 1556760 protocol: TCP
node_expo 9801 root 429u sock 0,7 0t0 1557709 protocol: TCP
node_expo 9801 root 430u sock 0,7 0t0 1551957 protocol: TCP
node_expo 9801 root 431u sock 0,7 0t0 1557732 protocol: TCP
node_expo 9801 root 432u sock 0,7 0t0 1556840 protocol: TCP
node_expo 9801 root 433u sock 0,7 0t0 1554969 protocol: TCP
node_expo 9801 root 434u sock 0,7 0t0 1555842 protocol: TCP
node_expo 9801 root 435u sock 0,7 0t0 1555864 protocol: TCP
node_expo 9801 root 436u sock 0,7 0t0 1558703 protocol: TCP
node_expo 9801 root 437u sock 0,7 0t0 1556925 protocol: TCP
node_expo 9801 root 438u sock 0,7 0t0 1557805 protocol: TCP
node_expo 9801 root 439u sock 0,7 0t0 1554229 protocol: TCP
node_expo 9801 root 440u sock 0,7 0t0 1556974 protocol: TCP
node_expo 9801 root 441u sock 0,7 0t0 1556988 protocol: TCP
node_expo 9801 root 442u sock 0,7 0t0 1557908 protocol: TCP
node_expo 9801 root 443u sock 0,7 0t0 1552206 protocol: TCP
node_expo 9801 root 444u sock 0,7 0t0 1558012 protocol: TCP
node_expo 9801 root 445u sock 0,7 0t0 1555189 protocol: TCP
node_expo 9801 root 446u sock 0,7 0t0 1555198 protocol: TCP
node_expo 9801 root 447u sock 0,7 0t0 1556004 protocol: TCP
node_expo 9801 root 448u sock 0,7 0t0 1556006 protocol: TCP
node_expo 9801 root 449u sock 0,7 0t0 1558908 protocol: TCP
node_expo 9801 root 450u sock 0,7 0t0 1557096 protocol: TCP
node_expo 9801 root 451u sock 0,7 0t0 1558214 protocol: TCP
node_expo 9801 root 452u sock 0,7 0t0 1558218 protocol: TCP
node_expo 9801 root 453u sock 0,7 0t0 1557139 protocol: TCP
node_expo 9801 root 454u sock 0,7 0t0 1558229 protocol: TCP
node_expo 9801 root 455u sock 0,7 0t0 1556085 protocol: TCP
node_expo 9801 root 456u sock 0,7 0t0 1552291 protocol: TCP
node_expo 9801 root 457u sock 0,7 0t0 1558265 protocol: TCP
node_expo 9801 root 458u sock 0,7 0t0 1556096 protocol: TCP
node_expo 9801 root 459u sock 0,7 0t0 1561638 protocol: TCP
node_expo 9801 root 460u sock 0,7 0t0 1556137 protocol: TCP
node_expo 9801 root 461u sock 0,7 0t0 1556145 protocol: TCP
node_expo 9801 root 462u sock 0,7 0t0 1558964 protocol: TCP
node_expo 9801 root 463u sock 0,7 0t0 1552354 protocol: TCP
node_expo 9801 root 464u sock 0,7 0t0 1552356 protocol: TCP
node_expo 9801 root 465u sock 0,7 0t0 1561696 protocol: TCP
node_expo 9801 root 466u sock 0,7 0t0 1557334 protocol: TCP
node_expo 9801 root 467u sock 0,7 0t0 1559829 protocol: TCP
node_expo 9801 root 468u sock 0,7 0t0 1556214 protocol: TCP
node_expo 9801 root 469u sock 0,7 0t0 1562640 protocol: TCP
node_expo 9801 root 470u sock 0,7 0t0 1560956 protocol: TCP
node_expo 9801 root 471u sock 0,7 0t0 1556256 protocol: TCP
node_expo 9801 root 472u sock 0,7 0t0 1558522 protocol: TCP
node_expo 9801 root 473u sock 0,7 0t0 1556261 protocol: TCP
node_expo 9801 root 474u sock 0,7 0t0 1556263 protocol: TCP
node_expo 9801 root 475u sock 0,7 0t0 1559204 protocol: TCP
node_expo 9801 root 476u sock 0,7 0t0 1562771 protocol: TCP
node_expo 9801 root 477u sock 0,7 0t0 1562805 protocol: TCP
node_expo 9801 root 478u sock 0,7 0t0 1561114 protocol: TCP
node_expo 9801 root 479u sock 0,7 0t0 1564694 protocol: TCP
node_expo 9801 root 480u sock 0,7 0t0 1564698 protocol: TCP
node_expo 9801 root 481u sock 0,7 0t0 1556333 protocol: TCP
node_expo 9801 root 482u sock 0,7 0t0 1561191 protocol: TCP
node_expo 9801 root 483u sock 0,7 0t0 1564747 protocol: TCP
node_expo 9801 root 484u sock 0,7 0t0 1556393 protocol: TCP
node_expo 9801 root 485u sock 0,7 0t0 1563846 protocol: TCP
node_expo 9801 root 486u sock 0,7 0t0 1556405 protocol: TCP
node_expo 9801 root 487u sock 0,7 0t0 1556477 protocol: TCP
node_expo 9801 root 488u sock 0,7 0t0 1560473 protocol: TCP
node_expo 9801 root 489u sock 0,7 0t0 1563030 protocol: TCP
node_expo 9801 root 490u sock 0,7 0t0 1563037 protocol: TCP
node_expo 9801 root 491u sock 0,7 0t0 1563957 protocol: TCP
node_expo 9801 root 492u sock 0,7 0t0 1564957 protocol: TCP
node_expo 9801 root 493u sock 0,7 0t0 1560549 protocol: TCP
node_expo 9801 root 494u sock 0,7 0t0 1565918 protocol: TCP
node_expo 9801 root 495u sock 0,7 0t0 1562042 protocol: TCP
node_expo 9801 root 496u sock 0,7 0t0 1566736 protocol: TCP
node_expo 9801 root 497u sock 0,7 0t0 1565962 protocol: TCP
node_expo 9801 root 498u sock 0,7 0t0 1564150 protocol: TCP
node_expo 9801 root 499u sock 0,7 0t0 1566018 protocol: TCP
node_expo 9801 root 500u sock 0,7 0t0 1563094 protocol: TCP
node_expo 9801 root 501u sock 0,7 0t0 1566837 protocol: TCP
node_expo 9801 root 502u sock 0,7 0t0 1563180 protocol: TCP
node_expo 9801 root 503u sock 0,7 0t0 1563202 protocol: TCP
node_expo 9801 root 504u sock 0,7 0t0 1564207 protocol: TCP
node_expo 9801 root 505u sock 0,7 0t0 1565139 protocol: TCP
node_expo 9801 root 506u sock 0,7 0t0 1562196 protocol: TCP
node_expo 9801 root 507u sock 0,7 0t0 1566112 protocol: TCP
node_expo 9801 root 508u sock 0,7 0t0 1562239 protocol: TCP
node_expo 9801 root 509u sock 0,7 0t0 1567781 protocol: TCP
node_expo 9801 root 510u sock 0,7 0t0 1566125 protocol: TCP
node_expo 9801 root 511u sock 0,7 0t0 1567010 protocol: TCP
node_expo 9801 root 512u sock 0,7 0t0 1566197 protocol: TCP
node_expo 9801 root 513u sock 0,7 0t0 1563355 protocol: TCP
node_expo 9801 root 514u sock 0,7 0t0 1567054 protocol: TCP
node_expo 9801 root 515u sock 0,7 0t0 1563384 protocol: TCP
node_expo 9801 root 516u sock 0,7 0t0 1563386 protocol: TCP
node_expo 9801 root 517u sock 0,7 0t0 1564287 protocol: TCP
node_expo 9801 root 518u sock 0,7 0t0 1565361 protocol: TCP
node_expo 9801 root 519u sock 0,7 0t0 1568028 protocol: TCP
node_expo 9801 root 520u sock 0,7 0t0 1566283 protocol: TCP
node_expo 9801 root 521u sock 0,7 0t0 1559504 protocol: TCP
node_expo 9801 root 522u sock 0,7 0t0 1568060 protocol: TCP
node_expo 9801 root 523u sock 0,7 0t0 1566354 protocol: TCP
node_expo 9801 root 524u sock 0,7 0t0 1562444 protocol: TCP
node_expo 9801 root 525u sock 0,7 0t0 1566358 protocol: TCP
node_expo 9801 root 526u sock 0,7 0t0 1563482 protocol: TCP
node_expo 9801 root 527u sock 0,7 0t0 1563495 protocol: TCP
node_expo 9801 root 528u sock 0,7 0t0 1563498 protocol: TCP
node_expo 9801 root 529u sock 0,7 0t0 1563505 protocol: TCP
node_expo 9801 root 530u sock 0,7 0t0 1564349 protocol: TCP
node_expo 9801 root 531u sock 0,7 0t0 1565643 protocol: TCP
node_expo 9801 root 532u sock 0,7 0t0 1568284 protocol: TCP
node_expo 9801 root 533u sock 0,7 0t0 1566442 protocol: TCP
node_expo 9801 root 534u sock 0,7 0t0 1567672 protocol: TCP
node_expo 9801 root 535u sock 0,7 0t0 1568316 protocol: TCP
node_expo 9801 root 536u sock 0,7 0t0 1566490 protocol: TCP
node_expo 9801 root 537u sock 0,7 0t0 1566497 protocol: TCP
node_expo 9801 root 538u sock 0,7 0t0 1566535 protocol: TCP
node_expo 9801 root 539u sock 0,7 0t0 1563587 protocol: TCP
node_expo 9801 root 540u sock 0,7 0t0 1563602 protocol: TCP
node_expo 9801 root 541u sock 0,7 0t0 1563605 protocol: TCP
node_expo 9801 root 542u sock 0,7 0t0 1563609 protocol: TCP
node_expo 9801 root 543u sock 0,7 0t0 1564496 protocol: TCP
node_expo 9801 root 544u sock 0,7 0t0 1569985 protocol: TCP
node_expo 9801 root 545u sock 0,7 0t0 1564512 protocol: TCP
node_expo 9801 root 546u sock 0,7 0t0 1573905 protocol: TCP
node_expo 9801 root 547u sock 0,7 0t0 1571003 protocol: TCP
node_expo 9801 root 548u sock 0,7 0t0 1570029 protocol: TCP
node_expo 9801 root 549u sock 0,7 0t0 1573989 protocol: TCP
node_expo 9801 root 550u sock 0,7 0t0 1573998 protocol: TCP
node_expo 9801 root 551u sock 0,7 0t0 1571931 protocol: TCP
node_expo 9801 root 552u sock 0,7 0t0 1572939 protocol: TCP
node_expo 9801 root 553u sock 0,7 0t0 1572971 protocol: TCP
node_expo 9801 root 554u sock 0,7 0t0 1572986 protocol: TCP
node_expo 9801 root 555u sock 0,7 0t0 1572998 protocol: TCP
node_expo 9801 root 556u sock 0,7 0t0 1564626 protocol: TCP
node_expo 9801 root 557u sock 0,7 0t0 1570283 protocol: TCP
node_expo 9801 root 558u sock 0,7 0t0 1564629 protocol: TCP
node_expo 9801 root 559u sock 0,7 0t0 1568741 protocol: TCP
node_expo 9801 root 560u sock 0,7 0t0 1571204 protocol: TCP
node_expo 9801 root 561u sock 0,7 0t0 1570393 protocol: TCP
node_expo 9801 root 562u sock 0,7 0t0 1574076 protocol: TCP
node_expo 9801 root 563u sock 0,7 0t0 1574096 protocol: TCP
node_expo 9801 root 564u sock 0,7 0t0 1564666 protocol: TCP
node_expo 9801 root 565u sock 0,7 0t0 1573198 protocol: TCP
node_expo 9801 root 566u sock 0,7 0t0 1573202 protocol: TCP
node_expo 9801 root 567u sock 0,7 0t0 1573227 protocol: TCP
node_expo 9801 root 568u sock 0,7 0t0 1572106 protocol: TCP
node_expo 9801 root 569u sock 0,7 0t0 1576085 protocol: TCP
node_expo 9801 root 570u sock 0,7 0t0 1571456 protocol: TCP
node_expo 9801 root 571u sock 0,7 0t0 1572148 protocol: TCP
node_expo 9801 root 572u sock 0,7 0t0 1572153 protocol: TCP
node_expo 9801 root 573u sock 0,7 0t0 1575056 protocol: TCP
node_expo 9801 root 574u sock 0,7 0t0 1570578 protocol: TCP
node_expo 9801 root 575u sock 0,7 0t0 1574223 protocol: TCP
node_expo 9801 root 576u sock 0,7 0t0 1574227 protocol: TCP
node_expo 9801 root 577u sock 0,7 0t0 1576321 protocol: TCP
node_expo 9801 root 578u sock 0,7 0t0 1573328 protocol: TCP
node_expo 9801 root 579u sock 0,7 0t0 1573346 protocol: TCP
node_expo 9801 root 580u sock 0,7 0t0 1570680 protocol: TCP
node_expo 9801 root 581u sock 0,7 0t0 1573357 protocol: TCP
node_expo 9801 root 582u sock 0,7 0t0 1576487 protocol: TCP
node_expo 9801 root 583u sock 0,7 0t0 1571647 protocol: TCP
node_expo 9801 root 584u sock 0,7 0t0 1575263 protocol: TCP
node_expo 9801 root 585u sock 0,7 0t0 1572324 protocol: TCP
node_expo 9801 root 586u sock 0,7 0t0 1575276 protocol: TCP
node_expo 9801 root 587u sock 0,7 0t0 1571728 protocol: TCP
node_expo 9801 root 588u sock 0,7 0t0 1572402 protocol: TCP
node_expo 9801 root 589u sock 0,7 0t0 1574384 protocol: TCP
node_expo 9801 root 590u sock 0,7 0t0 1571806 protocol: TCP
node_expo 9801 root 591u sock 0,7 0t0 1573542 protocol: TCP
node_expo 9801 root 592u sock 0,7 0t0 1573545 protocol: TCP
node_expo 9801 root 593u sock 0,7 0t0 1572565 protocol: TCP
node_expo 9801 root 594u sock 0,7 0t0 1573550 protocol: TCP
node_expo 9801 root 595u sock 0,7 0t0 1578057 protocol: TCP
node_expo 9801 root 596u sock 0,7 0t0 1575411 protocol: TCP
node_expo 9801 root 597u sock 0,7 0t0 1572639 protocol: TCP
node_expo 9801 root 598u sock 0,7 0t0 1575426 protocol: TCP
node_expo 9801 root 599u sock 0,7 0t0 1576810 protocol: TCP
node_expo 9801 root 600u sock 0,7 0t0 1578140 protocol: TCP
node_expo 9801 root 601u sock 0,7 0t0 1572729 protocol: TCP
node_expo 9801 root 602u sock 0,7 0t0 1574483 protocol: TCP
node_expo 9801 root 603u sock 0,7 0t0 1578192 protocol: TCP
node_expo 9801 root 604u sock 0,7 0t0 1577101 protocol: TCP
node_expo 9801 root 605u sock 0,7 0t0 1575521 protocol: TCP
node_expo 9801 root 606u sock 0,7 0t0 1579017 protocol: TCP
node_expo 9801 root 607u sock 0,7 0t0 1579040 protocol: TCP
node_expo 9801 root 608u sock 0,7 0t0 1578300 protocol: TCP
node_expo 9801 root 609u sock 0,7 0t0 1575673 protocol: TCP
node_expo 9801 root 610u sock 0,7 0t0 1574545 protocol: TCP
node_expo 9801 root 611u sock 0,7 0t0 1575734 protocol: TCP
node_expo 9801 root 612u sock 0,7 0t0 1576927 protocol: TCP
node_expo 9801 root 613u sock 0,7 0t0 1578409 protocol: TCP
node_expo 9801 root 614u sock 0,7 0t0 1580070 protocol: TCP
node_expo 9801 root 615u sock 0,7 0t0 1574582 protocol: TCP
node_expo 9801 root 616u sock 0,7 0t0 1578442 protocol: TCP
node_expo 9801 root 617u sock 0,7 0t0 1574603 protocol: TCP
node_expo 9801 root 618u sock 0,7 0t0 1575850 protocol: TCP
node_expo 9801 root 619u sock 0,7 0t0 1579151 protocol: TCP
node_expo 9801 root 620u sock 0,7 0t0 1579170 protocol: TCP
node_expo 9801 root 621u sock 0,7 0t0 1578551 protocol: TCP
node_expo 9801 root 622u sock 0,7 0t0 1575904 protocol: TCP
node_expo 9801 root 623u sock 0,7 0t0 1574669 protocol: TCP
node_expo 9801 root 624u sock 0,7 0t0 1580277 protocol: TCP
node_expo 9801 root 625u sock 0,7 0t0 1579234 protocol: TCP
node_expo 9801 root 626u sock 0,7 0t0 1580313 protocol: TCP
node_expo 9801 root 627u sock 0,7 0t0 1580369 protocol: TCP
node_expo 9801 root 628u sock 0,7 0t0 1574700 protocol: TCP
node_expo 9801 root 629u sock 0,7 0t0 1569753 protocol: TCP
node_expo 9801 root 630u sock 0,7 0t0 1574704 protocol: TCP
node_expo 9801 root 631u sock 0,7 0t0 1582269 protocol: TCP
node_expo 9801 root 632u sock 0,7 0t0 1579346 protocol: TCP
node_expo 9801 root 633u sock 0,7 0t0 1579350 protocol: TCP
node_expo 9801 root 634u sock 0,7 0t0 1580458 protocol: TCP
node_expo 9801 root 635u sock 0,7 0t0 1582351 protocol: TCP
node_expo 9801 root 636u sock 0,7 0t0 1574835 protocol: TCP
node_expo 9801 root 637u sock 0,7 0t0 1580539 protocol: TCP
node_expo 9801 root 638u sock 0,7 0t0 1579407 protocol: TCP
node_expo 9801 root 639u sock 0,7 0t0 1580585 protocol: TCP
node_expo 9801 root 640u sock 0,7 0t0 1580597 protocol: TCP
node_expo 9801 root 641u sock 0,7 0t0 1574869 protocol: TCP
node_expo 9801 root 642u sock 0,7 0t0 1583235 protocol: TCP
node_expo 9801 root 643u sock 0,7 0t0 1585192 protocol: TCP
node_expo 9801 root 644u sock 0,7 0t0 1582542 protocol: TCP
node_expo 9801 root 645u sock 0,7 0t0 1579487 protocol: TCP
node_expo 9801 root 646u sock 0,7 0t0 1577614 protocol: TCP
node_expo 9801 root 647u sock 0,7 0t0 1582592 protocol: TCP
node_expo 9801 root 648u sock 0,7 0t0 1582605 protocol: TCP
node_expo 9801 root 649u sock 0,7 0t0 1585353 protocol: TCP
node_expo 9801 root 650u sock 0,7 0t0 1583294 protocol: TCP
node_expo 9801 root 651u sock 0,7 0t0 1579532 protocol: TCP
node_expo 9801 root 652u sock 0,7 0t0 1580764 protocol: TCP
node_expo 9801 root 653u sock 0,7 0t0 1582651 protocol: TCP
node_expo 9801 root 654u sock 0,7 0t0 1580780 protocol: TCP
node_expo 9801 root 655u sock 0,7 0t0 1577696 protocol: TCP
node_expo 9801 root 656u sock 0,7 0t0 1577700 protocol: TCP
node_expo 9801 root 657u sock 0,7 0t0 1582824 protocol: TCP
node_expo 9801 root 658u sock 0,7 0t0 1579587 protocol: TCP
node_expo 9801 root 659u sock 0,7 0t0 1580880 protocol: TCP
node_expo 9801 root 660u sock 0,7 0t0 1582881 protocol: TCP
node_expo 9801 root 661u sock 0,7 0t0 1577759 protocol: TCP
node_expo 9801 root 662u sock 0,7 0t0 1585785 protocol: TCP
node_expo 9801 root 663u sock 0,7 0t0 1584521 protocol: TCP
node_expo 9801 root 664u sock 0,7 0t0 1579663 protocol: TCP
node_expo 9801 root 665u sock 0,7 0t0 1582936 protocol: TCP
node_expo 9801 root 666u sock 0,7 0t0 1582944 protocol: TCP
node_expo 9801 root 667u sock 0,7 0t0 1581540 protocol: TCP
node_expo 9801 root 668u sock 0,7 0t0 1577868 protocol: TCP
node_expo 9801 root 669u sock 0,7 0t0 1577871 protocol: TCP
node_expo 9801 root 670u sock 0,7 0t0 1582987 protocol: TCP
node_expo 9801 root 671u sock 0,7 0t0 1586106 protocol: TCP
node_expo 9801 root 672u sock 0,7 0t0 1586202 protocol: TCP
node_expo 9801 root 673u sock 0,7 0t0 1583477 protocol: TCP
node_expo 9801 root 674u sock 0,7 0t0 1583483 protocol: TCP
node_expo 9801 root 675u sock 0,7 0t0 1586127 protocol: TCP
node_expo 9801 root 676u sock 0,7 0t0 1584746 protocol: TCP
node_expo 9801 root 677u sock 0,7 0t0 1579757 protocol: TCP
node_expo 9801 root 678u sock 0,7 0t0 1587272 protocol: TCP
node_expo 9801 root 679u sock 0,7 0t0 1587290 protocol: TCP
node_expo 9801 root 680u sock 0,7 0t0 1588363 protocol: TCP
node_expo 9801 root 681u sock 0,7 0t0 1583571 protocol: TCP
node_expo 9801 root 682u sock 0,7 0t0 1584858 protocol: TCP
node_expo 9801 root 683u sock 0,7 0t0 1589258 protocol: TCP
node_expo 9801 root 684u sock 0,7 0t0 1586408 protocol: TCP
node_expo 9801 root 685u sock 0,7 0t0 1579815 protocol: TCP
node_expo 9801 root 686u sock 0,7 0t0 1587346 protocol: TCP
node_expo 9801 root 687u sock 0,7 0t0 1583688 protocol: TCP
node_expo 9801 root 688u sock 0,7 0t0 1588543 protocol: TCP
node_expo 9801 root 689u sock 0,7 0t0 1583735 protocol: TCP
node_expo 9801 root 690u sock 0,7 0t0 1579831 protocol: TCP
node_expo 9801 root 691u sock 0,7 0t0 1587419 protocol: TCP
node_expo 9801 root 692u sock 0,7 0t0 1589332 protocol: TCP
node_expo 9801 root 693u sock 0,7 0t0 1588586 protocol: TCP
node_expo 9801 root 694u sock 0,7 0t0 1583803 protocol: TCP
node_expo 9801 root 695u sock 0,7 0t0 1590364 protocol: TCP
node_expo 9801 root 696u sock 0,7 0t0 1590378 protocol: TCP
node_expo 9801 root 697u sock 0,7 0t0 1586735 protocol: TCP
node_expo 9801 root 698u sock 0,7 0t0 1587500 protocol: TCP
node_expo 9801 root 699u sock 0,7 0t0 1587526 protocol: TCP
node_expo 9801 root 700u sock 0,7 0t0 1583906 protocol: TCP
node_expo 9801 root 701u sock 0,7 0t0 1586784 protocol: TCP
node_expo 9801 root 702u sock 0,7 0t0 1583908 protocol: TCP
node_expo 9801 root 703u sock 0,7 0t0 1589472 protocol: TCP
node_expo 9801 root 704u sock 0,7 0t0 1587579 protocol: TCP
node_expo 9801 root 705u sock 0,7 0t0 1587591 protocol: TCP
node_expo 9801 root 706u sock 0,7 0t0 1588832 protocol: TCP
node_expo 9801 root 707u sock 0,7 0t0 1589495 protocol: TCP
node_expo 9801 root 708u sock 0,7 0t0 1586970 protocol: TCP
node_expo 9801 root 709u sock 0,7 0t0 1587025 protocol: TCP
node_expo 9801 root 710u sock 0,7 0t0 1587041 protocol: TCP
node_expo 9801 root 711u sock 0,7 0t0 1581861 protocol: TCP
node_expo 9801 root 712u sock 0,7 0t0 1587722 protocol: TCP
node_expo 9801 root 713u sock 0,7 0t0 1584053 protocol: TCP
node_expo 9801 root 714u sock 0,7 0t0 1587125 protocol: TCP
node_expo 9801 root 715u sock 0,7 0t0 1584086 protocol: TCP
node_expo 9801 root 716u sock 0,7 0t0 1592327 protocol: TCP
node_expo 9801 root 717u sock 0,7 0t0 1592338 protocol: TCP
node_expo 9801 root 718u sock 0,7 0t0 1587823 protocol: TCP
node_expo 9801 root 719u sock 0,7 0t0 1588979 protocol: TCP
node_expo 9801 root 720u sock 0,7 0t0 1581981 protocol: TCP
node_expo 9801 root 721u sock 0,7 0t0 1592472 protocol: TCP
node_expo 9801 root 722u sock 0,7 0t0 1592490 protocol: TCP
node_expo 9801 root 723u sock 0,7 0t0 1589736 protocol: TCP
node_expo 9801 root 724u sock 0,7 0t0 1581992 protocol: TCP
node_expo 9801 root 725u sock 0,7 0t0 1587963 protocol: TCP
node_expo 9801 root 726u sock 0,7 0t0 1587968 protocol: TCP
node_expo 9801 root 727u sock 0,7 0t0 1592577 protocol: TCP
node_expo 9801 root 728u sock 0,7 0t0 1593440 protocol: TCP
node_expo 9801 root 729u sock 0,7 0t0 1591022 protocol: TCP
node_expo 9801 root 730u sock 0,7 0t0 1592626 protocol: TCP
node_expo 9801 root 731u sock 0,7 0t0 1592660 protocol: TCP
node_expo 9801 root 732u sock 0,7 0t0 1591095 protocol: TCP
node_expo 9801 root 733u sock 0,7 0t0 1592671 protocol: TCP
node_expo 9801 root 734u sock 0,7 0t0 1592678 protocol: TCP
node_expo 9801 root 735u sock 0,7 0t0 1588217 protocol: TCP
node_expo 9801 root 736u sock 0,7 0t0 1589889 protocol: TCP
node_expo 9801 root 737u sock 0,7 0t0 1594398 protocol: TCP
node_expo 9801 root 738u sock 0,7 0t0 1595401 protocol: TCP
node_expo 9801 root 739u sock 0,7 0t0 1595420 protocol: TCP
node_expo 9801 root 740u sock 0,7 0t0 1595454 protocol: TCP
node_expo 9801 root 741u sock 0,7 0t0 1593561 protocol: TCP
node_expo 9801 root 742u sock 0,7 0t0 1596500 protocol: TCP
node_expo 9801 root 743u sock 0,7 0t0 1595506 protocol: TCP
node_expo 9801 root 744u sock 0,7 0t0 1592852 protocol: TCP
node_expo 9801 root 745u sock 0,7 0t0 1592858 protocol: TCP
node_expo 9801 root 746u sock 0,7 0t0 1592861 protocol: TCP
node_expo 9801 root 747u sock 0,7 0t0 1592895 protocol: TCP
node_expo 9801 root 748u sock 0,7 0t0 1595620 protocol: TCP
node_expo 9801 root 749u sock 0,7 0t0 1595640 protocol: TCP
node_expo 9801 root 750u sock 0,7 0t0 1594477 protocol: TCP
node_expo 9801 root 751u sock 0,7 0t0 1595743 protocol: TCP
node_expo 9801 root 752u sock 0,7 0t0 1595765 protocol: TCP
node_expo 9801 root 753u sock 0,7 0t0 1595769 protocol: TCP
node_expo 9801 root 754u sock 0,7 0t0 1593727 protocol: TCP
node_expo 9801 root 755u sock 0,7 0t0 1593080 protocol: TCP
node_expo 9801 root 756u sock 0,7 0t0 1595817 protocol: TCP
node_expo 9801 root 757u sock 0,7 0t0 1593100 protocol: TCP
node_expo 9801 root 758u sock 0,7 0t0 1593106 protocol: TCP
node_expo 9801 root 759u sock 0,7 0t0 1595923 protocol: TCP
node_expo 9801 root 760u sock 0,7 0t0 1593167 protocol: TCP
node_expo 9801 root 761u sock 0,7 0t0 1599533 protocol: TCP
node_expo 9801 root 762u sock 0,7 0t0 1598577 protocol: TCP
node_expo 9801 root 763u sock 0,7 0t0 1597036 protocol: TCP
node_expo 9801 root 764u sock 0,7 0t0 1597051 protocol: TCP
node_expo 9801 root 765u sock 0,7 0t0 1596062 protocol: TCP
node_expo 9801 root 766u sock 0,7 0t0 1599628 protocol: TCP
node_expo 9801 root 767u sock 0,7 0t0 1599677 protocol: TCP
node_expo 9801 root 768u sock 0,7 0t0 1596145 protocol: TCP
node_expo 9801 root 769u sock 0,7 0t0 1596156 protocol: TCP
node_expo 9801 root 770u sock 0,7 0t0 1593331 protocol: TCP
node_expo 9801 root 771u sock 0,7 0t0 1599758 protocol: TCP
node_expo 9801 root 772u sock 0,7 0t0 1596190 protocol: TCP
node_expo 9801 root 773u sock 0,7 0t0 1600529 protocol: TCP
node_expo 9801 root 774u sock 0,7 0t0 1599796 protocol: TCP
node_expo 9801 root 775u sock 0,7 0t0 1598745 protocol: TCP
node_expo 9801 root 776u sock 0,7 0t0 1597403 protocol: TCP
node_expo 9801 root 777u sock 0,7 0t0 1595070 protocol: TCP
node_expo 9801 root 778u sock 0,7 0t0 1600637 protocol: TCP
node_expo 9801 root 779u sock 0,7 0t0 1601564 protocol: TCP
node_expo 9801 root 780u sock 0,7 0t0 1597855 protocol: TCP
node_expo 9801 root 781u sock 0,7 0t0 1601581 protocol: TCP
node_expo 9801 root 782u sock 0,7 0t0 1596383 protocol: TCP
node_expo 9801 root 783u sock 0,7 0t0 1598845 protocol: TCP
node_expo 9801 root 784u sock 0,7 0t0 1600106 protocol: TCP
node_expo 9801 root 785u sock 0,7 0t0 1602576 protocol: TCP
node_expo 9801 root 786u sock 0,7 0t0 1600897 protocol: TCP
node_expo 9801 root 787u sock 0,7 0t0 1595263 protocol: TCP
node_expo 9801 root 788u sock 0,7 0t0 1602638 protocol: TCP
node_expo 9801 root 789u sock 0,7 0t0 1595269 protocol: TCP
node_expo 9801 root 790u sock 0,7 0t0 1595274 protocol: TCP
node_expo 9801 root 791u sock 0,7 0t0 1598907 protocol: TCP
node_expo 9801 root 792u sock 0,7 0t0 1601818 protocol: TCP
node_expo 9801 root 793u sock 0,7 0t0 1598115 protocol: TCP
node_expo 9801 root 794u sock 0,7 0t0 1601847 protocol: TCP
node_expo 9801 root 795u sock 0,7 0t0 1602744 protocol: TCP
node_expo 9801 root 796u sock 0,7 0t0 1598997 protocol: TCP
node_expo 9801 root 797u sock 0,7 0t0 1601092 protocol: TCP
node_expo 9801 root 798u sock 0,7 0t0 1602854 protocol: TCP
node_expo 9801 root 799u sock 0,7 0t0 1601157 protocol: TCP
node_expo 9801 root 800u sock 0,7 0t0 1595355 protocol: TCP
node_expo 9801 root 801u sock 0,7 0t0 1602908 protocol: TCP
node_expo 9801 root 802u sock 0,7 0t0 1595358 protocol: TCP
node_expo 9801 root 803u sock 0,7 0t0 1603586 protocol: TCP
node_expo 9801 root 804u sock 0,7 0t0 1599120 protocol: TCP
node_expo 9801 root 805u sock 0,7 0t0 1602177 protocol: TCP
node_expo 9801 root 806u sock 0,7 0t0 1598234 protocol: TCP
node_expo 9801 root 807u sock 0,7 0t0 1602230 protocol: TCP
node_expo 9801 root 808u sock 0,7 0t0 1602240 protocol: TCP
node_expo 9801 root 809u sock 0,7 0t0 1599166 protocol: TCP
node_expo 9801 root 810u sock 0,7 0t0 1605664 protocol: TCP
node_expo 9801 root 811u sock 0,7 0t0 1605699 protocol: TCP
node_expo 9801 root 812u sock 0,7 0t0 1601413 protocol: TCP
node_expo 9801 root 813u sock 0,7 0t0 1603656 protocol: TCP
node_expo 9801 root 814u sock 0,7 0t0 1598328 protocol: TCP
node_expo 9801 root 815u sock 0,7 0t0 1603124 protocol: TCP
node_expo 9801 root 816u sock 0,7 0t0 1603712 protocol: TCP
node_expo 9801 root 817u sock 0,7 0t0 1599273 protocol: TCP
node_expo 9801 root 818u sock 0,7 0t0 1601485 protocol: TCP
node_expo 9801 root 819u sock 0,7 0t0 1603192 protocol: TCP
node_expo 9801 root 820u sock 0,7 0t0 1606675 protocol: TCP
node_expo 9801 root 821u sock 0,7 0t0 1603245 protocol: TCP
node_expo 9801 root 822u sock 0,7 0t0 1599407 protocol: TCP
node_expo 9801 root 823u sock 0,7 0t0 1607788 protocol: TCP
node_expo 9801 root 824u sock 0,7 0t0 1606116 protocol: TCP
node_expo 9801 root 825u sock 0,7 0t0 1603326 protocol: TCP
node_expo 9801 root 826u sock 0,7 0t0 1603810 protocol: TCP
node_expo 9801 root 827u sock 0,7 0t0 1608735 protocol: TCP
node_expo 9801 root 828u sock 0,7 0t0 1603355 protocol: TCP
node_expo 9801 root 829u sock 0,7 0t0 1603848 protocol: TCP
node_expo 9801 root 830u sock 0,7 0t0 1606993 protocol: TCP
node_expo 9801 root 831u sock 0,7 0t0 1607990 protocol: TCP
node_expo 9801 root 832u sock 0,7 0t0 1603409 protocol: TCP
node_expo 9801 root 833u sock 0,7 0t0 1607058 protocol: TCP
node_expo 9801 root 834u sock 0,7 0t0 1603436 protocol: TCP
node_expo 9801 root 835u sock 0,7 0t0 1608093 protocol: TCP
node_expo 9801 root 836u sock 0,7 0t0 1603456 protocol: TCP
node_expo 9801 root 837u sock 0,7 0t0 1606404 protocol: TCP
node_expo 9801 root 838u sock 0,7 0t0 1603511 protocol: TCP
node_expo 9801 root 839u sock 0,7 0t0 1604066 protocol: TCP
node_expo 9801 root 840u sock 0,7 0t0 1608947 protocol: TCP
node_expo 9801 root 841u sock 0,7 0t0 1608178 protocol: TCP
node_expo 9801 root 842u sock 0,7 0t0 1605204 protocol: TCP
node_expo 9801 root 843u sock 0,7 0t0 1606527 protocol: TCP
node_expo 9801 root 844u sock 0,7 0t0 1607211 protocol: TCP
node_expo 9801 root 845u sock 0,7 0t0 1610820 protocol: TCP
node_expo 9801 root 846u sock 0,7 0t0 1607224 protocol: TCP
node_expo 9801 root 847u sock 0,7 0t0 1610884 protocol: TCP
node_expo 9801 root 848u sock 0,7 0t0 1610902 protocol: TCP
node_expo 9801 root 849u sock 0,7 0t0 1610913 protocol: TCP
node_expo 9801 root 850u sock 0,7 0t0 1606602 protocol: TCP
node_expo 9801 root 851u sock 0,7 0t0 1607309 protocol: TCP
node_expo 9801 root 852u sock 0,7 0t0 1604165 protocol: TCP
node_expo 9801 root 853u sock 0,7 0t0 1609093 protocol: TCP
node_expo 9801 root 854u sock 0,7 0t0 1608523 protocol: TCP
node_expo 9801 root 855u sock 0,7 0t0 1611813 protocol: TCP
node_expo 9801 root 856u sock 0,7 0t0 1608577 protocol: TCP
node_expo 9801 root 857u sock 0,7 0t0 1607407 protocol: TCP
node_expo 9801 root 858u sock 0,7 0t0 1611076 protocol: TCP
node_expo 9801 root 859u sock 0,7 0t0 1609129 protocol: TCP
node_expo 9801 root 860u sock 0,7 0t0 1611113 protocol: TCP
node_expo 9801 root 861u sock 0,7 0t0 1611139 protocol: TCP
node_expo 9801 root 862u sock 0,7 0t0 1611930 protocol: TCP
node_expo 9801 root 863u sock 0,7 0t0 1608694 protocol: TCP
node_expo 9801 root 864u sock 0,7 0t0 1607526 protocol: TCP
node_expo 9801 root 865u sock 0,7 0t0 1604387 protocol: TCP
node_expo 9801 root 866u sock 0,7 0t0 1609210 protocol: TCP
node_expo 9801 root 867u sock 0,7 0t0 1610157 protocol: TCP
node_expo 9801 root 868u sock 0,7 0t0 1612067 protocol: TCP
node_expo 9801 root 869u sock 0,7 0t0 1610179 protocol: TCP
node_expo 9801 root 870u sock 0,7 0t0 1607610 protocol: TCP
node_expo 9801 root 871u sock 0,7 0t0 1612158 protocol: TCP
node_expo 9801 root 872u sock 0,7 0t0 1609274 protocol: TCP
node_expo 9801 root 873u sock 0,7 0t0 1611390 protocol: TCP
node_expo 9801 root 874u sock 0,7 0t0 1613863 protocol: TCP
node_expo 9801 root 875u sock 0,7 0t0 1612236 protocol: TCP
node_expo 9801 root 876u sock 0,7 0t0 1613916 protocol: TCP
node_expo 9801 root 877u sock 0,7 0t0 1612276 protocol: TCP
node_expo 9801 root 878u sock 0,7 0t0 1613148 protocol: TCP
node_expo 9801 root 879u sock 0,7 0t0 1614029 protocol: TCP
node_expo 9801 root 880u sock 0,7 0t0 1612309 protocol: TCP
node_expo 9801 root 881u sock 0,7 0t0 1611506 protocol: TCP
node_expo 9801 root 882u sock 0,7 0t0 1610322 protocol: TCP
node_expo 9801 root 883u sock 0,7 0t0 1612343 protocol: TCP
node_expo 9801 root 884u sock 0,7 0t0 1613272 protocol: TCP
node_expo 9801 root 885u sock 0,7 0t0 1614109 protocol: TCP
node_expo 9801 root 886u sock 0,7 0t0 1614930 protocol: TCP
node_expo 9801 root 887u sock 0,7 0t0 1610403 protocol: TCP
node_expo 9801 root 888u sock 0,7 0t0 1613330 protocol: TCP
node_expo 9801 root 889u sock 0,7 0t0 1614201 protocol: TCP
node_expo 9801 root 890u sock 0,7 0t0 1612481 protocol: TCP
node_expo 9801 root 891u sock 0,7 0t0 1613390 protocol: TCP
node_expo 9801 root 892u sock 0,7 0t0 1611697 protocol: TCP
node_expo 9801 root 893u sock 0,7 0t0 1612569 protocol: TCP
node_expo 9801 root 894u sock 0,7 0t0 1610450 protocol: TCP
node_expo 9801 root 895u sock 0,7 0t0 1610461 protocol: TCP
node_expo 9801 root 896u sock 0,7 0t0 1616903 protocol: TCP
node_expo 9801 root 897u sock 0,7 0t0 1613575 protocol: TCP
node_expo 9801 root 898u sock 0,7 0t0 1610509 protocol: TCP
node_expo 9801 root 899u sock 0,7 0t0 1614352 protocol: TCP
node_expo 9801 root 900u sock 0,7 0t0 1614358 protocol: TCP
node_expo 9801 root 901u sock 0,7 0t0 1615118 protocol: TCP
node_expo 9801 root 902u sock 0,7 0t0 1612729 protocol: TCP
node_expo 9801 root 903u sock 0,7 0t0 1616074 protocol: TCP
node_expo 9801 root 904u sock 0,7 0t0 1617067 protocol: TCP
node_expo 9801 root 905u sock 0,7 0t0 1617991 protocol: TCP
node_expo 9801 root 906u sock 0,7 0t0 1610638 protocol: TCP
node_expo 9801 root 907u sock 0,7 0t0 1610654 protocol: TCP
node_expo 9801 root 908u sock 0,7 0t0 1610669 protocol: TCP
node_expo 9801 root 909u sock 0,7 0t0 1619037 protocol: TCP
node_expo 9801 root 910u sock 0,7 0t0 1619072 protocol: TCP
node_expo 9801 root 911u sock 0,7 0t0 1610726 protocol: TCP
node_expo 9801 root 912u sock 0,7 0t0 1614729 protocol: TCP
node_expo 9801 root 913u sock 0,7 0t0 1617252 protocol: TCP
node_expo 9801 root 914u sock 0,7 0t0 1618166 protocol: TCP
node_expo 9801 root 915u sock 0,7 0t0 1614769 protocol: TCP
node_expo 9801 root 916u sock 0,7 0t0 1616215 protocol: TCP
node_expo 9801 root 917u sock 0,7 0t0 1609707 protocol: TCP
node_expo 9801 root 918u sock 0,7 0t0 1614827 protocol: TCP
node_expo 9801 root 919u sock 0,7 0t0 1620017 protocol: TCP
node_expo 9801 root 920u sock 0,7 0t0 1620023 protocol: TCP
node_expo 9801 root 921u sock 0,7 0t0 1620027 protocol: TCP
node_expo 9801 root 922u sock 0,7 0t0 1619300 protocol: TCP
node_expo 9801 root 923u sock 0,7 0t0 1619323 protocol: TCP
node_expo 9801 root 924u sock 0,7 0t0 1619326 protocol: TCP
node_expo 9801 root 925u sock 0,7 0t0 1622117 protocol: TCP
node_expo 9801 root 926u sock 0,7 0t0 1620085 protocol: TCP
node_expo 9801 root 927u sock 0,7 0t0 1615675 protocol: TCP
node_expo 9801 root 928u sock 0,7 0t0 1617486 protocol: TCP
node_expo 9801 root 929u sock 0,7 0t0 1616313 protocol: TCP
node_expo 9801 root 930u sock 0,7 0t0 1621069 protocol: TCP
node_expo 9801 root 931u sock 0,7 0t0 1617620 protocol: TCP
node_expo 9801 root 932u sock 0,7 0t0 1617650 protocol: TCP
node_expo 9801 root 933u sock 0,7 0t0 1616367 protocol: TCP
node_expo 9801 root 934u sock 0,7 0t0 1620232 protocol: TCP
node_expo 9801 root 935u sock 0,7 0t0 1619464 protocol: TCP
node_expo 9801 root 936u sock 0,7 0t0 1622286 protocol: TCP
node_expo 9801 root 937u sock 0,7 0t0 1619476 protocol: TCP
node_expo 9801 root 938u sock 0,7 0t0 1618645 protocol: TCP
node_expo 9801 root 939u sock 0,7 0t0 1620353 protocol: TCP
node_expo 9801 root 940u sock 0,7 0t0 1615768 protocol: TCP
node_expo 9801 root 941u sock 0,7 0t0 1617791 protocol: TCP
node_expo 9801 root 942u sock 0,7 0t0 1618710 protocol: TCP
node_expo 9801 root 943u sock 0,7 0t0 1621267 protocol: TCP
node_expo 9801 root 944u sock 0,7 0t0 1617853 protocol: TCP
node_expo 9801 root 945u sock 0,7 0t0 1618784 protocol: TCP
node_expo 9801 root 946u sock 0,7 0t0 1616442 protocol: TCP
node_expo 9801 root 947u sock 0,7 0t0 1619653 protocol: TCP
node_expo 9801 root 948u sock 0,7 0t0 1619661 protocol: TCP
node_expo 9801 root 949u sock 0,7 0t0 1622479 protocol: TCP
node_expo 9801 root 950u sock 0,7 0t0 1619679 protocol: TCP
node_expo 9801 root 951u sock 0,7 0t0 1618890 protocol: TCP
node_expo 9801 root 952u sock 0,7 0t0 1620743 protocol: TCP
node_expo 9801 root 953u sock 0,7 0t0 1624068 protocol: TCP
node_expo 9801 root 954u sock 0,7 0t0 1622526 protocol: TCP
node_expo 9801 root 955u sock 0,7 0t0 1621360 protocol: TCP
node_expo 9801 root 956u sock 0,7 0t0 1616496 protocol: TCP
node_expo 9801 root 957u sock 0,7 0t0 1623189 protocol: TCP
node_expo 9801 root 958u sock 0,7 0t0 1621365 protocol: TCP
node_expo 9801 root 959u sock 0,7 0t0 1625171 protocol: TCP
node_expo 9801 root 960u sock 0,7 0t0 1619826 protocol: TCP
node_expo 9801 root 961u sock 0,7 0t0 1619836 protocol: TCP
node_expo 9801 root 962u sock 0,7 0t0 1623232 protocol: TCP
node_expo 9801 root 963u sock 0,7 0t0 1621433 protocol: TCP
node_expo 9801 root 964u sock 0,7 0t0 1619868 protocol: TCP
node_expo 9801 root 965u sock 0,7 0t0 1619870 protocol: TCP
node_expo 9801 root 966u sock 0,7 0t0 1624367 protocol: TCP
node_expo 9801 root 967u sock 0,7 0t0 1622677 protocol: TCP
node_expo 9801 root 968u sock 0,7 0t0 1619947 protocol: TCP
node_expo 9801 root 969u sock 0,7 0t0 1616582 protocol: TCP
node_expo 9801 root 970u sock 0,7 0t0 1624495 protocol: TCP
node_expo 9801 root 971u sock 0,7 0t0 1624502 protocol: TCP
node_expo 9801 root 972u sock 0,7 0t0 1624504 protocol: TCP
node_expo 9801 root 973u sock 0,7 0t0 1627157 protocol: TCP
node_expo 9801 root 974u sock 0,7 0t0 1627174 protocol: TCP
node_expo 9801 root 975u sock 0,7 0t0 1623427 protocol: TCP
node_expo 9801 root 976u sock 0,7 0t0 1621498 protocol: TCP
node_expo 9801 root 977u sock 0,7 0t0 1624591 protocol: TCP
node_expo 9801 root 978u sock 0,7 0t0 1627266 protocol: TCP
node_expo 9801 root 979u sock 0,7 0t0 1624617 protocol: TCP
node_expo 9801 root 980u sock 0,7 0t0 1622976 protocol: TCP
node_expo 9801 root 981u sock 0,7 0t0 1623485 protocol: TCP
node_expo 9801 root 982u sock 0,7 0t0 1623488 protocol: TCP
node_expo 9801 root 983u sock 0,7 0t0 1624674 protocol: TCP
node_expo 9801 root 984u sock 0,7 0t0 1624691 protocol: TCP
node_expo 9801 root 985u sock 0,7 0t0 1624693 protocol: TCP
node_expo 9801 root 986u sock 0,7 0t0 1624734 protocol: TCP
node_expo 9801 root 987u sock 0,7 0t0 1627413 protocol: TCP
node_expo 9801 root 988u sock 0,7 0t0 1627424 protocol: TCP
node_expo 9801 root 989u sock 0,7 0t0 1621611 protocol: TCP
node_expo 9801 root 990u sock 0,7 0t0 1624780 protocol: TCP
node_expo 9801 root 991u sock 0,7 0t0 1627515 protocol: TCP
node_expo 9801 root 992u sock 0,7 0t0 1624795 protocol: TCP
node_expo 9801 root 993u sock 0,7 0t0 1628358 protocol: TCP
node_expo 9801 root 994u sock 0,7 0t0 1624799 protocol: TCP
node_expo 9801 root 995u sock 0,7 0t0 1627564 protocol: TCP
node_expo 9801 root 996u sock 0,7 0t0 1623696 protocol: TCP
node_expo 9801 root 997u sock 0,7 0t0 1624823 protocol: TCP
node_expo 9801 root 998u sock 0,7 0t0 1624825 protocol: TCP
node_expo 9801 root 999u sock 0,7 0t0 1624857 protocol: TCP
node_expo 9801 root 1000u sock 0,7 0t0 1625958 protocol: TCP
node_expo 9801 root 1001u sock 0,7 0t0 1625978 protocol: TCP
node_expo 9801 root 1002u sock 0,7 0t0 1621727 protocol: TCP
node_expo 9801 root 1003u sock 0,7 0t0 1623819 protocol: TCP
node_expo 9801 root 1004u sock 0,7 0t0 1627695 protocol: TCP
node_expo 9801 root 1005u sock 0,7 0t0 1624877 protocol: TCP
node_expo 9801 root 1006u sock 0,7 0t0 1627766 protocol: TCP
node_expo 9801 root 1007u sock 0,7 0t0 1624910 protocol: TCP
node_expo 9801 root 1008u sock 0,7 0t0 1630226 protocol: TCP
node_expo 9801 root 1009u sock 0,7 0t0 1623915 protocol: TCP
node_expo 9801 root 1010u sock 0,7 0t0 1624919 protocol: TCP
node_expo 9801 root 1011u sock 0,7 0t0 1624932 protocol: TCP
node_expo 9801 root 1012u sock 0,7 0t0 1624939 protocol: TCP
node_expo 9801 root 1013u sock 0,7 0t0 1630351 protocol: TCP
node_expo 9801 root 1014u sock 0,7 0t0 1624962 protocol: TCP
node_expo 9801 root 1015u sock 0,7 0t0 1621832 protocol: TCP
node_expo 9801 root 1016u sock 0,7 0t0 1630435 protocol: TCP
node_expo 9801 root 1017u sock 0,7 0t0 1625040 protocol: TCP
node_expo 9801 root 1018u sock 0,7 0t0 1625083 protocol: TCP
node_expo 9801 root 1019u sock 0,7 0t0 1630565 protocol: TCP
node_expo 9801 root 1020u sock 0,7 0t0 1632257 protocol: TCP
node_expo 9801 root 1021u sock 0,7 0t0 1630658 protocol: TCP
node_expo 9801 root 1022u sock 0,7 0t0 1632367 protocol: TCP
In case you wonder about
node_expo 9801 root 3u FIFO 8,5 0t0 8650993 /var/log/all/.log
then it's because of my configuration for runit services - all of them are configured in same way, so it's unlikely the reason of this issue. But just in case here is full story:
# cat /service/node_exporter/run
#!/bin/sh
exec 2>&1
exec /home/powerman/gocode/bin/node_exporter -web.listen-address 127.0.0.1:9100 \
-collectors.enabled "conntrack,diskstats,entropy,filefd,filesystem,loadavg,mdadm,meminfo,netdev,netstat,sockstat,stat,textfile,time,uname,version,vmstat,runit,tcpstat"
# cat /service/node_exporter/log/run
#!/bin/sh
exec &>/var/log/all/.log
exec chpst -u log svlogd -tt /var/log/node_exporter/*/
# ls -l /var/log/all/.log
prw------- 1 root root 0 屑邪泄 24 09:06 /var/log/all/.log
# cat /service/log-all/run
#!/bin/bash
exec 1<>/var/log/.notify 2>&1
exec chpst -u log svlogd /var/log/all/ <>/var/log/all/.log
This way I can use epattern svlogd command in /var/log/ANYSERVICE/*/config to copy selected log records to that service's svlogd's STDERR, which is redirected to /var/log/all/.log FIFO, which is read by another one svlogd. In any case, there are no epattern commands in /var/log/node_exporter/*/config, so this FIFO won't get any output from node_exporter and can't break it.
powerman
on 24 May 2016
Can you use netstat -ntp to find out where those TCP connections are going?
brian-brazil
on 24 May 2016
There are no 1000 TCP connections related to these open sockets. For example, here is current output:
# netstat -antp | grep 9100
tcp 0 0 127.0.0.1:9100 0.0.0.0:* LISTEN 9801/node_exporter
tcp 0 0 127.0.0.1:9100 127.0.0.1:57614 TIME_WAIT -
tcp 0 0 127.0.0.1:9100 127.0.0.1:57548 TIME_WAIT -
tcp 0 0 127.0.0.1:9100 127.0.0.1:57538 TIME_WAIT -
tcp 0 0 127.0.0.1:9100 127.0.0.1:57598 TIME_WAIT -
And this is output for a couple of minutes ago:
# netstat -antp | grep 9100
tcp 0 0 127.0.0.1:9100 0.0.0.0:* LISTEN 9801/node_exporter
tcp 0 0 127.0.0.1:9100 127.0.0.1:57498 TIME_WAIT -
tcp 0 0 127.0.0.1:9100 127.0.0.1:57524 TIME_WAIT -
tcp 0 0 127.0.0.1:9100 127.0.0.1:57538 TIME_WAIT -
tcp 0 0 127.0.0.1:9100 127.0.0.1:57510 TIME_WAIT -
That output just means the sockets aren't on port 9100. Look at the rest of the netstat output.
brian-brazil
on 24 May 2016
That output just means the sockets aren't on port 9100. Look at the rest of the netstat output.
There are no such connections in netstat output. I've only about 100 connections, including listening sockets, while node_exporter has 1000 open sockets:
# netstat -antp | wc -l
106
Can you strace and figure out what these are talking to? Might also help to binary search modules, though we've very few that do network.
brian-brazil
on 24 May 2016
Looks like it somehow partially restored from that state, right now there are only few sockets open:
# ls -l /proc/$(pidof node_exporter)/fd
total 0
l-wx------ 1 root root 64 May 24 19:28 0 -> /dev/console
l-wx------ 1 root root 64 May 24 19:28 1 -> pipe:[708]
lrwx------ 1 root root 64 May 24 19:28 10 -> socket:[3137974]
lrwx------ 1 root root 64 May 24 19:28 11 -> socket:[3136139]
lrwx------ 1 root root 64 May 24 19:28 12 -> socket:[3141913]
lrwx------ 1 root root 64 May 24 19:28 13 -> socket:[3136220]
lrwx------ 1 root root 64 May 24 19:28 14 -> socket:[3141932]
lrwx------ 1 root root 64 May 24 19:28 15 -> socket:[3145427]
lrwx------ 1 root root 64 May 24 19:28 16 -> socket:[3138084]
lrwx------ 1 root root 64 May 24 19:28 17 -> socket:[3135201]
lrwx------ 1 root root 64 May 24 19:28 18 -> socket:[3139079]
l-wx------ 1 root root 64 May 24 19:28 2 -> pipe:[708]
lrwx------ 1 root root 64 May 24 19:28 3 -> /var/log/all/.log
lrwx------ 1 root root 64 May 24 19:28 4 -> socket:[1499975]
lrwx------ 1 root root 64 May 24 19:28 5 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 May 24 19:28 6 -> socket:[3141784]
lrwx------ 1 root root 64 May 24 19:28 7 -> socket:[3137939]
lrwx------ 1 root root 64 May 24 19:28 8 -> socket:[3137942]
lrwx------ 1 root root 64 May 24 19:28 9 -> socket:[3137971]
But memory usage is too high, I'll have to kill it soon if it continue to eat RAM that fast:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
9801 root 20 0 13.4G 934M 3032 S 0.0 11.8 16:43.94 /home/powerman/gocode/bin/node_export
BTW, while I was writing this comment amount of open sockets increased:
# ls -l /proc/$(pidof node_exporter)/fd
total 0
l-wx------ 1 root root 64 May 24 19:28 0 -> /dev/console
l-wx------ 1 root root 64 May 24 19:28 1 -> pipe:[708]
lrwx------ 1 root root 64 May 24 19:28 10 -> socket:[3137974]
lrwx------ 1 root root 64 May 24 19:28 11 -> socket:[3136139]
lrwx------ 1 root root 64 May 24 19:28 12 -> socket:[3141913]
lrwx------ 1 root root 64 May 24 19:28 13 -> socket:[3136220]
lrwx------ 1 root root 64 May 24 19:28 14 -> socket:[3141932]
lrwx------ 1 root root 64 May 24 19:28 15 -> socket:[3145427]
lrwx------ 1 root root 64 May 24 19:28 16 -> socket:[3138084]
lrwx------ 1 root root 64 May 24 19:28 17 -> socket:[3135201]
lrwx------ 1 root root 64 May 24 19:28 18 -> socket:[3139079]
lrwx------ 1 root root 64 May 24 19:28 19 -> socket:[3142083]
l-wx------ 1 root root 64 May 24 19:28 2 -> pipe:[708]
lrwx------ 1 root root 64 May 24 19:28 20 -> socket:[3138135]
lrwx------ 1 root root 64 May 24 19:28 21 -> socket:[3138149]
lrwx------ 1 root root 64 May 24 19:28 22 -> socket:[3138155]
lrwx------ 1 root root 64 May 24 19:28 23 -> socket:[3142129]
lrwx------ 1 root root 64 May 24 19:28 24 -> socket:[3136458]
lrwx------ 1 root root 64 May 24 19:28 25 -> socket:[3142153]
lrwx------ 1 root root 64 May 24 19:28 26 -> socket:[3145599]
lrwx------ 1 root root 64 May 24 19:28 27 -> socket:[3139380]
lrwx------ 1 root root 64 May 24 19:28 3 -> /var/log/all/.log
lrwx------ 1 root root 64 May 24 19:28 4 -> socket:[1499975]
lrwx------ 1 root root 64 May 24 19:28 5 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 May 24 19:28 6 -> socket:[3141784]
lrwx------ 1 root root 64 May 24 19:28 7 -> socket:[3137939]
lrwx------ 1 root root 64 May 24 19:28 8 -> socket:[3137942]
lrwx------ 1 root root 64 May 24 19:28 9 -> socket:[3137971]
And according to ss output some of these FD are not related to any open connection:
# ss -anp | grep node_exporter
tcp LISTEN 0 128 127.0.0.1:9100 *:* users:(("node_exporter",pid=9801,fd=4))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56470 users:(( node_exporter",pid=9801,fd=24))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56300 users:(( node_exporter",pid=9801,fd=18))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56462 users:(( node_exporter",pid=9801,fd=23))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56438 users:(( node_exporter",pid=9801,fd=21))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56546 users:(( node_exporter",pid=9801,fd=29))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56510 users:(( node_exporter",pid=9801,fd=27))
tcp ESTAB 0 0 127.0.0.1:9100 127.0.0.1:56554 users:(( node_exporter",pid=9801,fd=30))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56318 users:(( node_exporter",pid=9801,fd=19))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56446 users:(( node_exporter",pid=9801,fd=22))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56518 users:(( node_exporter",pid=9801,fd=28))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56424 users:(( node_exporter",pid=9801,fd=20))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56484 users:(( node_exporter",pid=9801,fd=25))
tcp CLOSE-WAIT 1 0 127.0.0.1:9100 127.0.0.1:56494 users:(( node_exporter",pid=9801,fd=26))
Also I've just checked memory usage once again and now it's very low:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
9801 root 20 0 13.4G 47016 3032 S 0.0 0.6 16:44.17 /home/powerman/gocode/bin/node_export
I've tried to open http://localhost:9100/metrics in browser but don't get any output (http://localhost:9100/ opens ok).
I don't really understand what you want me to find in strace output, can you provide more precise instructions?
powerman
on 24 May 2016
Where are the sockets connecting to?
brian-brazil
on 24 May 2016
It looks like when I run strace it somehow "unblock" node_exporter. Probably when I run it for the first time after your first request this result in closing that 1000 sockets. For example, here is strace -ff -p $(pidof node_exporter) output for one of such fd (same happens for all of them):
[pid 8972] write(15, "HTTP/1.1 200 OK\r\nContent-Encodin"..., 4096) = -1 EPIPE (Broken pipe)
[pid 8972] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=9801, si_uid=0} ---
[pid 8972] rt_sigreturn() = -1 EPIPE (Broken pipe)
[pid 30753] futex(0xc830d0e508, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 8972] epoll_ctl(5, EPOLL_CTL_DEL, 15, c840c894b4) = 0
[pid 8972] close(15) = 0
[pid 8972] futex(0xc8385ca908, FUTEX_WAIT, 0, NULL <unfinished ...>
I don't know how to find out "Where are the sockets connecting to?" using strace. It looks like they didn't connected to anything - ss output is authoritative here, their connections was closed some time ago, but node_exporter didn't closed these fd. Probably strace output above means node_exporter (or Go itself) hangs just before or in the middle of writing response to the socket. And when strace attached to the process it somehow unblocked these write() and they get EPIPE because these connections was already closed by linux kernel long time ago.
powerman
on 24 May 2016
I've run strace for long enough time to get more info. When strace attached to process all leaked socket FDs was released, but it was running long enough to leak new sockets. Here is what I get in strace output for leaked socket FD 91:
...
[pid 12551] <... epoll_wait resumed> {{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=3559631104, u64=11953679444
6080}}}, 128, -1) = 1
[pid 12551] epoll_wait(5, {{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP|EPOLLRDHUP, {u32=3605501352, u64=1195
36840316328}}}, 128, -1) = 1
[pid 12551] epoll_wait(5, {{EPOLLIN, {u32=3605504808, u64=119536840319784}}}, 128, -1) = 1
[pid 12551] futex(0xd80230, FUTEX_WAKE, 1) = 1
[pid 12551] accept4(4, {sa_family=AF_INET, sin_port=htons(34372), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_CLOEXEC|SOCK_NONBLOCK) = 91
[pid 12551] epoll_ctl(5, EPOLL_CTL_ADD, 91, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3604228832, u64=119536839043808}}) = 0
[pid 12551] getsockname(91, {sa_family=AF_INET, sin_port=htons(9100), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
[pid 12551] setsockopt(91, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 12551] setsockopt(91, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 12551] setsockopt(91, SOL_TCP, TCP_KEEPINTVL, [180], 4) = 0
[pid 12551] setsockopt(91, SOL_TCP, TCP_KEEPIDLE, [180], 4) = 0
[pid 12551] futex(0xc82147cd08, FUTEX_WAKE, 1) = 1
[pid 12551] accept4(4, 0xc83cee57d8, [112], SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12160] <... futex resumed> ) = 0
[pid 12160] epoll_wait(5, <unfinished ...>
[pid 12551] read(91, "GET /metrics HTTP/1.1\r\nHost: loc"..., 4096) = 335
[pid 12160] <... epoll_wait resumed> {{EPOLLIN|EPOLLOUT, {u32=3604228832, u64=119536839043808}}}, 128, 0) = 1
[pid 12551] futex(0xc820208508, FUTEX_WAKE, 1) = 1
[pid 12160] epoll_wait(5, <unfinished ...>
[pid 9811] <... futex resumed> ) = 0
[pid 9802] <... futex resumed> ) = 0
[pid 12551] stat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
[pid 12551] stat("/proc/9801", {st_mode=S_IFDIR|0550, st_size=0, ...}) = 0
[pid 12551] openat(AT_FDCWD, "/proc/9801/stat", O_RDONLY|O_CLOEXEC) = 92
...
FD 91 was never mentioned again in strace output. I.e. after reading HTTP request from this socket node_exporter didn't call write() into this socket.
powerman
on 24 May 2016
Is there anything in the logs? Which go version did you use to compile node_exporter?
grobie
on 24 May 2016
Is there anything in the logs?
All I have in logs is errors about "too many open files". Older logs was rotated and removed. But right now same issue is happening again:
# ls -l /proc/$(pidof node_exporter)/fd | wc -l
535
and there is nothing in logs for last several hours (since I've run strace for the first time, I think). So chances are there will be nothing in logs until all 1024 fd will be opened.
Which go version did you use to compile node_exporter?
1.6.1.
powerman
on 24 May 2016
I've updated node_exporter to latest master, recompiled node_exporter using go 1.6.2, restarted it, noticed issue still exists, did binary search on -collectors.enabled and found one which trigger this issue: filesystem.
Which gives me idea what's going on:
- when Prometheus request http://localhost:9100/metrics node_exporter run collectors and one of them (filesystem) hangs
- because of this node_exporter didn't respond to Prometheus and Prometheus closes connection by timeout while node_exporter still have open socket fd for that connection
- after 1000 of such hang requests node_exporter runs out of available fd
Now why filesystem collector hangs. I'm using webdav to 4shared.com:
# mount | grep webdav
https://webdav.4shared.com/ on /mnt/webdav.4shared type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=16384,uid=0,gid=0,helper=davfs)
I don't know why, but when this webdav is mounted some filesystem-related operations hang, for example df command:
# time df
^C
real 0m5.144s
user 0m0.000s
sys 0m0.000s
Strace shows df hangs on stat() syscall:
# strace -ff df
...
stat("/mnt/tmp", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/var/tmp/portage", {st_mode=S_IFDIR|0775, st_size=60, ...}) = 0
stat("/run/user/1000", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
stat("/mnt/webdav.4shared",
Probably this is because of some issue with davfs2-1.5.2 (and older versions) or kernel or fuse bug.
The reason why using strace affect this issue is probably this - when I kill strace attached to df this result in completing df command this way:
df: '/mnt/webdav.4shared': Interrupted system call
Filesystem 1K-blocks Used Available Use% Mounted on
...
Is it possible to take this (stat() may hangs) in account and fix in node_exporter?
powerman
on 26 May 2016
May be related: https://savannah.nongnu.org/support/?107064
powerman
on 26 May 2016
@powerman im seeing the exact same behavior. Filesystem collector hanging (for example due to a hung NFS or other shared network mount) will cause the leak of sockets.
node_exporter, version 0.12.0 (branch: master, revision: df8dcd2)
build user: root@ff68505a5469
build date: 20160505-22:14:18
go version: go1.6.2```
byxorna
on 2 Oct 2016
There's nothing we can really do about stuck filesystems.
brian-brazil
on 2 Oct 2016
There's nothing we can really do about stuck filesystems.
Move leaking part of code to another short-living subprocess?
powerman
on 2 Oct 2016
That'd just case process exhaustion.
brian-brazil
on 2 Oct 2016
Detect hang and stop probing that mount point?
powerman
on 2 Oct 2016
Exporters are meant to be stateless, also how would we start monitoring again once the filesystem was healthy?
brian-brazil
on 2 Oct 2016
For most situations, you don't want to monitor network filesystems with the node_exporter. The next question is why don't we ignore them by default. We chose to provide the most flexible default, which is all "valid" filesystems. This limits the debate over what to include or not include in the default configuration.
SuperQ
on 2 Oct 2016
@SuperQ thats a good idea for defaults (exclude network filesystems). However, a single collector could hang for whatever reason, and prevent the exporter from ever reporting metrics again. Perhaps all collectors should be bounded in execution time and launched in a separate goroutine? That way a single unforseen event like this doesn't wedge the whole exporter. Rather, it would only fail to report the metrics collected by the stalled collector.
byxorna
on 2 Oct 2016
@byxorna The node exporter already supports concurrent scrapes and every collector is run in its own goroutine. So a single stuck scrape wouldn't block other scrapes, but would just leak goroutines / sockets, like reported here. And there's no way to abort e.g. a read from the filesystem after some timeout. There's also no other general way of stopping collector goroutines other than threading Contexts through everything and inserting code points into each collector where the collector can terminate itself prematurely if the context has been cancelled.
juliusv
on 3 Oct 2016
👍1
So! What to do about this issue? As I see it there is nothing we can do about stale nfs mounts. The syscalls end up in deep sleep and there is no way to cancel them, so those will pile up and cause issue anyway.
But we could solve this problem for other cases where a collect might take longer than the scrape interval and stuff piles up, just the question if it's worth it.
I don't have a finial opinion on that but since it never caused me issues, we can't prevent it in the nfs case and it's easy to monitor for this (arguably rare) case, I tend toward keeping things as they are for now. But open for arguments.
discordianfish
on 22 Dec 2016
@discordianfish I think excluding any common network filesystems from being monitored by node_exporter by default would be a good move. (nfs, lustre, gpfs, ....?)
byxorna
on 23 Dec 2016
Our goal was to monitor all valid filesystems by default. Network filesystems can be tricky, but I think they're still valid enough to keep them in the defaults.
SuperQ
on 24 Dec 2016
I'm leaning toward keeping them too. I prefer it because it's the more strict: Better monitor your stale nfs mount, causing your scrape to timeout which you can alert on, then not monitoring your stale nfs mount and therefor not realizing that something is wrong. Arguably you should monitor your nfs differently, but as a default catching this problem the the node-exporter not responding is IMO preferably.
discordianfish
on 25 Dec 2016
We're already adding NFS mount/rpc stats. These will be in the next release.
SuperQ
on 25 Dec 2016
@SuperQ Right.. though you won't be able to access them if the node-exporter isn't responding because the mount is stale. That's the tricky issue. Wondering how hard it would be to implement a deadline for the collectors.
discordianfish
on 25 Dec 2016
I'm strongly in favour of setting a timeout for collectors.
We're using the up metric for the Node Exporter as a canary for checking that hosts are up (replacing the Nagios' active checks that use ICMP ping), and we're seeing false positives because the Node Exporter occasionally stops responding to HTTP requests (even on /, which is odd).
When the Node Exporter starts responding to requests again (without us taking any remedial action), there's a log line showing that a connection to dbus (used by the systemd collector) timed out after 400+ seconds.
There's an issue with dbus/systemd to be investigated though I think a timeout is necessary to better isolate failures in an individual collector, especially when using the Node Exporter as a canary.
Would a PR be accepted that sets a global timeout (applicable to all collectors), configured using a commandline flag? What would the default be, 60 seconds (being generous to start with)?
mattbostock
on 18 Mar 2017
👍2
@mattbostock I think a timeout in general is ok, but we can't always do something about the timeouts, as things are getting stuck on IO. Out of curiosity, what version of node_exporter are you seeing issues?
SuperQ
on 18 Mar 2017
@SuperQ Version 0.12.0. We're currently rolling out 0.14.0.
mattbostock
on 29 Mar 2017
FWIW I suspect that the cause of the issues we're seeing with the systemd collector maybe this open dbus bug:
https://bugs.freedesktop.org/show_bug.cgi?id=95263#c37
So I think a timeout should help in our case.
mattbostock
on 29 Mar 2017
As for the implemenation of a timeout, I'm thinking of using a context with a timeout set. That means that we're reliant on every collector to implement the timeout correctly, though I don't see any other way of enforcing a timeout using goroutines (you have to rely on the goroutines to clean up their own resources).
I was thinking of passing a context to the Update function on the Collector interface. All collectors would need updating to accept the additional argument but they wouldn't actually use it yet. I was thinking of making the systemd collector respect the timeout first (as an example), then the other collectors can be modified one by one to support it also.
mattbostock
on 1 Apr 2017
Now that we have nfs rpc and mountstats collectors, we can probably add nfs.* to the filesystems ignored list. @discordianfish / @grobie opinions?
SuperQ
on 9 May 2017
They're still real filesystems, I think we should have the usual block/inode usage stats for them.
brian-brazil
on 9 May 2017
But do we need those exact same stats on every client? Normally I would get those stats from the exporting server.
SuperQ
on 9 May 2017
That's a question for the person running the machine, we can't presume any particular monitoring strategy.
brian-brazil
on 9 May 2017
@SuperQ I agree, we should have sane and safe defaults and excluding NFS by default seems reasonable to me.
discordianfish
on 9 May 2017
One issue with NFS is is that since it's a network filesystem it's considerably more likely to get in an error state where any operation on the mountpoint will hang. I've had a couple of cases where a network glitch meant that the node exporter started returning partial results truncated at the point where it started collecting mount information.
acdha
on 9 May 2017
We had similar issue, sockets left open till it hit the max of 1024 and made the Node Exporter inoperable, but for what it's worth, after we restored access to the broken network mount and performed a mount -a on the box, the Node Exporter resumed without a restart and all the previously opened sockets were cleared on their own. So the Node Exporter itself was pretty robust in that regard. :+1:
nthanes
on 26 Jun 2017
Are any fixes here going to make it into the next tagged release? This would be a great improvement!
pmb311
on 23 Jul 2018
@pmb311 Yes, we're currently working on some other issues. We don't have a strict release schedule, but I may put together a bugfix release sometime soon.
SuperQ
on 23 Jul 2018
👍1
after read #244 I do not think 244 is the same problem while same effect , In my #1161 case it is slow to get cpu information not any thing about disk or network. and it caused sever damage to our online server
renchili
on 19 Nov 2018
node-exporter info
node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70f4363dced6b77d8fc311ea57b63387e4f)
build user: root@a67a9bc13a69
build date: 20180515-15:52:42
go version: go1.9.6
linux version
Linux 4.15.0-21 #22~16.04.1+3 SMP Mon Aug 6 20:13:57 +08 2018 x86_64 x86_64 x86_64 GNU/Linux
what happened
after restart node-exporter under systemctl/systemd the machine stuck
log snapshot
renchili
on 19 Nov 2018
@renchili This issue here is about general timeout mechanisms for collectors, so it could be used for the cpu collector as well.
That being said, you issue looks more like a lowe level issue, very unlikely to be triggered by the node-exporter.
discordianfish
on 19 Nov 2018
I'm considering a different tactic to avoid the problem of stuck collectors.
Rather than try and implement timeouts on collectors that generally get stuck on IO, and no timeout will solve the issue, we can implement a max-in-flight scrape flag. This will cause the node_exporter to start returning 503 errors when it gets stuck.
SuperQ
on 20 Nov 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
mhansen
路
4Comments
Dlorite
路
3Comments
ridvankaratas
路
3Comments
prologic
路
4Comments
fchiorascu
路
3Comments
Most helpful comment
I'm strongly in favour of setting a timeout for collectors.
We're using the
upmetric for the Node Exporter as a canary for checking that hosts are up (replacing the Nagios' active checks that use ICMP ping), and we're seeing false positives because the Node Exporter occasionally stops responding to HTTP requests (even on/, which is odd).When the Node Exporter starts responding to requests again (without us taking any remedial action), there's a log line showing that a connection to dbus (used by the systemd collector) timed out after 400+ seconds.
There's an issue with dbus/systemd to be investigated though I think a timeout is necessary to better isolate failures in an individual collector, especially when using the Node Exporter as a canary.
Would a PR be accepted that sets a global timeout (applicable to all collectors), configured using a commandline flag? What would the default be, 60 seconds (being generous to start with)?