Kube-router: IPVS: rr: TCP 10.254.159.251:9004 - no destination available

Created on 28 Apr 2018  Â·  18Comments  Â·  Source: cloudnativelabs/kube-router

Why I use kube-route in kubernetes,

After running 20 web, kubernetes found that kubernetes node could not connect.

Open the host to find the following error information.

20180428163023

bug

Most helpful comment

@Jacky2 Hey! How did u solve it?

All 18 comments

@Jacky2 That just looks like sevice endpoints which do not have any underlying pods to forward requests to? Seems like a legit error message to me?

It looks a bit scary with so many kernel messages, but on the other hand - in a real production scenario it would be a pretty serious error.

I have the same on ALL my node logs! And the ip it's always the same as the host itself.

I wonder if the service is marked with externalTrafficPolicy=Local.

@murali-reddy not in my case :)

@tonidas so you mean, there was IPVS service configured on the node with endpoints, and still you saw this error message?

@murali-reddy I didn't put any externalTrafficPolicy=Local configuration. That's what I meant.

How do I check if there is a "IPVS service configured on the node with endpoints"?

Either you use pod toolbox or just loging to the kube-router pods on the concerned node and do ipvsadm.

You should see IPVS services and server details.

Here is the output:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.5.4:9090              Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> ip-10-0-62-92.ec2.internal:9 Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.18:9109             Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.125.10:http           Masq    1      2          14
  -> 100.99.0.16:http             Masq    1      0          14
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.22:http             Masq    1      1          1
  -> 100.97.16.8:http             Masq    1      1          1
  -> 100.97.142.16:http           Masq    1      0          1
  -> 100.98.180.22:http           Masq    1      0          1
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.99.43.2:bacula-fd        Masq    1      0          0
  -> 100.99.70.6:bacula-fd        Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.99.76.7:bacula-fd        Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.99.43.2:9125             Masq    1      0          0
  -> 100.99.70.6:9125             Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.104.84.136:http          Masq    1      2          36
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.16.10:http            Masq    1      1          54
  -> 100.97.142.28:http           Masq    1      1          54
  -> 100.97.142.29:http           Masq    1      1          54
  -> 100.98.125.30:http           Masq    1      1          54
  -> 100.98.125.31:http           Masq    1      0          54
  -> 100.98.126.42:http           Masq    1      0          54
  -> 100.98.126.43:http           Masq    1      1          54
  -> 100.98.126.44:http           Masq    1      3          51
  -> 100.98.151.17:http           Masq    1      2          55
  -> 100.98.151.18:http           Masq    1      1          54
  -> 100.98.162.24:http           Masq    1      1          54
  -> 100.98.162.25:http           Masq    1      1          54
  -> 100.98.162.26:http           Masq    1      2          53
  -> 100.98.162.27:http           Masq    1      0          54
  -> 100.98.180.24:http           Masq    1      1          54
  -> 100.98.180.25:http           Masq    1      1          54
  -> 100.98.191.29:http           Masq    1      2          52
  -> 100.98.205.3:http            Masq    1      1          53
  -> 100.98.237.21:http           Masq    1      0          54
  -> 100.98.237.22:http           Masq    1      0          54
  -> 100.98.237.23:http           Masq    1      1          54
  -> 100.98.240.8:http            Masq    1      1          53
  -> 100.98.240.9:http            Masq    1      1          54
  -> 100.98.240.10:http           Masq    1      1          54
  -> 100.98.240.11:http           Masq    1      3          52
  -> 100.98.240.12:http           Masq    1      1          53
  -> 100.98.253.23:http           Masq    1      1          53
  -> 100.98.253.24:http           Masq    1      0          55
  -> 100.98.253.25:http           Masq    1      0          55
  -> 100.98.253.26:http           Masq    1      2          53
  -> 100.99.0.21:http             Masq    1      1          53
  -> 100.99.0.22:http             Masq    1      2          50
  -> 100.99.5.53:http             Masq    1      1          54
  -> 100.99.5.54:http             Masq    1      1          54
  -> 100.99.5.55:http             Masq    1      1          54
  -> 100.99.16.17:http            Masq    1      1          53
  -> 100.99.16.18:http            Masq    1      1          53
  -> 100.99.16.19:http            Masq    1      0          54
  -> 100.99.25.14:http            Masq    1      0          53
  -> 100.99.25.15:http            Masq    1      1          53
  -> 100.99.43.5:http             Masq    1      0          54
  -> 100.99.51.3:http             Masq    1      1          54
  -> 100.99.51.6:http             Masq    1      1          54
  -> 100.99.63.2:http             Masq    1      1          53
  -> 100.99.63.3:http             Masq    1      2          53
  -> 100.99.68.2:http             Masq    1      1          54
  -> 100.99.68.3:http             Masq    1      0          55
  -> 100.99.68.4:http             Masq    1      2          53
  -> 100.99.68.10:http            Masq    1      1          53
  -> 100.99.68.11:http            Masq    1      0          54
  -> 100.99.70.2:http             Masq    1      0          54
  -> 100.99.70.3:http             Masq    1      1          53
  -> 100.99.72.2:http             Masq    1      0          54
  -> 100.99.77.2:http             Masq    1      1          53
  -> 100.99.77.4:http             Masq    1      0          54
  -> 100.99.77.6:http             Masq    1      1          53
  -> 100.99.78.2:http             Masq    1      0          55
  -> 100.99.78.4:http             Masq    1      2          52
  -> 100.99.81.5:http             Masq    1      0          54
  -> 100.99.83.3:http             Masq    1      1          52
  -> 100.99.83.5:http             Masq    1      0          54
  -> 100.99.83.6:http             Masq    1      0          55
  -> 100.99.87.2:http             Masq    1      0          54
  -> 100.99.87.3:http             Masq    1      0          54
  -> 100.99.87.9:http             Masq    1      2          52
  -> 100.99.91.4:http             Masq    1      0          54
  -> 100.99.91.5:http             Masq    1      0          54
  -> 100.99.101.8:http            Masq    1      0          53
  -> 100.99.101.9:http            Masq    1      0          52
  -> 100.99.102.3:http            Masq    1      1          52
  -> 100.99.102.4:http            Masq    1      0          53
  -> 100.99.102.5:http            Masq    1      0          54
  -> 100.99.102.6:http            Masq    1      0          54
  -> 100.99.104.2:http            Masq    1      0          54
  -> 100.99.104.3:http            Masq    1      0          55
  -> 100.99.104.4:http            Masq    1      1          53
  -> 100.99.106.5:http            Masq    1      0          54
  -> 100.99.107.2:http            Masq    1      1          53
  -> 100.99.107.3:http            Masq    1      0          54
  -> 100.99.107.4:http            Masq    1      1          54
  -> 100.99.109.2:http            Masq    1      0          54
  -> 100.99.109.3:http            Masq    1      1          53
  -> 100.99.109.4:http            Masq    1      0          54
  -> 100.99.109.5:http            Masq    1      2          52
  -> 100.99.110.3:http            Masq    1      0          54
  -> 100.99.110.4:http            Masq    1      1          53
  -> 100.99.110.5:http            Masq    1      1          53
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.126.28:postgresql     Masq    1      0          0
  -> 100.98.253.19:postgresql     Masq    1      0          0
  -> 100.99.106.8:postgresql      Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.22:http             Masq    1      1          7
  -> 100.97.16.8:http             Masq    1      0          7
  -> 100.97.142.16:http           Masq    1      1          7
  -> 100.98.180.22:http           Masq    1      0          7
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.99.56.13:5601            Masq    1      2          28
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.12.71:http            Masq    1      0          4
  -> 100.97.228.242:http          Masq    1      1          4
  -> 100.97.229.186:http          Masq    1      1          4
TCP  ip-10-0-62-121.ec2.internal: rr
TCP  ip-10-0-62-121.ec2.internal: rr
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.104.84.22:http           Masq    1      2          12
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.23:9696             Masq    1      1          2
  -> 100.97.142.26:9696           Masq    1      0          2
  -> 100.98.44.26:9696            Masq    1      1          1
  -> 100.98.126.34:9696           Masq    1      0          2
  -> 100.98.233.3:9696            Masq    1      0          3
  -> 100.99.5.49:9696             Masq    1      0          2
  -> 100.99.5.50:9696             Masq    1      0          2
  -> 100.99.43.3:9696             Masq    1      0          2
  -> 100.99.76.8:9696             Masq    1      0          2
  -> 100.99.77.3:9696             Masq    1      0          1
  -> 100.99.78.3:9696             Masq    1      0          2
  -> 100.99.81.2:9696             Masq    1      0          1
  -> 100.99.87.7:9696             Masq    1      0          2
  -> 100.99.106.10:9696           Masq    1      0          2
  -> 100.99.110.2:9696            Masq    1      0          2
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.12.71:http            Masq    1      0          12
  -> 100.97.228.242:http          Masq    1      2          12
  -> 100.97.229.186:http          Masq    1      0          12
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.44.9:postgresql       Masq    1      0          0
  -> 100.98.151.16:postgresql     Masq    1      0          0
  -> 100.98.240.7:postgresql      Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.228.238:http          Masq    1      2          18
  -> 100.97.229.184:http          Masq    1      0          18
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.125.10:http           Masq    1      1          2
  -> 100.99.0.16:http             Masq    1      1          2
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.229.163:http          Masq    1      1          6
  -> 100.104.84.23:http           Masq    1      1          6
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.104.84.22:http           Masq    1      2          36
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.12.71:http            Masq    1      1          12
  -> 100.97.228.242:http          Masq    1      1          12
  -> 100.97.229.186:http          Masq    1      0          12
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.44.9:3000             Masq    1      0          8
  -> 100.98.151.16:3000           Masq    1      0          8
  -> 100.98.240.7:3000            Masq    1      0          8
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.18:9108             Masq    1      0          0
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.12.33:http            Masq    1      1          18
  -> 100.97.228.200:http          Masq    1      1          18
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.44.25:3000            Masq    1      1          14
  -> 100.99.0.15:3000             Masq    1      1          14
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.229.188:http          Masq    1      1          19
  -> 100.104.84.45:http           Masq    1      1          20
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.191.21:5601           Masq    1      2          28
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.13:http-alt         Masq    1      0          14
  -> 100.98.233.4:http-alt        Masq    1      2          14
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.16.10:http            Masq    1      1          54
  -> 100.97.142.28:http           Masq    1      1          55
  -> 100.97.142.29:http           Masq    1      2          54
  -> 100.98.125.30:http           Masq    1      1          55
  -> 100.98.125.31:http           Masq    1      1          55
  -> 100.98.126.42:http           Masq    1      1          54
  -> 100.98.126.43:http           Masq    1      2          53
  -> 100.98.126.44:http           Masq    1      0          55
  -> 100.98.151.17:http           Masq    1      0          53
  -> 100.98.151.18:http           Masq    1      1          54
  -> 100.98.162.24:http           Masq    1      3          50
  -> 100.98.162.25:http           Masq    1      1          55
  -> 100.98.162.26:http           Masq    1      2          54
  -> 100.98.162.27:http           Masq    1      0          56
  -> 100.98.180.24:http           Masq    1      0          53
  -> 100.98.180.25:http           Masq    1      0          55
  -> 100.98.191.29:http           Masq    1      1          53
  -> 100.98.205.3:http            Masq    1      1          55
  -> 100.98.237.21:http           Masq    1      1          54
  -> 100.98.237.22:http           Masq    1      1          54
  -> 100.98.237.23:http           Masq    1      1          55
  -> 100.98.240.8:http            Masq    1      1          56
  -> 100.98.240.9:http            Masq    1      2          53
  -> 100.98.240.10:http           Masq    1      1          54
  -> 100.98.240.11:http           Masq    1      1          54
  -> 100.98.240.12:http           Masq    1      1          55
  -> 100.98.253.23:http           Masq    1      1          53
  -> 100.98.253.24:http           Masq    1      1          55
  -> 100.98.253.25:http           Masq    1      0          56
  -> 100.98.253.26:http           Masq    1      1          54
  -> 100.99.0.21:http             Masq    1      0          55
  -> 100.99.0.22:http             Masq    1      1          53
  -> 100.99.5.53:http             Masq    1      1          54
  -> 100.99.5.54:http             Masq    1      1          54
  -> 100.99.5.55:http             Masq    1      1          53
  -> 100.99.16.17:http            Masq    1      2          54
  -> 100.99.16.18:http            Masq    1      1          54
  -> 100.99.16.19:http            Masq    1      1          54
  -> 100.99.25.14:http            Masq    1      0          55
  -> 100.99.25.15:http            Masq    1      2          54
  -> 100.99.43.5:http             Masq    1      1          54
  -> 100.99.51.3:http             Masq    1      1          54
  -> 100.99.51.6:http             Masq    1      2          53
  -> 100.99.63.2:http             Masq    1      1          54
  -> 100.99.63.3:http             Masq    1      1          52
  -> 100.99.68.2:http             Masq    1      1          54
  -> 100.99.68.3:http             Masq    1      2          53
  -> 100.99.68.4:http             Masq    1      1          55
  -> 100.99.68.10:http            Masq    1      2          53
  -> 100.99.68.11:http            Masq    1      1          53
  -> 100.99.70.2:http             Masq    1      0          55
  -> 100.99.70.3:http             Masq    1      0          56
  -> 100.99.72.2:http             Masq    1      0          54
  -> 100.99.77.2:http             Masq    1      0          55
  -> 100.99.77.4:http             Masq    1      2          53
  -> 100.99.77.6:http             Masq    1      1          54
  -> 100.99.78.2:http             Masq    1      0          54
  -> 100.99.78.4:http             Masq    1      0          54
  -> 100.99.81.5:http             Masq    1      0          54
  -> 100.99.83.3:http             Masq    1      0          56
  -> 100.99.83.5:http             Masq    1      0          53
  -> 100.99.83.6:http             Masq    1      1          53
  -> 100.99.87.2:http             Masq    1      0          55
  -> 100.99.87.3:http             Masq    1      0          55
  -> 100.99.87.9:http             Masq    1      0          55
  -> 100.99.91.4:http             Masq    1      0          53
  -> 100.99.91.5:http             Masq    1      0          55
  -> 100.99.101.8:http            Masq    1      0          55
  -> 100.99.101.9:http            Masq    1      1          54
  -> 100.99.102.3:http            Masq    1      1          54
  -> 100.99.102.4:http            Masq    1      1          53
  -> 100.99.102.5:http            Masq    1      0          56
  -> 100.99.102.6:http            Masq    1      0          54
  -> 100.99.104.2:http            Masq    1      0          55
  -> 100.99.104.3:http            Masq    1      0          54
  -> 100.99.104.4:http            Masq    1      0          56
  -> 100.99.106.5:http            Masq    1      0          55
  -> 100.99.107.2:http            Masq    1      1          54
  -> 100.99.107.3:http            Masq    1      1          56
  -> 100.99.107.4:http            Masq    1      0          54
  -> 100.99.109.2:http            Masq    1      0          56
  -> 100.99.109.3:http            Masq    1      0          55
  -> 100.99.109.4:http            Masq    1      0          55
  -> 100.99.109.5:http            Masq    1      1          53
  -> 100.99.110.3:http            Masq    1      1          55
  -> 100.99.110.4:http            Masq    1      2          51
  -> 100.99.110.5:http            Masq    1      0          54
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.126.28:3000           Masq    1      0          8
  -> 100.98.253.19:3000           Masq    1      0          8
  -> 100.99.106.8:3000            Masq    1      0          8
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.16.6:3000             Masq    1      1          14
  -> 100.99.102.2:3000            Masq    1      1          14
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.125.26:postgresql     Masq    1      0          0
  -> 100.98.253.20:postgresql     Masq    1      0          1
  -> 100.99.0.12:postgresql       Masq    1      0          1
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.12.67:http            Masq    1      2          12
  -> 100.96.12.68:http            Masq    1      0          12
  -> 100.97.229.182:http          Masq    1      0          12
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.142.30:http-alt       Masq    1      1          1
  -> 100.98.125.28:http-alt       Masq    1      0          3
  -> 100.98.125.29:http-alt       Masq    1      0          2
  -> 100.98.126.45:http-alt       Masq    1      0          2
  -> 100.98.151.20:http-alt       Masq    1      1          2
  -> 100.98.151.21:http-alt       Masq    1      0          3
  -> 100.98.162.28:http-alt       Masq    1      0          3
  -> 100.98.180.23:http-alt       Masq    1      0          3
  -> 100.98.180.26:http-alt       Masq    1      0          2
  -> 100.98.191.22:http-alt       Masq    1      1          2
  -> 100.98.191.23:http-alt       Masq    1      0          2
  -> 100.98.191.24:http-alt       Masq    1      1          2
  -> 100.98.237.19:http-alt       Masq    1      0          2
  -> 100.98.237.20:http-alt       Masq    1      0          2
  -> 100.99.0.13:http-alt         Masq    1      0          3
  -> 100.99.0.20:http-alt         Masq    1      0          3
  -> 100.99.0.23:http-alt         Masq    1      0          3
  -> 100.99.16.16:http-alt        Masq    1      0          2
  -> 100.99.25.11:http-alt        Masq    1      0          2
  -> 100.99.25.12:http-alt        Masq    1      0          2
  -> 100.99.25.13:http-alt        Masq    1      0          2
  -> 100.99.29.2:http-alt         Masq    1      0          2
  -> 100.99.29.3:http-alt         Masq    1      0          2
  -> 100.99.29.4:http-alt         Masq    1      0          2
  -> 100.99.29.5:http-alt         Masq    1      1          2
  -> 100.99.29.6:http-alt         Masq    1      0          2
  -> 100.99.29.7:http-alt         Masq    1      0          3
  -> 100.99.31.2:http-alt         Masq    1      0          2
  -> 100.99.31.3:http-alt         Masq    1      0          3
  -> 100.99.31.4:http-alt         Masq    1      0          2
  -> 100.99.31.5:http-alt         Masq    1      0          2
  -> 100.99.31.6:http-alt         Masq    1      1          2
  -> 100.99.31.7:http-alt         Masq    1      0          2
  -> 100.99.34.2:http-alt         Masq    1      0          3
  -> 100.99.34.3:http-alt         Masq    1      0          2
  -> 100.99.34.4:http-alt         Masq    1      0          2
  -> 100.99.34.5:http-alt         Masq    1      0          2
  -> 100.99.34.6:http-alt         Masq    1      0          3
  -> 100.99.34.7:http-alt         Masq    1      0          2
  -> 100.99.43.4:http-alt         Masq    1      0          2
  -> 100.99.43.7:http-alt         Masq    1      1          2
  -> 100.99.51.2:http-alt         Masq    1      0          2
  -> 100.99.51.5:http-alt         Masq    1      1          2
  -> 100.99.63.20:http-alt        Masq    1      0          2
  -> 100.99.63.21:http-alt        Masq    1      0          2
  -> 100.99.72.14:http-alt        Masq    1      0          2
  -> 100.99.72.15:http-alt        Masq    1      0          2
  -> 100.99.76.2:http-alt         Masq    1      0          2
  -> 100.99.76.3:http-alt         Masq    1      0          2
  -> 100.99.76.4:http-alt         Masq    1      0          2
  -> 100.99.76.9:http-alt         Masq    1      0          2
  -> 100.99.77.5:http-alt         Masq    1      0          2
  -> 100.99.81.3:http-alt         Masq    1      0          2
  -> 100.99.81.4:http-alt         Masq    1      0          2
  -> 100.99.87.5:http-alt         Masq    1      0          2
  -> 100.99.101.11:http-alt       Masq    1      0          2
  -> 100.99.101.12:http-alt       Masq    1      0          2
  -> 100.99.104.30:http-alt       Masq    1      1          1
  -> 100.99.104.31:http-alt       Masq    1      0          2
  -> 100.99.106.2:http-alt        Masq    1      0          2
  -> 100.99.106.3:http-alt        Masq    1      0          2
  -> 100.99.106.4:http-alt        Masq    1      0          2
  -> 100.99.107.6:http-alt        Masq    1      0          2
  -> 100.102.230.17:http-alt      Masq    1      0          3
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.142.30:http-alt       Masq    1      1          9
  -> 100.98.125.28:http-alt       Masq    1      0          10
  -> 100.98.125.29:http-alt       Masq    1      0          10
  -> 100.98.126.45:http-alt       Masq    1      1          9
  -> 100.98.151.20:http-alt       Masq    1      1          9
  -> 100.98.151.21:http-alt       Masq    1      1          9
  -> 100.98.162.28:http-alt       Masq    1      1          9
  -> 100.98.180.23:http-alt       Masq    1      1          9
  -> 100.98.180.26:http-alt       Masq    1      1          9
  -> 100.98.191.22:http-alt       Masq    1      0          9
  -> 100.98.191.23:http-alt       Masq    1      1          9
  -> 100.98.191.24:http-alt       Masq    1      1          9
  -> 100.98.237.19:http-alt       Masq    1      1          9
  -> 100.98.237.20:http-alt       Masq    1      0          10
  -> 100.99.0.13:http-alt         Masq    1      1          9
  -> 100.99.0.20:http-alt         Masq    1      1          9
  -> 100.99.0.23:http-alt         Masq    1      0          10
  -> 100.99.16.16:http-alt        Masq    1      1          9
  -> 100.99.25.11:http-alt        Masq    1      1          9
  -> 100.99.25.12:http-alt        Masq    1      1          9
  -> 100.99.25.13:http-alt        Masq    1      1          9
  -> 100.99.29.2:http-alt         Masq    1      0          10
  -> 100.99.29.3:http-alt         Masq    1      1          9
  -> 100.99.29.4:http-alt         Masq    1      1          9
  -> 100.99.29.5:http-alt         Masq    1      1          9
  -> 100.99.29.6:http-alt         Masq    1      0          10
  -> 100.99.29.7:http-alt         Masq    1      1          9
  -> 100.99.31.2:http-alt         Masq    1      1          9
  -> 100.99.31.3:http-alt         Masq    1      0          9
  -> 100.99.31.4:http-alt         Masq    1      1          9
  -> 100.99.31.5:http-alt         Masq    1      1          9
  -> 100.99.31.6:http-alt         Masq    1      0          9
  -> 100.99.31.7:http-alt         Masq    1      1          9
  -> 100.99.34.2:http-alt         Masq    1      1          9
  -> 100.99.34.3:http-alt         Masq    1      1          9
  -> 100.99.34.4:http-alt         Masq    1      0          10
  -> 100.99.34.5:http-alt         Masq    1      1          9
  -> 100.99.34.6:http-alt         Masq    1      1          9
  -> 100.99.34.7:http-alt         Masq    1      1          9
  -> 100.99.43.4:http-alt         Masq    1      0          10
  -> 100.99.43.7:http-alt         Masq    1      1          9
  -> 100.99.51.2:http-alt         Masq    1      0          10
  -> 100.99.51.5:http-alt         Masq    1      1          9
  -> 100.99.63.20:http-alt        Masq    1      1          9
  -> 100.99.63.21:http-alt        Masq    1      0          10
  -> 100.99.72.14:http-alt        Masq    1      0          9
  -> 100.99.72.15:http-alt        Masq    1      1          9
  -> 100.99.76.2:http-alt         Masq    1      0          10
  -> 100.99.76.3:http-alt         Masq    1      1          8
  -> 100.99.76.4:http-alt         Masq    1      0          10
  -> 100.99.76.9:http-alt         Masq    1      0          10
  -> 100.99.77.5:http-alt         Masq    1      0          10
  -> 100.99.81.3:http-alt         Masq    1      0          10
  -> 100.99.81.4:http-alt         Masq    1      0          9
  -> 100.99.87.5:http-alt         Masq    1      0          9
  -> 100.99.101.11:http-alt       Masq    1      0          9
  -> 100.99.101.12:http-alt       Masq    1      0          9
  -> 100.99.104.30:http-alt       Masq    1      0          9
  -> 100.99.104.31:http-alt       Masq    1      0          9
  -> 100.99.106.2:http-alt        Masq    1      0          9
  -> 100.99.106.3:http-alt        Masq    1      0          9
  -> 100.99.106.4:http-alt        Masq    1      0          9
  -> 100.99.107.6:http-alt        Masq    1      0          9
  -> 100.102.230.17:http-alt      Masq    1      0          10
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.12.71:http            Masq    1      0          4
  -> 100.97.228.242:http          Masq    1      1          4
  -> 100.97.229.186:http          Masq    1      1          4
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.125.26:3000           Masq    1      1          8
  -> 100.98.253.20:3000           Masq    1      0          10
  -> 100.99.0.12:3000             Masq    1      1          10
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.12.67:http            Masq    1      1          4
  -> 100.96.12.68:http            Masq    1      1          4
  -> 100.97.229.182:http          Masq    1      0          4
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.97.229.163:http          Masq    1      1          19
  -> 100.104.84.23:http           Masq    1      1          18
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.23:9696             Masq    1      2          4
  -> 100.97.142.26:9696           Masq    1      3          4
  -> 100.98.44.26:9696            Masq    1      3          4
  -> 100.98.126.34:9696           Masq    1      2          5
  -> 100.98.233.3:9696            Masq    1      2          4
  -> 100.99.5.49:9696             Masq    1      2          5
  -> 100.99.5.50:9696             Masq    1      2          4
  -> 100.99.43.3:9696             Masq    1      2          5
  -> 100.99.76.8:9696             Masq    1      2          4
  -> 100.99.77.3:9696             Masq    1      2          4
  -> 100.99.78.3:9696             Masq    1      2          4
  -> 100.99.81.2:9696             Masq    1      2          4
  -> 100.99.87.7:9696             Masq    1      2          4
  -> 100.99.106.10:9696           Masq    1      3          4
  -> 100.99.110.2:9696            Masq    1      3          5
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.18:9109             Masq    1      0          24
TCP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.44.25:postgresql      Masq    1      0          1
  -> 100.99.0.15:postgresql       Masq    1      0          2
TCP  100.64.0.1:https rr persistent 10800
  -> ip-10-0-62-36.ec2.internal:h Masq    1      1          0
  -> ip-10-0-62-114.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-162.ec2.internal: Masq    1      0          0
TCP  100.64.0.10:domain rr
  -> 100.98.125.27:domain         Masq    1      0          0
  -> 100.98.162.22:domain         Masq    1      0          0
  -> 100.99.5.48:domain           Masq    1      0          0
  -> 100.99.76.6:domain           Masq    1      0          0
  -> 100.99.77.8:domain           Masq    1      0          0
  -> 100.99.78.5:domain           Masq    1      0          0
  -> 100.99.101.10:domain         Masq    1      0          0
  -> 100.99.106.9:domain          Masq    1      0          0
TCP  100.64.3.193:http rr
TCP  100.64.3.193:postgresql rr
TCP  100.64.105.182:9300 rr
  -> 100.98.125.16:9300           Masq    1      0          0
TCP  100.64.108.21:http rr
  -> 100.98.125.10:http           Masq    1      0          0
  -> 100.99.0.16:http             Masq    1      0          0
TCP  100.64.108.21:https rr
  -> 100.98.125.10:http           Masq    1      0          0
  -> 100.99.0.16:http             Masq    1      0          0
TCP  100.64.129.81:http rr
  -> 100.104.84.28:3000           Masq    1      0          0
  -> 100.104.84.29:3000           Masq    1      0          0
TCP  100.64.134.163:http rr
  -> 100.98.126.28:3000           Masq    1      0          0
  -> 100.98.253.19:3000           Masq    1      0          0
  -> 100.99.106.8:3000            Masq    1      0          0
TCP  100.64.134.163:postgresql rr
  -> 100.98.126.28:postgresql     Masq    1      0          0
  -> 100.98.253.19:postgresql     Masq    1      0          0
  -> 100.99.106.8:postgresql      Masq    1      0          0
TCP  100.64.146.70:http rr
  -> 100.96.12.74:3000            Masq    1      0          0
  -> 100.97.229.187:3000          Masq    1      0          0
  -> 100.104.84.44:3000           Masq    1      0          0
TCP  100.64.151.195:https rr
  -> 100.99.56.13:5601            Masq    1      0          0
TCP  100.64.242.48:bacula-fd rr
  -> 100.99.76.7:bacula-fd        Masq    1      0          0
TCP  100.65.17.84:http rr
  -> 100.97.142.17:http-alt       Masq    1      0          0
TCP  100.65.17.84:https rr
  -> 100.97.142.17:http-alt       Masq    1      0          0
TCP  100.65.59.70:http rr
  -> 100.97.16.6:3000             Masq    1      0          0
  -> 100.99.102.2:3000            Masq    1      0          0
TCP  100.65.82.182:http-alt rr
  -> 100.98.125.32:http-alt       Masq    1      0          0
  -> 100.99.16.20:http-alt        Masq    1      0          0
TCP  100.65.95.158:https rr
  -> 100.97.228.238:http          Masq    1      0          0
  -> 100.97.229.184:http          Masq    1      0          0
TCP  100.65.208.89:http rr
  -> 100.96.6.23:9696             Masq    1      0          0
  -> 100.97.142.26:9696           Masq    1      0          0
  -> 100.98.44.26:9696            Masq    1      0          0
  -> 100.98.126.34:9696           Masq    1      0          0
  -> 100.98.233.3:9696            Masq    1      0          0
  -> 100.99.5.49:9696             Masq    1      0          0
  -> 100.99.5.50:9696             Masq    1      0          0
  -> 100.99.43.3:9696             Masq    1      0          0
  -> 100.99.76.8:9696             Masq    1      0          0
  -> 100.99.77.3:9696             Masq    1      0          0
  -> 100.99.78.3:9696             Masq    1      0          0
  -> 100.99.81.2:9696             Masq    1      0          0
  -> 100.99.87.7:9696             Masq    1      0          0
  -> 100.99.106.10:9696           Masq    1      0          0
  -> 100.99.110.2:9696            Masq    1      0          0
TCP  100.65.208.89:https rr
  -> 100.96.6.23:9696             Masq    1      0          0
  -> 100.97.142.26:9696           Masq    1      0          0
  -> 100.98.44.26:9696            Masq    1      0          0
  -> 100.98.126.34:9696           Masq    1      0          0
  -> 100.98.233.3:9696            Masq    1      0          0
  -> 100.99.5.49:9696             Masq    1      0          0
  -> 100.99.5.50:9696             Masq    1      0          0
  -> 100.99.43.3:9696             Masq    1      0          0
  -> 100.99.76.8:9696             Masq    1      0          0
  -> 100.99.77.3:9696             Masq    1      0          0
  -> 100.99.78.3:9696             Masq    1      0          0
  -> 100.99.81.2:9696             Masq    1      0          0
  -> 100.99.87.7:9696             Masq    1      0          0
  -> 100.99.106.10:9696           Masq    1      0          0
  -> 100.99.110.2:9696            Masq    1      0          0
TCP  100.65.234.176:59858 rr
TCP  100.65.248.83:5201 rr
TCP  100.65.248.83:12865 rr
TCP  100.66.53.115:9300 rr
  -> 100.98.125.16:9300           Masq    1      0          0
TCP  100.66.89.115:http rr
  -> 100.97.142.15:9115           Masq    1      0          0
TCP  100.66.140.117:http rr
  -> 100.99.5.47:8082             Masq    1      0          0
TCP  100.66.193.52:http rr
  -> 100.98.44.25:3000            Masq    1      0          0
  -> 100.99.0.15:3000             Masq    1      0          0
TCP  100.66.193.52:postgresql rr
  -> 100.98.44.25:postgresql      Masq    1      0          0
  -> 100.99.0.15:postgresql       Masq    1      0          0
TCP  100.66.199.17:http rr
  -> 100.96.6.22:http             Masq    1      0          0
  -> 100.97.16.8:http             Masq    1      0          0
  -> 100.97.142.16:http           Masq    1      0          0
  -> 100.98.180.22:http           Masq    1      0          0
TCP  100.66.199.17:https rr
  -> 100.96.6.22:http             Masq    1      0          0
  -> 100.97.16.8:http             Masq    1      0          0
  -> 100.97.142.16:http           Masq    1      0          0
  -> 100.98.180.22:http           Masq    1      0          0
TCP  100.66.225.170:http rr
  -> 100.96.12.67:http            Masq    1      0          0
  -> 100.96.12.68:http            Masq    1      0          0
  -> 100.97.229.182:http          Masq    1      0          0
TCP  100.66.225.170:https rr
  -> 100.96.12.67:http            Masq    1      0          0
  -> 100.96.12.68:http            Masq    1      0          0
  -> 100.97.229.182:http          Masq    1      0          0
TCP  100.66.249.244:5202 rr
TCP  100.67.44.228:9109 rr
  -> 100.96.6.18:9109             Masq    1      0          0
TCP  100.67.52.38:http rr
  -> 100.104.84.22:http           Masq    1      0          0
TCP  100.67.52.38:https rr
  -> 100.104.84.22:http           Masq    1      0          0
TCP  100.67.93.66:9090 rr
  -> 100.96.5.4:9090              Masq    1      0          0
TCP  100.67.99.191:http rr
  -> 100.96.12.71:http            Masq    1      0          0
  -> 100.97.228.242:http          Masq    1      0          0
  -> 100.97.229.186:http          Masq    1      0          0
TCP  100.67.99.191:https rr
  -> 100.96.12.71:http            Masq    1      0          0
  -> 100.97.228.242:http          Masq    1      0          0
  -> 100.97.229.186:http          Masq    1      0          0
TCP  100.67.121.197:http rr
  -> 100.97.228.222:3000          Masq    1      0          0
  -> 100.97.229.162:3000          Masq    1      0          0
  -> 100.104.84.24:3000           Masq    1      0          0
TCP  100.67.139.109:https rr
  -> 100.98.191.21:5601           Masq    1      0          0
TCP  100.68.54.76:http rr
  -> 100.96.1.2:9090              Masq    1      0          0
TCP  100.68.89.1:4140 rr
  -> ip-10-0-62-10.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-11.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-12.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-15.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-25.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-53.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-54.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-62.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-65.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-66.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-67.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-72.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-81.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-82.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-92.ec2.internal:4 Masq    1      1          0
  -> ip-10-0-62-93.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-94.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-110.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-121.ec2.internal: Masq    1      1          0
  -> ip-10-0-62-124.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-125.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-131.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-132.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-142.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-151.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-152.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-158.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-180.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-191.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-197.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-200.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-204.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-205.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-207.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-215.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-227.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-229.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-231.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-236.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-241.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-242.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-247.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-252.ec2.internal: Masq    1      0          0
TCP  100.68.89.1:4141 rr
  -> ip-10-0-62-10.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-11.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-12.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-15.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-25.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-53.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-54.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-62.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-65.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-66.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-67.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-72.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-81.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-82.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-92.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-93.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-94.ec2.internal:4 Masq    1      0          0
  -> ip-10-0-62-110.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-121.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-124.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-125.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-131.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-132.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-142.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-151.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-152.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-158.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-180.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-191.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-197.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-200.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-204.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-205.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-207.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-215.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-227.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-229.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-231.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-236.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-241.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-242.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-247.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-252.ec2.internal: Masq    1      0          0
TCP  100.68.89.1:9990 rr
  -> ip-10-0-62-10.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-11.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-12.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-15.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-25.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-53.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-54.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-62.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-65.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-66.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-67.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-72.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-81.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-82.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-92.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-93.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-94.ec2.internal:9 Masq    1      0          0
  -> ip-10-0-62-110.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-121.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-124.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-125.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-131.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-132.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-142.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-151.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-152.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-158.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-180.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-191.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-197.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-200.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-204.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-205.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-207.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-215.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-227.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-229.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-231.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-236.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-241.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-242.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-247.ec2.internal: Masq    1      0          0
  -> ip-10-0-62-252.ec2.internal: Masq    1      0          0
TCP  100.68.198.73:http rr
  -> 100.97.16.10:http            Masq    1      0          0
  -> 100.97.142.28:http           Masq    1      0          0
  -> 100.97.142.29:http           Masq    1      0          0
  -> 100.98.125.30:http           Masq    1      0          0
  -> 100.98.125.31:http           Masq    1      0          0
  -> 100.98.126.42:http           Masq    1      0          0
  -> 100.98.126.43:http           Masq    1      0          0
  -> 100.98.126.44:http           Masq    1      0          0
  -> 100.98.151.17:http           Masq    1      0          0
  -> 100.98.151.18:http           Masq    1      0          0
  -> 100.98.162.24:http           Masq    1      0          0
  -> 100.98.162.25:http           Masq    1      0          0
  -> 100.98.162.26:http           Masq    1      0          0
  -> 100.98.162.27:http           Masq    1      0          0
  -> 100.98.180.24:http           Masq    1      0          0
  -> 100.98.180.25:http           Masq    1      0          0
  -> 100.98.191.29:http           Masq    1      0          0
  -> 100.98.205.3:http            Masq    1      0          0
  -> 100.98.237.21:http           Masq    1      0          0
  -> 100.98.237.22:http           Masq    1      0          0
  -> 100.98.237.23:http           Masq    1      0          0
  -> 100.98.240.8:http            Masq    1      0          0
  -> 100.98.240.9:http            Masq    1      0          0
  -> 100.98.240.10:http           Masq    1      0          0
  -> 100.98.240.11:http           Masq    1      0          0
  -> 100.98.240.12:http           Masq    1      0          0
  -> 100.98.253.23:http           Masq    1      0          0
  -> 100.98.253.24:http           Masq    1      0          0
  -> 100.98.253.25:http           Masq    1      0          0
  -> 100.98.253.26:http           Masq    1      0          0
  -> 100.99.0.21:http             Masq    1      0          0
  -> 100.99.0.22:http             Masq    1      0          0
  -> 100.99.5.53:http             Masq    1      0          0
  -> 100.99.5.54:http             Masq    1      0          0
  -> 100.99.5.55:http             Masq    1      0          0
  -> 100.99.16.17:http            Masq    1      0          0
  -> 100.99.16.18:http            Masq    1      0          0
  -> 100.99.16.19:http            Masq    1      0          0
  -> 100.99.25.14:http            Masq    1      0          0
  -> 100.99.25.15:http            Masq    1      0          0
  -> 100.99.43.5:http             Masq    1      0          0
  -> 100.99.51.3:http             Masq    1      0          0
  -> 100.99.51.6:http             Masq    1      0          0
  -> 100.99.63.2:http             Masq    1      0          0
  -> 100.99.63.3:http             Masq    1      0          0
  -> 100.99.68.2:http             Masq    1      0          0
  -> 100.99.68.3:http             Masq    1      0          0
  -> 100.99.68.4:http             Masq    1      0          0
  -> 100.99.68.10:http            Masq    1      0          0
  -> 100.99.68.11:http            Masq    1      0          0
  -> 100.99.70.2:http             Masq    1      0          0
  -> 100.99.70.3:http             Masq    1      0          0
  -> 100.99.72.2:http             Masq    1      0          0
  -> 100.99.77.2:http             Masq    1      0          0
  -> 100.99.77.4:http             Masq    1      0          0
  -> 100.99.77.6:http             Masq    1      0          0
  -> 100.99.78.2:http             Masq    1      0          0
  -> 100.99.78.4:http             Masq    1      0          0
  -> 100.99.81.5:http             Masq    1      0          0
  -> 100.99.83.3:http             Masq    1      0          0
  -> 100.99.83.5:http             Masq    1      0          0
  -> 100.99.83.6:http             Masq    1      0          0
  -> 100.99.87.2:http             Masq    1      0          0
  -> 100.99.87.3:http             Masq    1      0          0
  -> 100.99.87.9:http             Masq    1      0          0
  -> 100.99.91.4:http             Masq    1      0          0
  -> 100.99.91.5:http             Masq    1      0          0
  -> 100.99.101.8:http            Masq    1      0          0
  -> 100.99.101.9:http            Masq    1      0          0
  -> 100.99.102.3:http            Masq    1      0          0
  -> 100.99.102.4:http            Masq    1      0          0
  -> 100.99.102.5:http            Masq    1      0          0
  -> 100.99.102.6:http            Masq    1      0          0
  -> 100.99.104.2:http            Masq    1      0          0
  -> 100.99.104.3:http            Masq    1      0          0
  -> 100.99.104.4:http            Masq    1      0          0
  -> 100.99.106.5:http            Masq    1      0          0
  -> 100.99.107.2:http            Masq    1      0          0
  -> 100.99.107.3:http            Masq    1      0          0
  -> 100.99.107.4:http            Masq    1      0          0
  -> 100.99.109.2:http            Masq    1      0          0
  -> 100.99.109.3:http            Masq    1      0          0
  -> 100.99.109.4:http            Masq    1      0          0
  -> 100.99.109.5:http            Masq    1      0          0
  -> 100.99.110.3:http            Masq    1      0          0
  -> 100.99.110.4:http            Masq    1      0          0
  -> 100.99.110.5:http            Masq    1      0          0
TCP  100.68.198.73:https rr
  -> 100.97.16.10:http            Masq    1      0          0
  -> 100.97.142.28:http           Masq    1      0          0
  -> 100.97.142.29:http           Masq    1      0          0
  -> 100.98.125.30:http           Masq    1      0          0
  -> 100.98.125.31:http           Masq    1      0          0
  -> 100.98.126.42:http           Masq    1      0          0
  -> 100.98.126.43:http           Masq    1      0          0
  -> 100.98.126.44:http           Masq    1      0          0
  -> 100.98.151.17:http           Masq    1      0          0
  -> 100.98.151.18:http           Masq    1      0          0
  -> 100.98.162.24:http           Masq    1      0          0
  -> 100.98.162.25:http           Masq    1      0          0
  -> 100.98.162.26:http           Masq    1      0          0
  -> 100.98.162.27:http           Masq    1      0          0
  -> 100.98.180.24:http           Masq    1      0          0
  -> 100.98.180.25:http           Masq    1      0          0
  -> 100.98.191.29:http           Masq    1      0          0
  -> 100.98.205.3:http            Masq    1      0          0
  -> 100.98.237.21:http           Masq    1      0          0
  -> 100.98.237.22:http           Masq    1      0          0
  -> 100.98.237.23:http           Masq    1      0          0
  -> 100.98.240.8:http            Masq    1      0          0
  -> 100.98.240.9:http            Masq    1      0          0
  -> 100.98.240.10:http           Masq    1      0          0
  -> 100.98.240.11:http           Masq    1      0          0
  -> 100.98.240.12:http           Masq    1      0          0
  -> 100.98.253.23:http           Masq    1      0          0
  -> 100.98.253.24:http           Masq    1      0          0
  -> 100.98.253.25:http           Masq    1      0          0
  -> 100.98.253.26:http           Masq    1      0          0
  -> 100.99.0.21:http             Masq    1      0          0
  -> 100.99.0.22:http             Masq    1      0          0
  -> 100.99.5.53:http             Masq    1      0          0
  -> 100.99.5.54:http             Masq    1      0          0
  -> 100.99.5.55:http             Masq    1      0          0
  -> 100.99.16.17:http            Masq    1      0          0
  -> 100.99.16.18:http            Masq    1      0          0
  -> 100.99.16.19:http            Masq    1      0          0
  -> 100.99.25.14:http            Masq    1      0          0
  -> 100.99.25.15:http            Masq    1      0          0
  -> 100.99.43.5:http             Masq    1      0          0
  -> 100.99.51.3:http             Masq    1      0          0
  -> 100.99.51.6:http             Masq    1      0          0
  -> 100.99.63.2:http             Masq    1      0          0
  -> 100.99.63.3:http             Masq    1      0          0
  -> 100.99.68.2:http             Masq    1      0          0
  -> 100.99.68.3:http             Masq    1      0          0
  -> 100.99.68.4:http             Masq    1      0          0
  -> 100.99.68.10:http            Masq    1      0          0
  -> 100.99.68.11:http            Masq    1      0          0
  -> 100.99.70.2:http             Masq    1      0          0
  -> 100.99.70.3:http             Masq    1      0          0
  -> 100.99.72.2:http             Masq    1      0          0
  -> 100.99.77.2:http             Masq    1      0          0
  -> 100.99.77.4:http             Masq    1      0          0
  -> 100.99.77.6:http             Masq    1      0          0
  -> 100.99.78.2:http             Masq    1      0          0
  -> 100.99.78.4:http             Masq    1      0          0
  -> 100.99.81.5:http             Masq    1      0          0
  -> 100.99.83.3:http             Masq    1      0          0
  -> 100.99.83.5:http             Masq    1      0          0
  -> 100.99.83.6:http             Masq    1      0          0
  -> 100.99.87.2:http             Masq    1      0          0
  -> 100.99.87.3:http             Masq    1      0          0
  -> 100.99.87.9:http             Masq    1      0          0
  -> 100.99.91.4:http             Masq    1      0          0
  -> 100.99.91.5:http             Masq    1      0          0
  -> 100.99.101.8:http            Masq    1      0          0
  -> 100.99.101.9:http            Masq    1      0          0
  -> 100.99.102.3:http            Masq    1      0          0
  -> 100.99.102.4:http            Masq    1      0          0
  -> 100.99.102.5:http            Masq    1      0          0
  -> 100.99.102.6:http            Masq    1      0          0
  -> 100.99.104.2:http            Masq    1      0          0
  -> 100.99.104.3:http            Masq    1      0          0
  -> 100.99.104.4:http            Masq    1      0          0
  -> 100.99.106.5:http            Masq    1      0          0
  -> 100.99.107.2:http            Masq    1      0          0
  -> 100.99.107.3:http            Masq    1      0          0
  -> 100.99.107.4:http            Masq    1      0          0
  -> 100.99.109.2:http            Masq    1      0          0
  -> 100.99.109.3:http            Masq    1      0          0
  -> 100.99.109.4:http            Masq    1      0          0
  -> 100.99.109.5:http            Masq    1      0          0
  -> 100.99.110.3:http            Masq    1      0          0
  -> 100.99.110.4:http            Masq    1      0          0
  -> 100.99.110.5:http            Masq    1      0          0
TCP  100.69.56.176:bacula-fd rr
  -> 100.99.43.2:bacula-fd        Masq    1      0          0
  -> 100.99.70.6:bacula-fd        Masq    1      0          0
TCP  100.69.56.176:9125 rr
  -> 100.99.43.2:9125             Masq    1      0          0
  -> 100.99.70.6:9125             Masq    1      0          0
TCP  100.69.113.25:http rr
  -> 100.97.228.240:3000          Masq    1      0          0
  -> 100.97.229.185:3000          Masq    1      0          0
TCP  100.69.151.226:http rr
  -> 100.104.84.136:http          Masq    1      0          0
TCP  100.69.161.88:http rr
  -> 100.96.12.33:http            Masq    1      0          0
  -> 100.97.228.200:http          Masq    1      0          0
TCP  100.69.187.230:3000 rr
  -> 100.98.125.26:3000           Masq    1      0          0
  -> 100.98.253.20:3000           Masq    1      0          0
  -> 100.99.0.12:3000             Masq    1      0          0
TCP  100.69.187.230:postgresql rr
  -> 100.98.125.26:postgresql     Masq    1      0          0
  -> 100.98.253.20:postgresql     Masq    1      0          0
  -> 100.99.0.12:postgresql       Masq    1      0          0
TCP  100.69.216.160:9092 rr
  -> ip-10-0-62-92.ec2.internal:9 Masq    1      0          0
TCP  100.69.218.7:http rr
  -> 100.96.12.52:3000            Masq    1      0          0
TCP  100.70.25.235:24224 rr
  -> 100.98.191.27:24224          Masq    1      0          0
  -> 100.99.70.7:24224            Masq    1      0          0
  -> 100.99.78.6:24224            Masq    1      0          0
TCP  100.70.25.235:24284 rr
  -> 100.98.191.27:24284          Masq    1      0          0
  -> 100.99.70.7:24284            Masq    1      0          0
  -> 100.99.78.6:24284            Masq    1      0          0
TCP  100.70.65.61:9108 rr
  -> 100.96.6.18:9108             Masq    1      0          0
TCP  100.70.65.61:9109 rr
  -> 100.96.6.18:9109             Masq    1      0          0
TCP  100.70.77.196:http rr
  -> 100.97.228.243:3000          Masq    1      0          0
  -> 100.104.84.46:3000           Masq    1      0          0
TCP  100.70.83.192:24224 rr
  -> 100.99.51.4:24224            Masq    1      0          0
  -> 100.99.81.6:24224            Masq    1      0          0
TCP  100.70.83.192:24284 rr
  -> 100.99.51.4:24284            Masq    1      0          0
  -> 100.99.81.6:24284            Masq    1      0          0
TCP  100.70.130.224:http rr
  -> 100.99.68.9:http-alt         Masq    1      0          0
TCP  100.70.130.224:https rr
  -> 100.99.68.9:http-alt         Masq    1      0          0
TCP  100.70.155.232:http rr
  -> 100.96.12.71:http            Masq    1      0          0
  -> 100.97.228.242:http          Masq    1      0          0
  -> 100.97.229.186:http          Masq    1      0          0
TCP  100.70.155.232:https rr
  -> 100.96.12.71:http            Masq    1      0          0
  -> 100.97.228.242:http          Masq    1      0          0
  -> 100.97.229.186:http          Masq    1      0          0
TCP  100.70.158.21:http rr
  -> 100.96.12.34:3000            Masq    1      0          0
  -> 100.97.229.154:3000          Masq    1      0          0
  -> 100.104.84.6:3000            Masq    1      0          0
TCP  100.70.208.157:https rr
  -> 100.96.6.13:http-alt         Masq    1      0          0
  -> 100.98.233.4:http-alt        Masq    1      0          0
TCP  100.71.12.39:https rr
  -> 100.97.229.188:http          Masq    1      0          0
  -> 100.104.84.45:http           Masq    1      0          0
TCP  100.71.70.221:http rr
  -> 100.104.84.137:3000          Masq    1      0          0
TCP  100.71.87.26:http rr
  -> 100.97.142.30:http-alt       Masq    1      0          0
  -> 100.98.125.28:http-alt       Masq    1      0          0
  -> 100.98.125.29:http-alt       Masq    1      0          0
  -> 100.98.126.45:http-alt       Masq    1      0          0
  -> 100.98.151.20:http-alt       Masq    1      0          0
  -> 100.98.151.21:http-alt       Masq    1      0          0
  -> 100.98.162.28:http-alt       Masq    1      0          0
  -> 100.98.180.23:http-alt       Masq    1      0          0
  -> 100.98.180.26:http-alt       Masq    1      0          0
  -> 100.98.191.22:http-alt       Masq    1      0          0
  -> 100.98.191.23:http-alt       Masq    1      0          0
  -> 100.98.191.24:http-alt       Masq    1      0          0
  -> 100.98.237.19:http-alt       Masq    1      0          0
  -> 100.98.237.20:http-alt       Masq    1      0          0
  -> 100.99.0.13:http-alt         Masq    1      0          0
  -> 100.99.0.20:http-alt         Masq    1      0          0
  -> 100.99.0.23:http-alt         Masq    1      0          0
  -> 100.99.16.16:http-alt        Masq    1      0          0
  -> 100.99.25.11:http-alt        Masq    1      0          0
  -> 100.99.25.12:http-alt        Masq    1      0          0
  -> 100.99.25.13:http-alt        Masq    1      0          0
  -> 100.99.29.2:http-alt         Masq    1      0          0
  -> 100.99.29.3:http-alt         Masq    1      0          0
  -> 100.99.29.4:http-alt         Masq    1      0          0
  -> 100.99.29.5:http-alt         Masq    1      0          0
  -> 100.99.29.6:http-alt         Masq    1      0          0
  -> 100.99.29.7:http-alt         Masq    1      0          0
  -> 100.99.31.2:http-alt         Masq    1      0          0
  -> 100.99.31.3:http-alt         Masq    1      0          0
  -> 100.99.31.4:http-alt         Masq    1      0          0
  -> 100.99.31.5:http-alt         Masq    1      0          0
  -> 100.99.31.6:http-alt         Masq    1      0          0
  -> 100.99.31.7:http-alt         Masq    1      0          0
  -> 100.99.34.2:http-alt         Masq    1      0          0
  -> 100.99.34.3:http-alt         Masq    1      0          0
  -> 100.99.34.4:http-alt         Masq    1      0          0
  -> 100.99.34.5:http-alt         Masq    1      0          0
  -> 100.99.34.6:http-alt         Masq    1      0          0
  -> 100.99.34.7:http-alt         Masq    1      0          0
  -> 100.99.43.4:http-alt         Masq    1      0          0
  -> 100.99.43.7:http-alt         Masq    1      0          0
  -> 100.99.51.2:http-alt         Masq    1      0          0
  -> 100.99.51.5:http-alt         Masq    1      0          0
  -> 100.99.63.20:http-alt        Masq    1      0          0
  -> 100.99.63.21:http-alt        Masq    1      0          0
  -> 100.99.72.14:http-alt        Masq    1      0          0
  -> 100.99.72.15:http-alt        Masq    1      0          0
  -> 100.99.76.2:http-alt         Masq    1      0          0
  -> 100.99.76.3:http-alt         Masq    1      0          0
  -> 100.99.76.4:http-alt         Masq    1      0          0
  -> 100.99.76.9:http-alt         Masq    1      0          0
  -> 100.99.77.5:http-alt         Masq    1      0          0
  -> 100.99.81.3:http-alt         Masq    1      0          0
  -> 100.99.81.4:http-alt         Masq    1      0          0
  -> 100.99.87.5:http-alt         Masq    1      0          0
  -> 100.99.101.11:http-alt       Masq    1      0          0
  -> 100.99.101.12:http-alt       Masq    1      0          0
  -> 100.99.104.30:http-alt       Masq    1      0          0
  -> 100.99.104.31:http-alt       Masq    1      0          0
  -> 100.99.106.2:http-alt        Masq    1      0          0
  -> 100.99.106.3:http-alt        Masq    1      0          0
  -> 100.99.106.4:http-alt        Masq    1      0          0
  -> 100.99.107.6:http-alt        Masq    1      0          0
  -> 100.102.230.17:http-alt      Masq    1      0          0
TCP  100.71.87.26:https rr
  -> 100.97.142.30:http-alt       Masq    1      0          0
  -> 100.98.125.28:http-alt       Masq    1      0          0
  -> 100.98.125.29:http-alt       Masq    1      0          0
  -> 100.98.126.45:http-alt       Masq    1      0          0
  -> 100.98.151.20:http-alt       Masq    1      0          0
  -> 100.98.151.21:http-alt       Masq    1      0          0
  -> 100.98.162.28:http-alt       Masq    1      0          0
  -> 100.98.180.23:http-alt       Masq    1      0          0
  -> 100.98.180.26:http-alt       Masq    1      0          0
  -> 100.98.191.22:http-alt       Masq    1      0          0
  -> 100.98.191.23:http-alt       Masq    1      0          0
  -> 100.98.191.24:http-alt       Masq    1      0          0
  -> 100.98.237.19:http-alt       Masq    1      0          0
  -> 100.98.237.20:http-alt       Masq    1      0          0
  -> 100.99.0.13:http-alt         Masq    1      0          0
  -> 100.99.0.20:http-alt         Masq    1      0          0
  -> 100.99.0.23:http-alt         Masq    1      0          0
  -> 100.99.16.16:http-alt        Masq    1      0          0
  -> 100.99.25.11:http-alt        Masq    1      0          0
  -> 100.99.25.12:http-alt        Masq    1      0          0
  -> 100.99.25.13:http-alt        Masq    1      0          0
  -> 100.99.29.2:http-alt         Masq    1      0          0
  -> 100.99.29.3:http-alt         Masq    1      0          0
  -> 100.99.29.4:http-alt         Masq    1      0          0
  -> 100.99.29.5:http-alt         Masq    1      0          0
  -> 100.99.29.6:http-alt         Masq    1      0          0
  -> 100.99.29.7:http-alt         Masq    1      0          0
  -> 100.99.31.2:http-alt         Masq    1      0          0
  -> 100.99.31.3:http-alt         Masq    1      0          0
  -> 100.99.31.4:http-alt         Masq    1      0          0
  -> 100.99.31.5:http-alt         Masq    1      0          0
  -> 100.99.31.6:http-alt         Masq    1      0          0
  -> 100.99.31.7:http-alt         Masq    1      0          0
  -> 100.99.34.2:http-alt         Masq    1      0          0
  -> 100.99.34.3:http-alt         Masq    1      0          0
  -> 100.99.34.4:http-alt         Masq    1      0          0
  -> 100.99.34.5:http-alt         Masq    1      0          0
  -> 100.99.34.6:http-alt         Masq    1      0          0
  -> 100.99.34.7:http-alt         Masq    1      0          0
  -> 100.99.43.4:http-alt         Masq    1      0          0
  -> 100.99.43.7:http-alt         Masq    1      0          0
  -> 100.99.51.2:http-alt         Masq    1      0          0
  -> 100.99.51.5:http-alt         Masq    1      0          0
  -> 100.99.63.20:http-alt        Masq    1      0          0
  -> 100.99.63.21:http-alt        Masq    1      0          0
  -> 100.99.72.14:http-alt        Masq    1      0          0
  -> 100.99.72.15:http-alt        Masq    1      0          0
  -> 100.99.76.2:http-alt         Masq    1      0          0
  -> 100.99.76.3:http-alt         Masq    1      0          0
  -> 100.99.76.4:http-alt         Masq    1      0          0
  -> 100.99.76.9:http-alt         Masq    1      0          0
  -> 100.99.77.5:http-alt         Masq    1      0          0
  -> 100.99.81.3:http-alt         Masq    1      0          0
  -> 100.99.81.4:http-alt         Masq    1      0          0
  -> 100.99.87.5:http-alt         Masq    1      0          0
  -> 100.99.101.11:http-alt       Masq    1      0          0
  -> 100.99.101.12:http-alt       Masq    1      0          0
  -> 100.99.104.30:http-alt       Masq    1      0          0
  -> 100.99.104.31:http-alt       Masq    1      0          0
  -> 100.99.106.2:http-alt        Masq    1      0          0
  -> 100.99.106.3:http-alt        Masq    1      0          0
  -> 100.99.106.4:http-alt        Masq    1      0          0
  -> 100.99.107.6:http-alt        Masq    1      0          0
  -> 100.102.230.17:http-alt      Masq    1      0          0
TCP  100.71.122.4:http rr
  -> 100.98.44.9:3000             Masq    1      0          0
  -> 100.98.151.16:3000           Masq    1      0          0
  -> 100.98.240.7:3000            Masq    1      0          0
TCP  100.71.122.4:postgresql rr
  -> 100.98.44.9:postgresql       Masq    1      0          0
  -> 100.98.151.16:postgresql     Masq    1      0          0
  -> 100.98.240.7:postgresql      Masq    1      0          0
TCP  100.71.152.70:2181 rr
  -> 100.99.43.6:2181             Masq    1      0          0
  -> 100.99.72.4:2181             Masq    1      0          0
  -> 100.99.107.5:2181            Masq    1      0          0
TCP  100.71.233.110:wap-wsp rr
  -> 100.98.125.16:wap-wsp        Masq    1      0          0
TCP  100.71.240.251:http rr
  -> 100.97.229.163:http          Masq    1      0          0
  -> 100.104.84.23:http           Masq    1      0          0
TCP  100.71.240.251:https rr
  -> 100.97.229.163:http          Masq    1      0          0
  -> 100.104.84.23:http           Masq    1      0          0
UDP  ip-10-0-62-121.ec2.internal: rr
  -> 100.96.6.18:9109             Masq    1      0          5
UDP  ip-10-0-62-121.ec2.internal: rr
  -> 100.98.253.21:8125           Masq    1      0          0
  -> 100.99.76.10:8125            Masq    1      0          0
UDP  ip-10-0-62-121.ec2.internal: rr
  -> 100.99.43.2:9125             Masq    1      0          0
  -> 100.99.70.6:9125             Masq    1      0          0
UDP  ip-10-0-62-121.ec2.internal: rr
UDP  ip-10-0-62-121.ec2.internal: rr
  -> 100.99.76.7:9125             Masq    1      0          0
UDP  100.64.0.10:domain rr
  -> 100.98.125.27:domain         Masq    1      0          28
  -> 100.98.162.22:domain         Masq    1      0          28
  -> 100.99.5.48:domain           Masq    1      0          28
  -> 100.99.76.6:domain           Masq    1      0          28
  -> 100.99.77.8:domain           Masq    1      0          28
  -> 100.99.78.5:domain           Masq    1      0          28
  -> 100.99.101.10:domain         Masq    1      0          28
  -> 100.99.106.9:domain          Masq    1      0          28
UDP  100.64.242.48:9125 rr
  -> 100.99.76.7:9125             Masq    1      0          0
UDP  100.65.248.83:5201 rr
UDP  100.66.79.253:9125 rr
UDP  100.69.56.176:9125 rr
  -> 100.99.43.2:9125             Masq    1      0          0
  -> 100.99.70.6:9125             Masq    1      0          0
UDP  100.70.26.165:8125 rr
  -> 100.98.253.21:8125           Masq    1      0          0
  -> 100.99.76.10:8125            Masq    1      0          0
UDP  100.70.65.61:9109 rr
  -> 100.96.6.18:9109             Masq    1      0          0

The node in question is 10.0.62.10 (ip-10-0-62-10).

It logs on its kernel log:

May 02 13:59:42 ip-10-0-62-10 kernel:  IPVS: rr: TCP 10.0.62.10:30472 - no destination available

Sorry if I borrowed the @Jacky2 issue. Maybe it's not related to his case.

@johanot
Yes, it will deplete the memory of my server node, making it impossible for me to log on to my node server.

@murali-reddy no, i do not configure the service is marked with externalTrafficPolicy=Local.

@tonidas Maybe your situation is the same as mine

@murali-reddy This is the ipvsadm situation of the node server with problems.
20180507085951

@Jacky2 I do see many services with 0 endpoints (or pods backing the service). Are you sure the service that you were trying to access has the endpoints?

@murali-reddy Yes,I can access has the services。
But it always seems to print these logs, which takes up memory.

@Jacky2 is this still a problem, does any of the new kube-router releases solve your problem?. Feel free to get back to us if you still have any issues

@Jacky2 Hey! How did u solve it?

I get the same error when using docker swarm (and looking for logs), so it's probably more docker related than kubernetes-specific. In particular what had happened was that there are 2 gateways in my network, and there was an outage on the gw1. After switching default route on host to gw2, no logs were anymore visible on docker swarm (but locally yes), and I got this error message.

@Jacky2
what's the fix here for "no destination available" ??

Was this page helpful?
0 / 5 - 0 ratings