Zerotierone: ZeroTier One service listening 9993 but not replying to anything

[bish0polis]

Describe the bug

1. zerotier-cli commands - all of them, given the dreaded
   ````
   Error connecting to the ZeroTier service:

   Please check that the service is running and that TCP port 9993 can be contacted via 127.0.0.1.
   ````

2. curl -Lv localhost:9993 also gives a bogus response:
   <ul> <li>About to connect() to localhost port 9993 (#0)</li> <li>Trying 127.0.0.1...</li> <li>Connected to localhost (127.0.0.1) port 9993 (#0)<br /> &gt; GET / HTTP/1.1<br /> &gt; User-Agent: curl/7.29.0<br /> &gt; Host: localhost:9993<br /> &gt; Accept: <em>/</em><br /> &gt;</li> <li>Empty reply from server</li> <li>Connection #0 to host localhost left intact<br /> curl: (52) Empty reply from server<br />



3. curl -Lv localhost:9993 the same

4. tshark gives a weird output:
   ````

   57 7054.762425574 172.30.3.91 -> 127.0.0.1 TCP 76 39840 > 9993 [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=2342716343 TSecr=0 WS=128
   58 7054.762445749 127.0.0.1 -> 127.0.0.1 TCP 76 9993 > 39840 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=2342716343 TSecr=2342716343 WS=128
   59 7054.762461596 172.30.3.91 -> 127.0.0.1 TCP 68 39840 > 9993 [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=2342716343 TSecr=2342716343
   60 7054.762534934 172.30.3.91 -> 127.0.0.1 HTTP 146 GET / HTTP/1.1
   61 7054.762543828 127.0.0.1 -> 127.0.0.1 TCP 68 9993 > 39840 [ACK] Seq=1 Ack=79 Win=43776 Len=0 TSval=2342716343 TSecr=2342716343
   62 7054.762572109 127.0.0.1 -> 127.0.0.1 TCP 68 9993 > 39840 [FIN, ACK] Seq=1 Ack=79 Win=43776 Len=0 TSval=2342716343 TSecr=2342716343
   63 7054.762664762 172.30.3.91 -> 127.0.0.1 TCP 68 39840 > 9993 [FIN, ACK] Seq=79 Ack=2 Win=43776 Len=0 TSval=2342716343 TSecr=2342716343
   64 7054.762681644 127.0.0.1 -> 127.0.0.1 TCP 68 9993 > 39840 [ACK] Seq=2 Ack=80 Win=43776 Len=0 TSval=2342716343 TSecr=2342716343
   ````

To Reproduce
Steps to reproduce the behavior:

1. rhel7 AWS host used for VPN connections between SOHOs and the AWS resources.
2. Install zerotier-one via RPM/Yum (/chef here via excellent chef-zerotier cookbook)
3. on https://my.zerotier.com/network/ we enable this host
4. on the host's CLI, we issue the commands above; e.g. zerotier-cli info

Expected behavior
````

zerotier-cli info

200 info fa5lOc8379 1.4.6 ONLINE
````
Desktop (please complete the following information):

• OS: Linux / CentOS7 (fully updated
• ZT version 1.4.6
• AWS T3.small VM

Additional context

• I can't vapourize all the nat rules because this host already is running other VPNs to connect hosts and networks

Add any other context about the problem here.

• firewall config (this new host has a far-too-simple itpables setup right now) :

````

Generated by iptables-save v1.4.21 on Mon Dec 9 22:15:36 2019

*nat
:PREROUTING ACCEPT [874831:47356693]
:INPUT ACCEPT [18920:1084863]
:OUTPUT ACCEPT [51237:4844138]
:POSTROUTING ACCEPT [41690:3567567]
-A POSTROUTING ! -o eth+ -j MASQUERADE
-A POSTROUTING -d 172.30.3.0/24 -j MASQUERADE
COMMIT

Completed on Mon Dec 9 22:15:36 2019

Generated by iptables-save v1.4.21 on Mon Dec 9 22:15:36 2019

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [155092:28075626]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 9993 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o tun+ -j ACCEPT
-A FORWARD -i tun+ -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Completed on Mon Dec 9 22:15:36 2019

````

[bish0polis]

I'm posting this here for others to discover. It was iptables/masquerade, again.

Look at the weird tshark again:
89 7956.026533306 172.30.3.91 -> 127.0.0.1 TCP 76 39848 > 9993 [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=2343617607 TSecr=0 WS=128 90 7956.026551276 127.0.0.1 -> 127.0.0.1 TCP 76 9993 > 39848 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=2343617607 TSecr=2343617607 WS=128 91 7956.026562674 172.30.3.91 -> 127.0.0.1 TCP 68 39848 > 9993 [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=2343617607 TSecr=2343617607 92 7956.026634394 172.30.3.91 -> 127.0.0.1 HTTP 152 GET /status HTTP/1.1 93 7956.026643648 127.0.0.1 -> 127.0.0.1 TCP 68 9993 > 39848 [ACK] Seq=1 Ack=85 Win=43776 Len=0 TSval=2343617607 TSecr=2343617607 94 7956.026662243 127.0.0.1 -> 127.0.0.1 TCP 68 9993 > 39848 [FIN, ACK] Seq=1 Ack=85 Win=43776 Len=0 TSval=2343617607 TSecr=2343617607 95 7956.026758030 172.30.3.91 -> 127.0.0.1 TCP 68 39848 > 9993 [FIN, ACK] Seq=85 Ack=2 Win=43776 Len=0 TSval=2343617607 TSecr=2343617607 96 7956.026765933 127.0.0.1 -> 127.0.0.1 TCP 68 9993 > 39848 [ACK] Seq=2 Ack=86 Win=43776 Len=0 TSval=2343617607 TSecr=2343617607
You see it?

Initial SYNs are coming from the eth0 IP and not the lo IP. This isn't too big a deal so far, but since ZT was acting ooky we decided to chase this one eventually.

Connections passing nat/POSTROUTING are -j MASQUERADE, which, yes, rewrites the packet's source address as it should. I'm guessing there's something funky in the authorization check to ensure that only lo connects to lo, and it does it by IP checking.

Here's the fix to fix the IPTables without killing other MASQ rules:
````

iptables -t nat -I POSTROUTING -o lo -j ACCEPT

service iptables save

````
Yep, just squeeze an ACCEPT rule at the head of your other rules and we're saved:

````

zerotier-cli info

200 info deadbeefa1 1.4.6 ONLINE
````
There. If you're getting what I got, maybe this will save you time on the fix.

[Atrate]

Thanks so much for this! Masquerade is a powerful tool, sometimes too powerful