Go-ipfs: Should have NAT Traversal

[whyrusleeping]

Peers behind nats need a way for other peers in the network to connect to them. It would be nice to also have webrtc, but until a go library for that exists, this will have to do.

[jbenet]

The good news is I found a very promising approach for nat traversal, which takes its ICE implementation directly from the chromium src (like i want to do in https://github.com/jbenet/random-ideas/issues/13). It's https://github.com/getlantern/natty with go bindings at https://github.com/getlantern/go-natty.

I can get OSX processes to connect to each other using their https://github.com/getlantern/waddell signaling server, and the https://github.com/getlantern/nattywad client, using a very crappy test script: https://gist.github.com/jbenet/632d9459414b996e9457 i wrote (but when I went to test linux I got https://github.com/getlantern/nattywad/issues/1).

Pros:

• same ICE as in chromium
• signaling server in Go, so we can embed it into our nodes
• of all the approaches I've seen so far, this is the most promising long term.

Cons

• seems to be only udp? (some nats stop UDP, so can always try: https://github.com/jbenet/random-ideas/issues/2)
• embeds a binary (natty) bigger than all of built ipfs. not a problem now, but will make implementations for embedded devices harder.

[jbenet]

Got farther with natty: https://github.com/jbenet/go-netcatnat/

[jbenet]

Ok the roadmap for NAT traversal is:

1. For now

1. add a signaling Service to ipfs nodes, to do what waddell does.
2. use the bootstrap nodes as the only signalers (fine for now)
3. (maybe) if a node discovers it is behind a nat, it can report its address as requiring nat traversal[1]

[1] this is a convoluted example, and there's probably simpler ways, but maybe:

<multiaddr of signaler used>/ipfsstun/<peer.ID>/ipfs/<peer.ID>`
// or something nicer than ipfsstun
// the peer.ID is repeated because encapsulation. ipfsstun could use arbitrary identifiers

This says "connect to <multiaddr of signaler used> and ask for connection to /ipfsstun/<peer.ID>, it will give you addresses, attempt to connect to those. behind them is an /ipfs/<peer.ID>.

2. Down the road

1. signaling integrated with routing
2. e.g. in kademlia, nodes in each others' routing table have open connections + can help signal.
3. in better routing systems, should be able to deliver a message through the network even if node not directly accessible, so can do signaling that way.
4. also look into pwnat

[jbenet]

I reiterate here that we cannot rely on others' systems for nat traversal. It's an important enough piece and should not rely on public stun servers, as our traffic needs to not depend on pieces of infrastructure that might {fail, rate limit, block ipfs} unexpectedly. ipfs must sustain itself.

[whyrusleeping]

Where do you put UDP on our priority list?

[jbenet]

@whyrusleeping ah right, natty is currently for udp only, so i guess we're switching now. We can find the simplest reliability protocol on top of udp, and aim for uTP or SCTP, and then QUIC once its implementations settle.

[jbenet]

Update: now that we have UDT (#58), I can move ahead on this. My next thing will be adding Natty and UDT. these are two C++ libs, that will add significant size (and noise) to the binary. These are stopgaps to get the right functionality, in the end native Go impls, or Cgo compiled things will probably be less problematic to use.

[whyrusleeping]

this makes me kinda sad, since now we cant just distribute binaries...

[jbenet]

@whyrusleeping oh we can. cross compilation :)

[whyrusleeping]

ehhh, not quite what i was hoping for though, with pure go, you can build once for each system and go. Once you mix in cgo and other linking stuff, you have to worry more about the specific system and their version of libc

[jbenet]

@whyrusleeping yeah, it's painful. Pure Go impl of {UTP, UDT, SCTP, and QUIC} in the horizon. in due time. For now, I much prefer my ipfs node able to talk yours, (almost) no matter where it is.

[whyrusleeping]

Oh yeah, im much more happy about UDP and NAT working than i am sad about not being pure Go

[jbenet]

Alpha: NAT or Relay

NAT traversal is non trivial, the roadmap looks like this:

• [x] relay service for ICE comm -- our p2p/protocols/relay
• [x] allow ipfs nodes to find suitable relays -- i.e. nodes directly connected to the node we wish to reach (implemented with (*IpfsDht).FindPeersConnectedToPeer)
• [ ] use something to perform ICE (current candidate: natty, which gives us only UDPAddr pairs)
• [ ] use reliable transport over UDP.

The blocker is reliable transport. Our options so far have been:

• utp - needs to be implemented robustly. h2so5/utp is not yet working, could go-wrap libutp.
• udt - go-wrapped, needs some ARM assembly
• sctp - needs go-wrapping, should not have assembly.
• quic - needs lib-izing, and go-wrapping.

Of these, quic is my favorite option for the future. It is a truly phenmenal transport, with tons of things done very, very well. However, though the dragon is ready to hatch from its egg, it must still be unearthed from the enormous Code Caverns, of the Chromium Mountains, a quest not to be taken lightly.

Reliable transport a really big deal for us, and hard to get right:

1. Buggy transports will cause us endless pain and suffering. They already have: I spent many hours debugging dht and bitswap problems only to find that utp was the problem all along.
2. "Congestion Control" and "Flow Control" are both very hard. It took the TCP, SCTP, UTP, and QUIC teams many months to perfect designs and implementations.
3. Low {memory, cpu} footprints, etc are hard to achieve well, while also achieving low bandwidth and latency overheads.

@whyrusleeping pre-empting your call for "Lets Implement UTP!", we should not roll our own right now-- it will take us _weeks_ to get reasonable robustness, speed, flow control, memory consumption, etc. We are good hackers, but it's taken the broader community _years_ to make the few implementations there are. this is not to be underestimated. I totally want to have native go implementations of these transports (see my related emails to go-nuts mailing list), but we cannot afford the implementation cost right now.

another option TCP/UDP tunnel

And, frankly, I'm seriously considering doing the following horrible hack that might actually work robustly and be much quicker to pull off that

• use the UDPAddr pairs from natty, BUT!
• setup a TCP/UDP tunnel with our own NAT:
  A-TCPConn <---> IPConn <---> UDPConn <--IPConn--> TCPConn-B

This is a mad science(tm) hack that gets around the awful realities of the network and the awful realities of software: simple things outside of specific rails is too fucking hard to do.

Looking at http://golang.org/pkg/net/#IPConn -- it should be possible, my concern is making sure DialIP doesn't interfere with other traffic:

Resources:

• https://godoc.org/golang.org/x/net/ipv4
• https://github.com/kdar/gorawtcpsyn/blob/master/main.go
• http://stackoverflow.com/questions/18427655/use-raw-sockets-in-go

Relay past NAT -- easy cop-out for now

The easiest thing to do for now is to just relay _all_ traffic. Would get us around NAT, but would certainly increase load of nodes.

We'll _have to_ relay some traffic, as not all NAT is hole punching. Sometimes we do have to use relaying. This could actually be done with Bitswap in mind, and be governed by the standard bitswap decision engine: relay for those nodes who are useful to us.

In lieu of using natty and hole-punching, we could start with the big guns (which will work for everything and we have to do anyway) and optimize later. My Knuth VM is telling me to optimize later.

[whyrusleeping]

I think a combination of relays, and NAT hole punching (NAT-PMP, uPnP) could be our best option (at least short term). I found a PMP library: https://github.com/jackpal/go-nat-pmp, and there was a few promising uPnP libraries around, which should get us through a decent number of networks, and the rest can be relayed.

[btc]

we could start with the big guns

the big guns being relay?

[jbenet]

the big guns being relay?

yeah

[jbenet]

So this happened (QUIC):

• https://groups.google.com/a/chromium.org/d/msg/proto-quic/MIogHT8wacA/69H8gchp0nMJ
• https://github.com/devsisters/libquic
• https://github.com/devsisters/goquic

[tv42]

Since I don't think I've seen this mentioned yet, and it's good overview:

https://tools.ietf.org/html/rfc5128
State of Peer-to-Peer (P2P) Communication across Network Address Translators (NATs)

[drwasho]

Any update on where you're at on this issue?

[whyrusleeping]

closing this for now, basic NAT traversal has been implemented as well as utp in dev0.4.0. Further discussion on relays and more advanced traversal should start a new issue

[drwasho]

Awesome, well done guys!

[myleshorton]

Hey @whyrusleeping and @jbenet can you describe the NAT traversal implementation in 0.4.0 or link to more details? Exciting stuff.

[makeworld-the-better-one]

@myleshorton @whyrusleeping @jbenet Yes, I am interested in how it works for the purposes of my own decentralized application. My application would require IPFS anyway, and I am wondering if I can just piggyback on the NAT traversal that IPFS uses, and then just specify a different a different port?

[makeworld-the-better-one]

I just came back to this link again. Does anyone have an answer?

[lidel]

@Cole128 check https://libp2p.io, the networking layer of IPFS that, among other things, provides solutions for NAT Traversal.
If you need support, ask at http://discuss.ipfs.io (it has bigger reach than a github issue)

[makeworld-the-better-one]

@lidel libp2p led me here which seems to say it is only kind of implemented now.

[Stebalien]

@Cole128 there are two implementations of libp2p, one in js and one in go (and a work in progress one in rust). The issue you linked was for js. NAT traversal works pretty well in go (although we're still missing a few pieces).

If you just want to piggyback on ipfs's NAT traversal, you may be interested in https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipfs-p2p

There's some improved documentation in #4894 but it's blocked on actually making this feature behave according to that documentation... (i.e., how one would expect).

If you're still having trouble with NAT traversal, you may also be interested in the circuit relay feature: https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#circuit-relay

[makeworld-the-better-one]

@Stebalien thanks a lot, I'll check all those links out.