Go-ipfs: Feature: Transparent HTTP proxy over p2p streams

[ianopolous]

After talking in depth with @Stebalien about what I was hoping for from #5256 we concluded that my idea is orthogonal to what @haadcode is requesting, hence this issue.

The idea is a transparent http proxy in the http api which pipes http requests over a p2p stream to another node.

E.g. the http api would include a new endpoint like
/api/v0/http_proxy/$name/$multiaddr/$http_path

$multiaddr would be something like /ipfs/$nodeid
The $http path is the path that is sent to the target server
@Stebalien could you explain the use of $name?

The cool thing about this is it would give ANY existing web app (including dynamic stuff like talking to a database etc.) a way to move their hosting from dns location based addressing to ipfs based trivially and everything would just work. (They would still run their back end as is but with an ipfs node proxying the http connections)

We would like to use this in @Peergos to allows us to tunnel writes (et al) to particular storage servers.

Note that this automatically includes websockets.

[Stebalien]

The endpoint would probably be /http/$peer_id/$name/$http_path as this will probably be separate from the HTTP API. We'd likely even provide a config option for mounting it on a separate port (but allow it to reside on the same host/port as the HTTP API).

In that path:

• $peer_id tells IPFS to open a stream to the peer at $peer_id.
• $name tells IPFS to use negotiate the /x/http/$name protocol when opening the stream. Note: /x/ is the new protocol prefix we're using for ipfs p2p protocols (to avoid conflicting with the /p2p multiaddr protocol name.
• $http_path is the path to the HTTP resource.

We could (possibly later) allow the client to specify multiaddrs for the target peer in HTTP headers.

---

@ianopolous when we discussed this, we talked about using a multiaddr instead of just a peer ID in the URL. Unfortunately, it turns out that that would introduce an ambiguity. For example, in /http/ipfs/QmRelay/p2p-circuit/ipfs/QmTarget/my_protocol/foo/bar, the multiaddr is /ipfs/QmRelay/p2p-circuit/ipfs/QmTarget.

Additionally, putting a full multiaddr in the GET path doesn't allow specifying more than one multiaddr.

---

To be explicit, when the IPFS node receives a request of the form:

$METHOD /http/$peer_id/$name/$http_path
...

The IPFS node would:

1. Open a stream to $peer_id.
2. Negotiate the /x/http/$name protocol.
3. Send the HTTP request:

$METHOD $http_path

X-Libp2p-Peer: QmClientPeer (maybe?)
...

~Note: the X-Libp2p-Peer header would allow authentication and sessions.~ We can't do that as we're just using p2p listeners on the server. We could eventually add websocket listeners with this header but that's a different issue.

[magik6k]

I like this approach, but I would like to have a way of listening for connections, so we'd probably want a prefix for what action we want to do:

• /http/proxy/$peer_id/$name/$http_path (http / ws just forwarded)
• /http/listen/$name (ws only, using some sort of muxer under the hood)

I'll try hack something together next week.

(If you want to do http requests over libp2p now, you should be able to with #4929)

[Stebalien]

@magik6k @ianopolous has actually volunteered to do this and it should be a good self-contained feature. The current plan is to use a normal p2p listener on the server.

The issue @ianopolous is having with #4929 is that it requires opening a new port per peer. With this version, we can listen on the same port as the API server.

We could even make it a part of the read-only API and expose it on the gateway. This would allow users to host static websites and then have their static websites make simple AJAX requests to services behind libp2p endpoints. We can even make this entirely client-side in the JS gateway using the service worker. The downside to this is that allowing semi-persistent connections on the public gateway may not be the best idea from a performance standpoint (for us).

[ianopolous]

If not me then another Peergossian (@cboddy or @kevodwyer) will have a crack at this soon.

[ianopolous]

Implemented by https://github.com/ipfs/go-ipfs/issues/5526