Go-ipfs: Let the gateway serve proofs of correct behavior.

[Bren2010]

Type:

feature

Description:

Motivation. Third-party IPFS gateways offer a lot of value to the ecosystem:

• They let clients that are constrained by bandwidth or power usage access data from the network in a more efficient way.
• A gateway can be used to answer requests if a client-side node hasn't had sufficient time to start-up yet or isn't working for unknown reasons (firewall?).
• A gateway can act as an anonymizing layer, like a VPN.

Given the above, I'd like to propose that gateways have the ability to serve proofs that they're behaving correctly. This way, gateways can't maliciously modify content before serving it to their clients. A gateway can only refuse to answer a request, in which case another gateway can be tried.

How the feature should work. Requests to a gateway with the header X-Ipfs-Secure-Gateway: 1 should switch to a binary format containing the proof.

The first part of the response, called a preamble, relates the request path domain.com/a/b/c to the hash of the response body. It may contain a DNSSEC proof mapping domain.com to dnslink=/ipfs/Qm... or an IPNS record, as well as the contents of any intermediate directories /a/b/c.

The second part of the response, called the body, is a depth-first traversal of the file's DAG. The client can hash the first block and see that it equals the output of the preamble, and then check that the second block corresponds to the first link of the first block, and so on.

Background.

• Cloudflare announcing their gateway supported this feature: https://blog.cloudflare.com/e2e-integrity/
• A browser extension that validates these proofs: https://addons.mozilla.org/en-US/firefox/addon/cloudflare-ipfs-validator/
• PR upstreaming feature: https://github.com/ipfs/go-ipfs/pull/6126

[Stebalien]

Thanks for writing this up, I'm currently traveling but I'll take a look when I get back (next Wednesday?) if not sooner.

(that is, this'll take some focus work to dig into so I'll need to allocate a chunk of distraction-free time)

[Stebalien]

(we're trying to cut a release but I'll still try to look at this ASAP)

[Stebalien]

Again, I apologize for the extensive delay. The release should be out shortly (https://github.com/ipfs/go-ipfs/pull/6223) and we'll re-enter the feature cycle.

[bertrandfalguiere]

What is the status on this?
Content-addressing is about "getting authentic data from any source", and this PR goes in the right direction. :)

[Stebalien]

The status:

1. I apologize for not communicating on this. I was over ambitious with my own time and should have managed expectations better.
2. The network has some critical performance issues at the moment which are taking up the majority of my (and my team's) time.
3. This is a pretty significant change and a large feature to maintain. It needs to be thoroughly reviewed, tested, documented, and discussed.

I'd like to revisit it when we get everything under control but I'm having trouble allocating time to it at the moment.

---

More generally, this feature doesn't really fit in with the overall goal of IPFS: to decentralize the internet. Basically, to make use of this feature, the user needs to install _something_. That leaves us with two choices:

1. The user installs the validator extension and uses a centralized gateway.
2. The user installs the IPFS companion extension and uses either a local gateway or the built-in, in-browser js-ipfs gateway.

The former helps provide better, secure access to the IPFS network but the _latter_ actually works towards our goal of decentralizing the internet. If the user needs to install something anyways, I'd rather move closer to our goal.

On the other hand, I do understand that running IPFS locally is not always possible for performance reasons. That's why I'd like to revisit this once we get the larger-scale network performance issues under control.

[lidel]

Additional design work needs to happen here, and probably some dev on the browser vendor side as well.

Open problems:

• PoC validator extension made by Cloudflare does not verify IPFS-based integrity correctly: it produces false-negatives if non-default parameter space was used during ipfs add (eg. different hash, trickle dag, chunk size etc)
• Chromium-based browsers have more than 80% of the market atm and they do not have WebExtension API that makes it possible to verify proofs, making it infeasible in real world – see initial analysis on Verifiable HTTP Gateways in https://github.com/ipfs/in-web-browsers/issues/128