Truffle: Ethpm registry package installation fails

[elenadimitrova]

Issue

Installing the packages below from the ethpm registry fails with Error: Could not find object at hash 'QmRoUvLLibQVBGVt4NMgWukjtt4TWVQQ12XBwpLFWRbKMm'

https://www.ethpm.com/registry/packages/25
https://www.ethpm.com/registry/packages/8

Steps to Reproduce

Run truffle install zeppelin or truffle install mintable-standard-token

Expected Behavior

Expected to install these packages

Actual Results

$ truffle install zeppelin
$ Error: Could not find object at hash 'QmexsaiGwyqruHyQic2KzWQDgZAFv4qB3MArgahW4WDjbT' in 5000ms
    at Timeout._onTimeout (/.nvm/versions/node/v6.10.0/lib/node_modules/truffle/node_modules/ethpm/lib/hosts/ipfshost.js:47:14)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)
$ truffle install multisig-wallet
$ truffle install mintable-standard-token
Error: Could not find object at hash 'QmRoUvLLibQVBGVt4NMgWukjtt4TWVQQ12XBwpLFWRbKMm' in 5000ms
    at Timeout._onTimeout (/.nvm/versions/node/v6.10.0/lib/node_modules/truffle/node_modules/ethpm/lib/hosts/ipfshost.js:47:14)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)

Environment

• Operating System: OSX
• Truffle version: 3.2.1
• node version: 6.10
• npm version: 3.10.10

[tcoulter]

@elenadimitrova EthPM is in flux right now. EthPM uses IPFS to store data, and IPFS inherently has no mechanism to ensure data stays on the network -- it has a concept of "pinning", which should keep it, but unlike SWARM there's no financial incentive and thus is prone to data loss. For some reason, the IPFS installation we used (via Infura) lost data for some packages. The only solution is to ask those packages to release new versions to EthPM so it updates the files stored on IPFS.

The only way to fix this completely is to have a distributed file storage mechanisms that provide incentives for keeping data on the network. As far as I can tell that will best be solved via SWARM.

[pooleja]

Is there a workaround for this while it gets sorted?

[jbenet]

Hey everyone, some pointed me to this issue. IPFS dev here.

EthPM is in flux right now. EthPM uses IPFS to store data, and IPFS inherently has no mechanism to ensure data stays on the network -- it has a concept of "pinning", which should keep it, but unlike SWARM there's no financial incentive and thus is prone to data loss.

This is not representing IPFS properly. IPFS does have a mechanism for ensuring data stays on the network, and you described it: pinning. Pinning relies on _external-to-ipfs_ reasons for keeping around content. This is how the majority of all data in the world is distributed today (through ipfs, http, bittorrent, ftp, smtp, ...). And _all_ of that distribution of data works just fine, without in-transport financial incentives. What you mean is that IPFS _does not include an internal financial incentive structure_, and that's right. It was separated out into a separate protocol (WIP now), Filecoin.

Compare to other registries. There is no financial incentive to keep around most registries of software packages in the world, from apt to pypi to npm. Linux sources and binaries, all programming languages, and so many more community infrastructures run _just fine_ without any kind of financial incentive structure in the transport. Of course, having easy ways to "incent the network" to store all the data, regardless of whether the network cares about the data, will be useful. Incentivized storage networks will be great. That said, you do not at all have to rely on those, nor to wait for such solutions to mature.

Package registries are trivially small amounts of data. even 2-5 different people offering to pin it as a community resource will be dramatically cheaper than the work required to get incentivized solutions to mature. (just measured in your dev time alone, integrating). All you need is 3-5 full copies, and a setup such that -- when people publish a package -- the ref is broadcast to every replica (eg a tool like ipfs-cluster).

Here, I offer to replicate all your data for free. If your registry is less than 5TB (which i highly suspect it is) you can just pin everything to our storage hosts. We already replicate lots of data for other orgs. We're happy to give people access to our storage nodes. @tcoulter email me.

For some reason, the IPFS installation we used (via Infura) lost data for some packages. The only solution is to ask those packages to release new versions to EthPM so it updates the files stored on IPFS.

That's a bigger issue, and we should look into it. @tcoulter email me as i'd like to get to the bottom of that. Curious to find out why content was dropped, and where the issue lies. If it was in any of the IPFS code, we'd love to figure out the bug that caused it and fix it. We run very large archives (>TB) without any data loss, so this is quite surprising.

[tcoulter]

Thanks @jbenet. I have no information on how the data was lost, but just that it was. Unfortunately I don't run the EthPM file storage infrastructure (that's Infura), so I can't pinpoint whether or not it was a bug or a lack of proper configuration, such as clustering (though I can put you in touch with the Infura team, if needed). What I can say is that if EthPM's promise is to provide a decentralized package registry on top of Ethereum, it eventually needs a file storage mechanism that's similarly decentralized. Pinning to your infrastructure could help (and I appreciate it); that would be a stop gap until we get proper financial mechanisms for persisting data.

I need to regroup with @pipermerriam to see what the way forward is. Will definitely reach out if I have questions. Thanks!

[tcoulter]

This has been fixed (more specifically, worked around) in the latest version of Truffle, v3.2.3.

To get the fix, uninstall and reinstall your version of Truffle.

Background: I was wrong. This issue wasn't an issue with data loss, although Infura has lost data (unfortunately I can't speak to that in more detail). Instead, this issue was an issue with response times, where files were taking > 30s to resolve.

I've isolated it to the specific API used against infura IPFS nodes, and have noticed similar behavior with gateway.ipfs.io. Here's an example:

1. Infura, IPFS port, /api/v0/cat -> slow. This example shows a request resolving in 2 seconds. I've seen upwards of 30 seconds.

$ curl https://ipfs.infura.io:5001/api/v0/cat/QmVceYuBZpVzgbjsYXi7Zkma3jE1wNp3gWY68u21GmX56v -o /dev/null -w "%{time_total}\n"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   283    0   283    0     0    128      0 --:--:--  0:00:02 --:--:--   128
2.207

1. Infura, local port (i.e., not 5001), /api/v0/cat -> slow. Seems to be the same response time as the above.

$ curl https://ipfs.infura.io/api/v0/cat/QmVceYuBZpVzgbjsYXi7Zkma3jE1wNp3gWY68u21GmX56v -o /dev/null -w "%{time_total}\n"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   283    0   283    0     0    138      0 --:--:--  0:00:02 --:--:--   138
2.039

1. Infura, local port (not 5001), /ipfs/ api -> much faster to resolve.

$ curl https://ipfs.infura.io/ipfs/QmVceYuBZpVzgbjsYXi7Zkma3jE1wNp3gWY68u21GmX56v -o /dev/null -w "%{time_total}\n"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   283  100   283    0     0    547      0 --:--:-- --:--:-- --:--:--   547
0.517

I've reworked Truffle's use of EthPM to use the third API, though I'm told this is requesting data from a specific node, and not from the IPFS network in general (correct me if I'm wrong). This is a workaround until we can get better response times over the full API.

Thanks for reporting the issue and my apologies for miscategorizing it above.