Go-ipfs: Content can sometimes only be found after hours or days

Created on 29 May 2019 · 19Comments · Source: ipfs/go-ipfs

See: https://github.com/ipfs/go-ipfs/issues/6382#issuecomment-496703161

This isn't the first time I've seen this and it warrants investigation. There _used_ to be an issue where bitswap would time out too quickly sending out provider records for new blocks. However, we now provide in a background queue.

It could be that this background queue is getting backed up.

kinbug topiprovider

Source

Stebalien

👍3

Most helpful comment

Would it be practical if most of the nodes without public ip can connect to some common nodes, e.g. ipfs.io, so those common nodes can rely the connections between the nodes without public ip and the rest of the network?

We have relays for this but we had to significantly reduce our relay deployment due to costs. There's a _lot_ of traffic on the IPFS network and relaying all of it was unsustainable.

However, you can still use them by enabling autorelay: ipfs config --bool Swarm.EnableAutoRelay true.

We're also slowly working on better NAT traversal (e.g., ICE) but the plan has been to use relays to coordinate things like ICE.

Stebalien on 9 Jul 2019

👍4

All 19 comments

@Stebalien is this the same issue that you are facing ?

I am adding a file and then pinning the hash.

ipfs add -r <filename>
ipfs pin add <hash>
https://ipfs.io/ipfs/<hash>

File doesn't seem to be available.

If this is the case. Then please make a page of disclaimer like information where you guys tell about the process and when will it be available. Its beating the purpose of using the system.

Please advise

warlockdn on 31 May 2019

any ideas ? how to make this work ???

warlockdn on 8 Jun 2019

There are more people having this issue (#6425). We also have a report on IPFS Desktop (https://github.com/ipfs-shipyard/ipfs-desktop/issues/955) and @olizilla and I have not been able to get our files through any public gateway.

@Stebalien could you confirm if this is only happening after 0.4.21?

hacdias on 10 Jun 2019

Regarding my problem report #955, it has been happening since new Win desktop ver 0.7.2 install on 23 May 19. Thought it might be something about my external/USB install or setup but didn't find anything. Then made problem report apparently in wrong place 15 days ago at https://discuss.ipfs.io/t/difficulty-accessing-ipfs-desktop-saved-files-on-external-hd/5503.

miguelito9 on 10 Jun 2019

@miguelito9 weird, 0.7.2 still uses go-ipfs 0.4.19. You shouldn't have any issues with that I think... I'll investigate it better

hacdias on 10 Jun 2019

Win Desktop app status screen states: VERSION go-ipfs v0.4.21 . Don't know what it was at time of install.

miguelito9 on 10 Jun 2019

👍1

I am seeing block activity averaging about 0.3 Mb/s with a couple of spikes to about 0.9 in my node over the last day with a small increase in storage used. Still getting 504 gateway time-outs when trying to access with browser (no ipfs companion) the small png file I uploaded to my node at
https://ipfs.io/ipfs/QmdMkoevxiyexeM21jAgKDrjrZ6G8DqvYpUBs6s3nVNeab

miguelito9 on 10 Jun 2019

We don't believe this has anything to do with the local IPFS version.

Stebalien on 12 Jun 2019

❤1

We don't believe this has anything to do with the local IPFS version.

Okay. Any thoughts on possible other causes?

miguelito9 on 13 Jun 2019

Two possible issues:

#6383
We've noticed that the gateways are bottle necking on _finding_ content (we severely limit the number of parallel content finding workers as they can impact performance). We're testing out a patch (https://github.com/ipfs/go-bitswap/pull/138) on the gateways to see if this is the case.

Stebalien on 13 Jun 2019

I remember in the early day, (not remember which version) I would publish content then use ipfs.io gateway to see the content without having a public IP.
But in the current version (0.4.20 or alike, I need to check when I back to home), the content is not accessible from ipfs.io if my node doesn't have a public IP.

beenotung on 8 Jul 2019

That was because:

The DHT would dial pretty much every node on the network due to a bug. This bug has partially been fixed.
The network was small enough that every node could be connected to every other node, all the time. That's no longer the case.

Stebalien on 8 Jul 2019

👍3

Would it be practical if most of the nodes without public ip can connect to some common nodes, e.g. ipfs.io, so those common nodes can rely the connections between the nodes without public ip and the rest of the network?

beenotung on 9 Jul 2019

As long as we're relying on TCP/IP, we need some nodes to act like router/relay to bridge the network, unless we're using protocol like NDN that doesn't rely on host address when sending packets.

beenotung on 9 Jul 2019

Would it be practical if most of the nodes without public ip can connect to some common nodes, e.g. ipfs.io, so those common nodes can rely the connections between the nodes without public ip and the rest of the network?

We have relays for this but we had to significantly reduce our relay deployment due to costs. There's a _lot_ of traffic on the IPFS network and relaying all of it was unsustainable.

However, you can still use them by enabling autorelay: ipfs config --bool Swarm.EnableAutoRelay true.

We're also slowly working on better NAT traversal (e.g., ICE) but the plan has been to use relays to coordinate things like ICE.

Stebalien on 9 Jul 2019

👍4

Maybe my ebservations helps: So I tried to share a 20GB-file between two Nodes (Desktop Computers). I've added the file on Node A and pined it. Afterwards I've downloaded it on Node B with get. It started well but got stuck after 657MB. I've figured out, that I've could solve the problem by directly connect the two Nodes over "ipfs swarm connect". So it seams that the file was after pinning uploaded only partially to ipfs and Node B was not able to find Node A and directly connect to Node A and download it from there. Seems to me that it should be important that "get" should try to find the source-Node, when it can't find the entire file in IPFS.

Kuurusch on 25 Oct 2019

Maybe my ebservations helps: So I tried to share a 20GB-file between two Nodes (Desktop Computers). I've added the file on Node A and pined it. Afterwards I've downloaded it on Node B with get. It started well but got stuck after 657MB. I've figured out, that I've could solve the problem by directly connect the two Nodes over "ipfs swarm connect". So it seams that the file was after pinning uploaded only partially to ipfs and Node B was not able to find Node A and directly connect to Node A and download it from there. Seems to me that it should be important that "get" should try to find the source-Node, when it can't find the entire file in IPFS.

Thanks for your report. Were either of the desktops using Auto-relay? Was one or both of them behind a NAT, or were they on the same local network? Thanks.