Go-ipfs: Performance indexing "big directories"

[kallisti5]

Version information:

$ ipfs version --all
go-ipfs version: 0.6.0
Repo version: 10
System version: amd64/linux
Golang version: go1.14.4

Description:

Usage:

• Add a folder containing 35GiB of files across ~7,108 files.
• Documents only exist on a single node with a 200 Mbit up/down internet connection
• Navigate to folder on a gateway such as gateway.ipfs.io (success)
• Enter the folder (index) with 7,108 files. Timeout on gateway.

Does the IPFS gateway need to transfer a large amount of data to get a directory index?

The IPFS daemon has been running about 30Mbit/s for a while now.

[kallisti5]

I connected a remote node (vm at external hosting provider) directly to my local home IPFS node:
ipfs swarm connect /ip4/XX.XX.XX.XX/tcp/4001/p2p/QmdDXXX

and ran an ipfs ls...

time ipfs ls /ipfs/QmWApXeXXX/current/packages

It took over 7 minutes to perform an ls:

real 7m17.463s

[utkarsh5k]

I'm willing to take a look at this, but I'm a first time contributor, so if someone could point me to the indexing logic, that'd be super helpful, thanks!

[aschmahmann]

Does the IPFS gateway need to transfer a large amount of data to get a directory index?

It currently needs the to get the first block of each file in the directory to decide if it's a file or a directory.

As for ipfs ls if you set --resolve-type and --size to false it will only need to get the directory data and not reach into the individual files. Also having --stream will give you outputs without having to wait for the full results.