Go-ipfs: Too many open files on add with daemon

[Kubuxu]

Version: 0.4.7

Just running: dd bs=128M if=/dev/urandom count=1 | pv -a | ipfs add causes too many files error in 80% of cases.

 23.00 MB / ? [-------------------------------------------------------------=--------------------] 0+1 records in
0+1 records out
33554431 bytes (34 MB, 32 MiB) copied, 0.785311 s, 42.7 MB/s
[40.8MiB/s]
 32.00 MB / ? [-------------------------------------------------------------=--------------------] 12:00:54.217 ERROR commands/h: open /home/kubuxu/.ipfs/blocks/WP/put-927501522: too many open files client.go:247
Error: open /home/kubuxu/.ipfs/blocks/WP/put-927501522: too many open files

[NeoTeo]

Also on
go-ipfs version: 0.4.7-
Repo version: 5
System version: amd64/darwin
Golang version: go1.8

I'm also getting a bunch (133 in a row last time) of
flatfs: too many open files, retrying in 500ms flatfs.go:180

It seemed to be caused by loading some new hash into my browser from outside (not already in my node) but cannot reproduce.

[Kubuxu]

The flatfs warning is not a problem (it is working) but in this case it fails the add process.

[whyrusleeping]

Lets go ahead and raise the default fd limit to 2048 for now. The number of nodes on the network is getting larger.

Longer term solutions will be adding QUIC support, and implementing connection closing and limiting strategies

[ghost]

Running the daemon with --offline should avoid this, since it's the same root cause as many other open bugs (flood of outgoing network traffic overloading something or other).

[whyrusleeping]

This should be temporarily resolved by #3828

We're working on several different solutions that will all help with this:

• QUIC, a udp based transport, will use only a single file descriptor for all connections
• sbs, a new block storage engine, will keep and use only a single file descriptor for all put operations
• connection closing, in the next few months we will be working on strategies for selecting open connections to close out. This will reduce the overall load on the daemon, including reducing the number of open file descriptors

[whyrusleeping]

We can either keep this open until other solutions are implemented, or close this once #3828 gets in.

Thoughts @Kubuxu ?

[Kubuxu]

I would keep it.

[ligi]

also stumbling into this right now with a ci-server I wrote that writes reports to IPFS:

14:40:14.196 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
14:40:14.302 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
14:40:14.502 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
14:40:14.802 ERROR     flatfs: too many open files, retrying in 400ms flatfs.go:180
14:40:15.272 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
14:40:15.373 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
14:40:15.573 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
14:40:16.109 ERROR commands/h: err: open /home/kontinuum/.ipfs/blocks/QK/put-044922444: too many open files handler.go:288

unfortunately "--offline" is not an option as far as I see - is there another workaround?

EDIT: just saw 0.4.8 is now out and I am on 0.4.7 - will try if the problem vanishes with 0.4.8

EDIT#2: Dam it also happens with 0.4.8:

15:03:57.094 ERROR commands/h: err: open /home/kontinuum/.ipfs/blocks/LP/put-915734068: too many open files handler.go:288
15:04:03.748 ERROR       mdns: mdns lookup error: failed to bind to any multicast udp port mdns.go:135
^C
Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)
kontinuum@ligi-tp ~> ipfs version
ipfs version 0.4.8

[whyrusleeping]

@ligi when trying 0.4.8, does the daemon output the message saying its raising the file descriptor limit to 2048 ?

[ligi]

yes - here the full output:

Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/172.17.0.1/tcp/4001
Swarm listening on /ip4/192.168.5.42/tcp/4001
Swarm listening on /ip4/82.119.11.152/tcp/4001
Swarm listening on /ip6/2a02:2450:102a:19d:1819:fa1c:abf1:586a/tcp/4001
Swarm listening on /ip6/2a02:2450:102a:19d:b963:bd27:4c80:529f/tcp/4001
Swarm listening on /ip6/2a02:2450:102a:19d:f2de:f1ff:fe9a:1365/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
15:03:57.094 ERROR commands/h: err: open
/home/kontinuum/.ipfs/blocks/LP/put-915734068: too many open files
handler.go:288
15:04:03.748 ERROR       mdns: mdns lookup error: failed to bind to any
multicast udp port mdns.go:135
^C
Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)
kontinuum@ligi-tp ~> ipfs version
ipfs version 0.4.8

--
friendly greetings,
ligi
http://ligi.de

[htor]

Getting the same error here on os x 10.11

client

ipfs version 0.4.10
...

 72.00 MB / 875.45 MB [========>------------------------------------------------------------------------------------------]   8.22% 14s22:14:29.876 ERROR commands/h: open /Users/herman/.ipfs/blocks/6R/put-228389154: too many open files client.go:247
Error: open /Users/herman/.ipfs/blocks/6R/put-228389154: too many open files

daemon:

...
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
21:53:22.811 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135
22:14:29.876 ERROR commands/h: err: open /Users/herman/.ipfs/blocks/6R/put-228389154: too many open files handler.go:285
22:14:32.811 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135

[alphaCTzo7G]

Got this as well today.. was following the tutorial here..

Probably related to the number of files added... happened while I was running the ipns example here..

http://www.atnnn.com/p/ipfs-hosting/

[magik6k]

Gave it a shoot, connection closing should help. Using this script to monitor deamon FD usage:

export DPID=26024; watch -n0 'printf "sockets: %s\nleveldb: %s\nflatfs: %s\n" $(ls /proc/${DPID}/fd/ -l | grep "socket:" | wc -l) $(ls /proc/${DPID}/fd/ -l | grep "\\/datastore\\/" | wc -l) $(ls /proc/${DPID}/fd/ -l | grep "\\/blocks\\/" | wc -l)'

where DPID is daemon PID, output looks like this:

sockets: 448
leveldb: 8
flatfs: 1

When adding a large file(dd bs=1M if=/dev/urandom count=1000 | ipfs add --pin=false), flatfs rarely gets up to 30, sockets sometimes 'explode' to 2k, though they can't be seen in ipfs swarm peers which is constantly at ~500 peers.

[ETeissonniere]

Got this on 0.4.10 today:

Version information:

go-ipfs version: 0.4.10-
Repo version: 5
System version: amd64/linux
Golang version: go1.8.3

Description:

Trying to add a big (70MB) file to ipfs make it crash.

Here is the output of the command:

$ ipfs add backup.tar.gz 
 72.00 MB / 78.10 MB [============================================================================================================================================================================================>----------------]  92.19% 1s18:55:08.136 ERROR commands/h: open /home/eliott/.ipfs/blocks/MX/put-110547071: too many open files client.go:247
Error: open /home/user/.ipfs/blocks/MX/put-110547071: too many open files

On the daemon side:

$ ipfs daemon
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
Swarm listening on /ip4/10.1.1.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/172.17.0.1/tcp/4001
Swarm listening on /ip4/172.18.0.1/tcp/4001
Swarm listening on /ip4/192.168.42.175/tcp/4001
Swarm listening on /ip4/78.233.22.41/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
18:55:08.136 ERROR commands/h: err: open /home/user/.ipfs/blocks/MX/put-110547071: too many open files handler.go:285

Some more infos:

$ ipfs diag sys
{"diskinfo":{"free_space":249762070528,"fstype":"61267","total_space":231902359552},"environment":{"GOPATH":"/home/user/Documents/GoLang","IPFS_PATH":""},"ipfs_commit":"","ipfs_version":"0.4.10","memory":{"swap":0,"virt":762644000},"net":{"interface_addresses":["/ip4/127.0.0.1","/ip4/10.1.1.1","/ip4/172.18.0.1","/ip4/172.17.0.1","/ip4/192.168.42.175","/ip6/::1","/ip6/fe80::4ef4:6811:8297:44d0"],"online":true},"runtime":{"arch":"amd64","compiler":"gc","gomaxprocs":4,"numcpu":4,"numgoroutines":2564,"os":"linux","version":"go1.8.3"}}

[magik6k]

As explained above the problem is caused by creating way too many new connections when a big file is added. Workaround is to run daemon with IPFS_FD_MAX env variable set to 4k.

IPFS_FD_MAX=4096 ipfs daemon

[magik6k]

Seems to have been fixed by fixing https://github.com/ipfs/go-ipfs/issues/4102, haven't ran into it since the fix was merged.

[kamaci]

Whenever I try to start a private network I get ulimit error:

Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
00:10:32.457 ERROR   cmd/ipfs: error from node construction: EOF daemon.go:320
Error: EOF

I've increased ulimit size via IPFS_FD_MAX, tried fresh install with declaring IPFS_PATH but didn't work. When I remove swarm.key (https://github.com/Kubuxu/go-ipfs-swarm-key-gen) I get no error.

[Stebalien]

I don't see any ulimit errors there. It looks like go-ipfs is prematurely hitting the end of a file (probably your swarm key) when starting. Let's open another issue, this should have a better error message.