go-ipfs: too many open files

[tidux]

I run a fairly high-traffic public IPFS node on a server. I received this error attempting to add a file:

$ ipfs add ./file.mp4
32.00 MB / 4.06 GB [>------------------------------------------------------------------------------------------------] 0.77 % 21m45s00:58:05.098 ERROR commands/h: open /home/$USER/.ipfs/blocks/1220b559/put-221257532: too many open files client.go:247
$

@whyrusleeping yes this is the same node that encountered the deadlock bug in an earlier version.

[RichardLitt]

Hi @tidux. The code snippet above pretty clearly mentions a copyrighted material. Please read our Code of Conduct, and, specifically, the section on Copyright Violations. We're here to help, but we take this subject very seriously. Please respect the Code of Conduct.

For now, this is just a warning; please edit the code comment to include a different file name. Thanks.

[whoizit]

There isn't violation issue in file name. This is a proper name, this is not a trademark. Anyone can write the name of a movie or song titles without restriction. If I'm wrong, you have to point to the law, in which it is clearly stated.

[jbenet]

What @RichardLitt is getting to is that it is _very_ important for our community to remain sound, that people dont accidentally make available content they should not make available through our communication channels, or imply behavior that is not in line with out Community Code of Conduct.

We have very strict community policies on this, because this kind of problem tends to hurt P2P community projects massively. And yeah, on the whole, i think names in general are fine. but this is actually a problematic piece of information for this issue tracker. The snippet tells me:

• a 4GB file
• name bearing the title and year of a well known movie
• video file extension

It raises questions. questions like: is this a copyrighted file? is it being added to ipfs? does the author have the copyright permissions necessary to distribute this file? Is this action legal or illegal? And look, @tidux may well have the copyright for this file and ability to move it. Or live in a jurisdiction where those copyrights do not apply. This may even be just a big file full of zeros. None of this is concrete, all of it is implied. But what is being implied? And is the IPFS community better or worse for it? There is also the subject of privacy violation, and incriminations of the people posting.

Let me be clear: implied behaviors carry risks with them, and may damage the community. and we have policies in place to avoid this sort of damage.

Consider a _worse_ case, hopefully more clear cut. What if the name said something like: ./revenge-porn.mp4. I hope you agree that this is damaging to the community, even if not illegal in many jurisdictions, and should not be here.

---

Please read our community code of conduct. In some ways, it is MORE STRICT than most legal jurisdictions. We opt into these restrictions to make make a broader and more accepting community, to avoid orthogonal problems, and we do this even if that means setting some of our views and beliefs to be discussed aside, elsewhere. If you disagree, that's fine, you can opt out of using our community communication channels.

A broader point is that this here is an issue tracker for the go-ipfs codebase. File names expressed should not put the community at any risk, whether through accidental implication, or misinterpretations, or even bad search indices. It is -- in general -- best to use innocuous example filenames that do not violate any privacy, or imply anything non-relevant.

[jbenet]

@tidux pls edit the filename.

Back on Topic

@tidux Have you raised your fd limit? what is it at on this node?

have you tried the ipfs daemon --manage-fdlimit flag?

[tidux]

Filename fixed. I haven't messed with the fd limit at all at any level, so whatever Debian Jessie has by default.

[whyrusleeping]

interesting, i'll have to take a look at how high contention affects the flatfs datastore code. It might just not be clearing the file descriptors fast enough

[RichardLitt]

@tidux Your edit to the filename also goes against the Code of Conduct. Please read this section on a harassment free space (that includes me and @jbenet, and anyone else in this repo).

[tidux]

It looks like this was due to an open-files-per-process limit at the OS level. The file in question was added successfully in under a minute on a similarly specced machine running the same binary on the same Linux distro, albeit with less IPFS network and disk load.

[whyrusleeping]

@tidux has this been resolved?

[Kubuxu]

I think it was resolved by backing-off and rising fd limit.

[tidux]

@whyrusleeping I only have the one system running IPFS with enough files to meaningfully test this, and I have not re-experienced it since then. I think it can stay closed.