Go-ipfs: Writeup of router kill issue

[whyrusleeping]

So we know that ipfs can kill people routers. We should do a quick write up of what the causes are, which routers are normally affected, and maybe propose a couple ideas for solutions.

@Kubuxu do you think you could handle doing this at some point?

[donothesitate]

My theory is, it does exhaust / overload NAT table, that on some routers does cause lockups.
UDP on the same routers can keep working without problem, as well as TCP connections that were already open when lockup occurred.

Possible solution: Have a switch to limit number of peers/connections.
Related #3311

[ghost]

That sounds highly likely. nf_conntrack_max on my edge router is set to 1024 by default and ipfs eats 700 of those on its own, per computer I'm running it on.

A lot of those are dead connections too: if I open the webui which tries to ping them it quickly drops to 90 or so.

[hsanjuan]

Running 5 daemons on local network with a well-known hash (they were pinning dist) kills my Fritzbox.

AFAIK everyone has high esteem for Fritzboxes as very good routers, and not some shitty hardware. Internet reports a NAT table size of around 7000. I find the problem is exacerbated when my nodes are pinning popular content (I suspect this not only consumes all the bandwidth but also increases the number of connections when other peers try to download these blocks?).

[Kubuxu]

So my idea of what happens is that conntracker table fills up (it is small in cheapo routers, bigger is good ones) and it starts trowing out other connections. @hsanjuan can you repeat the test, kill ipfs daemons and check if it comes back up online?

[hsanjuan]

@Kubuxu yeah yeah things are back up immediately when I kill them. Only once I had the router reboot itself, which worried me more.

[Kubuxu]

So other possibility is that cheapo routers have bigger conntracker limit than their RAM can handle and they kernel panics or lockups. Not sure how to check it.

[whyrusleeping]

Does UDP eat up conntracker entries? We're moving quickly towards having support for QUIC.

[Kubuxu]

AFAIK, yes. At least from the time my services were DDoSed with UDP packets and they were much more destructive because of low conntracker limits.

[hsanjuan]

Is it possible that this problem got much worse in the last releases (ie >=0.4.5). I used to be able to run 4 nodes without problems and now it seems I'm not, even after cleaning their contents.

[kakra]

I'm having issues, too. Maybe ipfs should take two connection pools and migrate peer connections from a bad quality pool to a good quality pool by applying some heuristics to the peers. Peers with higher delays, lower bandwidth and short lives would live in the "bad pool" and easily replaced by new peers if connection limits are hit. Better peers would migrate to the "good pool" and only be replaced by better peers if limits are hit. Having both pools gives slow peers a chance to be part of the network without being starved by higher quality peers, which is important for a p2p distributed network.

BTW, udp also needs connection tracking, this wouldn't help here, and usually udp tracking tables are much smaller and much more short-lived which adds a lot of new problems. But udp could probably lower the need for bandwidth as there's no implicit retransmission and no ack. Of course, the protocol has to be designed in a way to handle packet loss, and it must take into account that NAT gateways usually drop udp connection table entries much faster. It doesn't make sense to deploy udp and then reimplement retransfers and keep-alive, as this would replicate tcp with no benefit (probably it would even lower performance).

Also, ipfs should limit the amount of outstanding packets, not the amount of connections itself. If there are too many packets in-flight, it should throttle further communication with peers, maybe prioritizing some over others. This way, it could also auto-tune to the available bandwidth but I'm not sure.

Looking at what BBR does for network queues, it may be better to throw away some requests instead of queuing up a huge backlog. This can improve overall network performance, bloating buffers is a performance killer. I'd like to run ipfs 24/7 but if it increases my network latency, I simply cannot, which hurts widespread deployment.

Maybe ipfs needs to measure latency and throw away slowly responding peers. For this to properly work, it needs to auto-adjust to the bandwidth, because once network queues fill, latency will exponentially spike up and the former mentioned latency measurement is useless.

These big queues are also a problem with many routers as they tend to use huge queues to increase total bandwidth for benchmarks but it totally kills latency, and thus kills important services like DNS to properly work.

I'm running a 400/25mbps assymmetric link here, and as soon as "ipfs stats bw" get beyond a certain point, everything else chokes, browsers become unusable waiting for websites tens of seconds, or resulting in DNS errors. Once a web request comes through in such a situation, the website almost immediately completely appears (minus assets hosted on different hosts) so this is clearly an upstream issue with queues and buffers filled up and improper prioritizing (as ACKs still seem to pass early through the queues, otherwise download would be reduced, too).

I don't know if QUIC would really help here... It just reduces initial round-trip times (which HTTP/2 also does) which is not really an issue here as I consider ipfs a bulk-transfer tool, not a latency-sensitive one like web browsing.

Does ipfs properly use TOS/QoS flags in IP packets?

PS: ipfs should not try to avoid tcp/ips auto-tuning capabilities by moving to UDP. Instead it should be nice to competing traffic by keeping latency below a sane limit and let TCP do the bandwidth tuning. And it should be nice to edge-router equipment (which is most of the time cheap and cannot be avoided) by limiting outstanding requests and amount of total connections. I remembered when Windows XP tried to fix this in the TCP/IP stack by limiting outstanding TCP handshakes to ten, blocking everything else then globally. This was a silly idea but it was thinking in the right direction, I guess.

[dsvi]

I think you might as well not do anything at all, since routers are getting consistently better at supporting higher numbers of connections. My 5 years old struggled with supporting 2 ipfs nodes (about 600 connections each) + torrent (500 connections). I've just got cheap chinese one, and it works like a charm. Most of even cheap routers nowadays have hardware NAT. They don't much care how many connections you throw at them.
Also, switching to UDP doesn't help, since when i unleash torrent far beyond 500 connections limit, it used to kill the old router as good as ipfs. And torrent uses only UDP.

[ghost]

@dsvi: I'd rather not have to pay hard cash just to use IPFS on the pretence that it's fine to be badly behaved because some other software can be _misconfigured_ to crash routers. A lot of people don't even have the luxury of being allowed to connect to their ISP using their own hardware.

And what a strawman you've picked — _a Bittorrent client_! A system that evolved its defaults based on fifteen years real world experience for _precisely this reason_!

No thanks, just fix the code.

[kakra]

@dsvi I wonder if they use their own routers because the page times out upon request... ;-)

But please do not suggest that: Many people are stuck with what is delivered by their providers with no chance to swap that equipment for better stuff. Ipfs has not only to be nice to such equipment but to overall network traffic on that router, too: If it makes the rest of my traffic demands unusable, there's no chance for ipfs to evolve because nobody or only very few could run it 24/7. Ipfs won't reach its goal if it is started by people only on demand.

[dsvi]

Sorrry guys, should have expressed it better. I'll try this time from another direction ;)

1. Internet world is becoming decentralized in general. This is a natural trend which is everywhere, from secure instant messaging, filesharing, decentralized email systems and so on.
   And creating tons of connections is a natural part of such systems. They are distributed, and for effective work they have to support tons of connections (distributions channels). It's unavoidable in general. There can be improvements here and there, but its fundamentally "unfixible"
2. Hardware vendors have acknowledged that already. Modern router chipsets are way better in that regard nowadays, since all the hardware review sites have included at least torrent tests, in their review test suites. So you don't really need nowadays something 200$+ to work well with it. And a year from now, it will only get waay better, since they tend to offload a lot of routing work to hardware.
   So it already is not a problem, and will be even less so with every year.

And what about people who stuck with relic hardware for whatever reason? Well i feel sorry for some of them, but the progress will go on with them, or without.

[Calmarius]

@dsvi

"Internet world is becoming decentralized in general. "

Nope! It's becoming centralized. Almost the whole internet is served by a handful datacenter companies.
For most people search means Google, e-mail means Gmail, social interactions mean Facebook, videos mean Youtube, chat means Facebook Messenger, picture sharing means Instagram.
The rest of the web is hosted at the one of the several largest datacenter companies.

At the beginning we used to have Usenet and IRC servers running on our computers at home.
Then services got more and more centralized.

I don't see signs of any decentralization. But I see signs of further centralization.
For example some ISPs don't even give you public IP addresses anymore (for example 4G networks).

"And creating tons of connections is a natural part of such systems."

Having too many simultaneous connections makes the system inefficient.
If you have enough peers to saturate your bandwidth it's pointless to add more.

Currently my IPFS daemon opens 2048 connections within several hours to peers then runs out of file descriptors and becomes useless. This should be fixed.

[vext01]

I'm using a crappy TalkTalk router provided by the ISP and I've been unable to find a configuration where IPFS doesn't drag my internet connection to it's knees.

Using ifstat I see usually between 200kb/s and 1MB up and down whilst ipfs is connected to a couple of hundred peers.

I'd like to try connecting to fewer peers, but even with:

      "LowWater": 20,
      "HighWater": 30,

ipfs still connects to hundreds.

[vext01]

Perhaps this is a dumb question, but why don't you make it so that IPFS stops connecting to more peers once the high water mark is reached?

[Stebalien]

We should implement a max connections but high/low water are really designed to be target bounds.

The libp2p team is currently refactoring the "dialer" system in a way that'll make it easy for us to configure a maximum number of outbound connections. Unfortunately, there's really nothing we can do about inbound connections except kill them as soon as we can. On the other hand, having too many connections usually comes from dialing.

[Stebalien]

Note: there's actually another issue here. I'm not sure if limiting the max number of open connections will really fix this problem. I haven't tested this but I'm guessing that many routers have problems with connection velocity (the rate at which we (try to) establish connections) not simply having a bunch of connections. That's because routers often need to remember connections even after they've closed (for a period of time).

@vyzo's work on NAT detection and autorelay should help quite a bit, unless I'm mistaken.

[kakra]

A work-around could be to limit the number of opening connections (in contrast to opened connections) - thus reducing the number of connection attempts running at the same time. I think this could be much more important than limiting the number of total connections.

If such a change propagated through the network, it should also reduce the amount of overwhelming incoming connection attempts - especially those with slow handshaking because the sending side is not that busy with opening many connections at the same time.

[Stebalien]

We actually do that (mostly to avoid running out of file descriptors). We limit ourselves to opening at most 160 TCP connections at the same time.

[kakra]

@Stebalien Curious, since when? Because I noticed a while ago that running IPFS no longer chokes DNS resolution of my router...

[Stebalien]

We've been doing it for quite a while however, we may have changed some constants. I'm not sure.

[hsanjuan]

IPFS also chokes DNS resolution on my router @kakra (particularly when building dists, which downloads large files). I haven't checked if there are any improvements though, but will keep an eye. Do you have a Fritzbox too?

[kakra]

@hsanjuan No, it's a Vodafone DOCSIS 3.1 router, 1 gigait connection.

[raulk]

The connection manager currently works with hard bounds set by configuration. Do you think it’s feasible to “sense” appropriate limits from the environment by allowing connections to be alive until we hit an hiccup, at which point we back off?

[Kubuxu]

Some other mechanism of "router kill" I've observed is SYN and RST storms when ipfs starts working and is being shut down.
I had an ISP router go down sometimes when starting and stopping ipfs.

[magik6k]

SYN and RST storms

This was reported in libp2p-swarm some time ago: https://github.com/libp2p/go-libp2p-swarm/issues/86

[raulk]

This was reported in libp2p-swarm some time ago: libp2p/go-libp2p-swarm#86

With the new DHT dial queue, this should not happen any longer: https://github.com/libp2p/go-libp2p-kad-dht/pull/237

However I was more interested in knowing if there's a event (or set thereof) that we can interpret as a high-fidelity signal that we're tripping up the router, so we can back off and adjust our connection manager limits adaptively.

[Kubuxu]

However I was more interested in knowing if there's a event (or set thereof) that we can interpret as a high-fidelity signal that we're tripping up the router, so we can back off and adjust our connection manager limits adaptively.

Hard to tell, many of them will continue forwarding connections without any problem because they are on the fast path, until they give up.

[vext01]

With the new DHT dial queue, this should not happen any longer

Great! I'm going to try on my DSL router later!

[AndreaCensi]

On my Mac, IPFS kills (in addition to my router) also some applications --- e.g., Chrome, which I suspect cannot do a DNS query, but I am not sure.

What is the workaround currently?
Can IPFS be set up so that the daemon on my PCs connect only to my server in the cloud, and through it to the entire network? (I do not mean making a private network with the swarm key, but using one known "gateway").

[andrew]

This still regularly happens on my home network, killing both the router and the wifi extender.

[victorb]

Same here. Development is currently infeasible with anything involving go-ipfs (using ipfslite for Bolivar, but suffering from same problem).

Just running a node for about ~5 minutes (had time to connect to 30 peers) kills my router, which is standard issued router from my ISP.

[Stebalien]

@victorb could you try running in DHT client mode? I want to see if that makes any difference (relates to #6283).

[AndreaCensi]

@Stebalien FWIW, following the advice in other threads, I am using --routing dhclient, which makes the situation a bit better, but the problem persists.

Is there a previous version of go-ipfs where this problem is reduced? My problem like @victorb is also that I would like to set up a stable development environment, with the basic ipfs functionality.

[Stebalien]

No. The problem is likely to be significantly worse in previous versions. This is a long standing issue we've been whittling away at, not a regression.

[victorb]

Thanks @Stebalien. Did the following change: https://github.com/open-services/bolivar/pull/6/files

Seems to have stopped killing my router at home, at least after running the tests for a couple of times (without that change, it gets killed every time I run the tests).

While it's good that it's now working, the application does not work as wanted after that change, as it won't help local peers find content anymore, making it pretty useless.

Wondering if another DHT client could be introduced. One where the node doesn't respond to all DHT requests, only to ones coming from the local network.

[Stebalien]

Wondering if another DHT client could be introduced. One where the node doesn't respond to all DHT requests, only to ones coming from the local network.

Unfortunately, that won't work if any one of your local nodes is a member of the global DHT. That node will include you in their routing table and tell their peers about you.

as it won't help local peers find content anymore, making it pretty useless.

So, bitswap should still work. Are you trying to get this to work in an offline/disconnected context where local peers can't search the global DHT?

Honestly, a local peer exchange seems like the best solution for issues like this.

[victorb]

Unfortunately, that won't work if any one of your local nodes is a member of the global DHT. That node will include you in their routing table and tell their peers about you.

Hm, I see... I guess what I'm trying to figure out is how to get local (as in same local network) peers to collaborate on sending data, while avoiding the issue of ipfs/libp2p killing peoples router.

So, bitswap should still work. Are you trying to get this to work in an offline/disconnected context where local peers can't search the global DHT?

Exactly. The use case is that there are two peers, they are locally connected but might or might not have internet connection. Even if they do/do not have a internet connection, they should still be able to fetch content from each other, as they are both using the same hash to resolve links from within that hash.

I've done some tests with the DHT Client and my assumption seems correct that they won't be able to find content from each other, when both are running as DHT client. So while it's nice that the router doesn't die anymore, it kind of makes the use-case of p2p useless, as it needs to reach something that is not just a DHT client.

Honestly, a local peer exchange seems like the best solution for issues like this.

Is this something that already exists or you're talking about something that might exists in the future?

[Stebalien]

I've done some tests with the DHT Client and my assumption seems correct that they won't be able to find content from each other, when both are running as DHT client. So while it's nice that the router doesn't die anymore, it kind of makes the use-case of p2p useless, as it needs to reach something that is not just a DHT client.

MDNS _should_ cause these peers to connect to eachother automatically. However, we may need to improve MDNS support.

We could _also_ try using the DHT for local rendezvous (i.e., who else has my same public IP?).

[remon-nashid]

Has there been any recent efforts to tackle this issue. This issue still renders IPFS useless.

[Stebalien]

Has there been any recent efforts to tackle this issue.

Yes.

• There's experimental QUIC support (a UDP-based transport). _Some_ routers (not all) behave better when tracking UDP "connections" (UDP doesn't really have connections) instead of TCP connections.
• IPFS dials less (DHT fixes) and we're working on making it dial _even_ less (more DHT fixes on the way).
• IPFS now has a connection manager to try to keep the number of open connections below a sane limit.
• In my experience, consumer routers themselves are getting better (I'm seeing this issue less frequently).

[remon-nashid]

Thanks @Stebalien! That sounds quite promising. While IPFS keeps progressing, is there a set of settings that could help to avoid the issue, while not compromising on IPFS features? There are many suggestions in this thread but some could be outdated or just ineffective. A standard set of settings, preferably in IPFS documentation, would be awesome.

[Stebalien]

Unfortunately, the _best_ solution is to replace your router.

Baring that, you can try running IPFS in client-only mode by running the daemon with ipfs daemon --routing=dhtclient. This will lead to _significantly_ fewer inbound connections.

[chevdor]

@Stebalien any list anywhere of what you consider a bad/good router?

[Stebalien]

I've never found a router I've been truly happy with. I use a tp-link (TL-R600VPN) and it seems to hold up to IPFS just fine, but it has its own problems (we actually have a work-around in libp2p to deal with them).

[SunsBR]

Its 2020 this started in 2016, and it still crashes routers.
have a fiber ISP router (cannot change it). symmetric 100mbs up/down the connection number goes up really fast and I cannot even ping the router (local lan continues working fine). any other p2p software in a good number of machines never locked up the router. take a time to come back after kill the daemon, probably a watchdog reseting the router.
ipfs ver 0.5.1

[yxuco]

Same issue here using Linksys CG7500 cable modem/router on Comcast network. The router reboots frequently when ipfs daemon is running. The network is more stable after I set the following config properties, which had kept the number of peer connections below 500 most of the time.
However, the performance for curl https://ipfs.io/ipfs/mycid gradually degraded with time for a file pinned on the local node, and it returns only an error Couldn't connect to server after 15 minutes. Does it mean that the local daemon became non-responsive to inbound requests even though it was still running with no errors?

    "Swarm": {
        "ConnMgr": {
            "GracePeriod": "10s",
            "HighWater": 90,
            "LowWater": 60,
            "Type": "basic"
        },

[RubenKelevra]

I suspect that the issue is not the number of open connections, but the amount of connection attempts to private networks, which the router tries to push to his default route. Since the ISP has no route for them, the probably doesn't get answered with ICMP but just dropped. This causes the router to be unable to clear them up in a timely manner.

By default around 1/3 of all connection attempts go to private networks.

I wrote about this issue here.

[alexfisher]

So, I revisit this issue after trying to run an IPFS node every 6 months only to find my router/internet connection killed by it over and over again. I'm actually open to replaying my ISP's provided cable modem with something that IPFS doesn't break. Does anyone have a list of cable modems known to work well?

[kakra]

Chances are that you could ask your provider to switch the modem into bridge mode for attaching your own router. Bridge mode (the one which directly attaches you to a public IP on the LAN interface) will probably turn off a lot of stuff in the cable modem and thus reduce CPU and memory pressure. Then use a good router. Also, upgrading to a DOCSIS 3.1 capable modem may improve things, as those should be designed to support much higher cable speeds and more connections, but memory and NAT settings may still be an issue. Your best bet is probably going for a dedicated router and using the modem in bridge mode. I'm using a DOCSIS 3.0/3.1 modem with 1 Gbit/s downstream in bridge mode, connected to a Unifi UDM as a router. But I didn't have much problems since my provider swapped the modem for a DOCSIS 3.1 one, still switching to bridge mode improved things (latency went down from 40-50 to 20-30ms).

Also, you may want to make sure that you're blocking traffic to private networks from being routed to the WAN interface of the modem. Most ISP routers will try to route such traffic to external via the WAN interface because they don't blackhole private networks and just use the default route, thereby hogging the NAT tables of the router. You could blackhole such networks in either your dedicated router or your local machine (example for Linux):

ip route add blackhole 192.168.0.0/16
ip route add blackhole 172.16.0.0/12
ip route add blackhole 10.0.0.0/8

No fear, your directly attached network will still work as it has a longer network prefix usually (if it doesn't, don't blackhole it). Also, other networks like VPN tunnels should still work because they put routes with longer prefixes. IPv6 should have no such problems as routes have scopes there which restrict the routing decision.

BTW: On my UDM I've put firewall rules to answer such packets with ICMP unreachable instead, so the connection gets properly torn down again immediately.

[alexfisher]

Thank you for the thorough and helpful suggestion @kakra ! I just put in an order for a Unifi UDM and some other goodies. I had already been looking at a Peplink router for load balancing and more processing power, but looks like the UDM will do a lot more for me. Will report back how IPFS works for me after getting things setup.

[kakra]

You should use the UDM with a modem in bridge mode, I hope your provider allows that or you can directly switch it in the modem. Otherwise the modem will still do NAT, and your problem may not be solved.

[yxuco]

I hope that this problem will be fixed, or instructions on required modem types and ISPs will be documented. I ran Ethereum node without any problem, but IPFS will kill my cable modem/router within 20 minutes. I understand that Ethereum uses different p2p libs, but it indicates that such problem is not only an issue of the modem configuration.

[alexfisher]

@kakra Yep, I can switch the Arris modem into Bridge mode no problem. Thanks again.

@yxuco Ethereum currently uses devp2p, but Ethereum 2.0 will switch to using libp2p. I experienced this same issue running an early Prysm Eth 2.0 testnet client which also uses libp2p.

[kakra]

I understand that Ethereum uses different p2p libs, but it indicates that such problem is not only an issue of the modem configuration.

From what I understand, IPFS announces also LAN IPs for finding and connecting hosts locally. It cannot know if such IPs are behind a NAT router, they may just be attached locally in a different network segment via a local router. So it's just fair that it works this way. The problem comes from most NAT routers that try to route the traffic to their default gateway no matter if that IP belongs to a private range. This create a NAT table entry, and because there will be no reply, the connection eventually times out after 10 minutes or so - until then, a useless NAT entry hogs the table in the router, and probably not only one but hundreds of them. The result: The modem struggles to map any more valid connections, DNS dies because it gets no replies, new connections fail and time out, the modem seems to be killed and eventually recovers, or it experienced some memory/CPU overflow and has to be rebooted.

Since using a good router, I have no more problems. IPFS currently maintains 350-400 peer connections and I see absolutely no negative effect on other connections or throughput.

It's probably dumb in almost any situation to let a NAT router map connections on the egress WAN interface in the NAT table which have a private IP as destination. This only tells us that ISPs don't put much effort into making a stable router, they just use the default settings, make some pretty (or not so pretty) UI change, and lock down the advanced settings... :-(

I don't think that IPFS would kill any somewhat modern router with a somewhat modern broadband line if ISPs would pre-configure their routers in a sane way. But evidence shows otherwise - and then it's most probably a bad pre-configuration that is locked down so users cannot adjust it. Not IPFS' fault...

[alexfisher]

@kakra UniFi UDM Pro is amazing. Loving my new network set up. Can confirm, IPFS and network running great now. Arris cable modem in bridge mode + UDM Pro.

[kakra]

@alexfisher I'm using the Non-Pro UDM - works also great. And my cable modem is probably a rebranded Arris modem - I'm not sure. At least the OUI matches Arris.

[lkdmid]

10th Nov 2020 - I thought I was going crazy until I came across this 2016 thread - but I can confirm that IPFS (on my Macbook, latest everything) causes my Sky Q router to spontaneously reboot, which is rather unfortunate given this ISP's popularity in the UK at-least. Unfortunately, I'm not in the position to replace the hardware right now either (and that's not really a suitable "solution" if mass-adoption is the goal anyways, right?).

[kakra]

Try black-holing LAN networks on your machines running IPFS:

# http://www.uni-koeln.de/~pbogusze/posts/Blackhole_routing_with_linux.html
sudo ip route add blackhole 192.168.0.0/16
sudo ip route add blackhole 172.16.0.0/12
sudo ip route add blackhole 10.0.0.0/8

Any more specific routes (those with a larger mask, e.g. 192.168.1.0/24) have a higher weight, so you should be fine setting these routes without any unexpected issues. Now, IPFS won't be able to send any of those destinations to your router which is likely the problem here because it forwards those via the WAN interface, despite not being routable there anyways. It will only bog the NAT tables of the router, in turn making it unstable.

A more advanced option would be to actually reject those destinations at the router itself so IPFS will be notified about unreachable destinations immediately, otherwise you're just shifting the problem over to your PC - but at least your PC is able to handle this.

IPv6 should be safe because it uses scoped routing: The router (at least a sane implementation) won't route site and local scope addresses to the WAN interface. And it usually doesn't need NAT either.