monerod is auto killed

[dungdang95]

My ./monerod was killed. I start again and it exits again. Here my log:

[rblaine95]

There is an ongoing attack on the Monero Network where malicious nodes are creating memory bombs (very large packets that cause high memory usage) which causes the monerod process to get OOMKilled

Current mitigation is to update to the latest stable release of Monero (v0.17.1.7) and to apply the Tor Blocklist

wget https://gui.xmr.pm/files/block_tor.txt -O ~/.bitmonero/block_tor.txt
monerod --ban-list ~/.bitmonero/block_tor.txt

Or, add the following line to your bitmonero.conf

ban-list=/path/to/block_tor.txt

[janfiedler]

@rblaine95 Thanks, this must be massive attack, even with your solution it give me only few minutes more. Then my node is memory bombed and killed.

[teran-mckinney]

Are there any fixes in the works for this? I'm definitely running into this as well.

[selsta]

Yes, we tagged v0.17.1.8 and it should be released in the next hours.

[teran-mckinney]

Awesome, thank you!!

[selsta]

v0.17.1.8 is out: https://www.getmonero.org/downloads/

[teran-mckinney]

I just upgraded. I'll write back with my findings. It was being OOM killed every hour or two before, seemingly.

[selsta]

New attack, seems like the tor block list is required again: https://gui.xmr.pm/files/block_tor.txt

This does not seem related to the attack fixed in v0.17.1.8

[teran-mckinney]

My node has been up since I restarted it. No issues yet.

Edit: Also, I've never used the Tor blocklist.

[tempname1024]

New attack, seems like the tor block list is required again: https://gui.xmr.pm/files/block_tor.txt

This does not seem related to the attack fixed in v0.17.1.8

Indeed, had a crash today after ~36 hours of uptime running v0.17.1.8 without the block list.

2021-01-01 16:57:45.638 W There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Monero network or under attack, or your computer's time is off. Or it could be just sheer bad luck.

[selsta]

There were 0 blocks in the last 20 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Monero network or under attack, or your computer's time is off. Or it could be just sheer bad luck.

This is unrelated and happens sometimes.

[teran-mckinney]

No crashes but my block height got near 2265577 and the daemon is unresponsive, even to a block height command.

I just restarted it. Will see what it does.

[selsta]

@teran-mckinney

Next time this happens please don’t restart and use gdb:

gdb /path/to/monerod `pidof monerod`

thread apply all bt

so that we can see where it freezes.

[teran-mckinney]

Ok, will do! It's moving forward now at least. I wonder if anyone else will get this and they can try it. Or maybe I'll get it again.

[selsta]

I have only read of one other person having this freeze issue.

[Masterbob79]

I have noticed monerod got killed the other day, but I haven't checked the logs yet

[voidzero]

Unfortunately I'm still getting repeated OOM kills.

Monero 'Oxygen Orion' (v0.17.1.8-ef1ba5142)

Memory cgroup out of memory: Killed process 17408 (monerod) total-vm:127917672kB, anon-rss:7748416kB, file-rss:7668kB, shmem-rss:0kB, UID:xxxx pgtables:107460kB oom_score_adj:0

Monero is running in a cgroup with a 6GB memory soft limit and a 8GB hard limit.

[komatom]

Still kills itself, added tor block list to try with it

Monero 'Oxygen Orion' (v0.17.1.8-release)

[dEBRUYNE-1]

@komatom @voidzero - Can you try compiling the release branch?

https://github.com/monero-project/monero#compiling-monero-from-source

[voidzero]

@dEBRUYNE-1 you betcha. I'll check back in a few minutes.

[voidzero]

Built correctly; version v0.17.1.9 confirmed to be running and I've disabled using the tor blocklist. "Now we wait." If it crashes I will let you know and if not, well, I'll let you know too. It usually crashed after no more than an hour. If you can add a debug message "caught exploit such and so" then I can let you know more quickly.

[voidzero]

2 hosts were blocked so far, and I got a bunch of warnings saying

_there were (0|1|2) blocks in the last 30 minutes, there might be large hash rate changes (...)_

I guess that might actually indicate that stuff is working properly - no more DoS now. Just curious, Is the network under such a heavy attack or is something else going on? Anyway. Seems to be working now, if it does bug out, I'll let you know but it doesn't look like it will.

[selsta]

It just happens sometimes that blocks come in slower rates. Can have various reasons, does not seem attack related.

[komatom]

@dEBRUYNE-1 I have also compiled .9 version and had it run without the ban list, we will see if it will be ok for a day..

[voidzero]

Alright, hosts being blocked properly, no crashes, if it were up to me I'd say "push it!"

[komatom]

same here no crashes and hosts are blocked:

2021-01-04 22:43:19.061 I Host 142.44.144.199 blocked. 2021-01-05 01:08:28.374 I Host 51.91.33.151 blocked. 2021-01-05 01:11:12.678 I Host 51.77.192.92 blocked. 2021-01-05 02:10:51.069 I Host 54.36.150.238 blocked. 2021-01-05 02:47:45.731 I Host 54.38.217.119 blocked. 2021-01-05 02:58:04.829 I Host 51.178.251.7 blocked. 2021-01-05 03:03:38.514 I ----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 2267555 2021-01-05 03:03:38.514 I id: <050d9bc8248ac34e18445846899634d6f8380debf8cad59d4e63b731477214a1> 2021-01-05 03:03:38.514 I PoW: <3899e894f1f71a365d1995123857c5c6d560d71cc08f08c275c8600300000000> 2021-01-05 03:03:38.514 I difficulty: 206074715525 2021-01-05 03:46:57.432 I Host 51.77.227.128 blocked. 2021-01-05 04:00:24.403 I Host 188.165.17.204 blocked. 2021-01-05 05:12:59.958 I Host 145.239.118.5 blocked. 2021-01-05 05:23:37.785 I Host 146.59.211.239 blocked. 2021-01-05 05:50:06.436 I Host 87.98.224.123 blocked. 2021-01-05 06:10:33.432 I Host 178.32.215.155 blocked. 2021-01-05 06:35:47.146 I Host 217.182.9.71 blocked. 2021-01-05 06:44:09.102 I Host 51.255.233.156 blocked. 2021-01-05 06:47:38.347 I Host 51.255.126.219 blocked. 2021-01-05 07:21:16.827 I Host 79.137.53.34 blocked. 2021-01-05 07:54:55.539 I Host 5.196.124.241 blocked. 2021-01-05 07:56:42.273 I Host 51.91.8.178 blocked.

[selsta]

We are working on fixing more attack vectors before putting the release out.

[voidzero]

Okay, whatever you decide is best. I just want to make this one point - so far 76 hosts have been banned on my host (my in_peers and out_peers are both 32), and I don't know how many hosts on the network are crashing as a result but these crashes could have a significant impact. So if this critical one is fixed and legitimate nodes won't crash anymore, then perhaps push this one out, then fix the less critical attack vectors. Of course I have no idea which other vectors for attack there are so I hope I don't sound rude or imposing. Leaving this to your judgement.

[voidzero]

I'd say this is fixed!

[moneromooo-monero]

Thanks for testing, closing.