rippled v1.3.1 OOM (out of memory) crash on testnet

[gituser]

Hi!

Been running rippled v1.2.4 on testnet for quite some time without any issues on 10GB VM.

But 2 months or 3 months ago started to see massive OOMs and huge disk reads.

After updating to rippled v1.3.1 the situation is still the same, basically every hour there is an OOM crash.

Here is my rippled.cfg for testnet:

[server]
port_rpc_admin_local
port_peer
port_ws_admin_local
port_rpc_external
port_ws_external
#ssl_key = /etc/ssl/private/server.key
#ssl_cert = /etc/ssl/certs/server.crt

#enable signing
[signing_support]
true

[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer

[port_ws_admin_local]
port = 6006
ip = 127.0.0.1
admin = 127.0.0.1
protocol = ws

[port_rpc_external]
port = 51234
ip = 192.168.70.86
protocol = http
user = ripplerpc_testnet
password = pass
admin = 192.168.70.2
#admin_user = ripplerpc_testnet
#admin_password = adminpass

[port_ws_external]
port = 51233
ip = 192.168.70.86
protocol = http
user = ripplerpc_testnet
password = pass
admin = 192.168.70.2
#admin_user = ripplerpc_testnet
#admin_password = adminpass
protocol = ws

#-------------------------------------------------------------------------------

[node_size]
tiny

[node_db]
type=RocksDB
path=/home/ripple/.ripple/db_testnet/rocksdb
open_files=2000
filter_bits=12
cache_mb=256
file_size_mb=64
file_size_mult=2
online_delete=2000
advisory_delete=0

[database_path]
/home/ripple/.ripple/db_testnet

[debug_logfile]
/home/ripple/.ripple/debug_testnet.log

[sntp_servers]
time.windows.com
time.apple.com
time.nist.gov
pool.ntp.org

[ips]
r.altnet.rippletest.net 51235

[validator_list_sites]
https://vl.altnet.rippletest.net

[validator_list_keys]
ED264807102805220DA0F312E71FC2C69E1552C9C5790F6C25E3729DEB573D5860

[rpc_startup]
{ "command": "log_level", "severity": "warning" }

[ssl_verify]
1

The crashlog:

[9517417.712334] Out of memory in UB 785: OOM killed process 1136 (rippled: main) score 0 vm:12445792kB, rss:8542132kB, swap:2930984kB
[9523304.685633] Out of memory in UB 785: OOM killed process 1181 (rippled: main) score 0 vm:12600648kB, rss:10099244kB, swap:1374516kB
[9539038.259031] Out of memory in UB 785: OOM killed process 1214 (rippled: main) score 0 vm:12619392kB, rss:9740884kB, swap:1731872kB
[9547999.656154] Out of memory in UB 785: OOM killed process 1292 (rippled: main) score 0 vm:12631060kB, rss:8109076kB, swap:3363836kB
[9548834.788951] Out of memory in UB 785: OOM killed process 1374 (rippled: main) score 0 vm:12490436kB, rss:9968068kB, swap:1505032kB
[9554955.520002] Out of memory in UB 785: OOM killed process 1400 (rippled: main) score 0 vm:12546684kB, rss:10385836kB, swap:1079028kB
[9568000.591282] Out of memory in UB 785: OOM killed process 1433 (rippled: main) score 0 vm:12535856kB, rss:9847404kB, swap:1617708kB
[9586333.723690] Out of memory in UB 785: OOM killed process 1471 (rippled: main) score 0 vm:12518188kB, rss:8828636kB, swap:2628092kB
[9594660.762773] Out of memory in UB 785: OOM killed process 1551 (rippled: main) score 0 vm:12523148kB, rss:9851188kB, swap:1605008kB
[9600658.329453] Out of memory in UB 785: OOM killed process 1581 (rippled: main) score 0 vm:12575460kB, rss:9754816kB, swap:1702112kB
[9606191.889044] Out of memory in UB 785: OOM killed process 1614 (rippled: main) score 0 vm:12491040kB, rss:10405776kB, swap:1050684kB
[9618393.215965] Out of memory in UB 785: OOM killed process 1644 (rippled: main) score 0 vm:12501732kB, rss:8257096kB, swap:3199896kB
[9642889.916292] Out of memory in UB 785: OOM killed process 1682 (rippled: main) score 0 vm:12531844kB, rss:8513004kB, swap:2942972kB
[9649820.280216] Out of memory in UB 785: OOM killed process 1817 (rippled: main) score 0 vm:12605872kB, rss:9383708kB, swap:2072036kB
[9655551.374678] Out of memory in UB 785: OOM killed process 1851 (rippled: main) score 0 vm:12553380kB, rss:8081320kB, swap:3375464kB

[MarkusTeufelberger]

Use NuDB instead of RocksDB and make sure you have a fast SSD.

[gituser]

@MarkusTeufelberger how is this relevant?

There is an apparent memory issue (maybe a leak somewhere) in rippled and it has nothing to do with the underlying storage.

Both RocksDB and NuDB should work as they both worked before.

[mellery451]

see https://github.com/facebook/rocksdb/issues/4112 for example -- so It could be related to RocksDB. NuDB is preferred for performance and memory usage anyhow -- trying it could at least help determine if RocksDB is a problem in this case.

[gituser]

@mellery451 so you're saying right now RocksDB in rippled is not usable at all?
This should be reflected somewhere (e.g. documentation).

NuDB won't work on the server we use for testnet and as I pointed earlier some time ago (12th July this year to be exact) the exact same configuration was working just fine on the same amount of memory and same configuration.

Regarding, your mentioned issue - why there is no such problem in bitcoin or any other cryptocurrency using RocksDB and only in rippled? If it was such a serious issue it would affect all software using RocksDB.

And also as you can see from my configuration I don't keep whole database, I use online_delete=2000 which means rippled deletes old ledgers automatically and my db folder always about 3.4GB-3.6GB, so why does rippled consume more than 10G of memory?

[yxxyun]

https://www.xrpchat.com/topic/32268-skein-systems-validator-public-key/?do=findComment&comment=741620

The server does quite a bit of work to maintain some ledger indexes used to find paths for cross-currency payments , if you don't use path finding, you can disable it.

[path_search_max]
0

[gituser]

@yxxyun thanks for the tip! I'll try this config stanza and get back if it helps.

[gituser]

@yxxyun unfortunately that didn't help either :(

[miguelportilla]

Are there any other processes running on the VM?

Reiterating on what @MarkusTeufelberger and @mellery451 suggested, I also recommend trying NuDB as it requires considerably less memory than RocksDB.

[gituser]

@miguelportilla no, there is only rippled running and consuming all available memory at some point and being restarted by OOM.

[MarkusTeufelberger]

It could very well be that your node doesn't manage to write its files fast enough and fills caches until it runs out of memory. NuDB caches much less (and also does not rewrite/compact data into different levels), so that's one way you could try to find out what happens.

Other cryptocurrency clients also have their issues (I ran into some parity based servers that are hilariously bad at dealing with RocksDB), many don't have as much state or write/read load as XRPL so they can cache more or can take their time to do a compaction since the next block is still minutes away.

I'm not saying there isn't an issue within rippled potentially, it's just that a resource starved server (too little network IO, disk IO, CPU or RAM) tends to spiral out of control a bit in my experience. I had other issues with a spotty and slow internet connection, which lead to rippled constantly fully re-syncing which led to even more traffic. It might very well be the case that on your node it tries to write to the database but can't because it is compacting or still slowly writing so it will cache and cache and cache...

Running rippled without a fast SSD is anyways a futile task if you want any kind of historic data. Why do you need exactly 2000 ledgers anyways? Maybe you can increase/decrease that number to see if it helps. Also you could start logging below the "warning" level and also monitoring the machine so you have a bit more insight into what's happening.

[miguelportilla]

RocksDB will consume memory unbound if the storage subsystem cannot keep up with the write load. The load on the OS your VM runs on may be affecting your storage performance, for instance if several VMs are running. Most providers mention peak IOPS but fail to guarantee sustained metrics. Your VM must always have a sustained minimum of 10k IOPS or RocksDB will consume more memory. With NuDB, the OOMs may disappear but if your storage performance is sporadically dropping, you might desync from the network.

[gituser]

@MarkusTeufelberger

It could very well be that your node doesn't manage to write its files fast enough and fills caches until it runs out of memory. NuDB caches much less (and also does not rewrite/compact data into different levels), so that's one way you could try to find out what happens.

No, it's not. I've tested rippled on ssd server and getting same results - oom crash and HUGE memory consumption. I don't want NuDB I want RocksDB. Does it work at all?

Other cryptocurrency clients also have their issues (I ran into some parity based servers that are hilariously bad at dealing with RocksDB), many don't have as much state or write/read load as XRPL so they can cache more or can take their time to do a compaction since the next block is still minutes away.

parity is the next evil after rippled

I'm not saying there isn't an issue within rippled potentially, it's just that a resource starved server (too little network IO, disk IO, CPU or RAM) tends to spiral out of control a bit in my experience. I had other issues with a spotty and slow internet connection, which lead to rippled constantly fully re-syncing which led to even more traffic. It might very well be the case that on your node it tries to write to the database but can't because it is compacting or still slowly writing so it will cache and cache and cache...

See previous comment the same behavior is on the fast SSD backed storage.

Running rippled without a fast SSD is anyways a futile task if you want any kind of historic data. >Why do you need exactly 2000 ledgers anyways? Maybe you can increase/decrease that number to >see if it helps. Also you could start logging below the "warning" level and also monitoring the >machine so you have a bit more insight into what's happening.

I don't want full history, usually (before) issues started occuring if rippled's DB grew to 10G or more so I stick to 2000 ledgers (default configuration) for my tasks of receiving and sending transactions without history it has been working fine for last 2 years.

I'll try the logging later, but the rippled log is so messed up with huge amount of logs I'm not sure what to look for.

[gituser]

@miguelportilla

RocksDB will consume memory unbound if the storage subsystem cannot keep up with the write load. The load on the OS your VM runs on may be affecting your storage performance, for instance if several VMs are running. Most providers mention peak IOPS but fail to guarantee sustained metrics. Your VM must always have a sustained minimum of 10k IOPS or RocksDB will consume more memory. With NuDB, the OOMs may disappear but if your storage performance is sporadically dropping, you might desync from the network.

I understand that RocksDB should consume some memory, but as I said before it started to occur recently, so it must be something with the rippled software as before it was working just fine on the same configuration.

I also tried running rippled on fast storage (SSD) and the result is the same - rippled consumes a lot of memory even on testnet. In the guides there is outline that rippled should run just fine on 8GB of memory but in reality it can't.

[mDuo13]

I understand that RocksDB should consume some memory, but as I said before it started to occur recently, so it must be something with the rippled software as before it was working just fine on the same configuration.

This is not necessarily true. The size of the ledger's current state and the volume of transactions in the network both fluctuate with usage and market dynamics (generally trending bigger) regardless of any code changes. Both of those things contribute to RAM usage. It's entirely possible that the amount of working memory that's required to run your server simply increased because there's more data to track than there was before.

The Testnet hasn't been reset in a while. Given that the testnet faucet makes new accounts that are permanent (until the ledger is reset to scratch) very easily, and, for example, the transaction sender requests several new accounts every time someone loads the page, it seems possible that the main culprit is the number of accounts in the ledger.

The documentation states that 8GB is the _minimum_ memory requirement, but the _recommended_ number is 32 GB. To operate with less than 32 GB of RAM, you may want to change the server's node_size parameter based on the Capacity Planning recommendations. It's possible these settings need to be updated, but there are several factors involved in whether a server runs smoothly or not so it's best to do with several data points rather than just one instance. (And no, I don't have a farm of servers with different RAM requirements that I'm constantly watching to see when the minimum requirements need to be higher.)

[mDuo13]

Now that the Testnet has been reset, is this still happening? If not, maybe my hypothesis was correct. If it is still happening, then it bears further investigation into possible code causes.

[gituser]

@mDuo13 I had to switch the db engine to NuDB otherwise rippled was not usable.

I believe the issue is still there.

[luleigreat]

I have a similiar problem when using rippled:

There is a project using rippled0.80.2, and it have ran abount a year,with 180G data(sqlite+rocksdb)，
Environment: aws c5d.2xlarge 8cpu 16G NVMe SSD
and the sles count are below:

 {
         "account" : 2524069,
         "amendments" : 1,
         "directory" : 2518953,
         "escrow" : 0,
         "fee" : 0,
         "hashes" : 136,
         "offer" : 1,
         "payment_channel" : 0,
         "signer_list" : 0,
         "state" : 4738656,
         "table" : 0,
         "ticket" : 0
 }

I am puzzled about the memory in rippled process. The key point is when the treenodecache size returned from 'get_counts' get down from above 12000000 to below 2000000, and other params didn't grow, but the memory didn't get down at all(Always about 10G).
So I suspect there is a memory leak in rippled.

The json returned by get_counts the first time:

{
   "id" : 1,
   "result" : {
      "AL_hit_rate" : 15.70247936248779,
      "HashRouterEntry" : 1479,
      "InboundLedger" : 99,
      "Ledger" : 109,
      "NodeObject" : 17812,
      "RCLCxPeerPos::Data" : 40,
      "SLE_hit_rate" : 0.3076923076923077,
      "STArray" : 128,
      "STLedgerEntry" : 29,
      "STObject" : 1332,
      "STTx" : 244,
      "STValidation" : 392,
      "Transaction" : 173,
      "dbKBLedger" : 4204,
      "dbKBTotal" : 9604,
      "dbKBTransaction" : 4204,
      "fullbelow_size" : 2559678,
      "historical_perminute" : 22,
      "ledger_hit_rate" : 11.01928329467773,
      "node_hit_rate" : 28.22913932800293,
      "node_read_bytes" : 4163320653,
      "node_reads_hit" : 14754404,
      "node_reads_total" : 15643782,
      "node_writes" : 741054,
      "node_written_bytes" : 289595474,
      "status" : "success",
      "treenode_cache_size" : 12238,
      "treenode_track_size" : 12009843,
      "uptime" : "5 minutes, 5 seconds",
      "write_load" : 0
   }
}

The second:

{
   "id" : 1,
   "result" : {
      "AL_hit_rate" : 30.69767570495605,
      "HashRouterEntry" : 1509,
      "Ledger" : 14,
      "NodeObject" : 8539,
      "RCLCxPeerPos::Data" : 40,
      "SLE_hit_rate" : 0.5398058252427185,
      "STArray" : 102,
      "STLedgerEntry" : 141,
      "STObject" : 1301,
      "STTx" : 297,
      "STValidation" : 584,
      "Transaction" : 274,
      "dbKBLedger" : 4204,
      "dbKBTotal" : 9604,
      "dbKBTransaction" : 4204,
      "fullbelow_size" : 1374097,
      "historical_perminute" : 0,
      "ledger_hit_rate" : 15.36214923858643,
      "node_hit_rate" : 28.27434730529785,
      "node_read_bytes" : 4203252193,
      "node_reads_hit" : 14893786,
      "node_reads_total" : 15783684,
      "node_writes" : 743631,
      "node_written_bytes" : 291259310,
      "status" : "success",
      "treenode_cache_size" : 163,
      "treenode_track_size" : 1625090,
      "uptime" : "7 minutes, 38 seconds",
      "write_load" : 0
   }
}

[luleigreat]

Hello,every one ,I know the reason for my question, may be the same reason for yours @gituser .

The g_libc will hold the memory instead of free to OS after the destructor is called , you can use malloc_trim(0) to free the memory g_libc not freed,see :https://stackoverflow.com/questions/15529643/what-does-malloc-trim0-really-mean

[carlhua]

Closing.