Rippled: Rippled crash ripple::Keylet::check (Version: 1.6.0)

[gituser]

Issue Description

rippled v1.6.0 for some reason crashed on the mainnet. I do not see any OOM related messages in the dmesg.

The VM where rippled is run has 32 GB of RAM. It's running on NVME drives.

Configuration

rippled.cfg:

[server]
port_rpc_admin_local
port_peer
#port_ws_admin_local
port_rpc_external
port_ws_external
port_ws_public_admin
#ssl_key = /etc/ssl/private/server.key
#ssl_cert = /etc/ssl/certs/server.crt

[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer

[port_rpc_external]
port = 51234
ip = 1.2.3.4
protocol = http
user = ripplerpc
password = xxx
admin = 3.4.5.6
#admin_user = ripplerpc
#admin_password = xxx

[port_ws_external]
port = 51233
ip = 1.2.3.4
protocol = http
user = ripplerpc
password = xxx
admin = 3.4.5.6
#admin_user = ripplerpc
#admin_password = xxx
protocol = ws

[port_ws_public_admin]
port = 6006
ip = 1.2.3.4
admin = 3.4.5.6
protocol = ws

#[port_ws_public]
#port = 5005
#ip = 127.0.0.1
#protocol = wss

#-------------------------------------------------------------------------------

[node_size]
#
#   Tunes the servers based on the expected load and available memory. Legal
#   sizes are "tiny", "small", "medium", "large", and "huge". We recommend
#   you start at the default and raise the setting if you have extra memory.
#   The default is "tiny".
huge

[ledger_history]
256

# This is primary persistent datastore for rippled.  This includes transaction
# metadata, account states, and ledger headers.  Helpful information can be
# found here: https://ripple.com/wiki/NodeBackEnd
# delete old ledgers while maintaining at least 2000. Do not require an
# external administrative command to initiate deletion.
[node_db]
type=RocksDB
path=/home/ripple/.ripple/db/rocksdb
open_files=512
filter_bits=12
cache_mb=1024
file_size_mb=64
file_size_mult=2
online_delete=2000
advisory_delete=0

#type=NuDB
#path=/home/ripple/.ripple/db/nudb
#online_delete=2000
#advisory_delete=0

[database_path]
/home/ripple/.ripple/db

#[shard_db]
#type=RocksDB
#path=/home/ripple/.ripple/db/shards/rocksdb
#max_size_gb=5

# This needs to be an absolute directory reference, not a relative one.
# Modify this value as required.
[debug_logfile]
/home/ripple/.ripple/debug.log

[sntp_servers]
time.windows.com
time.apple.com
time.nist.gov
pool.ntp.org

# Where to find some other servers speaking the Ripple protocol.
#
# Where to find some other servers speaking the Ripple protocol.
#
[ips]
r.ripple.com 51235

[validator_list_sites]
https://vl.ripple.com

[validator_list_keys]
ED2677ABFFD1B33AC6FBC3062B71F1E8397C1505E1C42C64D11AD1B28FF73F4734

# Turn down default logging to save disk space in the long run.
# Valid values here are trace, debug, info, warning, error, and fatal
[rpc_startup]
{ "command": "log_level", "severity": "warning" }

# If ssl_verify is 1, certificates will be validated.
# To allow the use of self-signed certificates for development or internal use,
# set to ssl_verify to 0.
[ssl_verify]
1

Expected Result

Actual Result

Environment

$ ./rippled --version
rippled version 1.6.0

Ubuntu 18.04.3 LTS

Supporting Files

Sep 12 12:57:01 ripple-ac rippled[12059]: 2020-Sep-12 09:57:01.770515608 UTC SHAMapStore:WRN rotating  validatedSeq 58134184 lastRotated 58132184 deleteInterval 2000 canDelete_ 4294967295 state full age 0s
Sep 12 12:57:49 ripple-ac rippled[12059]: 2020-Sep-12 09:57:49.484066817 UTC LoadMonitor:WRN Job: acceptLedger run: 1005ms wait: 1ms
Sep 12 12:57:49 ripple-ac rippled[12059]: 2020-Sep-12 09:57:49.761838852 UTC LoadMonitor:WRN Job: sweep run: 1416ms wait: 0ms
Sep 12 12:58:28 ripple-ac rippled[12059]: 2020-Sep-12 09:58:28.396540286 UTC SHAMapStore:WRN finished rotation 58134184
Sep 12 12:58:33 ripple-ac rippled[12059]: 2020-Sep-12 09:58:33.457665417 UTC LoadMonitor:WRN Job: processLedgerData run: 2177ms wait: 0ms
Sep 12 12:59:51 ripple-ac rippled[12059]: 2020-Sep-12 09:59:51.186981022 UTC LoadMonitor:WRN Job: sweep run: 1425ms wait: 0ms
Sep 12 13:01:56 ripple-ac rippled[12059]: 2020-Sep-12 10:01:56.547085324 UTC LoadMonitor:WRN Job: sweep run: 5360ms wait: 0ms
Sep 12 13:39:33 ripple-ac rippled[12059]: rippled: /home/build/ripple/source/src/ripple/protocol/impl/Keylet.cpp:37: bool ripple::Keylet::check(const SLE&) const: Assertion `sle.getType() == type' failed.
Sep 12 13:39:33 ripple-ac systemd[1]: rippled.service: Main process exited, code=killed, status=6/ABRT
Sep 12 13:39:33 ripple-ac systemd[1]: rippled.service: Failed with result 'signal'.

[gituser]

I've got a crash on 3 separate rippled nodes simultaneously.

All are running v1.6.0.

Is it a bug affecting all v1.6.0 versions?

Ping @mellery451 @ximinez @nbougalis @seelabs @intelliot @mDuo13

[gituser]

It seems there is a fix in develop branch If I'm right?

https://github.com/ripple/rippled/commit/ab9f3fa42aed39477a8a9d27c46830ada9ab890a

Would it be a good idea to release v1.6.1 with this patch at least to fix this crash issue?

[seelabs]

@gituser Thanks for reporting this issue!

We take crashes very seriously. In this case, the problem is an assert is triggering. asserts are normally disabled in production code. They turned on only in debug builds or by explicitly turning them on in release builds. Since the vast majority of servers are safe from this error, I think it's OK to wait for the fix to land in the next release rather than a hot fix.

Thanks again for raising the issue!

Edit: I should also point out our bug bounty program: https://ripple.com/bug-bounty/ It's probably better to report potentially serious bugs there than in pubic (and we pay bounties!).

[gituser]

@seelabs so I guess it's because I have in my compilation build script this:
cmake -DCMAKE_BUILD_TYPE=Release -Dstatic=True -Dassert=On ../..

Guess I need to remove -Dassert=On and recompile rippled for now.

[seelabs]

@gituser Yes, removing that -Dassert=On will fix things in the meantime.