Grin: MDB_READERS_FULL: Environment maxreaders limit reached

Created on 24 Oct 2018 · 16Comments · Source: mimblewimble/grin

Wallet crashes with the following error message after sending grins every two minutes for a few hours. In this case it had been running like 18 hours I think.

2018-10-24T08:25:32.961287689+00:00 WARN grin - Using wallet configuration file at /root/.grin/grin-wallet.toml
2018-10-24T08:25:33.058494873+00:00 ERROR grin_util::logger -
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: LmdbErr(Error::Code(-30790, 'MDB_READERS_FULL: Environment maxreaders limit reached'))': libcore/result.rs:945stack backtrace:
   0:     0x559baeb36c0c - backtrace::backtrace::trace::h0d480c688c6e5983
   1:     0x559baeb369c2 - <backtrace::capture::Backtrace as core::default::Default>::default::hbf23cd498c62c039
   2:     0x559baeb36a38 - backtrace::capture::Backtrace::new::hb00f34110e8fb8d2
   3:     0x559baeac5dde - grin_util::logger::send_panic_to_log::{{closure}}::h40696617360cc809
   4:     0x559baec43f63 - std::panicking::rust_panic_with_hook::h542dacac3f3a20d8
                        at libstd/panicking.rs:479
   5:     0x559baec43ac9 - std::panicking::continue_panic_fmt::hd989cd729c0c5a4b
                        at libstd/panicking.rs:390
   6:     0x559baec439c5 - rust_begin_unwind
                        at libstd/panicking.rs:325
   7:     0x559baec9ecdb - core::panicking::panic_fmt::h26b7cf0675909ac6
                        at libcore/panicking.rs:77
   8:     0x559bae546e58 - core::result::unwrap_failed::h9eb2ee0cad4388b5
   9:     0x559bae55768f - <grin_wallet::lmdb_wallet::LMDBBackend<C, K> as grin_wallet::libwallet::types::WalletBackend<C, K>>::set_parent_key_id_by_name::h51f2edc7da53061c
  10:     0x559bae54b83b - grin::cmd::wallet::instantiate_wallet::h2c17a74e514ab58a
  11:     0x559bae54c2fb - grin::cmd::wallet::wallet_command::h9976daff0258656b
  12:     0x559bae568139 - grin::main::h4b201d9bfe7fe7c0
  13:     0x559bae4c4862 - std::rt::lang_start::{{closure}}::h26276e02d8776873
  14:     0x559baec43962 - std::rt::lang_start_internal::{{closure}}::h2d27f966df62f613
                        at libstd/rt.rs:59
                         - std::panicking::try::do_call::h0208f3354fcc3e87
                        at libstd/panicking.rs:310
  15:     0x559baec68269 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:105
  16:     0x559baec468c5 - std::panicking::try::h8cfae114cfc320aa
                        at libstd/panicking.rs:289
                         - std::panic::catch_unwind::h9e48b3e6dc9f9394
                        at libstd/panic.rs:392
                         - std::rt::lang_start_internal::h56481fc9d09c36f4
                        at libstd/rt.rs:58
  17:     0x559bae5693c3 - main
  18:     0x7fc6d619682f - __libc_start_main
  19:     0x559bae459fd8 - _start
  20:                0x0 - <unknown>


Thread 'main' panicked with message:
"called `Result::unwrap()` on an `Err` value: LmdbErr(Error::Code(-30790, 'MDB_READERS_FULL: Environment maxreaders limit reached'))"
See grin.log for further details.

Don't know if it could relate to #1820 in some way or not.

EDIT: It looks like the problem has already been reported here #1267, but no fix suggested.

Running my nodes on Linux 16.04 and 18.04, latest version of Rust.

bug

Source

mably

All 16 comments

FYI a grin wallet restore fixed the problem.

May be a simple grin wallet listen restart is enough, I'll try next time.

mably on 24 Oct 2018

Looks like something isn't being closed properly somewhere and/or some limit is being hit, but we'll probably have to find a quicker way of reproducing it. I'd be surprised if you need to restore your wallet to move it, this looks like a runtime issue not a data corruption issue. Could you verify restarting is enough if it happens again?

yeastplume on 24 Oct 2018

Yep I will try to just restart it next time. Will increase my send frequency for now.

mably on 24 Oct 2018

Can we use the -c 0 (zero confirmation) command line parameter when sending grin with current wallet state?
When used, transactions never show as confirmed when displayed using grin wallet txs command. Though the funds seems to be received on the other end, not completely sure though. At least the txs seem to be properly built and chained.

mably on 24 Oct 2018

Yes, -c 0 should work.

ignopeverell on 24 Oct 2018

For some reason, txs created with -c 0 seem to never confirm.

mably on 25 Oct 2018

Was hit again tonight. I can confirm that just restarting the grin wallet listen process fixes the problem.

FYI I have a shell script that tries to send 1.0 grin every 60 seconds.

mably on 1 Nov 2018

Looks like it happens when I stop the server while keeping the wallet listener running.

mably on 1 Nov 2018

Could reproduce it again after stopping the server for a few minutes while doing a source update and a cargo build.

mably on 1 Nov 2018

Could you elaborate? How you keep your wallet running while updating grin binary? What are exact steps, I was trying to reproduce it without luck.

hashmap on 1 Nov 2018

I have grin server and wallet listen processes running while a script is spending grins every few seconds.

I then stop my server process for a few minutes (for an update for example) while the script is still trying to send some grins and the wallet listen process is still running.

At first, wallet commands respond with: WARNING: Wallet failed to verify data. The above is from local cache and possibly invalid! (is your grin server offline or broken?) or Request error: Cannot make request: an error occurred trying to connect: Connection refused (os error 111) }

But after a few minutes (and few wallet send ~20-30) I get: Thread 'main' panicked with message: "called Result::unwrap() on an Err value: LmdbErr(Error::Code(-30790, 'MDB_READERS_FULL: Environment maxreaders limit reached'))"

So, it looks like the missing connection is poorly handled on the wallet side when trying to send grins.

mably on 1 Nov 2018

In fact, it looks like any error, like an 'not enough funds' error, can cause the problem after a few failing sends.

mably on 1 Nov 2018

@mably thanks! I reproduced it by constantly sending grins to myself, took 60-90 operations.

hashmap on 1 Nov 2018

👍1

I added logging to Store::open, transactions (readers) being leaked, after multiple wallet send I see:

 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 0
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 1
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 2
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 2
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 2
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 2
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 3
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 3
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 4
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 4
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 5
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 5
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 6
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 6
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 7
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 7
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 8
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 8
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 9
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 9
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 10
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 10
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 11
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 11
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 12
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 12
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 13
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 13
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 14
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 14
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 15
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 15
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 16
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 16
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 17
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 17
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 18
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 18
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 19
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 19
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 20
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 20
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 21
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 21
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 22
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 22
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 23
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 23
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 24
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 24
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 25
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 25
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 26
 Env "/home/arch/.grin/wallet_data/db/lmdb", DB db :  maxreaders 126, numreaders 26

the sequence is not random, but reproducible, including anomaly with 2.

hashmap on 2 Nov 2018

👍1

I found the reason, a proper fix requires a lot of typing, should be ready in a day or so

hashmap on 2 Nov 2018

👍1

Great news, thanks!

mably on 2 Nov 2018

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Investigate Curve25519-dalek for faster bulletproofs

lehnberg · 5Comments

Error while compiling GRIN wallet. error: Edition 2018 is unstable and only available for nightly builds of rustc.

black-phillip616 · 3Comments

Is there any good solution for slimming the block header?

garyyu · 6Comments

Peer banning should be temporary

ignopeverell · 3Comments

Use a working directory under the user home

ignopeverell · 4Comments