Monero: MDB_NOTFOUND: No matching key/data pair found

Created on 20 Jan 2017 · 18Comments · Source: monero-project/monero

Recently. i've been getting frequenty those errors.

2017-01-20 02:27:13.718     7fcf21340b80    WARN    blockchain.db.lmdb  src/blockchain_db/lmdb/db_lmdb.cpp:71   Error attempting to retrieve a hard fork version at height 1227592 from the db: MDB_NOTFOUND: No matching key/data pair found
terminate called after throwing an instance of 'cryptonote::DB_ERROR'
  what():  Error attempting to retrieve a hard fork version at height 1227592 from the db: MDB_NOTFOUND: No matching key/data pair found

The problem is that once this error occurs, daemon will not work anymore. It just crashes with the error shown below, and resycing of the blockchain is required, or recovering it from the backup, if someone has.

2017-01-20 02:29:31.384     7f0bc8290780    WARN    blockchain.db.lmdb  src/blockchain_db/lmdb/db_lmdb.cpp:71   Error attempting to retrieve a hard fork version at height 1227592 from the db: MDB_NOTFOUND: No matching key/data pair found
2017-01-20 02:29:31.387     7f0bc8290780    FATAL   daemon  src/daemon/daemon.cpp:147   Uncaught exception! Error attempting to retrieve a hard fork version at height 1227592 from the db: MDB_NOTFOUND: No matching key/data pair found

Has anyone experienced this issue? What is the cause of it?

Source

moneroexamples

Most helpful comment

To workaround monero run in docker, use -i flag on docker run; or stdin_open: true option for docker-compose

Kukunin on 4 Mar 2017

👍6

All 18 comments

Are you running this on Windows, and did your OS crash ?

moneromooo-monero on 20 Jan 2017

No. I use Linux (Majnaro now). Had this issue on two boxes. In all cases I just restored my blockchain from backup, and it was working for a while, till it crashed again. But after last time, I decided to resync it from scratch, and see how it goes with fresh blockchain.

Do you know what does it mean? Is it corrupted lmdb data files or something else?

moneroexamples on 21 Jan 2017

It seems corrupt. You might be able to restore by popping N blocks (blockchain_import --pop-blocks N) and restarting.

moneromooo-monero on 21 Jan 2017

Its same on freshly sync block from scrached. Worked for a one day, and then this error:

2017-01-22 08:15:53.960 [P2P0]  WARN    blockchain.db.lmdb  src/blockchain_db/lmdb/db_lmdb.cpp:71Failed to add tx blob to db transaction: MDB_KEYEXIST: Key/data pair already exists
2017-01-22 08:15:53.964 [P2P0]  ERROR   blockchain  src/cryptonote_core/blockchain.cpp:3309 Error adding block with hash: <d922236c6ac32625df8d245d85953ac575b4312162342abf49c8d4eb2e28860b> to blockchain, what = Failed to add tx blob to db transaction: MDB_KEYEXIST: Key/data pair already exists
2017-01-22 08:16:23.125 [RPC0]  ERROR   blockchain  src/cryptonote_core/blockchain.cpp:2029 internal error, transaction from block not found
2017-01-22 08:16:35.800 [P2P8]  WARN    blockchain.db.lmdb  src/blockchain_db/lmdb/db_lmdb.cpp:71Failed to add tx blob to db transaction: MDB_KEYEXIST: Key/data pair already exists
2017-01-22 08:16:35.803 [P2P8]  ERROR   blockchain  src/cryptonote_core/blockchain.cpp:3309 Error adding block with hash: <b75571fa17787ad0c0fb051f60e2db3fee8fe04cc6d78b25ab2955ab1375aa10> to blockchain, what = Failed to add tx blob to db transaction: MDB_KEYEXIST: Key/data pair already exists
2017-01-22 08:17:53.208 [RPC1]  ERROR   blockchain  src/cryptonote_core/blockchain.cpp:2029 internal error, transaction from block not found

And daemon wont work anymore at all like before.

Btw these errors started occurring maybe 1 week ago. So my guess is that some changes in marged pull requests at that time cause this.

moneroexamples on 22 Jan 2017

Do you have https://github.com/monero-project/monero/pull/1584 ?

moneromooo-monero on 22 Jan 2017

What git revision is the source you built from?

hyc on 22 Jan 2017

@moneromooo-monero

Yes I have #1584.

@hyc
I always run the latest monero development version.

moneroexamples on 22 Jan 2017

I cloned the git and built on 11jan2017 (no issues with that build) but did the same thing on the latest version of the github repo(new VM) but am getting these exact same errors MDB_NOTFOUND: No matching key/data pair found ... the timing and nature of these errors cannot be a coincidence: pretty sure something in the repo has fessed up around mid-Jan.

kumrzz on 31 Jan 2017

@moneroexamples That is not an answer. The "latest" version keeps changing. I asked you for a specific git revision ID.

hyc on 31 Jan 2017

It happened again. The error occured and lmdb seems to be corrupted as daemon wont start. Need to resync from scrach or get blockchain from backup.

Monero version: v0.10.1.0-cb54eeaa

2017-02-14 16:49:56.650     7ff41283bd80    WARN    blockchain.db.lmdb  src/blockchain_db/lmdb/db_lmdb.cpp:71   Error attempting to retrieve a hard fork version at height 1244280 from the db: MDB_NOTFOUND: No matching key/data pair found
2017-02-14 16:49:56.651     7ff41283bd80    FATAL   daemon  src/daemon/daemon.cpp:149   Uncaught exception! Error attempting to retrieve a hard fork version at height 1244280 from the db: MDB_NOTFOUND: No matching key/data pair found

moneroexamples on 14 Feb 2017

I just stumbled over the same error getting monerod running in a Docker container.

I'm trying to run the following docker-compose.yml:

version: '2'
services:
  monerod:
    build: ./monero
    entrypoint: ['monerod', '--testnet']

in this directory:
https://github.com/amiuhle/monero-docker/tree/fd1347573c8fd6a71f4c0c8b4f5d5e5227a82917

monerod_1  | 2017-02-28 00:31:07.265        7f7cd3265740        INFO    global  contrib/epee/src/mlog.cpp:145   New log categories: *:WARNING,net:FATAL,net.p2p:FATAL,global:INFO,verify:FATAL,stacktrace:INFO
monerod_1  | 2017-02-28 00:31:07.265        7f7cd3265740        INFO    global  src/daemon/main.cpp:282 Monero 'Wolfram Warptangent' (v0.10.2.1-release)
monerod_1  | 2017-02-28 00:31:07.266        7f7cd3265740        INFO    global  src/daemon/protocol.h:55        Initializing cryptonote protocol...
monerod_1  | 2017-02-28 00:31:07.266        7f7cd3265740        INFO    global  src/daemon/protocol.h:60        Cryptonote protocol initialized OK
monerod_1  | 2017-02-28 00:31:07.266        7f7cd3265740        INFO    global  src/daemon/p2p.h:63     Initializing p2p server...
monerod_1  | 2017-02-28 00:31:11.272        7f7cd3265740        INFO    global  src/daemon/p2p.h:68     P2p server initialized OK
monerod_1  | 2017-02-28 00:31:11.272        7f7cd3265740        INFO    global  src/daemon/rpc.h:58     Initializing core rpc server...
monerod_1  | 2017-02-28 00:31:11.272        7f7cd3265740        INFO    global  contrib/epee/include/net/http_server_impl_base.h:70     Binding on 127.0.0.1:28081
monerod_1  | 2017-02-28 00:31:11.273        7f7cd3265740        INFO    global  src/daemon/rpc.h:63     Core rpc server initialized OK on port: 28081
monerod_1  | 2017-02-28 00:31:11.273        7f7cd3265740        INFO    global  src/daemon/core.h:73    Initializing core...
monerod_1  | 2017-02-28 00:31:11.274        7f7cd3265740        INFO    global  src/cryptonote_core/cryptonote_core.cpp:316     Loading blockchain from folder /root/.bitmonero/testnet/lmdb ...
monerod_1  | 2017-02-28 00:31:11.277        7f7cd3265740        WARN    blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:71   Error attempting to retrieve a hard fork version at height 0 from the db: MDB_NOTFOUND: No matching key/data pair found
monerod_1  | 2017-02-28 00:31:11.410        7f7cd3265740        INFO    global  src/daemon/core.h:78    Core initialized OK
monerod_1  | 2017-02-28 00:31:11.410        7f7cd3265740        INFO    global  src/daemon/rpc.h:68     Starting core rpc server...
monerod_1  | 2017-02-28 00:31:11.410    [SRV_MAIN]      INFO    global  src/daemon/rpc.h:73     Core rpc server started ok
monerod_1  | 2017-02-28 00:31:11.411    [SRV_MAIN]      INFO    global  src/daemon/p2p.h:78     Starting p2p net loop...
monerod_1  | 2017-02-28 00:31:11.411    [SRV_MAIN]      INFO    global  src/daemon/p2p.h:80     p2p net loop stopped
monerod_1  | 2017-02-28 00:31:11.411    [SRV_MAIN]      INFO    global  src/daemon/rpc.h:78     Stopping core rpc server...
monerod_1  | 2017-02-28 00:31:11.411    [SRV_MAIN]      INFO    global  src/daemon/daemon.cpp:145       Node stopped.
monerod_1  | 2017-02-28 00:31:11.411    [SRV_MAIN]      INFO    global  src/daemon/rpc.h:90     Deinitializing rpc server...
monerod_1  | 2017-02-28 00:31:11.411    [SRV_MAIN]      INFO    global  src/daemon/p2p.h:90     Deinitializing p2p...
monerod_1  | 2017-02-28 00:31:12.412    [SRV_MAIN]      INFO    global  src/daemon/core.h:89    Deinitializing core...
monerod_1  | 2017-02-28 00:31:12.418    [SRV_MAIN]      INFO    global  src/daemon/protocol.h:77        Stopping cryptonote protocol...
monerod_1  | 2017-02-28 00:31:12.418    [SRV_MAIN]      INFO    global  src/daemon/protocol.h:81        Cryptonote protocol stopped successfully
monerod_1  | Daemon stopped successfully

I'm still struggling with Docker, so maybe I'm just doing something wrong, but I thought this might pin down the problem.

amiuhle on 28 Feb 2017

That's pretty weird, because if you ever get this error message "Error attempting to retrieve a hard fork version at height 0" the next thing you should see is "The DB has no hard fork info, reparsing from start" and that message isn't here.

hyc on 28 Feb 2017

I'm still struggling with Docker, so maybe I'm just doing something wrong

Yeah, I think that's the case. My problem doesn't seem to be related to this issue, so never mind.

amiuhle on 28 Feb 2017

I obtain the same issue on a fresh install on OpenBSD 6.0 (yes, with some patching)

2017-03-01 20:42:20.607 2017-03-01 20:42:20.608 2017-03-01 20:42:20.608 2017-03-01 20:42:20.608 2017-03-01 20:42:20.608 2017-03-01 20:42:20.609 2017-03-01 20:42:24.266 2017-03-01 20:42:24.268 2017-03-01 20:42:24.268 2017-03-01 20:42:24.268 2017-03-01 20:42:24.268 2017-03-01 20:42:24.269 2017-03-01 20:42:24.279 2017-03-01 20:42:24.337 2017-03-01 20:42:24.341 2017-03-01 20:42:24.342 2017-03-01 20:42:24.343 2017-03-01 20:42:24.344 2017-03-01 20:42:24.346 2017-03-01 20:42:24.351 2017-03-01 20:42:24.352 0x2705652f4e0 INFO global contrib/epee/src/mlog.cpp:145 New log categories: *:WARNING,net:FATAL,net.p2p:FATAL,global:INFO,verify:FATAL,stacktrace:INFO
0x2705652f4e0 ERROR default contrib/epee/src/mlog.cpp:179 Invalid numerical log level: 5
0x2705652f4e0 INFO global src/daemon/main.cpp:282 Monero 'Wolfram Warptangent' (v0.10.2.1-release)
0x2705652f4e0 INFO global src/daemon/protocol.h:55 Initializing cryptonote protocol...
0x2705652f4e0 INFO global src/daemon/protocol.h:60 Cryptonote protocol initialized OK
0x2705652f4e0 INFO global src/daemon/p2p.h:63 Initializing p2p server...
0x2705652f4e0 INFO global src/daemon/p2p.h:68 P2p server initialized OK
0x2705652f4e0 INFO global src/daemon/rpc.h:58 Initializing core rpc server...
0x2705652f4e0 INFO global contrib/epee/include/net/http_server_impl_base.h:70 Binding on 127.0.0.1:18081
0x2705652f4e0 INFO global src/daemon/rpc.h:63 Core rpc server initialized OK on port: 18081
0x2705652f4e0 INFO global src/daemon/core.h:73 Initializing core...
0x2705652f4e0 INFO global src/cryptonote_core/cryptonote_core.cpp:316 Loading blockchain from folder /var/monero/lmdb ...
0x2705652f4e0 WARN blockchain.db.lmdb src/blockchain_db/lmdb/db_lmdb.cpp:71 Error attempting to retrieve a hard fork version at height 0 from the db: MDB_NOTFOUND: No matching key/data pair found
0x2705652f4e0 ERROR blockchain src/cryptonote_core/blockchain.cpp:3360 Error adding block with hash: <418015bb9ae982a1975da7d79277c2705727a56894ba0fb246adaabb1f4632e3> to blockchain, what = Error adding hard fork version to db transaction: MDB_NOTFOUND: No matching key/data pair found
0x2705652f4e0 WARN blockchain.db.lmdb src/blockchain_db/lmdb/db_lmdb.cpp:71 Attempt to get hash from height 0 failed -- hash not in db
0x2705652f4e0 FATAL daemon src/daemon/daemon.cpp:150 Uncaught exception! Attempt to get hash from height 0 failed -- hash not in db
0x2705652f4e0 INFO global src/daemon/rpc.h:90 Deinitializing rpc server...
0x2705652f4e0 INFO global src/daemon/p2p.h:90 Deinitializing p2p...
0x2705652f4e0 INFO global src/daemon/core.h:89 Deinitializing core...
0x2705652f4e0 INFO global src/daemon/protocol.h:77 Stopping cryptonote protocol...
0x2705652f4e0 INFO global src/daemon/protocol.h:81 Cryptonote protocol stopped successfully

spaquin on 2 Mar 2017

To workaround monero run in docker, use -i flag on docker run; or stdin_open: true option for docker-compose

Kukunin on 4 Mar 2017

👍6

I close it, as it seems to be ok now. Havent had this problem for a while now.

moneroexamples on 5 Apr 2017

This is a request to re-open.
Compiled from git using commit fa489a2 and built using make -j 12 on 12 cores and 20GB of RAM, and have received the same issue.

Pardon the image, as I cannot paste from ESXi.
http://imgur.com/a/pZqAp
@moneroexamples @monero-project