Nano-node: Collect all intermittently failing tests

[rkeene]

If you have a test that fails intermittently let's collect them here for systematic investigation

[rkeene]

On Linux/x86_64 from Travis-CI:

[ RUN      ] rpc.wallet_destroy
core_test: /workspace/rai/secure/blockstore.cpp:102: rai::mdb_iterator<T, U>::mdb_iterator(MDB_txn*, MDB_dbi, const MDB_val&, rai::epoch) [with T = rai::uint256_union; U = rai::wallet_value; MDB_txn = MDB_txn; MDB_dbi = unsigned int; MDB_val = MDB_val]: Assertion `status == 0' failed.

FIXED - original error is not reproducible on develop, but there is a tsan race which is fixed with https://github.com/nanocurrency/nano-node/pull/2615

[rkeene]

On Linux/x86_64:

[ RUN      ] node.auto_bootstrap
/home/rkeene/devel/raiblocks/rai/core_test/node.cpp:227: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:

FIXED

[cryptocode]

On Travis (trusty):

[ RUN      ] wallet.create_open_receive
/workspace/rai/qt_test/qt.cpp:400: Failure
Value of: json1.empty ()
  Actual: true
Expected: false

FIXED

[cryptocode]

Failing on master: Fixed by @SergiySW

[ RUN      ] rpc.payment_begin_end
/workspace/rai/core_test/rpc.cpp:1162: Failure
Value of: ec
  Actual: Deadline expired
Expected: 
[  FAILED  ] rpc.payment_begin_end (10080 ms)

Quick when it succeeds, when it fails, it seems stuck forever (all threads are waiting on some condition variable)

[rkeene]

[ RUN      ] node.unlock_search
core_test: /workspace/rai/node/wallet.cpp:1385:
                     void rai::wallets::foreach_representative(
                         const rai::transaction&,
                         const std::function<void(
                             const rai::uint256_union&,
                             const rai::raw_key&
                         )>&
                     ): Assertion `!error' failed.

FIXED by https://github.com/nanocurrency/raiblocks/commit/b1d05d0ce83e6aa2950f8ecfa1a3c266cfcde2a0

[PlasmaPower]

EDIT: Fixed by https://github.com/nanocurrency/raiblocks/pull/1532

On local Linux box with some deadlock detection code that may exacerbate the race:

/var/hdd/programming/cpp/raiblocks/rai/core_test/network.cpp:92: Failure
Value of: peers1.size ()
  Actual: 0
Expected: 1
[  FAILED  ] network.send_node_id_handshake (238 ms)

[PlasmaPower]

on 051df7be9b4a9f755e6a94569da1f430f9982ca0

[ RUN      ] bulk.offline_send
/var/hdd/programming/cpp/raiblocks/rai/core_test/network.cpp:826: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:
[  FAILED  ] bulk.offline_send (20410 ms)

FIXED

[PlasmaPower]

On master 37de74f795e32daa02150da49b5f0140211a8c44

[ RUN      ] rpc.online_reps
/home/lee/programming/cpp/raiblocks2/rai/core_test/rpc.cpp:3662: Failure
Expected: (nullptr) != (receive), actual: 8-byte object <00-00 00-00 00-00 00-00> vs 16-byte object <00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>
[  FAILED  ] rpc.online_reps (238 ms)

EDIT: This test now fails with:

[ RUN      ] rpc.online_reps
nano/rpc_test/rpc.cpp:6121: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:

[cryptocode]

On Linux/x86_64 from Travis-CI:

[ RUN      ] rpc.wallet_destroy
core_test: /workspace/rai/secure/blockstore.cpp:102: rai::mdb_iterator<T, U>::mdb_iterator(MDB_txn*, MDB_dbi, const MDB_val&, rai::epoch) [with T = rai::uint256_union; U = rai::wallet_value; MDB_txn = MDB_txn; MDB_dbi = unsigned int; MDB_val = MDB_val]: Assertion `status == 0' failed.

This one happens because wallet_destroy sets handle = 0 after do_wallet_actions does the wallet->live () check, so a TOCTTOU it seems.

[rkeene]

[ RUN      ] node.fork_publish
/home/rkeene/devel/raiblocks/nano/core_test/node.cpp:643: Failure
Value of: node1.active.roots.size ()
  Actual: 0
Expected: 1
[  FAILED  ] node.fork_publish (668 ms)

and

[ RUN      ] node.fork_multi_flip
/home/rkeene/devel/raiblocks/nano/core_test/node.cpp:788: Failure
Value of: node2.active.roots.size ()
  Actual: 1
Expected: 2
[  FAILED  ] node.fork_multi_flip (1820 ms)

[cryptocode]

[ RUN      ] wallet.select_account
/workspace/rai/qt_test/qt.cpp:96: Failure
Value of: key4
  Actual: 32-byte object <52-E8 58-72 7C-65 C8-2F DE-A5 30-C8 EC-DF BC-B9 BA-14 06-D0 82-64 06-20 5C-03 92-67 06-DB 0F-F1>
Expected: key2
Which is: 32-byte object <2C-29 08-C1 5F-B9 6C-C6 6F-4E 2E-AB A6-3F 5F-C7 E2-5E CE-53 AC-0E 3C-09 01-A4 23-4E 5B-07 4B-AD>
[  FAILED  ] wallet.select_account (55 ms)

FIXED

[cryptocode]

node.unlock_search fails sporadically. When debugging it this sequence happens:

vote_generator::send -> foreach_representative -> wallet.store.fetch

...where fetch returns an error here:

    if (!result)
    {
        nano::public_key compare (nano::pub_key (prv.data));
        std::cout << pub.to_string() << " == " << compare.to_string() << std::endl;
        if (!(pub == compare))
        {
            result = true;  // <<======== oops
        }
    }

FIXED (https://github.com/nanocurrency/nano-node/issues/1121#issuecomment-450866102)

[SergiySW]

@cryptocode was fixed in 17.0, but not in master
https://github.com/nanocurrency/raiblocks/pull/1409/commits/b1d05d0ce83e6aa2950f8ecfa1a3c266cfcde2a0

[SergiySW]

Rare node.vote_republish test failure. node::process_confirmed () can start before rolling back of fork block

[ RUN      ] node.vote_republish
/root/rai_test/nano/core_test/node.cpp:1835: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:
[  FAILED  ] node.vote_republish (5307 ms)

FIXED

[cryptocode]

[ RUN      ] wallet.no_work
Expected equality of these values:
  0
  cached_work
    Which is: 4545208490332381752

Happened once, unable to reproduce

FIXED

[wezrule]

rpc.online_reps fails about 50% of the time for me (on MSVC at least)

[ RUN      ] rpc.online_reps
c:\users\wesley\documents\raiblocks\nano\core_test\rpc.cpp(4246): error: Value of: weight2
 Actual: "340281366920938463463374607431768211455"
Expected: system.nodes[1]->weight (nano::test_genesis_key.pub).convert_to<std::string> ()
Which is: "340282366920938463463374607431768211455"

FIXED #1848

[cryptocode]

[ RUN      ] work.eco_pow
work_pool.cpp:185: Failure
Expected: (future1.get ()) < (future2.get ()), actual: 8-byte object <A4-A5 57-00 00-00 00-00> vs 8-byte object <9E-2C 34-00 00-00 00-00>

FIXED

[wezrule]

@cryptocode This should be fixed in #1882 can you give it a try? I actually couldn't reproduce it on my Windows machine, but I do know it could have an issue (I had a comment in the test, but didn't really explain why).

FIXED

[wezrule]

[ RUN      ] node.broadcast_elected
Assertion (!error) failed c:\users\wesley\documents\raiblocks\nano\node\node.cpp:3114

Failing on this release_assert in nano::active_transactions::add:

auto error (nano::work_validate (*block_a, &difficulty));
release_assert (!error);

[wezrule]

[  RUN         ] conflicts.adjusted_difficulty
conflicts.cpp:247: Failure
Expected equality of these values:
  11
  nodes.active.size ()
    Which is: 10

Fails consistently when running TSAN on Ubuntu/GCC

FIXED (no longer able to reproduce locally, feel free to re-check @wezrule)

[wezrule]

[  RUN         ] network.send_node_id_handshake
nano/core_test/network:247: Failure
Expected equality of these values:
  0
  system.nodes[0]->network.size ()
    Which is: 1

Fails consistently when running TSAN on Ubuntu/GCC

FIXED in https://github.com/nanocurrency/nano-node/pull/2612

[wezrule]

[ RUN      ] receivable_processor.send_with_receive
/workspace/nano/core_test/network.cpp:337: Failure
Expected equality of these values:
  system.nodes[0]->config.receive_minimum.number ()
    Which is: 1000000000000000000000000
  system.nodes[1]->balance (key2.pub)
    Which is: 0

FIXED in #1940

[wezrule]

[ RUN      ] rpc.work_peer_bad
d:\agent\_work\2\s\src\vctools\crt\crtw32\stdcpp\thr\mutex.c(51): mutex destroyed while busy

Stack trace

[wezrule]

[ RUN      ] network.replace_port
c:\users\wesley\documents\raiblocks\nano\core_test\network.cpp(2037): error: Expected equality of these values:
  system.nodes[0]->network.udp_channels.size ()
    Which is: 1
  0
[  FAILED  ] network.replace_port (129 ms)

FIXED in #2630

[wezrule]

[ RUN      ] node.vote_by_hash_republish
/workspace/nano/core_test/node.cpp:2196: Failure
Value of: system.poll ()
  Actual: Deadline expired

[cryptocode]

[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from active_transactions
[ RUN      ] active_transactions.adjusted_difficulty_priority
nano/core_test/active_transactions.cpp:143: Failure
Expected: (i->adjusted_difficulty) < (last_adjusted), actual: 18446724124103881201 vs 18446724124103881201
[  FAILED  ] active_transactions.adjusted_difficulty_priority (3781 ms)
[----------] 1 test from active_transactions (3781 ms total)

FIXED

[cryptocode]

[ RUN      ] ledger.unchecked_epoch_invalid
core_test/ledger.cpp:2580: Failure
Value of: node1.active.empty ()
  Actual: false
Expected: true

FIXED

[cryptocode]

[ RUN      ] active_transactions.prioritize_chains
nano/core_test/active_transactions.cpp:211: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:

FIXED

[cryptocode]

confirmation_height.modified_chain sometimes get stuck (no timeout), presumably it's one of the while (!node->write_database_queue.contains (nano::writer::confirmation_height)) loops @wezrule ?

FIXED (was reproducible, but now survives thousands of iterations, so seems fixed by recent changes)

[guilhermelawless]

Ocasionally node.bootstrap_confirm_frontiers times out waiting for the election to start, in:

while (node1->active.empty ())
{
    ASSERT_NO_ERROR (system0.poll ());
    ASSERT_NO_ERROR (system1.poll ());
}

edit: No longer failing
```

[cryptocode]

[ RUN      ] node.local_votes_cache
node.cpp:2348: Failure
Expected equality of these values:
  current_vote->sequence
    Which is: 3
  2

Happens more frequently if running TSAN, so likely timing related

FIXED

[cryptocode]

There's an intermittent ASAN error with node.vote_by_hash_bundle
Looks like it's a lifetime issue with max_hashes - must maybe wait until all observer calls are done before exiting the test, or extend the lifetime by whatever means.

FIXED by https://github.com/nanocurrency/nano-node/pull/2356

[guilhermelawless]

STATUS: FIXED

[ RUN      ] confirmation_height.pending_observer_callbacks
core_test/confirmation_height.cpp:774: Failure
Expected equality of these values:
  0
  node->stats.count (nano::stat::type::http_callback, nano::stat::detail::http_callback, nano::stat::dir::out)
    Which is: 1

[cryptocode]

[ RUN      ] rpc.work_peer_many
unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
[  FAILED  ] rpc.work_peer_many (1842 ms)
[ RUN      ] rpc.block_count
Assertion failed: nano::thread_role::get () != nano::thread_role::name::io, file D:\a\raiblocks\raiblocks\nano\secure\blockstore.cpp, line 419

on CI. Wesley suggested it could be a nullptr dereference and the second (lmdb write on io thread) is a side effect. Possibly fixed by https://github.com/nanocurrency/nano-node/pull/2474 @guilhermelawless as it mentions fixing a TSAN issue with work_peer_many ?

Fixed

[guilhermelawless]

Hopefully, although I've never seen it actually fail myself, only TSAN race warnings. One due to work not being atomic in the test itself, the others due to not using a strand, which could have some side effects I imagine.

Ran #2474 a few hundred times under TSAN with no more issues.

[cryptocode]

[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from rpc
[ RUN      ] rpc.account_lazy_start
rpc.cpp:7534: Failure
Value of: account_error.is_initialized ()
  Actual: false
Expected: true
[  FAILED  ] rpc.account_lazy_start (216 ms)
[----------] 1 test from rpc (216 ms total)

Unable to reproduce, appears to be fixed by recent changes.

[cryptocode]

This hit a couple of times (debug build):

[ RUN      ] distributed_work.peer_malicious
nano/core_test/distributed_work.cpp:214: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:

Fixed by https://github.com/nanocurrency/nano-node/pull/2500

[cryptocode]

websocket.bootstrap_exited segfaults from time to time

Fixed (unable to reproduce, appears to be fixed by recent changes to websocket tests)

[cryptocode]

[ RUN      ] node_telemetry.batch_use_single_request_cache
nano/core_test/node_telemetry.cpp:772: Failure
Expected equality of these values:
  2
  node_client->stats.count (nano::stat::type::message, nano::stat::detail::telemetry_ack, nano::stat::dir::in)
    Which is: 1
[  FAILED  ] node_telemetry.batch_use_single_request_cache (7374 ms)

Seems to happen occasionally under TSAN, so likely timing related

Edit: This test is now called all_peers_use_single_request_cache Needs to be rechecked if issue still exists since the telemetry refactor

Edit2: This appears to be fixed with all_peers_use_single_request_cache

[cryptocode]

confirm_back failures, as seen on CI, is reproducible locally.

The active roots container sometimes never drains, so it times out.

Possibly solved by https://github.com/nanocurrency/nano-node/pull/2441

[cryptocode]

Moved convos/status into the corresponding comment and marked them resolved.

[cryptocode]

[ RUN      ] confirmation_height.gap_live
nano/core_test/confirmation_height.cpp:352: Failure
Expected equality of these values:
  1
  confirmation_height_info.height
    Which is: 2

Fairly easy to reproduce with TSAN

Fixed in #2621

[wezrule]

[ RUN      ] wallet.work_watcher_update
D:\a\raiblocks\raiblocks\nano\core_test\wallet.cpp(1171): error: Value of: system.poll ()
  Actual: Deadline expired
Expected: 
[  FAILED  ] wallet.work_watcher_update (20349 ms)

Fails a lot on Windows Actions

edit: Fixed by https://github.com/nanocurrency/nano-node/pull/2677

[wezrule]

Mac Actions (https://github.com/nanocurrency/nano-node/runs/617432809?check_suite_focus=true)

[ RUN      ] network.tcp_no_connect_excluded_peers
/Users/runner/runners/2.169.0/work/raiblocks/raiblocks/nano/core_test/network.cpp:1131: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected: 

[wezrule]

[ RUN      ] system.generate_send_new
/Users/runner/runners/2.169.0/work/raiblocks/raiblocks/nano/core_test/ledger.cpp:622: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected: 

Likely fixed by: https://github.com/nanocurrency/nano-node/pull/2742

[guilhermelawless]

[ RUN      ] node.fork_invalid_block_signature
nano-node/nano/core_test/node.cpp:3123: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:
[  FAILED  ] node.fork_invalid_block_signature (5195 ms)

EDIT: Bisected to 5dac531c911b0e19e2c615397fa38e94a9b34ecd

EDIT: Likely fixed by https://github.com/nanocurrency/nano-node/pull/2743

[wezrule]

Failed on MacOS actions:

[ RUN      ] node_telemetry.remove_peer_different_genesis
/Users/runner/runners/2.169.1/work/nano-node/nano-node/nano/core_test/node_telemetry.cpp:746: Failure
Expected equality of these values:
  0
  node0->network.size ()
    Which is: 1
[  FAILED  ] node_telemetry.remove_peer_different_genesis (396 ms)

Edit: Done in #2738

[wezrule]

Failed on Windows actions:

[ RUN      ] conflicts.adjusted_multiplier
D:\a\nano-node\nano-node\nano\core_test\conflicts.cpp(296): error: Expected equality of these values:
  node1.active.roots.get<1> ().begin ()->election->status.winner->hash ()
    Which is: 32-byte object <C0-A9 C2-5D BB-BA EE-B1 86-6A 47-85 23-FF E8-83 84-98 AB-B0 41-E9 BD-D3 AF-15 48-FC 2A-97 05-2E>
  open_epoch2->hash ()
    Which is: 32-byte object <A4-58 4D-27 50-79 D8-6D A3-1F 35-42 A9-4D 66-4D 17-EC 26-DB C7-09 17-01 48-CB 81-34 BD-3E 1E-6F>
[  FAILED  ] conflicts.adjusted_multiplier (296 ms)

Edit: Solved by https://github.com/nanocurrency/nano-node/pull/2739

[wezrule]

Failed with gcc actions (https://github.com/nanocurrency/nano-node/runs/617432731?check_suite_focus=true), also failed with macos (https://github.com/nanocurrency/nano-node/runs/617432809?check_suite_focus=true)

[ RUN      ] work_watcher.propagate
/workspace/nano/core_test/wallet.cpp:1259: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected: 
[  FAILED  ] work_watcher.propagate (10258 ms)

EDIT: Fixed in https://github.com/nanocurrency/nano-node/pull/2748

[wezrule]

Failed with gcc actions (https://github.com/nanocurrency/nano-node/runs/617432731?check_suite_focus=true)

[ RUN      ] websocket.bootstrap_exited
/workspace/nano/core_test/websocket.cpp:778: Failure
Expected: (nullptr) != (attempt), actual: (nullptr) vs 16-byte object <00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>
/workspace/nano/core_test/websocket.cpp:828: Failure
Expected equality of these values:
  contents.get<std::string> ("reason")
    Which is: "started"
  "exited"
[  FAILED  ] websocket.bootstrap_exited (5031 ms)

EDIT: Fixed in https://github.com/nanocurrency/nano-node/pull/2748

[wezrule]

Failed with MacOS actions (https://github.com/nanocurrency/nano-node/runs/617432809?check_suite_focus=true)

[ RUN      ] node.aggressive_flooding
/Users/runner/runners/2.169.1/work/nano-node/nano-node/nano/core_test/node.cpp:3727: Failure
Expected equality of these values:
  1 + 2 * nodes_wallets.size () + 2
    Which is: 13
  node1.ledger.cache.block_count
    Which is: 12
[  FAILED  ] node.aggressive_flooding (2362 ms)

Also failed locally on my Windows system

[wezrule]

Failed with MacOS actions (https://github.com/nanocurrency/nano-node/runs/616908313?check_suite_focus=true)

[ RUN      ] rpc.confirmation_height_currently_processing
libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

Windows Actions: https://github.com/nanocurrency/nano-node/runs/616908302?check_suite_focus=true

[ RUN      ] rpc.confirmation_height_currently_processing
unknown file: error: C++ exception with description "No such node (error)" thrown in the test body.

Edit: Should be solved by https://github.com/nanocurrency/nano-node/pull/2737

[wezrule]

Failed with gcc actions (https://github.com/nanocurrency/nano-node/runs/616908302?check_suite_focus=true), also clang & windows.

[ RUN      ] rpc.wallet_history
/workspace/nano/rpc_test/rpc.cpp:7278: Failure
Expected: (nullptr) != (receive), actual: (nullptr) vs 16-byte object <00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>
[  FAILED  ] rpc.wallet_history (1123 ms)

Fixed by https://github.com/nanocurrency/nano-node/pull/2739

[wezrule]

Failed with gcc actions (https://github.com/nanocurrency/nano-node/runs/616900835?check_suite_focus=true)

timeout 360 ./qt_test
QXcbConnection: Could not connect to display :2
timeout: the monitored command dumped core
./ci/test.sh: line 23: 42719 Aborted
DISPLAY=:2 ${TIMEOUT_CMD} ${TIMEOUT_TIME_ARG} ${TIMEOUT_SEC-${TIMEOUT_DEFAULT}} $@

[guilhermelawless]

Failed with clang actions

2020-04-28T13:25:19.6472621Z [ RUN      ] websocket.active_difficulty
2020-04-28T13:25:19.7226298Z /workspace/nano/core_test/websocket.cpp:114: Failure
2020-04-28T13:25:19.7227990Z Expected equality of these values:
2020-04-28T13:25:19.7228419Z   network_current
2020-04-28T13:25:19.7228608Z     Which is: 18428729675200069632
2020-04-28T13:25:19.7229397Z   node1->active.active_difficulty ()
2020-04-28T13:25:19.7229494Z     Which is: 18434320350599564040
2020-04-28T13:25:19.7674448Z [  FAILED  ] websocket.active_difficulty (125 ms)

EDIT: Fixed in https://github.com/nanocurrency/nano-node/pull/2748

[guilhermelawless]

Failed with clang actions

2020-04-28T13:30:04.6017646Z [ RUN      ] rpc.confirmation_active
2020-04-28T13:30:04.7142579Z unknown file: Failure
2020-04-28T13:30:04.7142983Z C++ exception with description "Address already in use" thrown in the test body.
2020-04-28T13:30:04.7143121Z [  FAILED  ] rpc.confirmation_active (114 ms)

[wezrule]

Failed with clang actions

[ RUN      ] active_transactions.activate_dependencies
/workspace/nano/core_test/active_transactions.cpp:671: Failure
Value of: node1->ledger.block_confirmed (node1->store.tx_begin_read (), block2->hash ())
  Actual: false
Expected: true
[  FAILED  ] active_transactions.activate_dependencies (5614 ms)

EDIT: Fixed in https://github.com/nanocurrency/nano-node/pull/2748

[wezrule]

active_transactions.vote_replays
/workspace/nano/core_test/active_transactions.cpp:774: Failure
Expected equality of these values:
  1
  node.active.size ()
    Which is: 0

[cryptocode]

I have a _core_test_ running in a loop to try reproduce a segfault, and came across this:

[ RUN      ] work_watcher.confirm_while_generating
wallet.cpp:1259: Failure
Expected equality of these values:
  0
  node.work.size ()
    Which is: 1

NOTE: This comment can be resolved with the merge of #3063

[cryptocode]

After another few hours of core_test looping:

[ RUN      ] active_transactions.pessimistic_elections
nano/core_test/active_transactions.cpp:1535: Failure
Expected equality of these values:
  1
  election_count
    Which is: 2

NOTE: A bit of investigation shows this is because confirm_expired_frontiers_pessimistically is not locked internally, so the test corrupts expired_optimistic_election_infos. Wesley points out it's not an issue for the production node as it's only called from the request loop thread. Possible fixes include internal locking, a separate mutex or some other workaround for the test.

[cryptocode]

[ RUN      ] node.bidirectional_tcp
Assertion (wallets.reps ().voting > 0) failed
void nano::request_aggregator::add(std::shared_ptr<nano::transport::channel> &, const std::vector<std::pair<nano::block_hash, nano::root> > &)
nano/node/request_aggregator.cpp:35

 0# nano::generate_stacktrace() in build/RelWithDebInfo/core_test
 1# assert_internal(char const*, char const*, char const*, unsigned int, bool) in build/RelWithDebInfo/core_test
 2# nano::request_aggregator::add(std::__1::shared_ptr<nano::transport::channel>&, std::__1::vector<std::__1::pair<nano::block_hash, nano::root>, std::__1::allocator<std::__1::pair<nano::block_hash, nano::root> > > const&) in build/RelWithDebInfo/core_test
 3# (anonymous namespace)::network_message_visitor::confirm_req(nano::confirm_req const&) in build/RelWithDebInfo/core_test
 4# nano::confirm_req::visit(nano::message_visitor&) const in build/RelWithDebInfo/core_test
 5# nano::network::process_message(nano::message const&, std::__1::shared_ptr<nano::transport::channel>) in build/RelWithDebInfo/core_test
 6# nano::transport::tcp_channels::process_message(nano::message const&, boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const&, nano::public_key const&, std::__1::shared_ptr<nano::socket>, nano::bootstrap_server_type) in build/RelWithDebInfo/core_test
 7# nano::transport::tcp_channels::process_messages() in build/RelWithDebInfo/core_test
 8# nano::network::network(nano::node&, unsigned short)::$_2::operator()() const in build/RelWithDebInfo/core_test
 9# boost::detail::thread_data<nano::network::network(nano::node&, unsigned short)::$_2>::run() in build/RelWithDebInfo/core_test

[cryptocode]

[ RUN      ] node.bootstrap_bulk_push
nano/core_test/node.cpp:1898: Failure
Value of: node1->active.empty ()
  Actual: true
Expected: false

NOTE: This comment can be resolved with the merge of https://github.com/nanocurrency/nano-node/pull/3062

[cryptocode]

[ RUN      ] node.dependency_graph_frontier
nano/core_test/node.cpp:4282: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:

[LenWilliamson]

I just run the test in the top level nano_node directory this way:
cmake -D NANO_TEST=ON
make
and then I got all the test files in nano_node. When running ./core_test I get

[wezrule]

Thanks, what OS is this? If you still have the output, would be nice to see where it's failing

[LenWilliamson]

Operating System: Linux Mint 20.1
Kernel: Linux 5.4.0-65-generic
Architecture: x86-64

After running the test again I didn't get the output. But I don't know (yet) how to check where it's failing. What should I do to check it?

[wezrule]

no worries, if it happens again you can scroll up (assuming there's enough lines configured in terminal) to where the test is and it should output something when it fails

[LenWilliamson]

I see, so this is the output I got

[ RUN      ] network.last_contacted
/home/len/Crypto/nano-node/nano/core_test/network.cpp:153: Failure
Expected: (timestamp_after_keepalive) > (timestamp_before_keepalive), actual: 8-byte object <71-AD 89-35 87-02 00-00> vs 8-byte object <71-AD 89-35 87-02 00-00>
[  FAILED  ] network.last_contacted (212 ms)
[ RUN      ] network.multi_keepalive

Btw, what is the goal of this test and why has the timestep_before_keepalive be strictly greater? What is the "keepalive"?

[cryptocode]

Happens on CI now in rpc_test:

[ RUN      ] rpc.confirmation_active
Assertion (block->has_sideband ()) failed
void nano::blocks_confirm(nano::node &, const std::vector<std::shared_ptr<nano::block> > &)
/workspace/nano/node/testing.cpp:212

 0# nano::generate_stacktrace[abi:cxx11]() at /workspace/nano/lib/utility.cpp:79
 1# assert_internal(char const*, char const*, char const*, unsigned int, bool, std::basic_string_view<char, std::char_traits<char> >) at /workspace/nano/lib/utility.cpp:125
 2# nano::blocks_confirm(nano::node&, std::vector<std::shared_ptr<nano::block>, std::allocator<std::shared_ptr<nano::block> > > const&) at /workspace/nano/node/testing.cpp:213
 3# rpc_confirmation_active_Test::TestBody() at /workspace/nano/rpc_test/rpc.cpp:8050
 4# void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2443
 5# void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2479
 6# testing::Test::Run() in ./rpc_test
 7# testing::TestInfo::Run() in ./rpc_test
 8# testing::TestCase::Run() at /workspace/gtest/googletest/src/gtest.cc:2810
 9# testing::internal::UnitTestImpl::RunAllTests() at /workspace/gtest/googletest/src/gtest.cc:5176
10# bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2443
11# bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2479
12# testing::UnitTest::Run() at /workspace/gtest/googletest/src/gtest.cc:4786
13# RUN_ALL_TESTS() at /workspace/gtest/googletest/include/gtest/gtest.h:2341
14# main at /workspace/nano/rpc_test/entry.cpp:17
15# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
16# _start in ./rpc_test

[LenWilliamson]

When I run the ./core_test I got

After running ./core_test again, this test passed.