Blockscout: failed to fetch blocks

[ssandeep]

There are:

• a few block numbers in the blocks table with consensus = false and
• a few missing block numbers (which are pretty old, created few days ago) that are missing from the blocks table

I see that these blocks are being attempted to be fetched repeatedly but the fetch is failing as suggested by the logs:

2020-10-22T11:58:26.856 application=indexer fetcher=block_catchup first_block_number=5585599
last_block_number=5585599 step=blocks [error] failed to fetch:
{:more, {:hparser, :response, 4096, 10, 0, :on_trailers, "",
{1, 1}, "", [], :undefined, "chunked", "", "application/json", "", :done}}. Retrying.

Verified that the same blocks are accessible and available from the same server through a curl request:

curl --location --request POST '<endpoint>:8545/' --header 'Content-Type: application/json' --data-raw '{
  "jsonrpc":"2.0",
  "method":"eth_getBlockByNumber",
  "params":[
    "0x553ABF",
    true
  ],
  "id":1
}'

Environment

• Elixir & Erlang/OTP versions (elixir -version):

Erlang/OTP 22 [erts-10.7.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

Elixir 1.10.3 (compiled with Erlang/OTP 21)

• Operating System: Ubuntu 18.04.4 LTS
• Blockscout Version/branch: v3.3.3-beta

Steps to reproduce

Just start the blockscout instance and let it catch up till the latest block. You can validate the blocks being retried are from:

• consensus = false blocks:
  select number from blocks where consensus = false except select number from blocks where consensus = true;
• missing blocks:
  SELECT count(s.i) AS missing_block_nos FROM generate_series( 0 , _latest_block_ ) s(i) LEFT OUTER JOIN blocks ON (blocks.number = s.i) WHERE blocks.number IS NULL;

Expected behaviour

The blocks should have been fetched and present in the database table.

Actual behaviour

Blocks are missing from the table.

[kai1412]

i got the same issues, do you already have solution to fix this error? :(

[ssandeep]

Not yet.

[vbaranov]

@ssandeep what is the response from eth_getBlockByNumber request for the block with number 5585599 from the node:
curl -H "Content-type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x553ABF", true],"id":1}' ?

[ssandeep]

It is returning fine:

{"jsonrpc":"2.0","id":1,"result":{"difficulty":"0x6","extraData":"0xd58301091083626f7286676f312e3133856c696e7578000000000000000000000375b2fc7140977c9c76d45421564e354ed4227700000000000000000000000000000000000000035973918275c01f50555d44e92c9d9b353cadad5400000000000000000000000000000000000000027fcd58c2d53d980b247f1612fdba93e9a76193e60000000000000000000000000000000000000002b702f1c9154ac9c08da247a8e30ee6f2f3373f410000000000000000000000000000000000000002b8bb158b93c94ed35c1970d610d1e2b34e26652c0000000000000000000000000000000000000001f84c74dea96df0ec22e11e7c33996c73fcc2d82200000000000000000000000000000000000000010464f8aac651a40a6a9edd64706e0926d39aecf8c975c4dc0b01979fede1df470a2e475128e99f6984522a113564d5c5748da96033a43b498cf8cc82a4861a6e00","gasLimit":"0x1312d00","gasUsed":"0x8e82","hash":"0x0f1173145d1399e9cf7d5ea8ef69419bbfc75d19375bf8dc41810162ab595285","logsBloom":"0x00000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000008000000800000000000100200000100000000000800000000020400000000000000000800000000000000000080000010000000000000000000000000000000000000000000000000000000008000000004000000200000000000000000000000000000000000000000200000000000000000004000000002000000000001000000000000000000000400000000100002000020000000002000000000000000000000000000000000200000000000000000100000","miner":"0x0000000000000000000000000000000000000000","mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000","nonce":"0x0000000000000000","number":"0x553abf","parentHash":"0x76dfb219f88d36d60e7600684924475d6c1f23b200c465b6fb7032a604da1ff8","receiptsRoot":"0x0efedacea0af897744e3837a2df777e7aab6358c6f936deac6415798fa8d6415","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x403","stateRoot":"0xdd565e4debb9b2112793b2e0e53a61423555597a3edab65519e2bbe077e66b17","timestamp":"0x5f825fe0","totalDifficulty":"0x2549c32","transactions":[{"blockHash":"0x0f1173145d1399e9cf7d5ea8ef69419bbfc75d19375bf8dc41810162ab595285","blockNumber":"0x553abf","from":"0x2883ed845726adacd3677e6b0065e9f6dfbb491b","gas":"0x7d000","gasPrice":"0x3b9aca00","hash":"0xe9742cb706d1cea605a462139f9637412b07e36d14209ce6b472b0cdcb73d23a","input":"0x40c10f1900000000000000000000000053270d44eee663e509f0b715525703b12c353e300000000000000000000000000000000000000000000000000000000000000002","nonce":"0xfb08","to":"0x371b97c779e8c5197426215225de0eeac7dd13af","transactionIndex":"0x0","value":"0x0","v":"0x1b","r":"0x10bdf517ce1370dcfaa04b5b1f7006f9292621905001c5dbcb8731e5616d4caa","s":"0x44ca1148e5922c0a5b10edb77fd86afa0ec011e404d73aa08c725d0d4ef16e3c"}],"transactionsRoot":"0xcbeee1b2a26c8b471b5811cb4ec836c7a98171d1759c704bf4c2fcbad2c9c28c","uncles":[]}}

[SwaroopH]

We compiled off the latest master but keep running into this issue with random blocks. Any quick way to re-index just the missing blocks?

EDIT: Now pulling the latest changes which includes https://github.com/poanetwork/blockscout/pull/3440

[vbaranov]

Try the latest master branch. It has a performance improvement for indexer https://github.com/poanetwork/blockscout/pull/3457

[ssandeep]

@vbaranov, Tried out the code with #3457. The no of missing blocks have reduced considerably but we're still seeing this error repeatedly for some blocks:
2020-12-03T09:46:03.071 application=indexer fetcher=block_catchup first_block_number=7589951 last_block_number=7589951 step=blocks [error] failed to fetch: {:more, {:hparser, :response, 4096, 10, 0, :on_trailers, "\r", {1, 1}, "", [], :undefined, "chunked", "", "application/json", "", :done}}. Retrying.

[vbaranov]

@ssandeep, try to pull the latest state of the master branch. We migrated to Erlang/OTP 23, Elixir@11 and updated most mix deps. Take a note, that you need rustc to compile ex_keccak. Please, let me know if it will resolve the issue.

[ssandeep]

Hi @vbaranov , still seeing the same issue with latest code from master branch and using:

Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

Elixir 1.11.2 (compiled with Erlang/OTP 23)

[ssandeep]

Also, seeing a lot of these after the upgrade:

2020-12-03T11:19:12.330 application=indexer fetcher=coin_balance
 [warn] BufferedTask #PID<0.24642.12> (Indexer.Fetcher.CoinBalance) bound queue is at
 maximum size (4564002) and 500 entries could not be added.

[vbaranov]

still seeing the same issue with latest code from master branch

Ok, try to downgrade hackney mix package to version 1.15.2 (the current one is 1.16.0) and please let me know the results.

Also, seeing a lot of these after the upgrade:

You can manage batch size for coin balances importing in the fetcher: https://github.com/poanetwork/blockscout/blob/master/apps/indexer/lib/indexer/fetcher/coin_balance.ex#L24. Default value is 500.

[ssandeep]

Is there any simple way to do this? I do not have any experience using hex/mix. I added this line in mix.exs under defp deps do:

{:hackney, "~> 1.15.2"}

and updated the hackney version in apps/ethereum_jsonrpc/mix.exs as well:

-      {:hackney, "~> 1.16"},
+      {:hackney, "~> 1.15.2"},

But now, it looks like some other package requires v 1.16 of hackney:

Failed to use "hackney" (version 1.15.2) because
  apps/ethereum_jsonrpc/mix.exs requires ~> 1.15.2
  excoveralls (version 0.13.3) requires ~> 1.16
  mix.exs specifies ~> 1.15.2

Seems non-trivial to downgrade a dependency that's required by many other packages.

[vbaranov]

@ssandeep try updates from the latest master branch. Hackney updated to the latest available version there https://github.com/blockscout/blockscout/pull/3697.

[unqjohnny]

i got the same issues (use latest master branch)

application=indexer fetcher=block_catchup first_block_number=142439 last_block_number=142430 step=blocks [error] failed to fetch: {:more, {:hparser, :response, 4096, 10, 0, :on_trailers, "\r", {1, 1}, "", [], :undefined, "chunked", "", "application/json", "", :done}}. Retrying.

Some blocks will be resolved after restart。
But there will still be problems with some new blocks

@vbaranov