Parity-ethereum: Thread 'jsonrpc-eventloop-1' panicked at 'index 56896 out of range for slice of length 608'

Created on 20 Jan 2019 · 18Comments · Source: openethereum/parity-ethereum

Parity Ethereum version: 2.2.7
Operating system: Debian 4.9.110
Installation: docker
Fully synchronized: yes
Network: ethereum
Restarted: yes

expected behavior

run jsonrpc for trace_replayBlockTransactions on block 7095460
get an actual result

actual behavior

run jsonrpc for trace_replayBlockTransactions on block 7095460
parity crashes with following error:

stack backtrace:
   0:     0x5644daf1c6dd - <no info>
   1:     0x5644daf1b8f2 - <no info>
   2:     0x5644daf1b969 - <no info>
   3:     0x5644daf1aaa6 - <no info>
   4:     0x5644d9638218 - <no info>
   5:     0x5644daf94c96 - <no info>
   6:     0x5644daf947e1 - <no info>
   7:     0x5644daf946c5 - <no info>
   8:     0x5644dafde25c - <no info>
   9:     0x5644dafd3921 - <no info>
  10:     0x5644da477253 - <no info>
  11:     0x5644dacdaac9 - <no info>
  12:     0x5644da3d5b63 - <no info>
  13:     0x5644da403e21 - <no info>
  14:     0x5644da3f80bf - <no info>
  15:     0x5644da433c4a - <no info>
  16:     0x5644da411563 - <no info>
  17:     0x5644da41dc3f - <no info>
  18:     0x5644d95dbd34 - <no info>
  19:     0x5644d95e54a1 - <no info>
  20:     0x5644d98568cb - <no info>
  21:     0x5644d96c2ba7 - <no info>
  22:     0x5644d9835c07 - <no info>
  23:     0x5644d9747840 - <no info>
  24:     0x5644d983b378 - <no info>
  25:     0x5644d990b424 - <no info>
  26:     0x5644d990e74c - <no info>
  27:     0x5644d996e4a2 - <no info>
  28:     0x5644d991f1f9 - <no info>
  29:     0x5644d95fc25d - <no info>
  30:     0x5644d95fd62b - <no info>
  31:     0x5644d95fd62b - <no info>
  32:     0x5644d9632790 - <no info>
  33:     0x5644d967ac02 - <no info>
  34:     0x5644d9678f39 - <no info>
  35:     0x5644d9764a66 - <no info>
  36:     0x5644d95f5b16 - <no info>
  37:     0x5644dacc3d41 - <no info>
  38:     0x5644dacc4795 - <no info>
  39:     0x5644dacc76b0 - <no info>
  40:     0x5644dafa6e79 - <no info>
  41:     0x5644dacc6a9b - <no info>
  42:     0x5644dacbe2b5 - <no info>
  43:     0x5644dacbd39d - <no info>
  44:     0x5644daca42d5 - <no info>
  45:     0x5644daca4611 - <no info>
  46:     0x5644daca44c8 - <no info>
  47:     0x5644dac9cf45 - <no info>
  48:     0x5644dacc55cc - <no info>
  49:     0x5644dacc5a5e - <no info>
  50:     0x5644dacc62b1 - <no info>
  51:     0x5644dacc76fb - <no info>
  52:     0x5644dafa6e79 - <no info>
  53:     0x5644dacc2bcf - <no info>
  54:     0x5644daf9112d - <no info>
  55:     0x5644daf7bce5 - <no info>
  56:     0x7f01d2c8d6b9 - <no info>
  57:     0x7f01d27ad41c - <no info>
  58:                0x0 - <no info>

Thread 'jsonrpc-eventloop-1' panicked at 'index 56896 out of range for slice of length 608', libcore/slice/mod.rs:2334

This is a bug. Please report it at:

    https://github.com/paritytech/parity-ethereum/issues/new

The same issue can be reproduced on the following blocks:

7095247
7095450
7095460
7095565
7096778

F1-panic 🔨 M6-rpcapi 📣

Source

lacasian

👍5

Most helpful comment

Found exact cause by by attaching debugger while tracing this transaction, the issue is caused by 'Out of gas' during specific instruction order (i. e. 'Out of gas' should occur exactly between specific opcodes for issue to occur). Working on a fix, will submit PR with details when ready.

evgeniuz on 23 Sep 2019

❤3

All 18 comments

Since I posted this issue, there have been a few more problematic blocks:

7097970
7103367
7108980
7109121
7118162

Same issue: parity crashes when requesting trace_replayBlockTransactions.
I also tried trace_replayTransaction but that resulted in the same issue.

trace_block on the other side seems to work just fine.

lacasian on 24 Jan 2019

The issue seems to be with the related to the vmTrace request. Here’s a sample request that crashes parity:

curl --data '{"method":"trace_replayBlockTransactions","params":["0x6c7a01",["vmTrace"]],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8545

Ran this against stable/v2.2.8. The debug build stacktrace is here:
https://gist.github.com/valer-cara/c1747acd88ea9fb52136eb6598e81e16

The crash seems to be caused by an out-of-range index in executive_tracer.rs. I believe the trace_executed function is called on each opcode?

The transaction in which it crashes is this one

This is where my rust-lessness beat me :) hints appreciated! (where’s that |o, s| in the map coming from?)

——

The last_mem_written is set by this function in interpreter/mod.rs

valer-cara on 31 Jan 2019

👍1

@valer-cara

This is where my rust-lessness beat me :) hints appreciated! (where’s that |o, s| in the map coming from?)

That is the output (offset, size) from last_mem_written (from the previous instruction) which is used to index into the mem slice which obviously is out-of-range

niklasad1 on 31 Jan 2019

👍1

thanks @niklasad1 ! I’ve been poking around, had traced the execution and it seems it’s crashing when reaching the CALL opcode that errors with no gas left (bottom of the page)

I see the offset & size are calculated here and the opcode looks like this

Currently having a hard time with gdb and accessing rust variables ( like the values stored in an Option<...>, etc.. ). Any further hints on what to try appreciated. Thanks!

valer-cara on 31 Jan 2019

@valer-cara

Yeah, debugging using gdb is indeed hard. I think the case for Option is that gdb can't determine the type sometimes (type-inference...)

However, I recommend using RUST_LOG instead otherwise you could also try annotating your important functions with:

#[inline(never)]
#[no_mangle]

to make gdb debugging easiser

niklasad1 on 2 Feb 2019

Any workaround?

karsrhyder on 4 Feb 2019

Same issue on Kovan Testnet! Stuck badly!!!
Image Link

sabbir360 on 4 Feb 2019

@sabbir360 you're seeing a different issue here: #10287 - the fix for that is in the latest release, please update!

joshua-mir on 4 Feb 2019

👍2

Hello @kwix, I just saw this post.

I met a similar parity crash problem (#10104) before trying to call trace_replayBlockTransactions and trace_replayTransaction. However, my problem was with block 1431919 and 2283417, and specific transaction 0x0c10fafe0cdbfff32abfe53d57ec861d09986cc1050c850481f79b1a862bb10a.

Could you help try these blocks and transaction in your node, and see how it behaves? I would be very grateful for that.

chaoliu92 on 5 Feb 2019

Hello @chaoliu92!

With my current configuration I'm getting a stack overflow crash for that transaction, same as you do.
Looking at the transaction here, I don't think the crash is related with this issue (stack overflow vs out of range).

The stack overflow is most probably caused by the depth the contract messages reach.
You might want to try increasing the cache size (the parity parameter) and also increasing the host operating system stack size.

lacasian on 5 Feb 2019

👍1

Just a small update here: the issue is still present even on newer versions of parity (tested with both 2.3.9-stable and 2.4.5-stable).
By now there are more than 100 blocks that crash parity when trying to get the transaction replays and they keep on appearing. The last one was block 7540712.

lacasian on 11 Apr 2019

👍2

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

__This issue now has a funding of 300.0 DAI (300.0 USD @ $1.0/DAI) attached to it as part of the Alethio fund.__

If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
- Want to chip in? Add your own contribution here.
- Questions? Checkout Gitcoin Help or the Gitcoin Slack
- $133,270.22 more funded OSS Work available on the Gitcoin Issue Explorer

gitcoinbot on 19 Jul 2019

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

__Work has been started__.

These users each claimed they can complete the work by 12 months from now.
Please review their action plans below:

1) evgeniuz has started work.

Would like to troubleshoot and fix this. Will start by reproducing, diving with debugger and go from there. Love tracing this kind of bugs :)

Learn more on the Gitcoin Issue Details page.

gitcoinbot on 20 Jul 2019

🎉2

This seems related to https://github.com/paritytech/parity-ethereum/pull/11002 , is it possible this was fixed as a byproduct of fixing the trace_call+vmTrace ?