Prysm: panic: invalid page type: 0: 4

[rfikki]

https://gist.github.com/rfikki/63acd4067c18ff909fdc5e2392366be5

[SjonHortensius]

I think you should have posted the complete logs here directly - however these invalid page type errors seem to happen so deep in the boltdb code that I actually suspect some form of (hardware) corruption - I take it you're not using ECC? Your first message seems to imply there isn't even a data-directory, can you confirm this?

Doesn't clearing the data-dir resolve this issue?

[rfikki]

So, it was mentioned that this could be a permissions issue. So, I added "sudo" to the command line and seem to be getting some what further, but error persists: see below:

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ sudo bazel run //beacon-chain -- --clear-db --datadir=/home/rfikki/prysm/prysm-data
[sudo] password for rfikki:
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
DEBUG: Rule 'com_google_protobuf' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1551387314 -0800"
INFO: Analyzed target //beacon-chain:beacon-chain (536 packages loaded, 15980 targets configured).
INFO: Found 1 target...
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:wrappers_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:descriptor_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:struct_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:compiler_plugin_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:field_mask_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:source_context_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:any_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:timestamp_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:empty_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:duration_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library //proto/beacon/p2p/v1:v1_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_github_libp2p_go_libp2p_autonat//pb:autonat_pb_proto:
[libprotobuf WARNING external/com_google_protobuf/src/google/protobuf/compiler/parser.cc:564] No syntax specified for the proto file: pb/autonat.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:type_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library //proto/sharding/p2p/v1:v1_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @go_googleapis//google/rpc:status_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:api_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library //proto/beacon/rpc/v1:v1_proto:
external/com_google_protobuf: warning: directory does not exist.
Target //beacon-chain:beacon-chain up-to-date:
bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain
INFO: Elapsed time: 874.733s, Critical Path: 111.27s
INFO: 1961 processes: 1961 processwrapper-sandbox.
INFO: Build completed successfully, 1992 total actions
INFO: Build completed successfully, 1992 total actions
[2019-05-10 15:20:06] INFO node: Using custom parameter configuration
[2019-05-10 15:20:06] INFO node: Checking db path=/home/rfikki/prysm/prysm-data/beaconchaindata
[2019-05-10 15:20:06] INFO node: Fetching testnet cluster address endpoint=https://beta.prylabs.net/contract
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/boltdb/bolt.(Cursor).search(0xc000371670, 0x20121d8, 0xa, 0xa, 0x4)
external/com_github_boltdb_bolt/cursor.go:256 +0x354
github.com/boltdb/bolt.(Cursor).seek(0xc000371670, 0x20121d8, 0xa, 0xa, 0x211de40, 0xc0002e5e50, 0x2, 0x2, 0x2, 0x2, ...)
external/com_github_boltdb_bolt/cursor.go:159 +0x7e
github.com/boltdb/bolt.(Bucket).Bucket(0xc000342558, 0x20121d8, 0xa, 0xa, 0x1487fa0)
external/com_github_boltdb_bolt/bucket.go:112 +0xd5
github.com/boltdb/bolt.(Tx).Bucket(...)
external/com_github_boltdb_bolt/tx.go:101
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).VerifyContractAddress.func1(0xc000342540, 0x1484520, 0xc000342540)
beacon-chain/db/verify_contract.go:24 +0x73
github.com/boltdb/bolt.(DB).Update(0xc0002223c0, 0xc0001d7a80, 0x0, 0x0)
external/com_github_boltdb_bolt/db.go:598 +0x90
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).update(...)
beacon-chain/db/db.go:52
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).VerifyContractAddress(0xc0001c23c0, 0x160bae0, 0xc000042100, 0xf8c541a3f3d806ab, 0xae7a47a38dd9d5a8, 0x742dfcab, 0x0, 0x0)
beacon-chain/db/verify_contract.go:23 +0xd9
github.com/prysmaticlabs/prysm/beacon-chain/node.(BeaconNode).registerPOWChainService(0xc0005af280, 0xc000278000, 0x0, 0x0)
beacon-chain/node/node.go:289 +0x5f4
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc000278000, 0x4, 0x4, 0x0)
beacon-chain/node/node.go:88 +0x278
main.startNode(0xc000278000, 0x0, 0x0)
beacon-chain/main.go:27 +0x8c
github.com/urfave/cli.HandleAction(0x11165c0, 0x14875f0, 0xc000278000, 0xc0001b28a0, 0x0)
external/com_github_urfave_cli/app.go:490 +0xc8
github.com/urfave/cli.(App).Run(0xc0001929c0, 0xc00003c090, 0x3, 0x3, 0x0, 0x0)
external/com_github_urfave_cli/app.go:264 +0x590
main.main()
beacon-chain/main.go:83 +0xa6d

[prestonvanloon]

Maybe related:
https://github.com/Microsoft/WSL/issues/3162
https://github.com/odeke-em/drive/wiki/Boltdb-breaks-drive-with-(panic:-invalid-page-type:)-or-(panic:--above-high-water-mark)

[prestonvanloon]

Just to be totally sure can you try this?

bazel build //beacon-chain 
bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain --clear-db --datadir=/tmp/prysm-data

If that doesn't work, try running the second command with sudo. I'm not entirely sure how this error occurs so consistently. :(

[rfikki]

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ bazel build //beacon-chain
Starting local Bazel server and connecting to it...
INFO: Analyzed target //beacon-chain:beacon-chain (536 packages loaded, 15979 targets configured).
INFO: Found 1 target...
Target //beacon-chain:beacon-chain up-to-date:
bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain
INFO: Elapsed time: 31.613s, Critical Path: 1.59s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action

then second command resulted in:

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain --clear-db --datadir=/tmp/prysm-data
[2019-05-11 14:02:53] INFO node: Using custom parameter configuration
[2019-05-11 14:02:53] INFO node: Checking db path=/tmp/prysm-data/beaconchaindata
[2019-05-11 14:02:53] INFO node: Fetching testnet cluster address endpoint=https://beta.prylabs.net/contract
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/boltdb/bolt.(Cursor).search(0xc000775670, 0x20121d8, 0xa, 0xa, 0x4)
external/com_github_boltdb_bolt/cursor.go:256 +0x354
github.com/boltdb/bolt.(Cursor).seek(0xc000775670, 0x20121d8, 0xa, 0xa, 0x211de40, 0xc00035e620, 0x2, 0x2, 0x2, 0x2, ...)
external/com_github_boltdb_bolt/cursor.go:159 +0x7e
github.com/boltdb/bolt.(Bucket).Bucket(0xc000337278, 0x20121d8, 0xa, 0xa, 0x1487fa0)
external/com_github_boltdb_bolt/bucket.go:112 +0xd5
github.com/boltdb/bolt.(Tx).Bucket(...)
external/com_github_boltdb_bolt/tx.go:101
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).VerifyContractAddress.func1(0xc000337260, 0x1484520, 0xc000337260)
beacon-chain/db/verify_contract.go:24 +0x73
github.com/boltdb/bolt.(DB).Update(0xc0002183c0, 0xc0005c2440, 0x0, 0x0)
external/com_github_boltdb_bolt/db.go:598 +0x90
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).update(...)
beacon-chain/db/db.go:52
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).VerifyContractAddress(0xc0001bc3c0, 0x160bae0, 0xc0000440f8, 0xf8c541a3f3d806ab, 0xae7a47a38dd9d5a8, 0x742dfcab, 0x0, 0x0)
beacon-chain/db/verify_contract.go:23 +0xd9
github.com/prysmaticlabs/prysm/beacon-chain/node.(BeaconNode).registerPOWChainService(0xc0004e10c0, 0xc000198000, 0x0, 0x0)
beacon-chain/node/node.go:289 +0x5f4
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc000198000, 0x4, 0x4, 0x0)
beacon-chain/node/node.go:88 +0x278
main.startNode(0xc000198000, 0x0, 0x0)
beacon-chain/main.go:27 +0x8c
github.com/urfave/cli.HandleAction(0x11165c0, 0x14875f0, 0xc000198000, 0xc00057e060, 0x0)
external/com_github_urfave_cli/app.go:490 +0xc8
github.com/urfave/cli.(App).Run(0xc00017c000, 0xc00003c090, 0x3, 0x3, 0x0, 0x0)
external/com_github_urfave_cli/app.go:264 +0x590
main.main()
beacon-chain/main.go:83 +0xa6d

Using the sudo command gave the same error................

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ sudo bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain --clear-db --datadir=/tmp/prysm-data
[sudo] password for rfikki:
[2019-05-11 14:03:12] INFO node: Using custom parameter configuration
[2019-05-11 14:03:13] INFO node: Checking db path=/tmp/prysm-data/beaconchaindata
[2019-05-11 14:03:13] INFO node: Fetching testnet cluster address endpoint=https://beta.prylabs.net/contract
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/boltdb/bolt.(Cursor).search(0xc000757670, 0x20121d8, 0xa, 0xa, 0x4)
external/com_github_boltdb_bolt/cursor.go:256 +0x354
github.com/boltdb/bolt.(Cursor).seek(0xc000757670, 0x20121d8, 0xa, 0xa, 0x211de40, 0xc000599230, 0x2, 0x2, 0x2, 0x2, ...)
external/com_github_boltdb_bolt/cursor.go:159 +0x7e
github.com/boltdb/bolt.(Bucket).Bucket(0xc00003e1d8, 0x20121d8, 0xa, 0xa, 0x1487fa0)
external/com_github_boltdb_bolt/bucket.go:112 +0xd5
github.com/boltdb/bolt.(Tx).Bucket(...)
external/com_github_boltdb_bolt/tx.go:101
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).VerifyContractAddress.func1(0xc00003e1c0, 0x1484520, 0xc00003e1c0)
beacon-chain/db/verify_contract.go:24 +0x73
github.com/boltdb/bolt.(DB).Update(0xc00000c1e0, 0xc0002797e0, 0x0, 0x0)
external/com_github_boltdb_bolt/db.go:598 +0x90
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).update(...)
beacon-chain/db/db.go:52
github.com/prysmaticlabs/prysm/beacon-chain/db.(BeaconDB).VerifyContractAddress(0xc0001148c0, 0x160bae0, 0xc0000c2068, 0xf8c541a3f3d806ab, 0xae7a47a38dd9d5a8, 0x742dfcab, 0x0, 0x0)
beacon-chain/db/verify_contract.go:23 +0xd9
github.com/prysmaticlabs/prysm/beacon-chain/node.(BeaconNode).registerPOWChainService(0xc0000cba00, 0xc000398000, 0x0, 0x0)
beacon-chain/node/node.go:289 +0x5f4
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc000398000, 0x4, 0x4, 0x0)
beacon-chain/node/node.go:88 +0x278
main.startNode(0xc000398000, 0x0, 0x0)
beacon-chain/main.go:27 +0x8c
github.com/urfave/cli.HandleAction(0x11165c0, 0x14875f0, 0xc000398000, 0xc0000c89c0, 0x0)
external/com_github_urfave_cli/app.go:490 +0xc8
github.com/urfave/cli.(App).Run(0xc00022a340, 0xc0000ba060, 0x3, 0x3, 0x0, 0x0)
external/com_github_urfave_cli/app.go:264 +0x590
main.main()
beacon-chain/main.go:83 +0xa6d

[SjonHortensius]

@rfikki can you clarify which OS and FS you use, is it also zfs (or maybe even WSL)? I've been trying to reproduce this but a zfs mount on a recent Archlinux machine does not show this - and boltdbs tests also run fine on it.

Given the related reports I'm pretty sure this is a go-specific issue with regards to filesystem interaction that might be worked around - but is not caused by anything prysm specific

[rfikki]

Windows Subsystem for Linux (WSL) is my OS setup.

[rfikki]

I wonder if this is somehow related to these issues:
https://github.com/prysmaticlabs/prysm/pull/2598

[SjonHortensius]

I have reproduced the testcase in https://github.com/microsoft/WSL/issues/3162 and apparently this was a WSL specific issue that has been fixed in https://github.com/microsoft/WSL/issues/3939 and also listed under https://docs.microsoft.com/en-us/windows/wsl/release-notes#build-18890 as

• Extending mmap’d files does not work properly [GH 3939]

I think @rfikki should retry this with a newer (at least 18890) WSL build, checkout the "Windows Insider Program" settings to download bleeding edge releases.

Also - I think in the future mentioning up-front that you're using WSL is pretty significant and useful for people attempting to reproduce your issue :)

[rfikki]

I agree, that it would have been helpful to mention WSL. However it might be easier to know what is pertinent to list/provide when submitting an issue. See the template provided by the Geth team here when submitting an issue. https://github.com/ethereum/go-ethereum/issues/new

[rfikki]

Thanks for all the assistance provided. I will look into the newer builds after 18890 which have not yet been made part of an official release.

[rfikki]

Can confirm that the new build works for me. Updated to 18898.

[SjonHortensius]

@rfikki can you close this issue as it is resolved - and wasn't caused by prysm? This issue has no use remaining open