Go-ethereum: crashed with lots of error response "authentication needed: password or unlock"

Created on 26 Oct 2019  路  3Comments  路  Source: ethereum/go-ethereum

Our ethereum network node crashed on oct 24, with lots of error response "authentication needed: password or unlock". The QPS is about 1~2, which is very low. According to our ab test, the node can support at most 60 QPS invalid request.

Here are the logs:

WARN [10-14|22:06:14.501] Served eth_signTransaction conn=144.48.63.243:50020 reqid=21 t=65.908碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:15.951] Served eth_signTransaction conn=144.48.63.243:52376 reqid=3 t=65.229碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:16.156] Served eth_signTransaction conn=45.76.106.157:51598 reqid=56 t=70.603碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:16.438] Served eth_signTransaction conn=167.86.104.202:49112 reqid=87 t=64.77碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:17.727] Served eth_signTransaction conn=144.48.63.243:54638 reqid=48 t=71.689碌s err="authentication needed: password or unlock"
INFO [10-14|22:06:18.017] Submitted transaction fullhash=0x26cf81dfdc13963f09862ab5e88d18e1b823dea2804263ae7fc2f92c43197853 recipient=0x2E65E12b5f0fD1D58738c6F38dA7D57F5F183d1c
WARN [10-14|22:06:18.093] Served eth_signTransaction conn=167.86.104.202:48914 reqid=1 t=65.019碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:18.093] Served eth_signTransaction conn=167.86.104.202:48914 reqid=2 t=17.121碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:18.093] Served eth_signTransaction conn=167.86.104.202:48914 reqid=3 t=16.642碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:18.095] Served eth_signTransaction conn=167.86.104.202:48914 reqid=4 t=2.145782ms err="authentication needed: password or unlock"
WARN [10-14|22:06:18.095] Served eth_signTransaction conn=167.86.104.202:48914 reqid=5 t=16.297碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:18.095] Served eth_signTransaction conn=167.86.104.202:48914 reqid=0 t=17.583碌s err="authentication needed: password or unlock"
WARN [10-14|22:06:18.402] Served eth_signTransaction conn=45.76.106.157:52550 reqid=77 t=59.605碌s err="authentication needed: password or unlock"

out of memory

WARN [10-24|18:29:47.975] Served eth_signTransaction               conn=45.76.106.157:6362    reqid=84   t=98.264碌s    err="authentication needed: password or unlock"
WARN [10-24|18:29:48.062] Served eth_signTransaction               conn=144.48.63.243:50830   reqid=51   t=64.547碌s    err="authentication needed: password or unlock"
INFO [10-24|18:29:48.224] Setting new local account                address=0x40A49838dd7f61666258254a2D9f611eFC408800
INFO [10-24|18:29:48.232] Submitted transaction                    fullhash=0x699e9c6d24198c0bff1edf147ee133eb6c2acc326bf8e1becff635b2035b6a78 recipient=0x2E65E12b5f0fD1D58738c6F38dA7D57F5F183d1c
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x10e662b, 0x16)
    /usr/lib/go-1.11/src/runtime/panic.go:608 +0x72
runtime.sysMap(0xc47c000000, 0x10000000, 0x23bc178)
    /usr/lib/go-1.11/src/runtime/mem_linux.go:156 +0xc7
runtime.(*mheap).sysAlloc(0x238d3e0, 0x10000000, 0x680031f220, 0x7fe5bdffacd8)
    /usr/lib/go-1.11/src/runtime/malloc.go:619 +0x1c7
runtime.(*mheap).grow(0x238d3e0, 0x8000, 0x0)
    /usr/lib/go-1.11/src/runtime/mheap.go:920 +0x42
runtime.(*mheap).allocSpanLocked(0x238d3e0, 0x8000, 0x23bc188, 0x0)
    /usr/lib/go-1.11/src/runtime/mheap.go:848 +0x337
runtime.(*mheap).alloc_m(0x238d3e0, 0x8000, 0x101, 0x149c0f8)
    /usr/lib/go-1.11/src/runtime/mheap.go:692 +0x119
runtime.(*mheap).alloc.func1()
    /usr/lib/go-1.11/src/runtime/mheap.go:759 +0x4c
runtime.(*mheap).alloc(0x238d3e0, 0x8000, 0x7fe5bd010101, 0x43ada0)
    /usr/lib/go-1.11/src/runtime/mheap.go:758 +0x8a
runtime.largeAlloc(0x10000000, 0x460101, 0xc46c000000)
    /usr/lib/go-1.11/src/runtime/malloc.go:1019 +0x97
runtime.mallocgc.func1()
    /usr/lib/go-1.11/src/runtime/malloc.go:914 +0x46
runtime.systemstack(0x0)
    /usr/lib/go-1.11/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
    /usr/lib/go-1.11/src/runtime/proc.go:1229

goroutine 3216928 [running]:
runtime.systemstack_switch()
    /usr/lib/go-1.11/src/runtime/asm_amd64.s:311 fp=0xc01922ef00 sp=0xc01922eef8 pc=0x4703a0
runtime.mallocgc(0x10000000, 0xf166c0, 0x1, 0xc0d1939800)
    /usr/lib/go-1.11/src/runtime/malloc.go:913 +0x896 fp=0xc01922efa0 sp=0xc01922ef00 pc=0x421e56
runtime.makeslice(0xf166c0, 0x4000000, 0x4000000, 0xc0d1939800, 0x200, 0x200)
    /usr/lib/go-1.11/src/runtime/slice.go:70 +0x77 fp=0xc01922efd0 sp=0xc01922efa0 pc=0x459207
github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/scrypt.Key(0x23ba020, 0x0, 0x0, 0xc0cc97b740, 0x20, 0x40, 0x40000, 0x8, 0x1, 0x20, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/scrypt/scrypt.go:236 +0x152 fp=0xc01922f090 sp=0xc01922efd0 pc=0x6dfe92
github.com/ethereum/go-ethereum/accounts/keystore.getKDFKey(0xc0bc42a1b0, 0xb, 0xc0cc97b600, 0x40, 0xc06bdff860, 0x20, 0xc0bc42a1c0, 0x6, 0xc05673aa20, 0xc0cc97b680, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/passphrase.go:332 +0x693 fp=0xc01922f198 sp=0xc01922f090 pc=0x6fabe3
github.com/ethereum/go-ethereum/accounts/keystore.DecryptDataV3(0xc0bc42a1b0, 0xb, 0xc0cc97b600, 0x40, 0xc06bdff860, 0x20, 0xc0bc42a1c0, 0x6, 0xc05673aa20, 0xc0cc97b680, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/passphrase.go:257 +0x280 fp=0xc01922f2d0 sp=0xc01922f198 pc=0x6f9930
github.com/ethereum/go-ethereum/accounts/keystore.decryptKeyV3(0xc01c0ba480, 0x0, 0x0, 0xed7b00, 0xc01c0ba480, 0x0, 0x0, 0xc036e26900, 0x0, 0x0, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/passphrase.go:279 +0xca fp=0xc01922f398 sp=0xc01922f2d0 pc=0x6f9d5a
github.com/ethereum/go-ethereum/accounts/keystore.DecryptKey(0xc09c190c00, 0x1eb, 0x3eb, 0x0, 0x0, 0x0, 0x0, 0x0)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/passphrase.go:223 +0x1b9 fp=0xc01922f450 sp=0xc01922f398 pc=0x6f93b9
github.com/ethereum/go-ethereum/accounts/keystore.keyStorePassphrase.GetKey(0xc000039fc0, 0x18, 0x40000, 0x1, 0x0, 0x7d547ac3ab1cbc23, 0xb126daaacf42bd, 0xb4c78200, 0xc0683a2500, 0x74, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/passphrase.go:89 +0x9c fp=0xc01922f4f8 sp=0xc01922f450 pc=0x6f7bdc
github.com/ethereum/go-ethereum/accounts/keystore.keyStorePassphrase.StoreKey(0xc000039fc0, 0x18, 0x40000, 0x1, 0x0, 0xc0083c6310, 0x66, 0xc080653c00, 0x0, 0x0, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/passphrase.go:118 +0x1b5 fp=0xc01922f5d0 sp=0xc01922f4f8 pc=0x6f81a5
github.com/ethereum/go-ethereum/accounts/keystore.(*keyStorePassphrase).StoreKey(0xc00056cba0, 0xc0083c6310, 0x66, 0xc080653c00, 0x0, 0x0, 0x6ef864, 0xc000386ac8)
    <autogenerated>:1 +0x98 fp=0xc01922f640 sp=0xc01922f5d0 pc=0x6fe718
github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).importKey(0xc000496d20, 0xc080653c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:460 +0x136 fp=0xc01922f6c8 sp=0xc01922f640 pc=0x6f7556
github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).ImportECDSA(0xc000496d20, 0xc07051bec0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:455 +0x192 fp=0xc01922f7b8 sp=0xc01922f6c8 pc=0x6f7342
github.com/ethereum/go-ethereum/internal/ethapi.(*PrivateAccountAPI).ImportRawKey(0xc023cd46e0, 0xc01fb33e80, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/internal/ethapi/api.go:305 +0x145 fp=0xc01922f8b8 sp=0xc01922f7b8 pc=0xabdef5
runtime.call128(0xc02a8742d0, 0xc010882180, 0xc056a38a50, 0x2800000050)
    /usr/lib/go-1.11/src/runtime/asm_amd64.s:524 +0x52 fp=0xc01922f948 sp=0xc01922f8b8 pc=0x470882
reflect.Value.call(0xc02ae5e900, 0xc010882180, 0x13, 0x10d1e77, 0x4, 0xc050562de0, 0x3, 0x4, 0xc01d4e0c88, 0x18, ...)
    /usr/lib/go-1.11/src/reflect/value.go:447 +0x454 fp=0xc01922fc10 sp=0xc01922f948 pc=0x4ced64
reflect.Value.Call(0xc02ae5e900, 0xc010882180, 0x13, 0xc050562de0, 0x3, 0x4, 0x2, 0x2, 0x0)
    /usr/lib/go-1.11/src/reflect/value.go:308 +0xa4 fp=0xc01922fc78 sp=0xc01922fc10 pc=0x4ce7f4
github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc02377cfc0, 0x137ec80, 0xc080653b40, 0xc0894d11e0, 0x15, 0xc07051bbc0, 0x2, 0x2, 0x0, 0x0, ...)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/service.go:209 +0x2cf fp=0xc01922fd58 sp=0xc01922fc78 pc=0x833aaf
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0xc09afb38c0, 0x137ec80, 0xc080653b40, 0xc03bc04540, 0xc02377cfc0, 0xc07051bbc0, 0x2, 0x2, 0x2)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/handler.go:369 +0x8a fp=0xc01922fdc8 sp=0xc01922fd58 pc=0x82cfca
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc09afb38c0, 0xc07051ba40, 0xc03bc04540, 0x88)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/handler.go:331 +0x1fc fp=0xc01922fe48 sp=0xc01922fdc8 pc=0x82c60c
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc09afb38c0, 0xc07051ba40, 0xc03bc04540, 0x88)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/handler.go:298 +0x1f4 fp=0xc01922ff10 sp=0xc01922fe48 pc=0x82bd14
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1(0xc07051ba40)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/handler.go:139 +0x46 fp=0xc01922ff80 sp=0xc01922ff10 pc=0x837b06
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1(0xc09afb38c0, 0xc006fb0980)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/handler.go:226 +0xdf fp=0xc01922ffd0 sp=0xc01922ff80 pc=0x837ccf
runtime.goexit()
    /usr/lib/go-1.11/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc01922ffd8 sp=0xc01922ffd0 pc=0x472481
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/handler.go:222 +0x66

goroutine 1 [chan receive, 51 minutes]:
github.com/ethereum/go-ethereum/node.(*Node).Wait(0xc000272000)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:492 +0x7d
main.geth(0xc000570580, 0x0, 0x0)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:312 +0x9e
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0xf31dc0, 0x11a0800, 0xc000570580, 0xc000570580, 0xc0004f5f58)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:490 +0xc8
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.(*App).Run(0xc000231520, 0xc0000301c0, 0x1c, 0x1c, 0x0, 0x0)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:264 +0x59d
main.main()
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:296 +0x55

goroutine 6 [chan receive]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc000098420, 0xc000098360)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:36 +0xb6
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:29 +0xdc

goroutine 7 [chan receive, 51 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc000098420, 0xc0000983c0)
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:81 +0x7b
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
    /build/ethereum-23hs5A/ethereum-1.9.2+build19708+bionic/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:30 +0x108

goroutine 8 [syscall, 51 minutes]:
os/signal.signal_recv(0x0)
    /usr/lib/go-1.11/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
    /usr/lib/go-1.11/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /usr/lib/go-1.11/src/os/signal/signal_unix.go:29 +0x41
...
picture flwwsg

Most helpful comment

Or are you stress testing the node yourself? An account decryption uses script, which is both time and memory hard, requiring about 1 second of CPU time and 256MB RAM. If you are hitting the node with concurrent decryption requests, you can easily overload it. Automated en-masse transaction sending should ideally use your own account management / key handling and signing. Alternatively perhaps check Clef.

picture karalabe on 29 Oct 2019
👍3

All 3 comments

You put your node on the public internet and opened the RPC port allowing anyone to send you malicious requests. That crash is caused by bots trying to brute force your password. Please immediately close all public access to your node. The RPC APIs are internal endpoints meant to be used only in trusted environments.

karalabe picture karalabe on 29 Oct 2019

Or are you stress testing the node yourself? An account decryption uses script, which is both time and memory hard, requiring about 1 second of CPU time and 256MB RAM. If you are hitting the node with concurrent decryption requests, you can easily overload it. Automated en-masse transaction sending should ideally use your own account management / key handling and signing. Alternatively perhaps check Clef.

karalabe picture karalabe on 29 Oct 2019
👍3

Or are you stress testing the node yourself? An account decryption uses script, which is both time and memory hard, requiring about 1 second of CPU time and 256MB RAM. If you are hitting the node with concurrent decryption requests, you can easily overload it. Automated en-masse transaction sending should ideally use your own account management / key handling and signing. Alternatively perhaps check Clef.

thinks, I will close port.

flwwsg picture flwwsg on 29 Oct 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

VenusHu picture VenusHu  路  3Comments
tymat picture tymat  路  3Comments
bgrieder picture bgrieder  路  3Comments
phpsamsb picture phpsamsb  路  3Comments
freshonline picture freshonline  路  3Comments