Cosmos-sdk: 504 Gateway Time-out error when querying tx for some addresses

Created on 30 Apr 2020 · 16Comments · Source: cosmos/cosmos-sdk

Summary of Bug

It seems that querying RPC for transactions doesn't work well for addresses with a high number of tx, triggering a 504 Gateway Time-out error response in RPC.

To reproduce:

https://api.cosmos.network/txs?transfer.recipient=cosmos1y6yvdel7zys8x60gz9067fjpcpygsn62ae9x46&limit=20&page=1

Version

Steps to Reproduce

https://api.cosmos.network/txs?transfer.recipient=cosmos1y6yvdel7zys8x60gz9067fjpcpygsn62ae9x46&limit=20&page=1

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

REST bug help wanted in progress

Source

mariopino

All 16 comments

Indeed there is something wrong with the server. /cc @helder-moreira

alexanderbez on 30 Apr 2020

@alexanderbez it results in a panic in gaia:

Apr 30 13:43:24 gaiad: I[2020-04-30|13:43:24.626] Committed state                              module=state height=1712543 txs=0 appHash=EC5733FAF7DBFD5D6EA31721703F7027BFFFF87E46311AECE08A7E3D9D6631AC
Apr 30 13:43:31 gaiad: I[2020-04-30|13:43:31.535] Executed block                               module=state height=1712544 validTxs=0 invalidTxs=0
Apr 30 13:43:31 gaiad: I[2020-04-30|13:43:31.666] Committed state                              module=state height=1712544 txs=0 appHash=0FF85D1DA5E30D18DDBC2954EFCC3FED89656305858DB98C0936B1F3901A1A57
Apr 30 13:43:38 gaiad: I[2020-04-30|13:43:38.516] Executed block                               module=state height=1712545 validTxs=0 invalidTxs=0
Apr 30 13:43:38 gaiad: I[2020-04-30|13:43:38.631] Committed state                              module=state height=1712545 txs=0 appHash=B541C600FAAFECA4313EC40307F550310C8C38A0E528A84B3A9AFD4EF291EEE3
Apr 30 13:43:45 gaiad: I[2020-04-30|13:43:45.941] Executed block                               module=state height=1712546 validTxs=0 invalidTxs=0
Apr 30 13:43:46 gaiad: I[2020-04-30|13:43:46.058] Committed state                              module=state height=1712546 txs=0 appHash=9E35C2C37B9E13CDF4B83F4654BE447ED574537015AC163697DCE4ED36782B21
Apr 30 13:43:53 gaiad: I[2020-04-30|13:43:53.367] Executed block                               module=state height=1712547 validTxs=0 invalidTxs=0
Apr 30 13:43:53 gaiad: I[2020-04-30|13:43:53.586] Committed state                              module=state height=1712547 txs=0 appHash=7A34E020F0DDA6011D9A95E8B08C94BBEB2760748082BC9DD947A673C364B355
Apr 30 13:43:57 gaiad: E[2020-04-30|13:43:57.144] Panic in RPC HTTP handler                    module=rpc-server err="write tcp 127.0.0.1:26657->127.0.0.1:41992: i/o timeout" stack="goroutine 2488388778 [running]:\nruntime/debug.Stack(0xc001867370, 0x10ac120, 0xc0921ce550)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1.1(0xc01195da60, 0x15aaaa0, 0xc000bbeee0, 0xbfa2d35a1e32cc35, 0x11652f5c776e77, 0x1ffa660, 0xc00c4e8f00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:161 +0x486\npanic(0x10ac120, 0xc0921ce550)\n\t/usr/local/go/src/runtime/panic.go:679 +0x1b2\ngithub.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseHTTP(0x15a50e0, 0xc01195da60, 0x11d0ad9, 0x3, 0x158d2a0, 0xc00fa56ae0, 0xc0497d8000, 0xe39c, 0xf022, 0x0)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:112 +0x281\ngithub.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseArrayHTTP(0x15a50e0, 0xc01195da60, 0xc00f08bf80, 0x1, 0x1)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:121 +0xca\ngithub.com/tendermint/tendermint/rpc/lib/server.makeJSONRPCHandler.func1(0x15a50e0, 0xc01195da60, 0xc00c4e8f00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:190 +0x49a\ngithub.com/tendermint/tendermint/rpc/lib/server.handleInvalidJSONRPCPaths.func1(0x15a50e0, 0xc01195da60, 0xc00c4e8f00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:204 +0x9c\nnet/http.HandlerFunc.ServeHTTP(0xc012c409d0, 0x15a50e0, 0xc01195da60, 0xc00c4e8f00)\n\t/usr/local/go/src/net/http/server.go:2007 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xc010716800, 0x15a50e0, 0xc01195da60, 0xc00c4e8f00)\n\t/usr/local/go/src/net/http/server.go:2387 +0x1bd\ngithub.com/tendermint/tendermint/rpc/lib/server.maxBytesHandler.ServeHTTP(0x158bc80, 0xc010716800, 0xf4240, 0x15a50e0, 0xc01195da60, 0xc00c4e8f00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:210 +0xcf\ngithub.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1(0x15a57e0, 0xc009baf7a0, 0xc00c4e8f00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:183 +0x33f\nnet/http.HandlerFunc.ServeHTTP(0xc001ba21e0, 0x15a57e0, 0xc009baf7a0, 0xc00c4e8f00)\n\t/usr/local/go/src/net/http/server.go:2007 +0x44\nnet/http.serverHandler.ServeHTTP(0xc0000dcee0, 0x15a57e0, 0xc009baf7a0, 0xc00c4e8f00)\n\t/usr/local/go/src/net/http/server.go:2802 +0xa4\nnet/http.(*conn).serve(0xc01548c640, 0x15aa0a0, 0xc0154a3080)\n\t/usr/local/go/src/net/http/server.go:1890 +0x875\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2928 +0x384\n"
Apr 30 13:43:57 gaiad: 2020/04/30 13:43:57 http: superfluous response.WriteHeader call from github.com/tendermint/tendermint/rpc/lib/server.(*ResponseWriterWrapper).WriteHeader (http_server.go:195)
Apr 30 13:43:57 gaiad: 2020/04/30 13:43:57 http: panic serving 127.0.0.1:41992: write tcp 127.0.0.1:26657->127.0.0.1:41992: i/o timeout
Apr 30 13:43:57 gaiad: goroutine 2488388778 [running]:
Apr 30 13:43:57 gaiad: net/http.(*conn).serve.func1(0xc01548c640)
Apr 30 13:43:57 gaiad: /usr/local/go/src/net/http/server.go:1767 +0x139
Apr 30 13:43:57 gaiad: panic(0x10ac120, 0xc0921ce550)
Apr 30 13:43:57 gaiad: /usr/local/go/src/runtime/panic.go:679 +0x1b2
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseHTTPError(0x15a50e0, 0xc01195da60, 0x1f4, 0x11d0ad9, 0x3, 0x158d2a0, 0x156f500, 0x0, 0x0, 0x0, ...)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:100 +0x285
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1.1(0xc01195da60, 0x15aaaa0, 0xc000bbeee0, 0xbfa2d35a1e32cc35, 0x11652f5c776e77, 0x1ffa660, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:163 +0x672
Apr 30 13:43:57 gaiad: panic(0x10ac120, 0xc0921ce550)
Apr 30 13:43:57 gaiad: /usr/local/go/src/runtime/panic.go:679 +0x1b2
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseHTTP(0x15a50e0, 0xc01195da60, 0x11d0ad9, 0x3, 0x158d2a0, 0xc00fa56ae0, 0xc0497d8000, 0xe39c, 0xf022, 0x0)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:112 +0x281
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseArrayHTTP(0x15a50e0, 0xc01195da60, 0xc00f08bf80, 0x1, 0x1)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:121 +0xca
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.makeJSONRPCHandler.func1(0x15a50e0, 0xc01195da60, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:190 +0x49a
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.handleInvalidJSONRPCPaths.func1(0x15a50e0, 0xc01195da60, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:204 +0x9c
Apr 30 13:43:57 gaiad: net/http.HandlerFunc.ServeHTTP(0xc012c409d0, 0x15a50e0, 0xc01195da60, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /usr/local/go/src/net/http/server.go:2007 +0x44
Apr 30 13:43:57 gaiad: net/http.(*ServeMux).ServeHTTP(0xc010716800, 0x15a50e0, 0xc01195da60, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /usr/local/go/src/net/http/server.go:2387 +0x1bd
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.maxBytesHandler.ServeHTTP(0x158bc80, 0xc010716800, 0xf4240, 0x15a50e0, 0xc01195da60, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:210 +0xcf
Apr 30 13:43:57 gaiad: github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1(0x15a57e0, 0xc009baf7a0, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:183 +0x33f
Apr 30 13:43:57 gaiad: net/http.HandlerFunc.ServeHTTP(0xc001ba21e0, 0x15a57e0, 0xc009baf7a0, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /usr/local/go/src/net/http/server.go:2007 +0x44
Apr 30 13:43:57 gaiad: net/http.serverHandler.ServeHTTP(0xc0000dcee0, 0x15a57e0, 0xc009baf7a0, 0xc00c4e8f00)
Apr 30 13:43:57 gaiad: /usr/local/go/src/net/http/server.go:2802 +0xa4
Apr 30 13:43:57 gaiad: net/http.(*conn).serve(0xc01548c640, 0x15aa0a0, 0xc0154a3080)
Apr 30 13:43:57 gaiad: /usr/local/go/src/net/http/server.go:1890 +0x875
Apr 30 13:43:57 gaiad: created by net/http.(*Server).Serve
Apr 30 13:43:57 gaiad: /usr/local/go/src/net/http/server.go:2928 +0x384
Apr 30 13:44:00 gaiad: I[2020-04-30|13:44:00.155] Executed block                               module=state height=1712548 validTxs=0 invalidTxs=0
Apr 30 13:44:00 gaiad: I[2020-04-30|13:44:00.208] Committed state                              module=state height=1712548 txs=0 appHash=F63B440B9A3B83D9D744574357D99499ABD83E37F4457BBCF4A92B8EE2D49D4D
Apr 30 13:44:02 gaiad: E[2020-04-30|13:44:02.845] Panic in RPC HTTP handler                    module=rpc-server err="write tcp 127.0.0.1:26657->127.0.0.1:42242: i/o timeout" stack="goroutine 2488399132 [running]:\nruntime/debug.Stack(0xc001df7370, 0x10ac120, 0xc026fa6e10)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1.1(0xc009c88860, 0x15aaaa0, 0xc000bbeee0, 0xbfa2d35bd081b913, 0x116530f001e169, 0x1ffa660, 0xc00b80ea00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:161 +0x486\npanic(0x10ac120, 0xc026fa6e10)\n\t/usr/local/go/src/runtime/panic.go:679 +0x1b2\ngithub.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseHTTP(0x15a50e0, 0xc009c88860, 0x11d0ad9, 0x3, 0x158d2a0, 0xc0588e0c00, 0xc0b9d8c000, 0xe39c, 0xf022, 0x0)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:112 +0x281\ngithub.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseArrayHTTP(0x15a50e0, 0xc009c88860, 0xc0b9629e80, 0x1, 0x1)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:121 +0xca\ngithub.com/tendermint/tendermint/rpc/lib/server.makeJSONRPCHandler.func1(0x15a50e0, 0xc009c88860, 0xc00b80ea00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:190 +0x49a\ngithub.com/tendermint/tendermint/rpc/lib/server.handleInvalidJSONRPCPaths.func1(0x15a50e0, 0xc009c88860, 0xc00b80ea00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:204 +0x9c\nnet/http.HandlerFunc.ServeHTTP(0xc012c409d0, 0x15a50e0, 0xc009c88860, 0xc00b80ea00)\n\t/usr/local/go/src/net/http/server.go:2007 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xc010716800, 0x15a50e0, 0xc009c88860, 0xc00b80ea00)\n\t/usr/local/go/src/net/http/server.go:2387 +0x1bd\ngithub.com/tendermint/tendermint/rpc/lib/server.maxBytesHandler.ServeHTTP(0x158bc80, 0xc010716800, 0xf4240, 0x15a50e0, 0xc009c88860, 0xc00b80ea00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:210 +0xcf\ngithub.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1(0x15a57e0, 0xc0010e6000, 0xc00b80ea00)\n\t/home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:183 +0x33f\nnet/http.HandlerFunc.ServeHTTP(0xc001ba21e0, 0x15a57e0, 0xc0010e6000, 0xc00b80ea00)\n\t/usr/local/go/src/net/http/server.go:2007 +0x44\nnet/http.serverHandler.ServeHTTP(0xc0000dcee0, 0x15a57e0, 0xc0010e6000, 0xc00b80ea00)\n\t/usr/local/go/src/net/http/server.go:2802 +0xa4\nnet/http.(*conn).serve(0xc01ee6f2c0, 0x15aa0a0, 0xc05b04c7c0)\n\t/usr/local/go/src/net/http/server.go:1890 +0x875\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2928 +0x384\n"
Apr 30 13:44:02 gaiad: 2020/04/30 13:44:02 http: superfluous response.WriteHeader call from github.com/tendermint/tendermint/rpc/lib/server.(*ResponseWriterWrapper).WriteHeader (http_server.go:195)
Apr 30 13:44:02 gaiad: 2020/04/30 13:44:02 http: panic serving 127.0.0.1:42242: write tcp 127.0.0.1:26657->127.0.0.1:42242: i/o timeout
Apr 30 13:44:02 gaiad: goroutine 2488399132 [running]:
Apr 30 13:44:02 gaiad: net/http.(*conn).serve.func1(0xc01ee6f2c0)
Apr 30 13:44:02 gaiad: /usr/local/go/src/net/http/server.go:1767 +0x139
Apr 30 13:44:02 gaiad: panic(0x10ac120, 0xc026fa6e10)
Apr 30 13:44:02 gaiad: /usr/local/go/src/runtime/panic.go:679 +0x1b2
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseHTTPError(0x15a50e0, 0xc009c88860, 0x1f4, 0x11d0ad9, 0x3, 0x158d2a0, 0x156f500, 0x0, 0x0, 0x0, ...)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:100 +0x285
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1.1(0xc009c88860, 0x15aaaa0, 0xc000bbeee0, 0xbfa2d35bd081b913, 0x116530f001e169, 0x1ffa660, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:163 +0x672
Apr 30 13:44:02 gaiad: panic(0x10ac120, 0xc026fa6e10)
Apr 30 13:44:02 gaiad: /usr/local/go/src/runtime/panic.go:679 +0x1b2
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseHTTP(0x15a50e0, 0xc009c88860, 0x11d0ad9, 0x3, 0x158d2a0, 0xc0588e0c00, 0xc0b9d8c000, 0xe39c, 0xf022, 0x0)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:112 +0x281
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.WriteRPCResponseArrayHTTP(0x15a50e0, 0xc009c88860, 0xc0b9629e80, 0x1, 0x1)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:121 +0xca
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.makeJSONRPCHandler.func1(0x15a50e0, 0xc009c88860, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:190 +0x49a
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.handleInvalidJSONRPCPaths.func1(0x15a50e0, 0xc009c88860, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/handlers.go:204 +0x9c
Apr 30 13:44:02 gaiad: net/http.HandlerFunc.ServeHTTP(0xc012c409d0, 0x15a50e0, 0xc009c88860, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /usr/local/go/src/net/http/server.go:2007 +0x44
Apr 30 13:44:02 gaiad: net/http.(*ServeMux).ServeHTTP(0xc010716800, 0x15a50e0, 0xc009c88860, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /usr/local/go/src/net/http/server.go:2387 +0x1bd
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.maxBytesHandler.ServeHTTP(0x158bc80, 0xc010716800, 0xf4240, 0x15a50e0, 0xc009c88860, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:210 +0xcf
Apr 30 13:44:02 gaiad: github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1(0x15a57e0, 0xc0010e6000, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /home/centos/go/pkg/mod/github.com/tendermint/[email protected]/rpc/lib/server/http_server.go:183 +0x33f
Apr 30 13:44:02 gaiad: net/http.HandlerFunc.ServeHTTP(0xc001ba21e0, 0x15a57e0, 0xc0010e6000, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /usr/local/go/src/net/http/server.go:2007 +0x44
Apr 30 13:44:02 gaiad: net/http.serverHandler.ServeHTTP(0xc0000dcee0, 0x15a57e0, 0xc0010e6000, 0xc00b80ea00)
Apr 30 13:44:02 gaiad: /usr/local/go/src/net/http/server.go:2802 +0xa4
Apr 30 13:44:02 gaiad: net/http.(*conn).serve(0xc01ee6f2c0, 0x15aa0a0, 0xc05b04c7c0)
Apr 30 13:44:02 gaiad: /usr/local/go/src/net/http/server.go:1890 +0x875
Apr 30 13:44:02 gaiad: created by net/http.(*Server).Serve
Apr 30 13:44:02 gaiad: /usr/local/go/src/net/http/server.go:2928 +0x384

I am not sure who is the best person to deal with this.

helder-moreira on 30 Apr 2020

@helder-moreira what are the resources like on that machine? Can you try restarting the service? Also, the Tendermint RPC node it's connected to, is that operating correctly and is healthy? The stacktrace indicates that it's an IO timeout. It doesn't look software related, looks infra related.

alexanderbez on 30 Apr 2020

@alexanderbez its a t3a.large machine (2 vCPUs, 8GB memory), and it is using around 45% of CPU and 21% of memory. There is also about 25% disk space available.

I have restarted both gaiad and gaiacli services, but same panic happens.

helder-moreira on 30 Apr 2020

👍1

Apr 30 13:44:02 gaiad: 2020/04/30 13:44:02 http: panic serving 127.0.0.1:42242: write tcp 127.0.0.1:26657->127.0.0.1:42242: i/o timeout

Is there nginx or another proxy in the mix? Maybe that could be the culprit?

alexanderbez on 30 Apr 2020

There is an nginx on that node yes, with this config. However, nothing from that config could be causing the issue I think.

I have just tried to run that request locally, and it also does not work:

$ curl 'http://localhost:1317/txs?transfer.recipient=cosmos1y6yvdel7zys8x60gz9067fjpcpygsn62ae9x46&limit=20&page=1'
curl: (52) Empty reply from server

helder-moreira on 30 Apr 2020

does not this look more like a problem into tendermint?

The panic is in this call:

https://github.com/tendermint/tendermint/blob/bc572217c07b90ad9cee851f193aaa8e9557cbc7/rpc/lib/server/http_server.go#L112

Probably it overflows or something like that that generates the timeout.

jgimeno on 6 May 2020

does not this look more like a problem into tendermint?

The panic is in this call:

https://github.com/tendermint/tendermint/blob/bc572217c07b90ad9cee851f193aaa8e9557cbc7/rpc/lib/server/http_server.go#L112

Probably it overflows or something like that that generates the timeout.

Maybe a size limit issue? From same file MaxBodyBytes is 1MB:

https://github.com/tendermint/tendermint/blob/bc572217c07b90ad9cee851f193aaa8e9557cbc7/rpc/lib/server/http_server.go#L42

mariopino on 6 May 2020

Yeah, the body could be too large, but what in the world are we responding with that's more than 1MB? In any case, we should allow the RPC server Config to be configured completely by the operator.

alexanderbez on 6 May 2020

Also, in this case, OP stated the following request was being used: /txs?transfer.recipient=cosmos1y6yvdel7zys8x60gz9067fjpcpygsn62ae9x46&limit=20&page=1. This request is paginated and results in 20 txs at most. There is no way that results in 1MB.

However, it could very well be since this RPC service is public, that the given request is not what causes this panic(s), but something else.

alexanderbez on 6 May 2020

👍1

I don't think the problem is related to the server, I can reproduce the same behaviour in other private nodes. That query gives the same timeout in every node I tested.

mariopino on 13 May 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] on 5 Jul 2020

@alexanderbez @helder-moreira Is there any update on this issue? I have tried to run rest-server to serve /txs and face this error log too.

taobun on 6 Aug 2020

I don't have any clues here apart from the response body size being too large, which is now configurable on master.

alexanderbez on 6 Aug 2020

Is this blocking a Stargate release?

robert-zaremba on 2 Dec 2020

We could try the equivalent endpoint on grpc-gateway, also with a large body size. If that doesn't 504, i propose to just close this issue (and not fix the legacy endpoint).

cc @anilCSE could you try this?