Victoriametrics: Errors on some queries - since 1.22.0

[EricAntoni]

Some queries that were working are failing on version >= 1.22.0.
I can provide the data used to reproduce if requested.

Here are some of the failing queries with associated error log

---

Query exemple 1 :
min by (item) (min_over_time(forms_daily_count[1d:1d]))

Logs :
2019-09-11T12:36:37.527+0000 panic app/vmselect/promql/eval.go:411 BUG: iafc must be nil for rollup "min_over_time" over subquery "default_rollup(forms_daily_count)[1d:1d]"
2019-09-11T12:36:37.527+0000 error /usr/local/go/src/net/http/server.go:3010 http: panic serving 127.0.0.1:45870: BUG: iafc must be nil for rollup "min_over_time" over subquery "default_rollup(forms_daily_count)[1d:1d]"
goroutine 800 [running]:
net/http.(conn).serve.func1(0xc000154460)
/usr/local/go/src/net/http/server.go:1769 +0x139
panic(0xa29bc0, 0xc000052f40)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage(0xab875f, 0x5, 0xc003ff8700, 0x69, 0x3)
/VictoriaMetrics/lib/logger/logger.go:124 +0x53d
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(0xab875f, 0x5, 0xad6ca3, 0x34, 0xc0008a53d8, 0x2, 0x2)
/VictoriaMetrics/lib/logger/logger.go:78 +0x113
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
/VictoriaMetrics/lib/logger/logger.go:62
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalRollupFunc(0xc004da3c00, 0xc00658354f, 0xd, 0xae4470, 0xc00515af00, 0xc004bd6b20, 0x0, 0x0, 0xc006582600, 0x40, ...)
/VictoriaMetrics/app/vmselect/promql/eval.go:411 +0x3d0
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalExpr(0xc004da3c00, 0xb7a6c0, 0xc0050d92c0, 0xc0050d92c0, 0x0, 0x0, 0xc006582640, 0x38)
/VictoriaMetrics/app/vmselect/promql/eval.go:212 +0xef3
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.Exec(0xc004da3c00, 0xc006582640, 0x38, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/VictoriaMetrics/app/vmselect/promql/exec.go:55 +0xc2
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus.QueryHandler(0xb84520, 0xc000020380, 0xc0000ee300, 0xd, 0x0)
/VictoriaMetrics/app/vmselect/prometheus/prometheus.go:431 +0x769
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect.RequestHandler(0xb84520, 0xc000020380, 0xc0000ee300, 0xc004a2cb00)
/VictoriaMetrics/app/vmselect/main.go:78 +0x4e5
main.requestHandler(0xb84520, 0xc000020380, 0xc0000ee300, 0x1169100)
/VictoriaMetrics/app/victoria-metrics/main.go:53 +0x6c
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xb84520, 0xc000020380, 0xc0000ee300, 0xae4ab0)
/VictoriaMetrics/lib/httpserver/httpserver.go:185 +0x1c6
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xb84520, 0xc000020380, 0xc0000ee300)
/VictoriaMetrics/lib/httpserver/httpserver.go:144 +0x88
net/http.HandlerFunc.ServeHTTP(0xc000114f60, 0xb84520, 0xc000020380, 0xc0000ee300)
/usr/local/go/src/net/http/server.go:1995 +0x44
net/http.serverHandler.ServeHTTP(0xc000786000, 0xb84520, 0xc000020380, 0xc0000ee300)
/usr/local/go/src/net/http/server.go:2774 +0xa8
net/http.(conn).serve(0xc000154460, 0xb85c60, 0xc005ea10c0)
/usr/local/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2884 +0x2f4

---

Query exemple 2 :
stddev_over_time( avg by (item) (avg_over_time(forms_daily_duration_avg_min[1d:1d]))[1d:1d])

Logs
2019-09-11T12:39:36.329+0000 panic app/vmselect/promql/eval.go:411 BUG: iafc must be nil for rollup "avg_over_time" over subquery "default_rollup(forms_daily_duration_avg_min)[1d:1d]"
2019-09-11T12:39:36.330+0000 error /usr/local/go/src/net/http/server.go:3010 http: panic serving 127.0.0.1:45928: BUG: iafc must be nil for rollup "avg_over_time" over subquery "default_rollup(forms_daily_duration_avg_min)[1d:1d]"
goroutine 844 [running]:
net/http.(conn).serve.func1(0xc000089180)
/usr/local/go/src/net/http/server.go:1769 +0x139
panic(0xa29bc0, 0xc000053080)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage(0xab875f, 0x5, 0xc000335380, 0x74, 0x3)
/VictoriaMetrics/lib/logger/logger.go:124 +0x53d
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(0xab875f, 0x5, 0xad6ca3, 0x34, 0xc005eace60, 0x2, 0x2)
/VictoriaMetrics/lib/logger/logger.go:78 +0x113
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
/VictoriaMetrics/lib/logger/logger.go:62
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalRollupFunc(0xc00543f7a0, 0xc00013b1c3, 0xd, 0xae43f8, 0xc005ea2460, 0xc0066b1480, 0x0, 0x0, 0x8020101, 0xc00013b1fa, ...)
/VictoriaMetrics/app/vmselect/promql/eval.go:411 +0x3d0
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalExpr(0xc00543f7a0, 0xb7a6c0, 0xc005ea23c0, 0x0, 0x0, 0x0, 0xc0001409a8, 0xc0000f8380)
/VictoriaMetrics/app/vmselect/promql/eval.go:212 +0xef3
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalRollupFuncWithSubquery(0xc00543f730, 0xc00013b1a0, 0x10, 0xae4480, 0xc005ea2410, 0xb80000c000052f50, 0xc005ea2370, 0x1, 0xc004382b70, 0x1)
/VictoriaMetrics/app/vmselect/promql/eval.go:460 +0x1a2
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalRollupFunc(0xc00543f730, 0xc00013b1a0, 0x10, 0xae4480, 0xc005ea2410, 0x0, 0x0, 0x0, 0x0, 0xc000052fc0, ...)
/VictoriaMetrics/app/vmselect/promql/eval.go:413 +0xba
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.evalExpr(0xc00543f730, 0xb7a700, 0xc004382cc0, 0xc004382cc0, 0x0, 0x0, 0xc00013b1a0, 0x5f)
/VictoriaMetrics/app/vmselect/promql/eval.go:191 +0x110a
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.Exec(0xc00543f730, 0xc00013b1a0, 0x5f, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/VictoriaMetrics/app/vmselect/promql/exec.go:55 +0xc2
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus.QueryHandler(0xb84520, 0xc004a407e0, 0xc0008a8500, 0xd, 0x0)
/VictoriaMetrics/app/vmselect/prometheus/prometheus.go:431 +0x769
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect.RequestHandler(0xb84520, 0xc004a407e0, 0xc0008a8500, 0xc004a30b00)
/VictoriaMetrics/app/vmselect/main.go:78 +0x4e5
main.requestHandler(0xb84520, 0xc004a407e0, 0xc0008a8500, 0x1169100)
/VictoriaMetrics/app/victoria-metrics/main.go:53 +0x6c
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xb84520, 0xc004a407e0, 0xc0008a8500, 0xae4ab0)
/VictoriaMetrics/lib/httpserver/httpserver.go:185 +0x1c6
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xb84520, 0xc004a407e0, 0xc0008a8500)
/VictoriaMetrics/lib/httpserver/httpserver.go:144 +0x88
net/http.HandlerFunc.ServeHTTP(0xc000114f60, 0xb84520, 0xc004a407e0, 0xc0008a8500)
/usr/local/go/src/net/http/server.go:1995 +0x44
net/http.serverHandler.ServeHTTP(0xc000786000, 0xb84520, 0xc004a407e0, 0xc0008a8500)
/usr/local/go/src/net/http/server.go:2774 +0xa8
net/http.(conn).serve(0xc000089180, 0xb85c60, 0xc0066ae0c0)
/usr/local/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2884 +0x2f4

Best regards

[valyala]

@EricAntoni , thanks for the bug report. The issue should be fixed in the commit https://github.com/VictoriaMetrics/VictoriaMetrics/commit/d4218d27e6121d4986c007fc41338b66588627f6 . Could you rebuild VictoriaMetrics from this commit according to these docs and verify whether the issue is fixed?

[valyala]

The issue should be fixed in v1.27.3. @EricAntoni , could you confirm this?

[EricAntoni]

version 1.27.3 fixed all the issues (even one I did not detect in the 1.21.2)
tks