Cockroach: sql: v20.1.0: assertion failed in ProcOutputHelper.Init: cannot determine type of empty array

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/1682100339/?referrer=webhooks_plugin

Panic message:

barriers.barrierError: -- details for github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgerror/pgerror.withCandidateCode::: (1)
*errutil.withMessage
*safedetails.withSafeDetails: %s (2)
expr.go:78: *withstack.withStack (3)
*assert.withAssertionFailure
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/execinfra/expr.go#L77-L79 in pkg/sql/execinfra.processExpression
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/execinfra/expr.go#L160-L162 in pkg/sql/execinfra.(ExprHelper).Init
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/execinfra/processorsbase.go#L107-L109 in pkg/sql/execinfra.(ProcOutputHelper).Init
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/execinfra/processorsbase.go#L799-L801 in pkg/sql/execinfra.(ProcessorBase).InitWithEvalCtx
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/execinfra/processorsbase.go#L775-L777 in pkg/sql/execinfra.(ProcessorBase).Init
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/rowexec/tablereader.go#L93-L95 in pkg/sql/rowexec.newTableReader
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/rowexec/processors.go#L137-L139 in pkg/sql/rowexec.NewProcessor
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/rowflow/row_based_flow.go#L227-L229 in pkg/sql/rowflow.(rowBasedFlow).makeProcessor
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/rowflow/row_based_flow.go#L80-L82 in pkg/sql/rowflow.(rowBasedFlow).setupProcessors
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/rowflow/row_based_flow.go#L63-L65 in pkg/sql/rowflow.(rowBasedFlow).Setup
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/distsql/server.go#L351-L353 in pkg/sql/distsql.(ServerImpl).setupFlow
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/distsql/server.go#L465-L467 in pkg/sql/distsql.(ServerImpl).SetupLocalSyncFlow
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/distsql_running.go#L255-L257 in pkg/sql.(DistSQLPlanner).setupFlows
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/distsql_running.go#L383-L385 in pkg/sql.(DistSQLPlanner).Run
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/distsql_running.go#L1018-L1020 in pkg/sql.(DistSQLPlanner).PlanAndRun
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/conn_executor_exec.go#L882-L884 in pkg/sql.(connExecutor).execWithDistSQLEngine
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/conn_executor_exec.go#L775-L777 in pkg/sql.(connExecutor).dispatchToExecutionEngine
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/conn_executor_exec.go#L480-L482 in pkg/sql.(connExecutor).execStmtInOpenState
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/conn_executor_exec.go#L95-L97 in pkg/sql.(connExecutor).execStmt
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/conn_executor.go#L1429-L1431 in pkg/sql.(connExecutor).execCmd
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/conn_executor.go#L1296-L1298 in pkg/sql.(connExecutor).run
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/conn_executor.go#L475-L477 in pkg/sql.(Server).ServeConn
https://github.com/cockroachdb/cockroach/blob/9d456b9ec82cbf9a740a092c0d9f56da48779689/pkg/sql/pgwire/conn.go#L589-L591 in pkg/sql/pgwire.(conn).processCommandsAsync.func1
/usr/local/go/src/runtime/asm_amd64.s#L1356-L1358 in runtime.goexit

pkg/sql/execinfra/expr.go in pkg/sql/execinfra.processExpression at line 78
pkg/sql/execinfra/expr.go in pkg/sql/execinfra.(*ExprHelper).Init at line 161
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcOutputHelper).Init at line 108
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBase).InitWithEvalCtx at line 800
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBase).Init at line 776
pkg/sql/rowexec/tablereader.go in pkg/sql/rowexec.newTableReader at line 94
pkg/sql/rowexec/processors.go in pkg/sql/rowexec.NewProcessor at line 138
pkg/sql/rowflow/row_based_flow.go in pkg/sql/rowflow.(*rowBasedFlow).makeProcessor at line 228
pkg/sql/rowflow/row_based_flow.go in pkg/sql/rowflow.(*rowBasedFlow).setupProcessors at line 81
pkg/sql/rowflow/row_based_flow.go in pkg/sql/rowflow.(*rowBasedFlow).Setup at line 64
pkg/sql/distsql/server.go in pkg/sql/distsql.(*ServerImpl).setupFlow at line 352
pkg/sql/distsql/server.go in pkg/sql/distsql.(*ServerImpl).SetupLocalSyncFlow at line 466
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).setupFlows at line 256
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 384
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1019
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 883
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 776
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 481
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 96
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1430
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1297
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 476
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 590
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1357

| Tag | Value |
|---|---|
| Cockroach Release | v20.1.0 |
| Cockroach SHA: | 9d456b9ec82cbf9a740a092c0d9f56da48779689 |
| Platform | windows amd64 |
| Distribution | CCL |
| Environment | v20.1.0 |
| Command | server |
| Go Version | go1.13.9|
| # of CPUs |8 |
| # of Goroutines |191 |

[asubiotto]

cannot determine type of empty array. Consider annotating with the desired type, for example ARRAY[]:::int[]

[asubiotto]

@rohany do you have any insight into this? I would expect there to be an error during type-checking. @yuzefovich thinks it might be related to losing information during expression serialization.

[rohany]

Yeah, it seems we somehow got an expression sent to a remote node that had an empty unannotated array. From the query it seems like the user was doing an inverted index containment query -- maybe thats a good place to start trying a repro?

I would expect there to be an error during type-checking.

Yeah, this is correct. We seem to have lost the information when serializing expressions.

[yuzefovich]

Hm, it seems that my suspicion that we do expression serialization on all nodes, including the gateway, if the flow is distributed: in MakeExpression we call ctx.IsLocal() to decide whether we should serialize the expression, and that call returns true only when PlanningCtx.isLocal is true which is the case only when the flow is not distributed. So although the error occurred on the gateway, it is likely we have serialized and deserialized the expression (I don't think we can get the information whether the query was distributed from the sentry report, can we?).

On an unrelated note, I think we could be smarter and avoid the serialization on the gateway node, but it doesn't seem trivial though, so I'll open up an issue for that.

I tried reproducing the error with the given statement signature SELECT * FROM _ WHERE _ <@ _ ORDER BY _ on 3 node local cluster (with data living on every node), but the problem didn't occur.

At this point I would deem this issue as unactionable and close it. Thoughts?

[rohany]

I thought we had some sort of LocalExpr thing in distsql, where we only serialize/deserialize expressions sent over the wire

[yuzefovich]

We do have that, but it happens only if the flow as a whole is not distributed.

[asubiotto]

SGTM