Cockroach: sql: panic due to UPSERT encountering NULL in non-nullable column

[tbg]

https://sentry.io/cockroach-labs/cockroachdb/issues/771690722/?referrer=webhooks_plugin

conn_executor.go:670: panic while executing 1 statements: UPSERT INTO _(_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _) VALUES ($1, $2, __more50__): caused by

github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1

stacktrace: {u'frames': [{u'function': u'func4', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql/pgwire.(conn).serveImpl', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go', u'lineno': 309, u'in_app': True}, {u'function': u'ServeConn', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(Server)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go', u'lineno': 420, u'in_app': True}, {u'function': u'run', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(connExecutor)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go', u'lineno': 1200, u'in_app': True}, {u'function': u'execStmt', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(connExecutor)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'lineno': 95, u'in_app': True}, {u'function': u'execStmtInOpenState', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(connExecutor)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'lineno': 403, u'in_app': True}, {u'function': u'dispatchToExecutionEngine', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(connExecutor)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'lineno': 825, u'in_app': True}, {u'function': u'execWithDistSQLEngine', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(connExecutor)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go', u'lineno': 983, u'in_app': True}, {u'function': u'PlanAndRun', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(DistSQLPlanner)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go', u'lineno': 746, u'in_app': True}, {u'function': u'Run', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(DistSQLPlanner)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go', u'lineno': 253, u'in_app': True}, {u'function': u'StartSync', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(Flow)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go', u'lineno': 602, u'in_app': True}, {u'function': u'Run', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(ProcessorBase)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go', u'lineno': 730, u'in_app': True}, {u'function': u'Start', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(planNodeToRowSource)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go', u'lineno': 139, u'in_app': True}, {u'function': u'startExec', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/plan.go', u'lineno': 645, u'in_app': True}, {u'function': u'walkPlan', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'lineno': 65, u'in_app': True}, {u'function': u'visit', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(planVisitor)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'lineno': 101, u'in_app': True}, {u'function': u'visitInternal', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(planVisitor)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'lineno': 543, u'in_app': True}, {u'function': u'func1', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(planVisitor).visitInternal', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/walk.go', u'lineno': 134, u'in_app': True}, {u'function': u'func2', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.startExec', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/plan.go', u'lineno': 640, u'in_app': True}, {u'function': u'startExec', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(rowCountNode)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go', u'lineno': 173, u'in_app': True}, {u'function': u'BatchedNext', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/upsert.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(upsertNode)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/upsert.go', u'lineno': 331, u'in_app': True}, {u'function': u'atBatchEnd', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(tableUpserter)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go', u'lineno': 326, u'in_app': True}, {u'function': u'fetchExisting', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(tableUpserter)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go', u'lineno': 851, u'in_app': True}, {u'function': u'NextRowDecoded', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(RowFetcher)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go', u'lineno': 1011, u'in_app': True}, {u'function': u'NextRow', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(RowFetcher)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go', u'lineno': 997, u'in_app': True}, {u'function': u'finalizeRow', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(RowFetcher)', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go', u'lineno': 1247, u'in_app': True}, {u'function': u'gopanic', u'abs_path': u'/usr/local/go/src/runtime/panic.go', u'module': u'runtime', u'filename': u'runtime/panic.go', u'lineno': 502, u'in_app': False}, {u'function': u'call32', u'abs_path': u'/usr/local/go/src/runtime/asm_amd64.s', u'module': u'runtime', u'filename': u'runtime/asm_amd64.s', u'lineno': 573, u'in_app': False}, {u'function': u'func1', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go', u'module': u'github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn', u'filename': u'github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go', u'lineno': 418, u'in_app': True}]}
type: *log.safeError
value: conn_executor.go:670: panic while executing 1 statements: UPSERT INTO _(_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _) VALUES ($1, $2, __more50__): caused by

[jordanlewis]

@knz @BramGruneir

[knz]

*log.safeError: conn_executor.go:670: panic while executing 1 statements: UPSERT INTO _(_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _) VALUES ($1, $2, __more50__): caused by <redacted>
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go", line 418, in func1
  File "runtime/asm_amd64.s", line 573, in call32
  File "runtime/panic.go", line 502, in gopanic
  File "github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go", line 1247, in finalizeRow
  File "github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go", line 997, in NextRow
  File "github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go", line 1011, in NextRowDecoded
  File "github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go", line 851, in fetchExisting
  File "github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go", line 326, in atBatchEnd
  File "github.com/cockroachdb/cockroach/pkg/sql/upsert.go", line 331, in BatchedNext
  File "github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go", line 173, in startExec
  File "github.com/cockroachdb/cockroach/pkg/sql/plan.go", line 640, in func2
  File "github.com/cockroachdb/cockroach/pkg/sql/walk.go", line 134, in func1
  File "github.com/cockroachdb/cockroach/pkg/sql/walk.go", line 543, in visitInternal
  File "github.com/cockroachdb/cockroach/pkg/sql/walk.go", line 101, in visit
  File "github.com/cockroachdb/cockroach/pkg/sql/walk.go", line 65, in walkPlan
  File "github.com/cockroachdb/cockroach/pkg/sql/plan.go", line 645, in startExec
  File "github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go", line 139, in Start
  File "github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go", line 730, in Run
  File "github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go", line 602, in StartSync
  File "github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go", line 253, in Run
  File "github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go", line 746, in PlanAndRun
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go", line 983, in execWithDistSQLEngine
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go", line 825, in dispatchToExecutionEngine
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go", line 403, in execStmtInOpenState
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go", line 95, in execStmt
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go", line 1200, in run
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go", line 420, in ServeConn
  File "github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go", line 309, in func4

[knz]

panic is this:

        panic(fmt.Sprintf(
          "Non-nullable column \"%s:%s\" with no value! Index scanned was %q with the index key columns (%s) and the values (%s)",
          table.desc.Name, table.cols[i].Name, table.index.Name,
          strings.Join(table.index.ColumnNames, ","), strings.Join(indexColValues, ",")))

[knz]

This needs to be converted to an internal error instead of a panic.

[jordanlewis]

Agreed, but the root cause bug also needs to be fixed, whatever it is.

[jordanlewis]

And, for the record, if the panic was converted into an error, we likely wouldn't have seen the report.

[knz]

Internal errors are channeled to the reg cluter. We'll be data mining this soon.

[vivekmenezes]

I've been able to reproduce this problem. It's concerned with running more than one UPSERT statement on the same session during a schema change. The second UPSERT crashes because the first UPSERT doesn't clean up some state properly. I'll work on a solution and send it out for review sometime today