Influxdb: nil pointer dereference with certain SHOW TAG VALUES qeuries

[e-dard]

Certain types of SHOW TAG VALUES queries, against certain types of schema, appear to be allowing either measurements without fields, or an empty set of indexes in an IndexSet. If a WHERE is present in the query, then seriesByBinaryExprIterator is called, and attempting to lookup the fields associated with a nil MeasurementFields results in a panic.

See comment here: https://github.com/influxdata/influxdb/issues/8786#issuecomment-371916894

SHOW TAG VALUES WITH KEY = room WHERE area =~ /^OG$/ [panic:runtime error: invalid memory address or nil pointer dereference] goroutine 133184 [running]:
runtime/debug.Stack(0xc43da1fda0, 0xc423bb0000, 0x34)
    /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
github.com/influxdata/influxdb/query.(*QueryExecutor).recover(0xc4201510b0, 0xc43da1fda0, 0xc42b42d860)
    /go/src/github.com/influxdata/influxdb/query/query_executor.go:451 +0xaf
panic(0xce4120, 0x1302970)
    /usr/local/go/src/runtime/panic.go:491 +0x283
github.com/influxdata/influxdb/tsdb.(*MeasurementFields).HasField(0x0, 0xc43da2774c, 0x4, 0xc4202f2910)
    /go/src/github.com/influxdata/influxdb/tsdb/shard.go:1408 +0x26
github.com/influxdata/influxdb/tsdb.IndexSet.seriesByBinaryExprIterator(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0xc421ac5ec8, 0x7, 0x8, 0xc42e1b44b0, 0x0, 0xd3ac00, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:1833 +0xdd2
github.com/influxdata/influxdb/tsdb.IndexSet.seriesByExprIterator(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0xc421ac5ec8, 0x7, 0x8, 0x12cae00, 0xc42e1b44b0, 0x0, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:1786 +0x476
github.com/influxdata/influxdb/tsdb.IndexSet.tagValuesByKeyAndExpr(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0x7fc5cdfc93a8, 0xc420184120, 0xc421ac5ec8, 0x7, 0x8, 0xc43150e310, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:2144 +0x1a0
github.com/influxdata/influxdb/tsdb.IndexSet.MeasurementTagKeyValuesByExpr(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0x7fc5cdfc93a8, 0xc420184120, 0xc421ac5ec8, 0x7, 0x8, 0xc43150e310, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:2293 +0xca1
github.com/influxdata/influxdb/tsdb.(*Store).TagValues(0xc420001980, 0x7fc5cdfc93a8, 0xc420184120, 0xc420b6b800, 0x99, 0x100, 0x12cae00, 0xc42e1b4360, 0x100, 0x400, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/store.go:1517 +0x946
github.com/influxdata/influxdb/coordinator.LocalTSDBStore.TagValues(0xc420001980, 0x7fc5cdfc93a8, 0xc420184120, 0xc420b6b800, 0x99, 0x100, 0x12cae00, 0xc42e1b4360, 0x32f2d7fe, 0x109d537404, ...)
    <autogenerated>:1 +0x97
github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeShowTagValues(0xc4201953b0, 0xc4300bca80, 0xc42153dc48, 0xdf7588, 0xf)
    /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:1066 +0x840
github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement(0xc4201953b0, 0x12ce780, 0xc4300bca80, 0x0, 0x14e, 0xc42e8cdf80, 0xc42b42d860, 0xc42b42d8c0, 0xc43da23e3e, 0x6, ...)
    /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:194 +0x2937
github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery(0xc4201510b0, 0xc43da1fda0, 0xc43da23e3e, 0x6, 0x7fc5cdfc93a8, 0xc420184120, 0x2710, 0x1, 0x0, 0x0, ...)
    /go/src/github.com/influxdata/influxdb/query/query_executor.go:389 +0xae8
created by github.com/influxdata/influxdb/query.(*QueryExecutor).ExecuteQuery
    /go/src/github.com/influxdata/influxdb/query/query_executor.go:285 +0xc3

Also seen in #9539.

[e-dard]

I have narrowed this down to fieldset.Fields(name) returning a nil MeasurementFieldSet for name here: https://github.com/influxdata/influxdb/blob/master/tsdb/index.go#L2144

Doesn't appear that there are any other issues higher up the stack, and the position of the nil pointers in the stack traces I've seen supports that.

[e-dard]

github.com/influxdata/influxdb/tsdb.IndexSet.seriesByExprIterator(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0xc421ac5ec8, 0x7, 0x8, 0x12cae00, 0xc42e1b44b0, 0x0, ...)

takes as the three arguments the pointer to the slice of tsdb.Indexes as well as its length and capacity. Then we have the SeriesFile pointer from the IndexSet. After that we have 0xc421ac5ec8, 0x7, 0x8, which suggests the name value has a length of 7 bytes.

This narrows the issue down further to a measurement for which there is a missing MeasurementFieldSet...

[e-dard]

Repro'd. When de-duplicating an inmem index during query time we only use the fields.idx from the first shard, which may not contain measurements from other shards.

[adamehirsch]

We're seeing much the same problem in 1.5.0 via the latest Docker image.

SHOW TAG VALUES WITH KEY = "page" WHERE "group" =~ /./
ERR: SHOW TAG VALUES WITH KEY = page WHERE "group" =~ /./ [panic:runtime error: invalid memory address or nil pointer dereference]

[e-dard]

@adamehirsch the fix is up for review. It should be in master soon, and in a 1.5.1 release.

[prasadasokan]

We upgraded to v1.5.2 and still see the error.

ts=2018-04-18T14:12:46.052431Z lvl=info msg="Executing query" log_id=079IFdql000 service=query query="SHOW TAG VALUES ON telegraf WITH KEY = assetType WHERE (_name = 'AssetObjectFileSize') AND ((environment = PROD) AND (_tagKey = 'assetType'))"
ts=2018-04-18T14:12:46.053771Z lvl=error msg="SHOW TAG VALUES FROM AssetObjectFileSize WITH KEY = assetType WHERE environment = PROD [panic:runtime error: invalid memory address or nil pointer dereference] goroutine 245325461 [running]:nruntime/debug.Stack(0xc458b68660, 0xc468df4780, 0x56)nt/usr/local/go/src/runtime/debug/stack.go:24 +0xa7ngithub.com/influxdata/influxdb/query.(QueryExecutor).recover(0xc420210690, 0xc458b68660, 0xc46f927f20)nt/go/src/github.com/influxdata/influxdb/query/query_executor.go:451 +0xafnpanic(0xce4120, 0x1302970)nt/usr/local/go/src/runtime/panic.go:491 +0x283ngithub.com/influxdata/influxdb/tsdb.(MeasurementFields).HasField(0x0, 0xc4522b2270, 0xb, 0xc423ff4680)nt/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1408 +0x26ngithub.com/influxdata/influxdb/tsdb.IndexSet.seriesByBinaryExprIterator(0xc425162d80, 0x6, 0x21, 0xc4201572c0, 0xc450ca5a04, 0x13, 0xc6, 0xc4588ab8c0, 0x0, 0x6600000000d4d240, ...)nt/go/src/github.com/influxdata/influxdb/tsdb/index.go:1833 +0xdd2ngithub.com/influxdata/influxdb/tsdb.IndexSet.seriesByExprIterator(0xc425162d80, 0x6, 0x21, 0xc4201572c0, 0xc450ca5a04, 0x13, 0xc6, 0x12cae00, 0xc4588ab8c0, 0x0, ...)nt/go/src/github.com/influxdata/influxdb/tsdb/index.go:1786 +0x476ngithub.com/influxdata/influxdb/tsdb.IndexSet.tagValuesByKeyAndExpr(0xc425162d80, 0x6, 0x21, 0xc4201572c0, 0x12ce9c0, 0x1336238, 0xc450ca5a04, 0x13, 0xc6, 0xc451fbf6b0, ...)nt/go/src/github.com/influxdata/influxdb/tsdb/index.go:2144 +0x1a0ngithub.com/influxdata/influxdb/tsdb.IndexSet.MeasurementTagKeyValuesByExpr(0xc425162d80, 0x6, 0x21, 0xc4201572c0, 0x12ce9c0, 0x1336238, 0xc450ca5a04, 0x13, 0xc6, 0xc451fbf6b0, ...)nt/go/src/github.com/influxdata/influxdb/tsdb/index.go:2293 +0xca1ngithub.com/influxdata/influxdb/tsdb.(Store).TagValues(0xc4201f2180, 0x12ce9c0, 0x1336238, 0xc436b77400, 0x21, 0x40, 0x12cae00, 0xc4588ab680, 0x40, 0x100, ...)nt/go/src/github.com/influxdata/influxdb/tsdb/store.go:1517 +0x946ngithub.com/influxdata/influxdb/coordinator.LocalTSDBStore.TagValues(0xc4201f2180, 0x12ce9c0, 0x1336238, 0xc436b77400, 0x21, 0x40, 0x12cae00, 0xc4588ab680, 0x32f2d7fe, 0x109d537404, ...)nt:1 +0x97ngithub.com/influxdata/influxdb/coordinator.(StatementExecutor).executeShowTagValues(0xc42021d490, 0xc45a775e80, 0xc4435e7c48, 0xdf7588, 0xf)nt/go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:1066 +0x840ngithub.com/influxdata/influxdb/coordinator.(StatementExecutor).ExecuteStatement(0xc42021d490, 0x12ce780, 0xc45a775e80, 0x0, 0x3557f, 0xc45a66d960, 0xc46f927f20, 0xc468df4660, 0xc43d08c0bc, 0x8, ...)nt/go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:194 +0x2937ngithub.com/influxdata/influxdb/query.(QueryExecutor).executeQuery(0xc420210690, 0xc458b68660, 0xc43d08c0bc, 0x8, 0x12ce9c0, 0x1336238, 0x2710, 0x0, 0x0, 0x0, ...)nt/go/src/github.com/influxdata/influxdb/query/query_executor.go:389 +0xae8ncreated by github.com/influxdata/influxdb/query.(*QueryExecutor).ExecuteQuerynt/go/src/github.com/influxdata/influxdb/query/query_executor.go:285 +0xc3n" log_id=079IFdql000 service=query

[e-dard]

@prasadasokan from that stacktrace it looks like you're running v1.5.0 not v1.5.2.

[prasadasokan]

Edd,

Sorry for the false alarm. After the upgrade, we failed to restart the
service.

Regards,
Prasad Asokan.

Prasad Asokan |

Voice: [phone] | Skype / Twitter / LinkedIn / Facebook / g+ :
prasadasokan |

On 19 April 2018 at 16:03, Edd Robinson [e-mail] wrote:

@prasadasokan https://github.com/prasadasokan from that stacktrace it
looks like you're running v1.5.0 not v1.5.2.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/influxdata/influxdb/issues/9548#issuecomment-382688409,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AgVgvT4iF0kMo28cgpw4GbK_5KlO6rRlks5tqGgEgaJpZM4SmZo6
.

[kalidasya]

I am using 1.6.2 and I still get invalid memory address or nul pointer dereference error for certain regex selects. What am I missing? And I do not see any stacktrace logged (checking the journal)

[e-dard]

@kalidasya can you log a new issue with reproduction steps please?

[kalidasya]

@e-dard sure can u tell me how to get the stacktraces? nothing in the journal, would be nice if I can attach that. Also is there a way to get the influx version from a running instance? a special query or something

[e-dard]

@kalidasya try running with GOTRACEBACK=all If that doesn't help then try GOTRACEBACK=crash, but note that running with crash will cause Influx to exit the first time the panic is hit.