Influxdb: panic: runtime error: invalid memory address or nil pointer dereference (PointsWriter)

[Apollon77]

Hi,

I had a crash of the InfluxDB process (1.3.4) on ubuntu 16.04 on amd64

Installed version:
Sep 4 16:38:09 nuc1 influxd[19775]: [I] 2017-09-04T14:38:09Z InfluxDB starting, version 1.3.4, branch HEAD, commit 0c54d8fc744220b8dc70f2987e04e78fec951285

Process was restarted. Only happend once till now.

Sep  4 16:38:09 nuc1 influxd[1670]: panic: runtime error: invalid memory address or nil pointer dereference
Sep  4 16:38:09 nuc1 influxd[1670]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xb8 pc=0x86f9a3]
Sep  4 16:38:09 nuc1 influxd[1670]: goroutine 1783 [running]:
Sep  4 16:38:09 nuc1 influxd[1670]: github.com/influxdata/influxdb/coordinator.(*PointsWriter).MapShards(0xc42005f700, 0xc4205c7c68, 0x8, 0x
c44395ff40, 0x4ed23b430)
Sep  4 16:38:09 nuc1 influxd[1670]: #011/root/go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:210 +0x1d3
Sep  4 16:38:09 nuc1 influxd[1670]: github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged(0xc42005f700, 0xb8b250,
 0x8, 0xc42016d338, 0x7, 0x0, 0xc439e1e000, 0x3e8, 0x3e8, 0x0, ...)
Sep  4 16:38:09 nuc1 influxd[1670]: #011/root/go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:305 +0x13c
Sep  4 16:38:09 nuc1 influxd[1670]: github.com/influxdata/influxdb/services/collectd.(*Service).writePoints(0xc4200c2600)
Sep  4 16:38:09 nuc1 influxd[1670]: #011/root/go/src/github.com/influxdata/influxdb/services/collectd/service.go:377 +0x32d
Sep  4 16:38:09 nuc1 influxd[1670]: github.com/influxdata/influxdb/services/collectd.(*Service).Open.func3(0xc4200c2600)
Sep  4 16:38:09 nuc1 influxd[1670]: #011/root/go/src/github.com/influxdata/influxdb/services/collectd/service.go:207 +0x57
Sep  4 16:38:09 nuc1 influxd[1670]: created by github.com/influxdata/influxdb/services/collectd.(*Service).Open
Sep  4 16:38:09 nuc1 influxd[1670]: #011/root/go/src/github.com/influxdata/influxdb/services/collectd/service.go:207 +0x78d

[timhallinflux]

You might want to grab 1.3.5 -- https://github.com/influxdata/influxdb/blob/1.3/CHANGELOG.md
Couple of issues like that were resolved.

[e-dard]

@Apollon77 did you have any success with a more recent version?

[Apollon77]

Problem did not appeared again (but also before it was only once ...)

[frederikhegger]

Unfortunately, the Problem still exists under Debain 9 (Stretch) with InfluxDB version 1.3.7-1

[e-dard]

@frederikhegger if you could provide more information / logs, or steps to reproduce we would be happy to look into this.

[frederikhegger]

Since our Influx instance (Non-cluster version) is running in a pre-production environment, where many values are written and read from each second, we can not really reproduce the problem.

While crawling through our logs, we realized that there are several panics which do not cause a crash of the influx service and one which did.

Here is one of the panics which did not cause a crash of the process;

panic:runtime error: invalid memory address or nil pointer dereference] goroutine 13456438627 [running]:
Nov 14 16:32:09 influxd[569]: runtime/debug.Stack(0xc603c3b2c0, 0xc54fdd6000, 0xa1e)
Nov 14 16:32:09 influxd[569]: #011/usr/local/go/src/runtime/debug/stack.go:24 +0x79
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*QueryExecutor).recover(0xc4201c4cf0, 0xc603c3b2c0, 0xc654cfb3e0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/query_executor.go:397 +0x96
Nov 14 16:32:09 influxd[569]: panic(0xac1c20, 0xeecf70)
Nov 14 16:32:09 influxd[569]: #011/usr/local/go/src/runtime/panic.go:489 +0x2cf
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*bufCursor).close(0xc59d129b00, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:54 +0x2d
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.cursorsAt.close(0xc6fd458f60, 0x3, 0x3)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.go:55 +0x49
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatIterator).Close(0xc694fe4200, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:264 +0x45
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc5e5a10240, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatLimitIterator).Close(0xc694fe6000, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:291 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatMergeIterator).Close(0xc601efe300, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:164 +0x54
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc6e17f5da0, 0xc43fd37710, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Close(0xc6e17f5dc0, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:66 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).Close(0xc6e0fd5a20, 0xc6e17f5e20, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1035 +0x2e
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatFinalizerIterator).Close(0xc6e17f5e20, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:139 +0x5f
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatMergeIterator).Close(0xc52dee78c0, 0x18, 0xa8e9c0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:164 +0x54
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc4d5620720, 0x18, 0xc43b494820)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Close(0xc4d5620740, 0x7f21a12974b0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:66 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).Close(0xc46d63fa20, 0x60, 0xc43fd37840)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1035 +0x2e
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc4d5620780, 0xc43fd37850, 0x7c6f27)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Close(0xc4d56207c0, 0x42b6e6, 0xc654cfb3e0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:66 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).Close(0xc46d63fce0, 0xc654cfb380, 0xc654cfb500)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1035 +0x2e
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc4d5620820, 0x6, 0xc43fd378c0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Close(0xc4d5620840, 0x78aab7, 0xc5eea08b18)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:66 +0x33
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatAuxIterator).Close(0xc4d5620980, 0x0, 0x0)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:883 +0x2e
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.Iterators.Close(0xc5118079c0, 0x4, 0x4, 0xc6110016d0, 0xc43fd37958)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.go:56 +0x49
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*Emitter).Close(0xc567ac3050, 0x5, 0x4)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/emitter.go:44 +0x41
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeSelectStatement(0xc4201d2770, 0xc5bca73420, 0xc43fd37cc0, 0xeac3a0, 0xc420015300)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:487 +0x475
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement(0xc4201d2770, 0xeb9f20, 0xc5bca73420, 0x0, 0x71c7fd, 0xc654cfb5c0, 0xc654cfb3e0, 0xebd9c0, 0xc4201d7620, 0xc654cfb500, ...)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:57 +0x2b13
Nov 14 16:32:09 influxd[569]: github.com/influxdata/influxdb/influxql.(*QueryExecutor).executeQuery(0xc4201c4cf0, 0xc603c3b2c0, 0xc54d5f590e, 0x6, 0xeba620, 0xf19df0, 0x2710, 0x1, 0x0, 0x0, ...)
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/query_executor.go:347 +0x577
Nov 14 16:32:09 influxd[569]: created by github.com/influxdata/influxdb/influxql.(*QueryExecutor).ExecuteQuery
Nov 14 16:32:09 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/query_executor.go:242 +0xc3
Nov 14 16:32:09 influxd[569]:  service=query

The following panic caused a crash of the influx service:

Nov 14 23:25:57 influxd[569]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 14 23:25:57 influxd[569]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x9da21d]
Nov 14 23:25:57 influxd[569]: goroutine 13552892750 [running]:
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*bufCursor).close(0xc6e529a0c0, 0x0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:54 +0x2d
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.cursorsAt.close(0xc589137e00, 0x3, 0x3)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.go:55 +0x49
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatIterator).Close(0xc4dc490000, 0x0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:264 +0x45
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc5891438c0, 0x0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatLimitIterator).Close(0xc589163320, 0x0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:291 +0x33
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatMergeIterator).Close(0xc5cfaf39e0, 0x0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:164 +0x54
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc5b87267a0, 0xc4b2cdb1b0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Close(0xc5b87267c0, 0x0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:66 +0x33
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).Close(0xc4dd7942c0, 0xc5b8726840, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1035 +0x2e
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatFinalizerIterator).Close(0xc5b8726840, 0x0, 0x0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:139 +0x5f
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatMergeIterator).Close(0xc47747bb60, 0xc4b2cdb200, 0x410001)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:164 +0x54
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Close(0xc488c1b1c0, 0x0, 0xc42b5a58c0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:779 +0x33
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Next(0xc488c1b1c0, 0x14f6f88e1e31ba40, 0x4063393480000000, 0xc6d127fa70)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:789 +0xbf
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Next(0xc488c1b1e0, 0x3, 0x4063393480000000, 0xc6d127fa70)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:95 +0x95
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).NextInWindow(0xc488c1b1e0, 0x8000000000000001, 0x14f7141f57bafb17, 0x17, 0xc4b2cdb6f8, 0x17)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:101 +0x2f
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).reduce(0xc446652dc0, 0xc58e8fd400, 0x455510, 0xc58e8fd400, 0xc6d127f620, 0xc6ca73bde0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1092 +0x230
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).Next(0xc446652dc0, 0xc495288601, 0xc6ca73bda0, 0xc4282cd928)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1042 +0xa1
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Next(0xc488c1b220, 0x14ad9f167f0a98c0, 0x4073000000000000, 0xc6d127f620)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:798 +0x4f
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Next(0xc488c1b240, 0x3, 0x4073000000000000, 0xc6d127f620)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:95 +0x95
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).NextInWindow(0xc488c1b240, 0x8000000000000001, 0x14f7141f57bafb17, 0xc4282cdd50, 0xf1abe0, 0x19)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:101 +0x2f
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).reduce(0xc446652f20, 0x1, 0x3, 0xc495288630, 0xa88460, 0xc49a3720e0)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1092 +0x230
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatReduceFloatIterator).Next(0xc446652f20, 0xc64e4fa144, 0xc6c47abcb0, 0xc488c1b2f8)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:1042 +0xa1
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Next(0xc488c1b280, 0xc517bcbf98, 0xc435131740, 0x19)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:798 +0x4f
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Next(0xc488c1b2a0, 0xebb100, 0xc49e039b60, 0x1)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:95 +0x95
Nov 14 23:25:57 influxd[569]: github.com/influxdata/influxdb/influxql.(*floatAuxIterator).stream(0xc488c1b320)
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:895 +0x32
Nov 14 23:25:57 influxd[569]: created by github.com/influxdata/influxdb/influxql.(*floatAuxIterator).Start
Nov 14 23:25:57 influxd[569]: #011/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:881 +0x3f
Nov 14 23:25:58 systemd[1]: influxdb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 14 23:25:58 systemd[1]: influxdb.service: Unit entered failed state.
Nov 14 23:25:58 systemd[1]: influxdb.service: Failed with result 'exit-code'.
Nov 14 23:25:58 systemd[1]: influxdb.service: Service hold-off time over, scheduling restart.
Nov 14 23:25:58 systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Nov 14 23:25:58 systemd[1]: Started InfluxDB is an open-source, distributed, time series database.

[frederikhegger]

We are still facing the two types of influx crashes approximately every two days on the recent released influx version 1.4.2 under Debian 9.3 (stretch).

Is this a known issue?

[frederikhegger]

We just discovered the possible cause for the crashes. For some reason the VM network adapter resetted exactly at the times of our crashes:

[Thu Jan 11 06:04:25 2018] e1000 0000:02:00.0 eth0: Reset adapter
[Thu Jan 11 06:04:25 2018] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sun Jan 14 07:01:03 2018] e1000 0000:02:00.0 eth0: Reset adapter
[Sun Jan 14 07:01:03 2018] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Tue Jan 16 22:13:56 2018] e1000 0000:02:00.0 eth0: Reset adapter
[Tue Jan 16 22:13:56 2018] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Thu Jan 18 22:07:49 2018] e1000 0000:02:00.0 eth0: Reset adapter
[Thu Jan 18 22:07:49 2018] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Fri Jan 19 22:19:14 2018] e1000 0000:02:00.0 eth0: Reset adapter
[Fri Jan 19 22:19:14 2018] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

Doesn't influx handle such kind of "network interface losses"?

[frederikhegger]

Unfortunately, the network adapter reset was only the cause the for the panics which can not be recovered and the service restarts. But we still have the following panic, occurring approx. every two days:

[panic:runtime error: invalid memory address or nil pointer dereference] goroutine 812902544 [running]:
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: runtime/debug.Stack(0xc88940f480, 0xca980d3000, 0xe28)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*QueryExecutor).recover(0xc42016faa0, 0xc88940f480, 0xc7ebd1eba0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/query_executor.go:443 +0xaf
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: panic(0xbcc6e0, 0x1116120)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/usr/local/go/src/runtime/panic.go:491 +0x283
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*bufCursor).close(0xc5d82a9880, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:58 +0x2d
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/tsdb/engine/tsm1.cursorsAt.close(0xc50b3e14a0, 0x3, 0x3)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.go:152 +0x41
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatIterator).Close(0xc735c83e00, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:300 +0x45
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatInterruptIterator).Close(0xc8f655ff00, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:823 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatLimitIterator).Close(0xc46fa70a20, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:327 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatMergeIterator).Close(0xc43b39be60, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:158 +0x51
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatInterruptIterator).Close(0xc994dfa400, 0xcaad24f648, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:823 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*bufFloatIterator).Close(0xc994dfa420, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:61 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatReduceFloatIterator).Close(0xc986d47080, 0xc994dfa460, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:1086 +0x2e
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*floatFinalizerIterator).Close(0xc994dfa460, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:145 +0x5f
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatMergeIterator).Close(0xc884f45da0, 0x114a160, 0x19)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:158 +0x51
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatInterruptIterator).Close(0xc8bc6efb00, 0x4123c7, 0xc665d35320)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:823 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*bufFloatIterator).Close(0xc8bc6efb40, 0xcaad24f868, 0x3)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:61 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatReduceFloatIterator).Close(0xc7711f9760, 0x3, 0xcaad24f8f8)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:1086 +0x2e
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatInterruptIterator).Close(0xc8bc6efb80, 0x0, 0xc67090a830)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:823 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*bufFloatIterator).Close(0xc8bc6efba0, 0xca6fcceec0, 0x412b28)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:61 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatReduceFloatIterator).Close(0xc7711f98c0, 0xc665d35320, 0xcaad24f908)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:1086 +0x2e
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatInterruptIterator).Close(0xc8bc6efbe0, 0x42bd69, 0xca6fcceef0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:823 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*bufFloatIterator).Close(0xc8bc6efc00, 0xcaad24f828, 0xcaad24f830)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:61 +0x33
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatAuxIterator).Close.func1()
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:931 +0x36
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: sync.(*Once).Do(0xc63c826d0c, 0xcaad24f880)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/usr/local/go/src/sync/once.go:44 +0xbe
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*floatAuxIterator).Close(0xc63c826cf0, 0x0, 0x0)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:931 +0x68
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.Iterators.Close(0xc889399580, 0x4, 0x4, 0xc65b5099a8, 0xcaad24f918)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/iterator.go:49 +0x41
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*Emitter).Close(0xc4320681e0, 0xca56a2fb40, 0x4)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/emitter.go:47 +0x41
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeSelectStatement(0xc420228a80, 0x10e24e0, 0xc420014048, 0xc889376500, 0xcaad24fcb0, 0x10d14a0, 0xc420043010)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:610 +0x4cc
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement(0xc420228a80, 0x10e38e0, 0xc889376500, 0x0, 0xdd0d9, 0xc8894c0310, 0xc7ebd1eba0, 0x10e7780, 0xc42022dce0, 0xc7ebd1ecc0, ...)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:62 +0x29a6
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery(0xc42016faa0, 0xc88940f480, 0xc6848c990e, 0x6, 0x10e4060, 0x1149230, 0x2710, 0x1, 0x0, 0x0, ...)
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/query_executor.go:381 +0xb27
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: created by github.com/influxdata/influxdb/query.(*QueryExecutor).ExecuteQuery
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: #011/go/src/github.com/influxdata/influxdb/query/query_executor.go:276 +0xc3
Feb 601:24:58 NXHQ-VLS-DEV-DB01 influxd[3692]: service=query

[frederikhegger]

Since we installed the new release 1.4.3 last week, we do not see these issues anymore, for now.

[jwilder]

The bufCursor panics will be fixed in #9470

[Programie]

I have the same issue in version 1.5.0-1 on Debian Stretch.

But I only get this error for a "SHOW TAG VALUES" query in a single database. A similar "SHOW TAG VALUES" query in another database works without any issues.

I didn't had those issues with InfluxDB 1.4.x before, but I'm not sure whether it occurred since the update to 1.5.0.

[benbjohnson]

@Programie Can you post a stack trace of the error you're seeing?

[Programie]

Here is the stack trace:

SHOW TAG VALUES WITH KEY = room WHERE area =~ /^OG$/ [panic:runtime error: invalid memory address or nil pointer dereference] goroutine 133184 [running]:
runtime/debug.Stack(0xc43da1fda0, 0xc423bb0000, 0x34)
    /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
github.com/influxdata/influxdb/query.(*QueryExecutor).recover(0xc4201510b0, 0xc43da1fda0, 0xc42b42d860)
    /go/src/github.com/influxdata/influxdb/query/query_executor.go:451 +0xaf
panic(0xce4120, 0x1302970)
    /usr/local/go/src/runtime/panic.go:491 +0x283
github.com/influxdata/influxdb/tsdb.(*MeasurementFields).HasField(0x0, 0xc43da2774c, 0x4, 0xc4202f2910)
    /go/src/github.com/influxdata/influxdb/tsdb/shard.go:1408 +0x26
github.com/influxdata/influxdb/tsdb.IndexSet.seriesByBinaryExprIterator(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0xc421ac5ec8, 0x7, 0x8, 0xc42e1b44b0, 0x0, 0xd3ac00, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:1833 +0xdd2
github.com/influxdata/influxdb/tsdb.IndexSet.seriesByExprIterator(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0xc421ac5ec8, 0x7, 0x8, 0x12cae00, 0xc42e1b44b0, 0x0, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:1786 +0x476
github.com/influxdata/influxdb/tsdb.IndexSet.tagValuesByKeyAndExpr(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0x7fc5cdfc93a8, 0xc420184120, 0xc421ac5ec8, 0x7, 0x8, 0xc43150e310, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:2144 +0x1a0
github.com/influxdata/influxdb/tsdb.IndexSet.MeasurementTagKeyValuesByExpr(0xc4200a9500, 0x1, 0x99, 0xc4202f2910, 0x7fc5cdfc93a8, 0xc420184120, 0xc421ac5ec8, 0x7, 0x8, 0xc43150e310, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/index.go:2293 +0xca1
github.com/influxdata/influxdb/tsdb.(*Store).TagValues(0xc420001980, 0x7fc5cdfc93a8, 0xc420184120, 0xc420b6b800, 0x99, 0x100, 0x12cae00, 0xc42e1b4360, 0x100, 0x400, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/store.go:1517 +0x946
github.com/influxdata/influxdb/coordinator.LocalTSDBStore.TagValues(0xc420001980, 0x7fc5cdfc93a8, 0xc420184120, 0xc420b6b800, 0x99, 0x100, 0x12cae00, 0xc42e1b4360, 0x32f2d7fe, 0x109d537404, ...)
    <autogenerated>:1 +0x97
github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeShowTagValues(0xc4201953b0, 0xc4300bca80, 0xc42153dc48, 0xdf7588, 0xf)
    /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:1066 +0x840
github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement(0xc4201953b0, 0x12ce780, 0xc4300bca80, 0x0, 0x14e, 0xc42e8cdf80, 0xc42b42d860, 0xc42b42d8c0, 0xc43da23e3e, 0x6, ...)
    /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:194 +0x2937
github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery(0xc4201510b0, 0xc43da1fda0, 0xc43da23e3e, 0x6, 0x7fc5cdfc93a8, 0xc420184120, 0x2710, 0x1, 0x0, 0x0, ...)
    /go/src/github.com/influxdata/influxdb/query/query_executor.go:389 +0xae8
created by github.com/influxdata/influxdb/query.(*QueryExecutor).ExecuteQuery
    /go/src/github.com/influxdata/influxdb/query/query_executor.go:285 +0xc3

[e-dard]

This issue is fixed in 1.5.0. @Programie, a fix for your issue is now in master; it will be part of the 1.5.1 maintenance release.

[hanynowsky]

I can confirm it was solved in 1.5.1. Thanks

[timhallinflux]

@Programie -- can you please confirm that your issue has been addressed as well (when you are able)? Thanks!

[Programie]

I just tried to update to InfluxDB 1.5.1 but this version doesn't seem to be in the Debian repository (https://repos.influxdata.com/debian) yet, right?

[hanynowsky]

@Programie No not yet. Just stop your influxdb instance, and then update manually:

wget https://dl.influxdata.com/influxdb/releases/influxdb_1.5.1_amd64.deb
sudo dpkg -i influxdb_1.5.1_amd64.deb

[Programie]

Just tested it and can confirm I also don't have the issues anymore. Thanks for the fix!

[timhallinflux]

@programie Deb repo now updated as well.