As an additional note, it seems that when this happens, and I restart InfluxDB, one of my measurements will be corrupt. The measurement appears empty. Any attempt to insert into the measurement returns something like [shard 17] field type conflict. And even when I drop the measurement and try to recreate it, the error still occurs.

phemmer on 16 Jun 2017

It just happened again. Working fine all day, and then just a minute ago it went unresponsive. The only thing I was doing (in addition to what's been going on all day long) was that I was skipping backwards in time a lot in grafana (the little left arrow thing next to the time selector in the top).

Here's another SIGQUIT dump: https://gist.githubusercontent.com/phemmer/251eab87914681d30f0e0435c664e9f7/raw/a63f6cdf6c54d89771630764a13d42aa07318f88/log2

Note however that this time I was running InfluxDB on a completely different host and OS. This time it's on Linux, and I was running a7ca7b5. I also had a completely brand new database as the crash last night completely corrupted my old one and I couldn't recover it.

Edit: And another... https://gist.githubusercontent.com/phemmer/251eab87914681d30f0e0435c664e9f7/raw/1008e1e1206e59501193d381fa99fc10af8972ae/log3

Edit: And just in case thats not enough SIGQUIT, here's some more: https://gist.githubusercontent.com/phemmer/251eab87914681d30f0e0435c664e9f7/raw/ac5fcb3757ff47f794dd8711bef51b1618e63b18/log4

phemmer on 17 Jun 2017

Looks like a deadlock in the inmem index.

jwilder on 21 Jun 2017

@phemmer Are you seeing any panics in your logs?

benbjohnson on 21 Jun 2017

no

phemmer on 21 Jun 2017

@phemmer Could you try #8518 to see if that helps?

jwilder on 21 Jun 2017

Perhaps. I'll see if I can reproduce it in an isolated environment. I rolled back to v1.2 because the issue started happening every few minutes, and was constantly corrupting my database. Though given how frequently it was occurring, it shouldn't be hard.

phemmer on 21 Jun 2017

@phemmer I would run show field keys on that measurement. [shard 17] field type conflict would indicate that a field type has switched somewhere.

jwilder on 21 Jun 2017

I could drop the entire measurement drop measurement foo and the issue would still occur. show measurements would indicate that it's not there, yet I would still get that error trying to recreate it.

phemmer on 21 Jun 2017

I just upgraded my system to 1.3.0 (downloaded from https://dl.influxdata.com/influxdb/releases/influxdb-1.3.0.x86_64.rpm)

and then ran into troubles. Things run fine for ~1 min, then the whole system locks up and dies. I have to shutdown the aws instance and bring it back up to get in. I can't find any logging that clearly says what happened -- but can try to dig further if this is helpful.

I then tried upgrading to the nightly 1.4.0~n201706240800 / e014dd012d9a66cce540f4a021ee0a4a177960a7 and I get the same behavior.

I then put back the 1.2.4 and things are OK again
sudo rpm -Uvh --oldpackage influxdb-1.2.4.x86_64.rpm

I don't know if this is related... let me know if there is any more info you want (and where to find it!)

ryantxu on 24 Jun 2017

@ryantxu it looks like 1.4.0~n201706240800 would contain the suggested possible fix in https://github.com/influxdata/influxdb/issues/8518, which means that #8518 probably hasn't done the trick.

Does the server only lock up under write load or can you literally start it up and leave it to lock up without any writes or queries going in?

e-dard on 28 Jun 2017

@phemmer did you have the same problems when trying #8518 or did you not get around to that?

e-dard on 28 Jun 2017

The setup has active writes and continuous queries running. It would take some work to test without that... but if useful, let me know.

ryantxu on 28 Jun 2017

@phemmer did you have the same problems when trying #8518 or did you not get around to that?

I have not had a chance. I'm not keen on doing this with my main InfluxDB and corrupting my DB again, so I need to duplicate the server and all the clients, which will take some work.

phemmer on 28 Jun 2017

👍1

@ryantxu I'm trying to think of a way for us to be able to locally reproduce your situations so we can track down the deadlock. I figured as much that writes/queries were needed. If not, we probably would have seen this within our tests. @phemmer are you running CQs by chance?

e-dard on 28 Jun 2017

No CQs

phemmer on 28 Jun 2017

@phemmer @ryantxu I just realised I didn't ask—do you know if profile endpoints are responding on 8086? A blocking profile could be useful.. curl -o blocking.txt "http://localhost:8086/debug/pprof/block?debug=1"

@ryantxu if you're using 1.3 onwards then the new archive profile endpoint would be available to you: curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all"

e-dard on 28 Jun 2017

I must say that as an InfluxDB user, the handling of this issue is concerning. With this issue, and #8533, we now have several people confirming the issue.
I created this issue well before InfluxDB 1.3.0 was released. An issue where at the very least InfluxDB becomes unresponsive, and at the worst results in DB corruption. How did a stable release get out without this issue being investigated and addressed? What is the purpose of the release candidates if they're not used to validate stability before general release?

phemmer on 29 Jun 2017

👍6

@phemmer I can assure you that we're looking into this. We had hoped that #8518 would fix the issue, but it appears not according to @ryantxu's testing of the 1.4.0~n201706240800 nightly. You mention that the server goes unresponsive but I just wanted to get a couple of clear clarifications to help me try and isolate the issue:

does the entire 8086 write/query endpoint become unresponsive, or is it just related to the database in question?
if it's not the entire port, are you able to get profiles when the database does become unresponsive? As I mentioned previously, a blocking profile may provide some insight.

e-dard on 29 Jun 2017

I understand we're working on it now, and I appreciate that, but how come it wasn't worked on until after 1.3.0 was released? That's my concern. As a user, I should be comfortable using stable releases.

I only had 1 database, so I was unable to determine if other databases were affected.
Other operations were responsive, such as I can confirm the ping endpoint was responsive.

I'm working on reproducing this in a lab right now. Hopefully I'll be able to do so.

phemmer on 29 Jun 2017

@phemmer Are you able to share your log files related to the hang?

benbjohnson on 29 Jun 2017

@phemmer We are running about 100 production instances of 1.3.0 and have been running 1.3.0 shadows internally for weeks; we have never reproduced this error. We are giving this issue a very high priority - it is blocking our 1.3 GA build and announcement as it stands. I apologize for the frustration it has caused you.

rbetts on 29 Jun 2017

When I last tried, i was in a bit of panic mode trying to get things back to working :)

If you guys are going to look at it, i can get things to reproduce again -- if you want, i can give you direct access if that helps your debugging

ryantxu on 29 Jun 2017

@ryantxu I just merged in a potential fix here: https://github.com/influxdata/influxdb/pull/8547

Are you able to pull that down and build it? I'd be interested to know if you can reproduce with 1.3 but not that branch.

benbjohnson on 29 Jun 2017

is there an rpm i can install with it? or do i need to wait for the nightly to get built and try tomorrow?

ryantxu on 29 Jun 2017

testing now with
"2f1134708925fd54a36b45feeaf48553cdc274e3",
"1.4.0~n201706290800"

Still freezes after a few mins -- this time, the last few lines of the logs before crash say something interesting:

[I] 2017-06-29T17:57:12Z Snapshot for path /mnt/db/data/_internal/monitor/14227 written in 1.295712536s engine=tsm1
[I] 2017-06-29T17:57:13Z beginning level 1 compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-06-29T17:57:13Z compacting level 1 group (0) /mnt/db/data/_internal/monitor/14227/000000915-000000001.tsm (#0) engine=tsm1
[I] 2017-06-29T17:57:13Z compacting level 1 group (0) /mnt/db/data/_internal/monitor/14227/000000916-000000001.tsm (#1) engine=tsm1
[I] 2017-06-29T17:57:16Z compacted level 1 group (0) into /mnt/db/data/_internal/monitor/14227/000000916-000000002.tsm.tmp (#0) engine=tsm1
[I] 2017-06-29T17:57:16Z compacted level 1 2 files into 1 files in 2.698048352s engine=tsm1
[I] 2017-06-29T17:57:16Z beginning level 2 compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-06-29T17:57:16Z compacting level 2 group (0) /mnt/db/data/_internal/monitor/14227/000000913-000000002.tsm (#0) engine=tsm1
[I] 2017-06-29T17:57:16Z compacting level 2 group (0) /mnt/db/data/_internal/monitor/14227/000000916-000000002.tsm (#1) engine=tsm1

ryantxu on 29 Jun 2017

I was wondering if it might be compaction related. This might help explain why my db was getting corrupted.

phemmer on 29 Jun 2017

I don't think my db is getting corrupted -- and the downgrade back to 1.2.4 seems to work without needing to reload the data. My problem is that the whole system freezes and i need to reboot the EC2 instance to get back on

ryantxu on 29 Jun 2017

Yeah, there might be something else going on in my environment that in combination triggers the corruption. Something like a new series being created, measurement being dropped & re-created, etc.

phemmer on 29 Jun 2017

I'm trying to get the output of blocking.txt and profiles.tar.gz for you...

This latest reboot and restart never got to a point I can query :( The system freezes after:

[I] 2017-06-29T18:13:09Z /mnt/db/data/usgs/autogen/12701 opened in 11.167452ms service=store
[I] 2017-06-29T18:13:09Z opened service service=subscriber
[I] 2017-06-29T18:13:09Z Starting monitor system service=monitor
[I] 2017-06-29T18:13:09Z 'build' registered for diagnostics monitoring service=monitor
[I] 2017-06-29T18:13:09Z 'runtime' registered for diagnostics monitoring service=monitor
[I] 2017-06-29T18:13:09Z 'network' registered for diagnostics monitoring service=monitor
[I] 2017-06-29T18:13:09Z 'system' registered for diagnostics monitoring service=monitor
[I] 2017-06-29T18:13:09Z Starting precreation service with check interval of 10m0s, advance period of 30m0s service=shard-precreation
[I] 2017-06-29T18:13:09Z Starting snapshot service service=snapshot
[I] 2017-06-29T18:13:09Z Starting continuous query service service=continuous_querier
[I] 2017-06-29T18:13:09Z Starting HTTP service service=httpd
[I] 2017-06-29T18:13:09Z Authentication enabled:false service=httpd
[I] 2017-06-29T18:13:09Z Listening on HTTP:[::]:8086 service=httpd
[I] 2017-06-29T18:13:09Z Starting retention policy enforcement service with check interval of 30m0s service=retention
[I] 2017-06-29T18:13:09Z Listening for signals
[I] 2017-06-29T18:13:09Z Storing statistics in database '_internal' retention policy 'monitor', at interval 10s service=monitor
[I] 2017-06-29T18:13:09Z Sending usage statistics to usage.influxdata.com
[I] 2017-06-29T18:13:10Z beginning full compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-06-29T18:13:10Z compacting full group (0) /mnt/db/data/_internal/monitor/14227/000000897-000000005.tsm (#0) engine=tsm1
[I] 2017-06-29T18:13:10Z compacting full group (0) /mnt/db/data/_internal/monitor/14227/000000916-000000004.tsm (#1) engine=tsm1

ryantxu on 29 Jun 2017

@ryantxu could you try SIGQUITing the process when it freezes?

$ kill -s SIGQUIT <pid of influxd>

Or, if you're running influxd in the foreground just hit ctrl \

e-dard on 29 Jun 2017

@ryantxu could you try SIGQUITing the process when it freezes?

When it freezes, i can't type in my terminal window. I can't reconnect via ssh, the only thing i can do is reboot the ec2 instance!

ryantxu on 29 Jun 2017

I am unable to login to the ec2 instance now... i will need to restore and try again. Maybe this afternoon, but more likely tomorrow morning. Let me know if there is an .rpm i should try.

ryantxu on 29 Jun 2017

@ryantxu maybe the system is becoming unhealthy due to limited RAM or other resources? Have you tried keeping a second terminal open tailing kern.log or running top?

cxreg on 29 Jun 2017

Yes, it appears that 1.4.0~n201706290800 pushes this machine outside its memory

top - 22:48:27 up  3:41,  2 users,  load average: 2.61, 0.77, 0.28
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.6%sy,  0.0%ni, 14.1%id, 85.1%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   7660676k total,  7544632k used,   116044k free,      192k buffers
Swap:        0k total,        0k used,        0k free,     2616k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                  
  658 root      20   0     0    0    0 R  7.2  0.0   1:10.62 kswapd0                                                                                                   
 3361 influxdb  20   0 31.5g 7.1g    0 D  6.7 96.6   1:36.44 influxd                                                                                                   
 2645 grafana   20   0  716m  10m    0 S  2.0  0.1   3:41.31 grafana-server                                                                                            
 3304 ec2-user  20   0 15308  524  204 R  0.9  0.0   0:00.27 top                                                                                                       
    1 root      20   0 19636  316    0 S  0.0  0.0   0:01.66 init                                                                                                      
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                  
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.02 ksoftirqd/0

When I revert back to 1.2.4, things run fine and look like this:
````
top - 23:28:38 up 33 min, 2 users, load average: 2.59, 2.56, 5.72
Tasks: 107 total, 1 running, 106 sleeping, 0 stopped, 0 zombie
Cpu(s): 45.7%us, 0.1%sy, 0.0%ni, 53.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.3%st
Mem: 7660676k total, 4187248k used, 3473428k free, 61340k buffers
Swap: 0k total, 0k used, 0k free, 1594792k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2966 influxdb 20 0 25.2g 3.5g 1.2g S 200.1 47.4 5:00.83 influxd
2380 root 20 0 4372 88 0 S 0.3 0.0 0:01.14 rngd
1 root 20 0 19632 824 508 S 0.0 0.0 0:01.00 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
````

So i guess this is really a memory use issue. Perhaps it is getting stuck on a one time upgrade? Or do you think the memory requirements really changed that much between 1.2.4 and 1.3?

this is on a c3.xlarge/us-east-1a

ryantxu on 30 Jun 2017

I'm still unable to reproduce the hanging issue. But I still haven't reproduced the original environment exactly, just mostly :-/
I am however seeing a plethora of other issues (won't go into right now). But I can confirm the memory thing @ryantxu is seeing. I have a test box where InfluxDB is completely idle (no queries, and no writes), and I can watch the memory usage go from ~1gb up to ~7gb, while using around 100% CPU (it bounces around, sometimes using 70%, sometimes using 150%). The memory plateaus at ~7gb, and sits there for for a few minutes, still using 100% CPU, and then drops back down to ~1gb and starts the process over again.

Every time the memory drops, a few seconds later it logs something like this:

[I] 2017-07-03T01:51:29Z aborted full compaction group (0). compactions disabled engine=tsm1
[I] 2017-07-03T01:51:29Z beginning full compaction of group 0, 5 TSM files engine=tsm1
[I] 2017-07-03T01:51:29Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000005.tsm (#0) engine=tsm1
[I] 2017-07-03T01:51:29Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000006.tsm (#1) engine=tsm1
[I] 2017-07-03T01:51:29Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000007.tsm (#2) engine=tsm1
[I] 2017-07-03T01:51:29Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000008.tsm (#3) engine=tsm1
[I] 2017-07-03T01:51:29Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000678-000000004.tsm (#4) engine=tsm1

And then occasionally it'll go nuts and start doing this, of which the only way to make it stop is to restart:

[I] 2017-07-03T01:57:21Z beginning full compaction of group 0, 5 TSM files engine=tsm1
[I] 2017-07-03T01:57:21Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000005.tsm (#0) engine=tsm1
[I] 2017-07-03T01:57:21Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000006.tsm (#1) engine=tsm1
[I] 2017-07-03T01:57:21Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000007.tsm (#2) engine=tsm1
[I] 2017-07-03T01:57:21Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000008.tsm (#3) engine=tsm1
[I] 2017-07-03T01:57:21Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000678-000000004.tsm (#4) engine=tsm1
[I] 2017-07-03T01:57:21Z aborted full compaction group (0). compaction in progress: open /mnt/influxdb/data/spec/autogen/317/000000678-000000005.tsm.tmp: file exists engine=tsm1
[I] 2017-07-03T01:57:22Z beginning full compaction of group 0, 5 TSM files engine=tsm1
[I] 2017-07-03T01:57:22Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000005.tsm (#0) engine=tsm1
[I] 2017-07-03T01:57:22Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000006.tsm (#1) engine=tsm1
[I] 2017-07-03T01:57:22Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000007.tsm (#2) engine=tsm1
[I] 2017-07-03T01:57:22Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000008.tsm (#3) engine=tsm1
[I] 2017-07-03T01:57:22Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000678-000000004.tsm (#4) engine=tsm1
[I] 2017-07-03T01:57:22Z aborted full compaction group (0). compaction in progress: open /mnt/influxdb/data/spec/autogen/317/000000678-000000005.tsm.tmp: file exists engine=tsm1
[I] 2017-07-03T01:57:23Z beginning full compaction of group 0, 5 TSM files engine=tsm1
[I] 2017-07-03T01:57:23Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000005.tsm (#0) engine=tsm1
[I] 2017-07-03T01:57:23Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000006.tsm (#1) engine=tsm1
[I] 2017-07-03T01:57:23Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000007.tsm (#2) engine=tsm1
[I] 2017-07-03T01:57:23Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000675-000000008.tsm (#3) engine=tsm1
[I] 2017-07-03T01:57:23Z compacting full group (0) /mnt/influxdb/data/spec/autogen/317/000000678-000000004.tsm (#4) engine=tsm1
[I] 2017-07-03T01:57:23Z aborted full compaction group (0). compaction in progress: open /mnt/influxdb/data/spec/autogen/317/000000678-000000005.tsm.tmp: file exists engine=tsm1

phemmer on 3 Jul 2017

@phemmer can you open another issue with the .tmp compaction problems? This is something that we identified internally on Friday but have not yet found the root cause of. I would love to hear more about the data you have in this instance and any other information you can provide.

e-dard on 3 Jul 2017

So the hanging thing has started happening for me again.

Version e3918e0
SIGQUIT: https://gist.githubusercontent.com/phemmer/251eab87914681d30f0e0435c664e9f7/raw/f4d6a9d33a680bd7ab3eef2dcbf13d8df9ca0135/log5 (unfortunately truncated at ~10k lines due to logging configuration issue)
curl /debug/pprof/block: https://gist.github.com/phemmer/251eab87914681d30f0e0435c664e9f7/raw/f4d6a9d33a680bd7ab3eef2dcbf13d8df9ca0135/log5-blocking

Version 8550fab
SIGQUIT https://gist.github.com/phemmer/251eab87914681d30f0e0435c664e9f7/raw/52859f853ca16ae42bcbd45b789eb86b56625e16/log6
curl /debug/pprof/block: https://gist.github.com/phemmer/251eab87914681d30f0e0435c664e9f7/raw/52859f853ca16ae42bcbd45b789eb86b56625e16/log6-blocking

There are no panics in the log prior to this happening. Completely normal POST calls with 204 responses, and then 500s & timeouts.

Edit: These SIGQUIT outputs have a very different profile to them than the earlier ones. So I suspect this might be a completely different issue, just with the same observable result.
In fact this one to me seems pretty obvious: https://github.com/influxdata/influxdb/blob/master/tsdb/engine/tsm1/wal.go#L446
^ We're blocked reading on that chan, but there is no code anywhere which sends anything to it.

phemmer on 5 Jul 2017

@phemmer syncErr is sent to the syncWaiters channel and then written to here

However, if the syncWaiters channel is full (1024), it will fail, but the closure will return an error and thus return early here

stuartcarnie on 5 Jul 2017

Ah, I misread line 424. I skimmed the code and read it as a read from the chan, not sending the chan to another chan o.O. Sorry bout that :-)
Something is causing it to block though.

phemmer on 5 Jul 2017

👍1

@phemmer yes something is up. I'm looking at it as a priority.

e-dard on 5 Jul 2017

@phemmer by any chance do you have the server logs from either of the above two SIGQUITs?

e-dard on 5 Jul 2017

Unfortunately not as the logs have already rotated out (it's a personal box, and I don't permanently archive the logs).
I can however tell you that for a long time prior, and after the issue started, it was nothing but writes with 204 responses, and then writes with 500 responses. There weren't even any queries being executed around the time. It's possible there might have been something way in the past (several minutes before the issue started), but nothing near the time of the event.
The database also only has a single writer. So no concurrent writes.

phemmer on 5 Jul 2017

OK no worries. Was the box under significant write load? Roughly how many writes per second do you think? And typically what would your batch sizes be?

e-dard on 5 Jul 2017

Average WPS is around 6. The writer flushes every 100ms, and when its buffer is full. Normal batch size is pretty small, probably less than 20 points. However once every 60 seconds a batch (or multiples if it can't fit in the write buffer, buffer size is 512KB, and the average point size is ~110B) will come through that is about 4k points.

phemmer on 5 Jul 2017

Oh, probably a very critical bit of information I forgot about, I set wal-fsync-delay = "100ms". This was in attempt to solve some major performance issues I was having. Unfortunately it didn't solve the issue and I instead had to drop the database (seems the issue starts happening once the DB gets large. not number of series, but data size, age, or number of points). But I left the setting around since it did seem to help.

phemmer on 5 Jul 2017

@phemmer What kind of disks do you have? SSDs? HDD? Can you run iostat -xdz 1 30 when you are getting the timeouts?

Would you be able to test #8567, it _may_ fix the timeouts.

For the higher memory usage, if you can grab a memory profile via:

curl -o heap.txt "http://localhost:8086/debug/pprof/heap?debug=1"

that might help identify what may be consuming more memory.

jwilder on 6 Jul 2017

Hybrid disks.
I don't think this is a disk issue. When it happens, influxdb stays stuck until I restart it. It does not recover own its own.

I'll see about reproducing this again (I downgraded back to 1.2.4 which does not have the issue), but probably not tonight. Maybe tomorrow.

phemmer on 6 Jul 2017

Can you elaborate on "Hybrid disks"? What are the disks and how are they setup? Are they RAIDed? Directly attached?

jwilder on 6 Jul 2017

Each disk is a combination of SSD & spinning. There's 2 of them in raid-1. Yes, direct attach.

phemmer on 6 Jul 2017

@phemmer could you clarify something for us? Throughout all of the this ticket have you been running InfluxDB on the same HW (with the raid-1 hybrid drives), and could you also confirm:

if you've been building influxdb or using nightlies/releases.
if you have been building influxdb: with what version of Go?
have you had wal-fsync-delay = "100ms" on the entire time? Any other non-default configuration values?

I just want to get a better understanding of your environment. Cheers.

e-dard on 6 Jul 2017

Yes, same hardware.
I've been building myself. Go 1.8.1.

phemmer on 6 Jul 2017

8577 should fix the case where writer is blocked indefinitely on the `syncErr` channel.

jwilder on 7 Jul 2017

@phemmer All of the pending fixes are on the 1.3 branch and master now. Would you be able to test off of 1.3 and see if the issue is resolved?

jwilder on 7 Jul 2017

@stuartcarnie and I have found the cause for the initial deadlock on Measurement reported in the issue description. We'll get a fix up for that as well.

jwilder on 8 Jul 2017

Would you be able to test off of 1.3 and see if the issue is resolved?

I might be able to tomorrow night. I'll be intermittently available to test things as starting yesterday I'm on vacation through next week. But that one should be really easy to test. It generally occurred within a few minutes of starting InfluxDB.

stuartcarnie and I have found the cause for the initial deadlock on Measurement reported in the issue description. We'll get a fix up for that as well.

🎆 I'm eager to see that PR, as when I skimmed the code, I couldn't see any obvious explanation for it. Everywhere I could see a lock being taken, all code paths released it.

phemmer on 8 Jul 2017

It's here: https://github.com/influxdata/influxdb/blob/master/tsdb/index/inmem/meta.go#L333 and https://github.com/influxdata/influxdb/blob/master/tsdb/index/inmem/meta.go#L338

The RLock is not released.

jwilder on 8 Jul 2017

👍1

Ahha. And that makes sense why I'm able to trigger the issue so easily. I have InfluxDB hooked up to splunk, so splunk can query InfluxDB directly. Splunk is quite fond of killing searches while they're in progress. I didn't even think of that atypical behavior.

phemmer on 8 Jul 2017

👍1

Seems to have same issue with influxdb:1.5-alpine

Blocking output.

--- contention:
cycles/second=3600064271
6252195909477307 1738355 @ 0x43d002 0xb3a653 0xb8467b 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                      /usr/local/go/src/runtime/select.go:628
#   0xb3a652    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact+0x292             /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1781
#   0xb8467a    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1+0x5a /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:355

3028675127701681 84130 @ 0x43d002 0x94c034 0x45cb41
#   0x43d001    runtime.selectgo+0x651                          /usr/local/go/src/runtime/select.go:628
#   0x94c033    github.com/influxdata/influxdb/tsdb.(*Store).monitorShards+0x213    /go/src/github.com/influxdata/influxdb/tsdb/store.go:1690

2298659648282734 640049 @ 0x43d002 0xb39e53 0xb84701 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                      /usr/local/go/src/runtime/select.go:628
#   0xb39e52    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache+0x152            /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1737
#   0xb84700    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1+0x50  /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:434

1244177082848213 4 @ 0x43d002 0xc03eaf 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                  /usr/local/go/src/runtime/select.go:628
#   0xc03eae    github.com/influxdata/influxdb/cmd/influxd/run.(*Server).startServerReporting+0x14e /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:497

1054094715046223 488 @ 0x43d002 0xaa998c 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                  /usr/local/go/src/runtime/select.go:628
#   0xaa998b    github.com/influxdata/influxdb/services/precreator.(*Service).runPrecreation+0x14b  /go/src/github.com/influxdata/influxdb/services/precreator/service.go:76

1054039248608588 292741 @ 0x43d002 0x9b6ee2 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                      /usr/local/go/src/runtime/select.go:628
#   0x9b6ee1    github.com/influxdata/influxdb/services/continuous_querier.(*Service).backgroundLoop+0x1e1  /go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:215

1051795865692636 33018 @ 0x43d002 0xbe8dd4 0xbeb197 0x45cb41
#   0x43d001    runtime.selectgo+0x651                              /usr/local/go/src/runtime/select.go:628
#   0xbe8dd3    github.com/influxdata/influxdb/services/subscriber.(*Service).run+0x2f3     /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:239
#   0xbeb196    github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func1+0x56   /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:98

1037523568823209 29280 @ 0x43d002 0x98a9a0 0x45cb41
#   0x43d001    runtime.selectgo+0x651                          /usr/local/go/src/runtime/select.go:628
#   0x98a99f    github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics+0xa0f /go/src/github.com/influxdata/influxdb/monitor/service.go:432

1030332930728819 159 @ 0x43d002 0xaac7e6 0xaacae7 0x45cb41
#   0x43d001    runtime.selectgo+0x651                              /usr/local/go/src/runtime/select.go:628
#   0xaac7e5    github.com/influxdata/influxdb/services/retention.(*Service).run+0x1c85     /go/src/github.com/influxdata/influxdb/services/retention/service.go:78
#   0xaacae6    github.com/influxdata/influxdb/services/retention.(*Service).Open.func1+0x56    /go/src/github.com/influxdata/influxdb/services/retention/service.go:51

751690876209887 7 @ 0x43d002 0xbe7c21 0xbeb217 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                  /usr/local/go/src/runtime/select.go:628
#   0xbe7c20    github.com/influxdata/influxdb/services/subscriber.(*Service).waitForMetaUpdates+0x110  /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:165
#   0xbeb216    github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func2+0x56       /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:102

160923935145483 133674 @ 0x43d002 0x98f6e8 0x98eff1 0xa88e2b 0xa9d1cc 0xa9b5f2 0x6c15e4 0xa9c01b 0x6c15e4 0xa9a7e5 0x6c15e4 0xa9b712 0x6c15e4 0xa9bb39 0x6c15e4 0xa9bc91 0x6c15e4 0xa9c59c 0x6c15e4 0x9c58cb 0xa8543d 0x6c42c4 0x6c04bd 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                                      /usr/local/go/src/runtime/select.go:628
#   0x98f6e7    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged+0x6c7                      /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:345
#   0x98eff0    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePoints+0x90                         /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:286
#   0xa88e2a    github.com/influxdata/influxdb/services/httpd.(*Handler).serveWrite+0x9fa                           /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:743
#   0xa9d1cb    github.com/influxdata/influxdb/services/httpd.(*Handler).(github.com/influxdata/influxdb/services/httpd.serveWrite)-fm+0x5b /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:150
#   0xa9b5f1    github.com/influxdata/influxdb/services/httpd.authenticate.func1+0x7d1                              /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1371
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9c01a    github.com/influxdata/influxdb/services/httpd.(*Handler).responseWriter.func1+0xaa                      /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1548
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9a7e4    github.com/influxdata/influxdb/services/httpd.gzipFilter.func1+0x214                                /go/src/github.com/influxdata/influxdb/services/httpd/gzip.go:39
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9b711    github.com/influxdata/influxdb/services/httpd.cors.func1+0x101                                  /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1493
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9bb38    github.com/influxdata/influxdb/services/httpd.requestID.func1+0x178                             /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1524
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9bc90    github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1+0xe0                         /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1532
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9c59b    github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1+0x14b                           /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1585
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0x9c58ca    github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP+0x15a                                 /go/src/github.com/bmizerany/pat/mux.go:117
#   0xa8543c    github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP+0x24c                            /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:315
#   0x6c42c3    net/http.serverHandler.ServeHTTP+0xb3                                               /usr/local/go/src/net/http/server.go:2619
#   0x6c04bc    net/http.(*conn).serve+0x71c                                                    /usr/local/go/src/net/http/server.go:1801

92082611943978 110426 @ 0x472b5e 0x473b5d 0xb8d92f 0xb76591 0xb759eb 0xb36f6d 0x93629a 0x94846b 0x98fa72 0x9a1bba 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed                                     /usr/local/go/src/sync/mutex.go:134
#   0x473b5c    sync.(*RWMutex).Lock+0x2c                                   /usr/local/go/src/sync/rwmutex.go:93
#   0xb8d92e    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).writeToLog.func1+0x6e            /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:417
#   0xb76590    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).writeToLog+0x230             /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:451
#   0xb759ea    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).WriteMulti+0x6a              /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:315
#   0xb36f6c    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).WritePoints+0xc1c         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1238
#   0x936299    github.com/influxdata/influxdb/tsdb.(*Shard).WritePoints+0x239                  /go/src/github.com/influxdata/influxdb/tsdb/shard.go:491
#   0x94846a    github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard+0xea                  /go/src/github.com/influxdata/influxdb/tsdb/store.go:1211
#   0x98fa71    github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard+0x91            /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:365
#   0x9a1bb9    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0x99 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311

35113403063630 69085 @ 0x40479b 0xb7665e 0xb759eb 0xb36f6d 0x93629a 0x94846b 0x98fa72 0x9a1bba 0x45cb41
#   0x40479a    runtime.chanrecv1+0x2a                                      /usr/local/go/src/runtime/chan.go:388
#   0xb7665d    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).writeToLog+0x2fd             /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:460
#   0xb759ea    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).WriteMulti+0x6a              /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:315
#   0xb36f6c    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).WritePoints+0xc1c         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1238
#   0x936299    github.com/influxdata/influxdb/tsdb.(*Shard).WritePoints+0x239                  /go/src/github.com/influxdata/influxdb/tsdb/shard.go:491
#   0x94846a    github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard+0xea                  /go/src/github.com/influxdata/influxdb/tsdb/store.go:1211
#   0x98fa71    github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard+0x91            /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:365
#   0x9a1bb9    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0x99 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311

4448621253275 6341 @ 0x43d002 0x98f6e8 0xc0517d 0x987657 0x98aabf 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                  /usr/local/go/src/runtime/select.go:628
#   0x98f6e7    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged+0x6c7  /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:345
#   0xc0517c    github.com/influxdata/influxdb/cmd/influxd/run.(*monitorPointsWriter).WritePoints+0x8c  /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:614
#   0x987656    github.com/influxdata/influxdb/monitor.(*Monitor).writePoints+0x106         /go/src/github.com/influxdata/influxdb/monitor/service.go:152
#   0x98aabe    github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics+0xb2e         /go/src/github.com/influxdata/influxdb/monitor/service.go:465

2214185814299 35 @ 0x43d002 0xa9aa01 0x45cb41
#   0x43d001    runtime.selectgo+0x651                              /usr/local/go/src/runtime/select.go:628
#   0xa9aa00    github.com/influxdata/influxdb/services/httpd.(*Handler).serveQuery.func2+0xd0  /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:474

2192897913608 29 @ 0x43d002 0x87f906 0x45cb41
#   0x43d001    runtime.selectgo+0x651                          /usr/local/go/src/runtime/select.go:628
#   0x87f905    github.com/influxdata/influxdb/query.(*TaskManager).waitForQuery+0x165  /go/src/github.com/influxdata/influxdb/query/task_manager.go:259

2029836674661 2826 @ 0x473a79 0xb762b5 0xb39e9b 0xb84701 0x45cb41
#   0x473a78    sync.(*RWMutex).RLock+0x48                                  /usr/local/go/src/sync/rwmutex.go:50
#   0xb762b4    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).LastWriteTime+0x54           /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:388
#   0xb39e9a    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache+0x19a            /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1743
#   0xb84700    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1+0x50  /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:434

2025483615528 2858 @ 0x473a79 0xb762b5 0xb3a767 0xb8467b 0x45cb41
#   0x473a78    sync.(*RWMutex).RLock+0x48                                  /usr/local/go/src/sync/rwmutex.go:50
#   0xb762b4    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).LastWriteTime+0x54           /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:388
#   0xb3a766    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact+0x3a6             /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1791
#   0xb8467a    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1+0x5a /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:355

224331090104 1328 @ 0x43d002 0x81f765 0x45cb41
#   0x43d001    runtime.selectgo+0x651                              /usr/local/go/src/runtime/select.go:628
#   0x81f764    github.com/influxdata/influxdb/query.(*integerParallelIterator).monitor+0x1a4   /go/src/github.com/influxdata/influxdb/query/iterator.gen.go:3959

127985436841 11 @ 0x403a53 0x94fa70 0x45cb41
#   0x403a52    runtime.chansend1+0x42                          /usr/local/go/src/runtime/chan.go:113
#   0x94fa6f    github.com/influxdata/influxdb/pkg/limiter.limiter.Fixed.Take+0x4f  /go/src/github.com/influxdata/influxdb/pkg/limiter/fixed.go:40
#   0x94fa6f    github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1+0x4f  /go/src/github.com/influxdata/influxdb/tsdb/store.go:265

72964305726 9 @ 0x40479b 0x94296a 0x9419db 0xc03456 0xbf918a 0xc065a5 0xc0631a 0x42d3b6 0x45cb41
#   0x40479a    runtime.chanrecv1+0x2a                          /usr/local/go/src/runtime/chan.go:388
#   0x942969    github.com/influxdata/influxdb/tsdb.(*Store).loadShards+0xe09       /go/src/github.com/influxdata/influxdb/tsdb/store.go:316
#   0x9419da    github.com/influxdata/influxdb/tsdb.(*Store).Open+0x28a         /go/src/github.com/influxdata/influxdb/tsdb/store.go:160
#   0xc03455    github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open+0xa55 /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:418
#   0xbf9189    github.com/influxdata/influxdb/cmd/influxd/run.(*Command).Run+0xd99 /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:140
#   0xc065a4    main.(*Main).Run+0x1d4                          /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:81
#   0xc06319    main.main+0x169                             /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45
#   0x42d3b5    runtime.main+0x225                          /usr/local/go/src/runtime/proc.go:195

68756918342 58 @ 0x473a79 0x93dde1 0xb301a0 0x937a78 0x936189 0x94846b 0x98fa72 0x9a1bba 0x45cb41
#   0x473a78    sync.(*RWMutex).RLock+0x48                                  /usr/local/go/src/sync/rwmutex.go:50
#   0x93dde0    github.com/influxdata/influxdb/tsdb.(*MeasurementFieldSet).CreateFieldsIfNotExists+0x30     /go/src/github.com/influxdata/influxdb/tsdb/shard.go:1492
#   0xb3019f    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).MeasurementFields+0x4f        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:516
#   0x937a77    github.com/influxdata/influxdb/tsdb.(*Shard).validateSeriesAndFields+0x1597         /go/src/github.com/influxdata/influxdb/tsdb/shard.go:592
#   0x936188    github.com/influxdata/influxdb/tsdb.(*Shard).WritePoints+0x128                  /go/src/github.com/influxdata/influxdb/tsdb/shard.go:474
#   0x94846a    github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard+0xea                  /go/src/github.com/influxdata/influxdb/tsdb/store.go:1211
#   0x98fa71    github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard+0x91            /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:365
#   0x9a1bb9    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0x99 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311

36277212845 17 @ 0x4047db 0xa85eb6 0xa9d15c 0xa9b5f2 0x6c15e4 0xa9c01b 0x6c15e4 0xa9a7e5 0x6c15e4 0xa9b712 0x6c15e4 0xa9bb39 0x6c15e4 0xa9bc91 0x6c15e4 0xa9c59c 0x6c15e4 0x9c58cb 0xa8543d 0x6c42c4 0x6c04bd 0x45cb41
#   0x4047da    runtime.chanrecv2+0x2a                                                      /usr/local/go/src/runtime/chan.go:393
#   0xa85eb5    github.com/influxdata/influxdb/services/httpd.(*Handler).serveQuery+0x895                           /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:510
#   0xa9d15b    github.com/influxdata/influxdb/services/httpd.(*Handler).(github.com/influxdata/influxdb/services/httpd.serveQuery)-fm+0x5b /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:138
#   0xa9b5f1    github.com/influxdata/influxdb/services/httpd.authenticate.func1+0x7d1                              /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1371
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9c01a    github.com/influxdata/influxdb/services/httpd.(*Handler).responseWriter.func1+0xaa                      /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1548
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9a7e4    github.com/influxdata/influxdb/services/httpd.gzipFilter.func1+0x214                                /go/src/github.com/influxdata/influxdb/services/httpd/gzip.go:39
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9b711    github.com/influxdata/influxdb/services/httpd.cors.func1+0x101                                  /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1493
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9bb38    github.com/influxdata/influxdb/services/httpd.requestID.func1+0x178                             /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1524
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9bc90    github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1+0xe0                         /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1532
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0xa9c59b    github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1+0x14b                           /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1585
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                                             /usr/local/go/src/net/http/server.go:1918
#   0x9c58ca    github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP+0x15a                                 /go/src/github.com/bmizerany/pat/mux.go:117
#   0xa8543c    github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP+0x24c                            /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:315
#   0x6c42c3    net/http.serverHandler.ServeHTTP+0xb3                                               /usr/local/go/src/net/http/server.go:2619
#   0x6c04bc    net/http.(*conn).serve+0x71c                                                    /usr/local/go/src/net/http/server.go:1801

22417259886 1 @ 0x43d002 0x989ea4 0x98a341 0x45cb41
#   0x43d001    runtime.selectgo+0x651                              /usr/local/go/src/runtime/select.go:628
#   0x989ea3    github.com/influxdata/influxdb/monitor.(*Monitor).waitUntilInterval+0x1e3   /go/src/github.com/influxdata/influxdb/monitor/service.go:409
#   0x98a340    github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics+0x3b0     /go/src/github.com/influxdata/influxdb/monitor/service.go:424

5584976171 14 @ 0x473f12 0xb3fd88 0xb88061 0xb3ed7c 0xb3e2b9 0x938c2d 0x93bea0 0x956ff5 0x991afe 0x887999 0x86f7d5 0x886880 0x86d2a3 0x86beff 0x885b90 0x86b962 0x869a20 0x86914a 0x868ee8 0x998be6 0x997b2f 0x995116 0x866b28 0x45cb41
#   0x473f11    sync.(*WaitGroup).Wait+0x71                                 /usr/local/go/src/sync/waitgroup.go:131
#   0xb3fd87    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetIterators+0x567       /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2382
#   0xb88060    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createCallIterator.func1+0x170    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2223
#   0xb3ed7b    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createCallIterator+0x3eb      /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2248
#   0xb3e2b8    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).CreateIterator+0x3d8          /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2156
#   0x938c2c    github.com/influxdata/influxdb/tsdb.(*Shard).CreateIterator+0x13c               /go/src/github.com/influxdata/influxdb/tsdb/shard.go:809
#   0x93be9f    github.com/influxdata/influxdb/tsdb.Shards.CreateIterator+0x12f                 /go/src/github.com/influxdata/influxdb/tsdb/shard.go:1212
#   0x956ff4    github.com/influxdata/influxdb/tsdb.(*Shards).CreateIterator+0xb4               <autogenerated>:1
#   0x991afd    github.com/influxdata/influxdb/coordinator.(*LocalShardMapping).CreateIterator+0x46d        /go/src/github.com/influxdata/influxdb/coordinator/shard_mapper.go:208
#   0x887998    github.com/influxdata/influxdb/query.(*exprIteratorBuilder).callIterator.func1+0x138        /go/src/github.com/influxdata/influxdb/query/select.go:814
#   0x86f7d4    github.com/influxdata/influxdb/query.(*exprIteratorBuilder).callIterator+0xd4           /go/src/github.com/influxdata/influxdb/query/select.go:838
#   0x88687f    github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildCallIterator.func1+0xef    /go/src/github.com/influxdata/influxdb/query/select.go:712
#   0x86d2a2    github.com/influxdata/influxdb/query.(*exprIteratorBuilder).buildCallIterator+0x1052        /go/src/github.com/influxdata/influxdb/query/select.go:756
#   0x86befe    github.com/influxdata/influxdb/query.buildExprIterator+0x46e                    /go/src/github.com/influxdata/influxdb/query/select.go:429
#   0x885b8f    github.com/influxdata/influxdb/query.buildFieldIterators.func1+0x22f                /go/src/github.com/influxdata/influxdb/query/select.go:359
#   0x86b961    github.com/influxdata/influxdb/query.buildFieldIterators+0x171                  /go/src/github.com/influxdata/influxdb/query/select.go:406
#   0x869a1f    github.com/influxdata/influxdb/query.buildIterators+0x75f                   /go/src/github.com/influxdata/influxdb/query/select.go:194
#   0x869149    github.com/influxdata/influxdb/query.(*preparedStatement).Select+0x159              /go/src/github.com/influxdata/influxdb/query/select.go:109
#   0x868ee7    github.com/influxdata/influxdb/query.Select+0x147                       /go/src/github.com/influxdata/influxdb/query/select.go:90
#   0x998be5    github.com/influxdata/influxdb/coordinator.(*StatementExecutor).createIterators+0x105       /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:652
#   0x997b2e    github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeSelectStatement+0x8e /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:548
#   0x995115    github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement+0x29a5     /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:62
#   0x866b27    github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery+0xae7            /go/src/github.com/influxdata/influxdb/query/query_executor.go:389

2780963030 172 @ 0x472b5e 0x5271cc 0x52770a 0xa9bd67 0x6c15e4 0xa9c59c 0x6c15e4 0x9c58cb 0xa8543d 0x6c42c4 0x6c04bd 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed                             /usr/local/go/src/sync/mutex.go:134
#   0x5271cb    log.(*Logger).Output+0x7b                           /usr/local/go/src/log/log.go:153
#   0x527709    log.(*Logger).Println+0x69                          /usr/local/go/src/log/log.go:188
#   0xa9bd66    github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1+0x1b6    /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1533
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                     /usr/local/go/src/net/http/server.go:1918
#   0xa9c59b    github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1+0x14b   /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1585
#   0x6c15e3    net/http.HandlerFunc.ServeHTTP+0x43                     /usr/local/go/src/net/http/server.go:1918
#   0x9c58ca    github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP+0x15a         /go/src/github.com/bmizerany/pat/mux.go:117
#   0xa8543c    github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP+0x24c    /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:315
#   0x6c42c3    net/http.serverHandler.ServeHTTP+0xb3                       /usr/local/go/src/net/http/server.go:2619
#   0x6c04bc    net/http.(*conn).serve+0x71c                            /usr/local/go/src/net/http/server.go:1801

2684136930 1 @ 0x40479b 0xb4e17c 0xb32b17 0x94ec2e 0x93571f 0x9501a4 0x45cb41
#   0x40479a    runtime.chanrecv1+0x2a                          /usr/local/go/src/runtime/chan.go:388
#   0xb4e17b    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open+0x7eb /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:501
#   0xb32b16    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).Open+0x256    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:668
#   0x94ec2d    github.com/influxdata/influxdb/tsdb.(*Shard).Open.func1+0x40d       /go/src/github.com/influxdata/influxdb/tsdb/shard.go:320
#   0x93571e    github.com/influxdata/influxdb/tsdb.(*Shard).Open+0x2e          /go/src/github.com/influxdata/influxdb/tsdb/shard.go:331
#   0x9501a3    github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1+0x783 /go/src/github.com/influxdata/influxdb/tsdb/store.go:299

2613804474 1 @ 0x473a79 0xb50eac 0xb1806c 0xb142bc 0xb3a694 0xb8467b 0x45cb41
#   0x473a78    sync.(*RWMutex).RLock+0x48                                  /usr/local/go/src/sync/rwmutex.go:50
#   0xb50eab    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).LastModified+0x4b          /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:788
#   0xb1806b    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*DefaultPlanner).findGenerations+0x87b     /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:591
#   0xb142bb    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*DefaultPlanner).PlanLevel+0x8b        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:210
#   0xb3a693    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact+0x2d3             /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1788
#   0xb8467a    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1+0x5a /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:355

2414902350 2 @ 0x43d002 0x6d04ff 0x6cc7b6 0x67c489 0x67c11d 0x67d7bd 0xbf6d22 0x45cb41
#   0x43d001    runtime.selectgo+0x651                      /usr/local/go/src/runtime/select.go:628
#   0x6d04fe    net/http.(*Transport).getConn+0x5be             /usr/local/go/src/net/http/transport.go:948
#   0x6cc7b5    net/http.(*Transport).RoundTrip+0x6a5               /usr/local/go/src/net/http/transport.go:400
#   0x67c488    net/http.send+0x1a8                     /usr/local/go/src/net/http/client.go:249
#   0x67c11c    net/http.(*Client).send+0xfc                    /usr/local/go/src/net/http/client.go:173
#   0x67d7bc    net/http.(*Client).Do+0x28c                 /usr/local/go/src/net/http/client.go:602
#   0xbf6d21    github.com/influxdata/usage-client/v1.(*Client).Save+0x451  /go/src/github.com/influxdata/usage-client/v1/client.go:77

1993111493 4 @ 0x40479b 0xb18e77 0xb395e9 0xb392ae 0xb3a01c 0xb84701 0x45cb41
#   0x40479a    runtime.chanrecv1+0x2a                                      /usr/local/go/src/runtime/chan.go:388
#   0xb18e76    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).WriteSnapshot+0x386        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:820
#   0xb395e8    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).writeSnapshotAndCommit+0xa8       /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1703
#   0xb392ad    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).WriteSnapshot+0x42d           /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1673
#   0xb3a01b    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache+0x31b            /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1746
#   0xb84700    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1+0x50  /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:434

1528101802 10 @ 0x472b5e 0x473b5d 0xb8d7e4 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed                             /usr/local/go/src/sync/mutex.go:134
#   0x473b5c    sync.(*RWMutex).Lock+0x2c                           /usr/local/go/src/sync/rwmutex.go:93
#   0xb8d7e3    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).scheduleSync.func1+0x163 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:280

1057004970 1 @ 0x4047db 0xb6c9f4 0xb0a74b 0xb39051 0xb3a01c 0xb84701 0x45cb41
#   0x4047da    runtime.chanrecv2+0x2a                                      /usr/local/go/src/runtime/chan.go:393
#   0xb6c9f3    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*ring).apply+0x163             /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/ring.go:176
#   0xb0a74a    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Cache).Deduplicate+0x7a           /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:430
#   0xb39050    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).WriteSnapshot+0x1d0           /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1668
#   0xb3a01b    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache+0x31b            /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1746
#   0xb84700    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1+0x50  /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:434

923592212 1 @ 0x40479b 0xb7665e 0xb759eb 0xb36f6d 0x93629a 0x94846b 0x98fb1f 0x9a1bba 0x45cb41
#   0x40479a    runtime.chanrecv1+0x2a                                      /usr/local/go/src/runtime/chan.go:388
#   0xb7665d    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).writeToLog+0x2fd             /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:460
#   0xb759ea    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).WriteMulti+0x6a              /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:315
#   0xb36f6c    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).WritePoints+0xc1c         /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1238
#   0x936299    github.com/influxdata/influxdb/tsdb.(*Shard).WritePoints+0x239                  /go/src/github.com/influxdata/influxdb/tsdb/shard.go:491
#   0x94846a    github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard+0xea                  /go/src/github.com/influxdata/influxdb/tsdb/store.go:1211
#   0x98fb1e    github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard+0x13e           /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:388
#   0x9a1bb9    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0x99 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311

848860478 1 @ 0x473b9e 0xb8771d 0xb38fc7 0xb3a01c 0xb84701 0x45cb41
#   0x473b9d    sync.(*RWMutex).Lock+0x6d                                   /usr/local/go/src/sync/rwmutex.go:98
#   0xb8771c    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).WriteSnapshot.func2+0x6c      /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1635
#   0xb38fc6    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).WriteSnapshot+0x146           /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1653
#   0xb3a01b    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache+0x31b            /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1746
#   0xb84700    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1+0x50  /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:434

617863816 1 @ 0x472b5e 0x473b5d 0x87eb9e 0x8661a5 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed                             /usr/local/go/src/sync/mutex.go:134
#   0x473b5c    sync.(*RWMutex).Lock+0x2c                           /usr/local/go/src/sync/rwmutex.go:93
#   0x87eb9d    github.com/influxdata/influxdb/query.(*TaskManager).AttachQuery+0x7d        /go/src/github.com/influxdata/influxdb/query/task_manager.go:154
#   0x8661a4    github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery+0x164    /go/src/github.com/influxdata/influxdb/query/query_executor.go:301

547975328 1 @ 0x40479b 0x6d1ac4 0x6dc548 0x45cb41
#   0x40479a    runtime.chanrecv1+0x2a              /usr/local/go/src/runtime/chan.go:388
#   0x6d1ac3    net/http.(*Transport).dialConn+0xdf3        /usr/local/go/src/net/http/transport.go:1157
#   0x6dc547    net/http.(*Transport).getConn.func4+0x77    /usr/local/go/src/net/http/transport.go:943

298170055 3 @ 0x43d002 0x8b46ca 0x8b535d 0x8b5a07 0x4ef6de 0x4ef999 0xb7ea10 0xb1ac65 0xb1a4d9 0xb81631 0x45cb41
#   0x43d001    runtime.selectgo+0x651                                  /usr/local/go/src/runtime/select.go:628
#   0x8b46c9    golang.org/x/time/rate.(*Limiter).waitN+0x2f9                       /go/src/golang.org/x/time/rate/rate.go:259
#   0x8b535c    golang.org/x/time/rate.(*Limiter).WaitN+0x6c                        /go/src/golang.org/x/time/rate/rate_go17.go:20
#   0x8b5a06    github.com/influxdata/influxdb/pkg/limiter.(*Writer).Write+0xb6             /go/src/github.com/influxdata/influxdb/pkg/limiter/writer.go:61
#   0x4ef6dd    bufio.(*Writer).Flush+0x7d                              /usr/local/go/src/bufio/bufio.go:567
#   0x4ef998    bufio.(*Writer).Write+0x108                             /usr/local/go/src/bufio/bufio.go:603
#   0xb7ea0f    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmWriter).WriteBlock+0x15f       /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/writer.go:679
#   0xb1ac64    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).write+0x264        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1099
#   0xb1a4d8    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).writeNewFiles+0x258    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:994
#   0xb81630    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).WriteSnapshot.func1+0xc0   /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:811

287561304 11 @ 0x472b5e 0x473b5d 0xab6f7b 0xab72ed 0xb38d1a 0x936bbe 0x936189 0x94846b 0x98fa72 0x9a1bba 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed                                     /usr/local/go/src/sync/mutex.go:134
#   0x473b5c    sync.(*RWMutex).Lock+0x2c                                   /usr/local/go/src/sync/rwmutex.go:93
#   0xab6f7a    github.com/influxdata/influxdb/tsdb/index/inmem.(*Index).assignExistingSeries+0x29a     /go/src/github.com/influxdata/influxdb/tsdb/index/inmem/inmem.go:1034
#   0xab72ec    github.com/influxdata/influxdb/tsdb/index/inmem.(*ShardIndex).CreateSeriesListIfNotExists+0xcc  /go/src/github.com/influxdata/influxdb/tsdb/index/inmem/inmem.go:1082
#   0xb38d19    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).CreateSeriesListIfNotExists+0xa9  /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1607
#   0x936bbd    github.com/influxdata/influxdb/tsdb.(*Shard).validateSeriesAndFields+0x6dd          /go/src/github.com/influxdata/influxdb/tsdb/shard.go:542
#   0x936188    github.com/influxdata/influxdb/tsdb.(*Shard).WritePoints+0x128                  /go/src/github.com/influxdata/influxdb/tsdb/shard.go:474
#   0x94846a    github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard+0xea                  /go/src/github.com/influxdata/influxdb/tsdb/store.go:1211
#   0x98fa71    github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard+0x91            /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:365
#   0x9a1bb9    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0x99 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311

224073111 1 @ 0x472b5e 0x6feff1 0x6f0576 0x6f21f7 0x70909f 0x866f23 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed                             /usr/local/go/src/sync/mutex.go:134
#   0x6feff0    go.uber.org/zap/zapcore.(*lockedWriteSyncer).Write+0x30             /go/src/go.uber.org/zap/zapcore/write_syncer.go:65
#   0x6f0575    go.uber.org/zap/zapcore.(*ioCore).Write+0x105                   /go/src/go.uber.org/zap/zapcore/core.go:90
#   0x6f21f6    go.uber.org/zap/zapcore.(*CheckedEntry).Write+0xe6              /go/src/go.uber.org/zap/zapcore/entry.go:215
#   0x70909e    go.uber.org/zap.(*Logger).Info+0x7e                     /go/src/go.uber.org/zap/logger.go:187
#   0x866f22    github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery+0xee2    /go/src/github.com/influxdata/influxdb/query/query_executor.go:385

42761540 2 @ 0x472b5e 0x473337 0x47322a 0x6b5171 0x6b5394 0x6bc4ac 0x6c02ae 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed         /usr/local/go/src/sync/mutex.go:134
#   0x473336    sync.(*Pool).getSlow+0xc6       /usr/local/go/src/sync/pool.go:165
#   0x473229    sync.(*Pool).Get+0x139          /usr/local/go/src/sync/pool.go:141
#   0x6b5170    net/http.newTextprotoReader+0x30    /usr/local/go/src/net/http/request.go:895
#   0x6b5393    net/http.readRequest+0x63       /usr/local/go/src/net/http/request.go:920
#   0x6bc4ab    net/http.(*conn).readRequest+0x17b  /usr/local/go/src/net/http/server.go:933
#   0x6c02ad    net/http.(*conn).serve+0x50d        /usr/local/go/src/net/http/server.go:1739

34697385 3 @ 0x473b9e 0xb8d7e4 0x45cb41
#   0x473b9d    sync.(*RWMutex).Lock+0x6d                           /usr/local/go/src/sync/rwmutex.go:98
#   0xb8d7e3    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).scheduleSync.func1+0x163 /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:280

24553421 2 @ 0x472b5e 0x6c517d 0x6bfb9c 0x6dbe34 0x6c06a4 0x45cb41
#   0x472b5d    sync.(*Mutex).Lock+0xed         /usr/local/go/src/sync/mutex.go:134
#   0x6c517c    net/http.(*Server).trackConn+0x3c   /usr/local/go/src/net/http/server.go:2786
#   0x6bfb9b    net/http.(*conn).setState+0xcb      /usr/local/go/src/net/http/server.go:1640
#   0x6dbe33    net/http.(*conn).serve.func1+0x243  /usr/local/go/src/net/http/server.go:1702
#   0x6c06a3    net/http.(*conn).serve+0x903        /usr/local/go/src/net/http/server.go:1811

8894856 1 @ 0x473a79 0xab6d11 0xab72ed 0xb38d1a 0x936bbe 0x936189 0x94846b 0x98fa72 0x9a1bba 0x45cb41
#   0x473a78    sync.(*RWMutex).RLock+0x48                                  /usr/local/go/src/sync/rwmutex.go:50
#   0xab6d10    github.com/influxdata/influxdb/tsdb/index/inmem.(*Index).assignExistingSeries+0x30      /go/src/github.com/influxdata/influxdb/tsdb/index/inmem/inmem.go:1023
#   0xab72ec    github.com/influxdata/influxdb/tsdb/index/inmem.(*ShardIndex).CreateSeriesListIfNotExists+0xcc  /go/src/github.com/influxdata/influxdb/tsdb/index/inmem/inmem.go:1082
#   0xb38d19    github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).CreateSeriesListIfNotExists+0xa9  /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1607
#   0x936bbd    github.com/influxdata/influxdb/tsdb.(*Shard).validateSeriesAndFields+0x6dd          /go/src/github.com/influxdata/influxdb/tsdb/shard.go:542
#   0x936188    github.com/influxdata/influxdb/tsdb.(*Shard).WritePoints+0x128                  /go/src/github.com/influxdata/influxdb/tsdb/shard.go:474
#   0x94846a    github.com/influxdata/influxdb/tsdb.(*Store).WriteToShard+0xea                  /go/src/github.com/influxdata/influxdb/tsdb/store.go:1211
#   0x98fa71    github.com/influxdata/influxdb/coordinator.(*PointsWriter).writeToShard+0x91            /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:365
#   0x9a1bb9    github.com/influxdata/influxdb/coordinator.(*PointsWriter).WritePointsPrivileged.func1+0x99 /go/src/github.com/influxdata/influxdb/coordinator/points_writer.go:311

1696503 8 @ 0x40479b 0x94607b 0x94c1b1 0x45cb41
#   0x40479a    runtime.chanrecv1+0x2a                          /usr/local/go/src/runtime/chan.go:388
#   0x94607a    github.com/influxdata/influxdb/tsdb.(*Store).walkShards+0x10a       /go/src/github.com/influxdata/influxdb/tsdb/store.go:827
#   0x94c1b0    github.com/influxdata/influxdb/tsdb.(*Store).monitorShards+0x390    /go/src/github.com/influxdata/influxdb/tsdb/store.go:1724

1636242 2 @ 0x470cb0 0x6baf36 0x6bf668 0x6c04fd 0x45cb41
#   0x470caf    sync.(*Cond).Wait+0x7f              /usr/local/go/src/sync/cond.go:56
#   0x6baf35    net/http.(*connReader).abortPendingRead+0xb5    /usr/local/go/src/net/http/server.go:690
#   0x6bf667    net/http.(*response).finishRequest+0x97     /usr/local/go/src/net/http/server.go:1518
#   0x6c04fc    net/http.(*conn).serve+0x75c            /usr/local/go/src/net/http/server.go:1806

50372 1 @ 0x403a53 0x95128a 0x45cb41
#   0x403a52    runtime.chansend1+0x42                          /usr/local/go/src/runtime/chan.go:113
#   0x951289    github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x189 /go/src/github.com/influxdata/influxdb/tsdb/store.go:821

xor22h on 2 Apr 2018

This issue is still happening in 1.5.2.

skburgart on 1 Aug 2018

👍5

Influxdb: InfluxDB goes unresponsive

Bug report

Most helpful comment

All 60 comments

8577 should fix the case where writer is blocked indefinitely on the `syncErr` channel.

Related issues

Influxdb: InfluxDB goes unresponsive

Bug report

Most helpful comment

All 60 comments

8577 should fix the case where writer is blocked indefinitely on the syncErr channel.

Related issues

8577 should fix the case where writer is blocked indefinitely on the `syncErr` channel.