Etcd: Umbrella issue for flaky tests

Created on 2 May 2019  路  26Comments  路  Source: etcd-io/etcd

The current test CI is very flaky. We need to start to fix them.

I will look over the recent CI failures and post the flaky test cases here.

Help Wanted good first issue
Source
picture xiang90
👍4

Most helpful comment

Noticed these failures in two different PRs today (basically same failure point for both tests):

--- FAIL: TestV3CurlAuth (1.00s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (52) Empty reply from server\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

--- FAIL: TestV3CurlAuthClientTLSCertAuth (0.26s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (56) Unexpected EOF\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10631/builds/14 for 386-e2e

Fixed by #10800.

picture jingyih on 7 Jun 2019
👍2

All 26 comments

As mentioned in etcd community meeting, I happen to have already collected the flaky tests. Here they are.

Flaky tests:

  • TestBalancerUnderBlackholeKeepAliveWatch
--- FAIL: TestBalancerUnderBlackholeKeepAliveWatch (5.72s)
    black_hole_test.go:83: took too long to receive watch events
  • TestBalancerUnderNetworkPartitionWatchLeader
--- FAIL: TestBalancerUnderNetworkPartitionWatchLeader (3.27s)
    network_partition_test.go:266: took too long to detect leader lost
  • TestRoundRobinBalancedResolvableFailoverFromRequestFail
--- FAIL: TestRoundRobinBalancedResolvableFailoverFromRequestFail (0.03s)
    balancer_test.go:308: expected balanced loads for 20 requests, got switches 2
  • TestLeasingTxnCancel (stack trace https://travis-ci.com/etcd-io/etcd/jobs/193773143#L1807)
=== RUN   TestLeasingTxnCancel
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xb0184d]
  • TestLeasingNonOwnerPutError (stack trace https://travis-ci.com/etcd-io/etcd/jobs/197250711#L2017)
=== RUN   TestLeasingNonOwnerPutError
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]
  • TestLeasingOwnerDeleteError (stack trace https://travis-ci.com/etcd-io/etcd/jobs/196938398#L1882)
=== RUN   TestLeasingOwnerDeleteError
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]

We fixed the following tests. If you see them in CI failure after the fix PR, please let us know.

  • TestResumeElection
    Fixed by #10699.
--- FAIL: TestResumeElection (0.04s)
    testing.go:771: race detected during execution of test
  • TestLeasingDeleteRangeContendTxn (stack trace https://travis-ci.com/etcd-io/etcd/jobs/197250712#L2277)
    Fixed by #10772.
--- FAIL: TestLeasingDeleteRangeContendTxn (0.94s)
    testing.go:771: race detected during execution of test

  • TestBarrierSingleNode
    Fixed by #10772.

  • TestBarrierMultiNode
    Fixed by #10772.

jingyih picture jingyih on 2 May 2019
👍1

another one txn cancel

=== RUN   TestLeasingTxnCancel
{"level":"warn","ts":"2019-05-14T14:00:38.384Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-aa069d4d-0556-44fb-b798-344b6508f99f/localhost:715405736098611950","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: leader changed"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]
goroutine 20054 [running]:
google.golang.org/grpc.(*csAttempt).sendMsg(0xc00025ba20, 0x12b23c0, 0xc00051c2e0, 0xc0019569c8, 0x5, 0x5, 0xc0019569d0, 0xb, 0xb, 0xc0019569d0, ...)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:713 +0x14d
google.golang.org/grpc.(*clientStream).SendMsg.func2(0xc00025ba20, 0x48, 0x48)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:637 +0x186
google.golang.org/grpc.(*clientStream).withRetry(0xc001d26480, 0xc0019cf900, 0xc0012cbaa0, 0xc0019569c8, 0x5)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:530 +0x431
google.golang.org/grpc.(*clientStream).SendMsg(0xc001d26480, 0x12b23c0, 0xc00051c2e0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:643 +0x80c
go.etcd.io/etcd/v3/etcdserver/etcdserverpb.(*watchWatchClient).Send(0xc00051c2d0, 0xc00051c2e0, 0x0, 0x1cd5a18)
    /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3618 +0x6b
go.etcd.io/etcd/v3/clientv3.(*watchGrpcStream).run(0xc001a496c0)
    /go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd8d
created by go.etcd.io/etcd/v3/clientv3.(*watcher).newWatcherGrpcStream
    /go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c
FAIL    go.etcd.io/etcd/v3/clientv3/integration 128.500s
xiang90 picture xiang90 on 14 May 2019

semaphoreci sometimes ran out of the 20m limit.

xiang90 picture xiang90 on 14 May 2019

another TestBalancerUnderBlackholeKeepAliveWatch 

=== RUN   TestBalancerUnderBlackholeKeepAliveWatch
--- FAIL: TestBalancerUnderBlackholeKeepAliveWatch (5.76s)
    black_hole_test.go:83: took too long to receive watch events
xiang90 picture xiang90 on 14 May 2019

TestCtlV3AuthFromKeyPerm

--- FAIL: TestCtlV3AuthFromKeyPerm (16.21s)
    testutil.go:55: goroutine 3497 [running]:
        go.etcd.io/etcd/pkg/testutil.FatalStack(0xacc2140, 0xae3c020, 0x18)
            /go/src/go.etcd.io/etcd/pkg/testutil/testutil.go:54 +0x60
        go.etcd.io/etcd/tests/e2e.testCtl(0xacc2140, 0x87592e4, 0x0, 0x0, 0x0)
            /go/src/go.etcd.io/etcd/tests/e2e/ctl_v3_test.go:184 +0x3c1
        go.etcd.io/etcd/tests/e2e.TestCtlV3AuthFromKeyPerm(0xacc2140)
            /go/src/go.etcd.io/etcd/tests/e2e/ctl_v3_auth_test.go:43 +0x43
        testing.tRunner(0xacc2140, 0x8758f7c)
            /usr/local/go/src/testing/testing.go:827 +0xa3
        created by testing.(*T).Run

https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10683/builds/5

xiang90 picture xiang90 on 14 May 2019

TestBarrierSingleNode

WARNING: DATA RACE
Write at 0x00c006787ba8 by goroutine 334:
  go.etcd.io/etcd/mvcc.(*keyIndex).put()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:101 +0x206
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:141 +0x2ea
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c006787ba8 by goroutine 415:
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:365 +0x6ae
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 334 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 415 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Write at 0x00c006787b90 by goroutine 334:
  go.etcd.io/etcd/mvcc.(*keyIndex).put()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:102 +0x27f
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:141 +0x2ea
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c006787b90 by goroutine 415:
  go.etcd.io/etcd/mvcc.(*keyIndex).findGeneration()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:313 +0x150
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:160 +0x255
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 334 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 415 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Write at 0x00c0031fe468 by goroutine 334:
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:142 +0x3b7
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c0031fe468 by goroutine 415:
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:298 +0x61
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 334 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 415 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Read at 0x00c00063bfd8 by goroutine 1012:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:618 +0x225
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Previous write at 0x00c00063bfd8 by goroutine 843:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:620 +0x33b
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 1012 (running) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
Goroutine 843 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Read at 0x00c014205820 by goroutine 1012:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:618 +0x23e
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Previous write at 0x00c014205820 by goroutine 843:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:95 +0x1b7
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 1012 (running) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
Goroutine 843 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
--- FAIL: TestBarrierSingleNode (0.25s)
    testing.go:771: race detected during execution of test
=== RUN   TestBarrierMultiNode
==================
WARNING: DATA RACE
Write at 0x00c00042d4b8 by goroutine 689:
  go.etcd.io/etcd/mvcc.(*keyIndex).put()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:101 +0x206
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:141 +0x2ea
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c00042d4b8 by goroutine 271:
  go.etcd.io/etcd/mvcc.(*keyIndex).findGeneration()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:313 +0x150
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:160 +0x255
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 689 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 271 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
--- FAIL: TestBarrierMultiNode (0.66s)
    testing.go:771: race detected during execution of test
xiang90 picture xiang90 on 14 May 2019 
--- FAIL: proxy-clientv3/integration
--- FAIL: tests/e2e-proxy
make: *** [docker-test-coverage] Error 1
travis_time:end:2b992fcc:start=1555527843145227846,finish=1555530481575826563,duration=2638430598717
The command "case "${TARGET}" in
  linux-amd64-fmt)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 PASSES='fmt bom dep' ./test"
    ;;
  linux-amd64-integration-1-cpu)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 CPU=1 PASSES='integration' ./test"
    ;;
  linux-amd64-integration-2-cpu)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 CPU=2 PASSES='integration' ./test"
    ;;
  linux-amd64-integration-4-cpu)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 CPU=4 PASSES='integration' ./test"
    ;;
  linux-amd64-functional)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "./build && GOARCH=amd64 PASSES='functional' ./test"
    ;;
  linux-amd64-unit)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 PASSES='unit' ./test"
    ;;
  all-build)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 PASSES='build' ./test \
        && GOARCH=386 PASSES='build' ./test \
        && GO_BUILD_FLAGS='-v' GOOS=darwin GOARCH=amd64 ./build \
        && GO_BUILD_FLAGS='-v' GOOS=windows GOARCH=amd64 ./build \
        && GO_BUILD_FLAGS='-v' GOARCH=arm ./build \
        && GO_BUILD_FLAGS='-v' GOARCH=arm64 ./build \
        && GO_BUILD_FLAGS='-v' GOARCH=ppc64le ./build"
    ;;
  linux-amd64-grpcproxy)
    sudo HOST_TMP_DIR=/tmp TEST_OPTS="PASSES='build grpcproxy'" make docker-test
    ;;
  linux-amd64-coverage)
    sudo HOST_TMP_DIR=/tmp make docker-test-coverage
    ;;
  linux-amd64-fmt-unit-go-tip)
    GOARCH=amd64 PASSES='fmt unit' ./test
    ;;
  linux-386-unit)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=386 PASSES='unit' ./test"
    ;;
esac
" exited with 2.
xiang90 picture xiang90 on 14 May 2019 
=== RUN   TestLeasingTxnOwnerIf
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]
goroutine 20358 [running]:
google.golang.org/grpc.(*csAttempt).sendMsg(0xc0017386e0, 0x12b7d20, 0xc0011ba620, 0xc00615e92b, 0x5, 0x5, 0xc00615e920, 0xb, 0xb, 0xc00615e920, ...)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:713 +0x14d
google.golang.org/grpc.(*clientStream).SendMsg.func2(0xc0017386e0, 0x48, 0x48)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:637 +0x186
google.golang.org/grpc.(*clientStream).withRetry(0xc0000bed80, 0xc001beaeb0, 0xc001781aa0, 0xc00615e92b, 0x5)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:530 +0x431
google.golang.org/grpc.(*clientStream).SendMsg(0xc0000bed80, 0x12b7d20, 0xc0011ba620, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/[email protected]/stream.go:643 +0x80c
go.etcd.io/etcd/v3/etcdserver/etcdserverpb.(*watchWatchClient).Send(0xc0011ba5f0, 0xc0011ba620, 0x0, 0x1cdfb18)
    /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3689 +0x6b
go.etcd.io/etcd/v3/clientv3.(*watchGrpcStream).run(0xc001757450)
    /go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd8d
created by go.etcd.io/etcd/v3/clientv3.(*watcher).newWatcherGrpcStream
    /go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c
xiang90 picture xiang90 on 15 May 2019 
--- FAIL: TestBalancerUnderNetworkPartitionWatchLeader (3.43s)
    network_partition_test.go:266: took too long to detect leader lost
xiang90 picture xiang90 on 16 May 2019 
--- FAIL: TestLeasingDeleteRangeContendTxn (0.73s)
    testing.go:771: race detected during execution of test
xiang90 picture xiang90 on 16 May 2019

--- FAIL: TestBalancerUnderNetworkPartitionWatchLeader (3.29s)
network_partition_test.go:266: took too long to detect leader lost

https://travis-ci.com/etcd-io/etcd/jobs/204590472

xiang90 picture xiang90 on 31 May 2019

Noticed these failures in two different PRs today (basically same failure point for both tests):

--- FAIL: TestV3CurlAuth (1.00s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (52) Empty reply from server\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

--- FAIL: TestV3CurlAuthClientTLSCertAuth (0.26s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (56) Unexpected EOF\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10631/builds/14 for 386-e2e

spzala picture spzala on 4 Jun 2019 
=== RUN   TestLeasingReconnectTxn
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xb5c7ca]
goroutine 30639 [running]:
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*csAttempt).sendMsg(0xc000150210, 0x13438c0, 0xc0011b7760, 0xc0016dc2f8, 0x5, 0x5, 0xc0016dc300, 0xb, 0xb, 0xc0016dc300, ...)
    /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:713 +0x14a
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).SendMsg.func2(0xc000150210, 0xb5b88f, 0xc00283ccd0)
    /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:637 +0x186
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00017a5a0, 0xc00283ccd0, 0xc0016abaa8, 0xc0016dc2f8, 0x0)
    /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:530 +0x43d
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).SendMsg(0xc00017a5a0, 0x13438c0, 0xc0011b7760, 0x0, 0x0)
    /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:643 +0x777
go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Send(0xc0011b7730, 0xc0011b7760, 0x0, 0x1e4cf98)
    /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3697 +0x6b
go.etcd.io/etcd/clientv3.(*watchGrpcStream).run(0xc000360410)
    /go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd9b
created by go.etcd.io/etcd/clientv3.(*watcher).newWatcherGrpcStream
    /go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c

https://travis-ci.com/etcd-io/etcd/jobs/205678389#L2183

spzala picture spzala on 5 Jun 2019

Noticed these failures in two different PRs today (basically same failure point for both tests):

--- FAIL: TestV3CurlAuth (1.00s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (52) Empty reply from server\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

--- FAIL: TestV3CurlAuthClientTLSCertAuth (0.26s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (56) Unexpected EOF\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10631/builds/14 for 386-e2e

Fixed by #10800.

jingyih picture jingyih on 7 Jun 2019
👍2

=== RUN TestLeasingGetWithOpts
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xb5c7ca]
goroutine 24216 [running]:
go.etcd.io/etcd/vendor/google.golang.org/grpc.(csAttempt).sendMsg(0xc00065c160, 0x1346c20, 0xc0003b0ec0, 0xc002edc378, 0x5, 0x5, 0xc002edc380, 0xb, 0xb, 0xc002edc380, ...)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:713 +0x14a
go.etcd.io/etcd/vendor/google.golang.org/grpc.(clientStream).SendMsg.func2(0xc00065c160, 0xb5b88f, 0xc00016f130)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:637 +0x186
go.etcd.io/etcd/vendor/google.golang.org/grpc.(clientStream).withRetry(0xc00183a240, 0xc00016f130, 0xc0003edaa8, 0xc002edc378, 0x0)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:530 +0x43d
go.etcd.io/etcd/vendor/google.golang.org/grpc.(clientStream).SendMsg(0xc00183a240, 0x1346c20, 0xc0003b0ec0, 0x0, 0x0)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:643 +0x777
go.etcd.io/etcd/etcdserver/etcdserverpb.(watchWatchClient).Send(0xc00088cbb0, 0xc0003b0ec0, 0x0, 0x1e53018)
/go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3697 +0x6b
go.etcd.io/etcd/clientv3.(watchGrpcStream).run(0xc0001275f0)
/go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd9b
created by go.etcd.io/etcd/clientv3.(*watcher).newWatcherGrpcStream
/go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c

jingyih picture jingyih on 14 Jun 2019

This one is pretty new, only observed it in two very recent tests (might be related to recently merged PRs?)
https://travis-ci.com/etcd-io/etcd/jobs/208110433#L1176
https://travis-ci.com/etcd-io/etcd/jobs/208860886#L1256

--- FAIL: TestKVPutError (0.18s)
kv_test.go:56: expected etcdserver: request is too large, got etcdserver: mvcc: database space exceeded

jingyih picture jingyih on 18 Jun 2019

As reported under https://github.com/etcd-io/etcd/issues/10787 (I believe we should close it and track the test failure here)

=== RUN   TestCtlV3AuthEndpointHealth
panic: test timed out after 20m0s

goroutine 6839 [running]:
testing.(*M).startAlarm.func1()
    /usr/local/go/src/testing/testing.go:1334 +0xdf
created by time.goFunc
    /usr/local/go/src/time/sleep.go:169 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0001ca100, 0xcba649, 0x1b, 0xce1508, 0x4b6c01)
    /usr/local/go/src/testing/testing.go:917 +0x381
make: *** [docker-test] Error 1
spzala picture spzala on 1 Jul 2019

Just as a heads up, the linux-amd64-integration-4-cpu is now skipped as a temporary measure. From what I saw, it is usually the target that incurred the failures.

In my book, this also increases the priority of resolving these flakes.

tbg picture tbg on 11 Jul 2019

@jingyih in response to https://github.com/etcd-io/etcd/issues/10700#issuecomment-499718579 these errors still appear today: https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10903/builds/2

--- FAIL: TestCtlV2BackupSnapshot (0.46s)
    ctl_v2_test.go:536: could not start etcd process cluster (read /dev/ptmx: input/output error)
=== RUN   TestCtlV2BackupV3
--- PASS: TestCtlV2BackupV3 (2.34s)
=== RUN   TestCtlV2BackupV3Snapshot
--- FAIL: TestCtlV2BackupV3Snapshot (0.31s)
    ctl_v2_test.go:536: could not start etcd process cluster (read /dev/ptmx: input/output error)

I found this issue which is perhaps related; https://github.com/creack/pty/issues/21

Edit (jingyih@, Fri 19 Jul 2019 05:47:20 PM PDT): fixed by #10908.

tbg picture tbg on 17 Jul 2019

@tbg This is not the same as those previously fixed by #10800. They may be related and we should investigate. Thanks for reporting!

jingyih picture jingyih on 17 Jul 2019

^- I pulled on this thread because these tests seem to somehow be related to version compatibility, and found that I broke this #10889. I probably didn't notice because the semaphore tests were failing for unrelated reasons.

Will investigate more. I assume there's a better error message that just gets lost somehow.

PS I am using HOST_TMP_DIR=/tmp TEST_OPTS="TESTCASE='TestCtlV2Backup.*Snap' GOARCH=386 PASSES='build e2e'" make docker-test on my linux machine to verify.

tbg picture tbg on 19 Jul 2019

Ha, took me 45 minutes to figure out how to get the output:

HOST_TMP_DIR=/tmp TEST_OPTS="EXPECT_DEBUG=true CPU=4 TESTCASE='TestCtlV2BackupV3' GOARCH=386 PASSES='e2e'" make docker-test

Now that I have it, here we go:

../../bin/etcd-12937: 2019-07-19 12:23:08.623487 I | etcdserver: 6c0a2f9e2301 as single-node; fast-forwarding 9 ticks (election ticks 10)
../../bin/etcd-12937: panic: removed all voters
../../bin/etcd-12937:
../../bin/etcd-12937: goroutine 141 [running]:
../../bin/etcd-12937: go.etcd.io/etcd/raft.(*raft).applyConfChange(0xb4ee000, 0x0, 0x0, 0x1, 0x7181d758, 0xca50e935, 0x0, 0x0, 0x0, 0x0, ...)
../../bin/etcd-12937:   /go/src/go.etcd.io/etcd/raft/raft.go:1406 +0x43f
../../bin/etcd-12937: go.etcd.io/etcd/raft.(*node).run(0xb49c180, 0xb4ee000)
../../bin/etcd-12937:   /go/src/go.etcd.io/etcd/raft/node.go:372 +0xcd6
../../bin/etcd-12937: created by go.etcd.io/etcd/raft.RestartNode
../../bin/etcd-12937:   /go/src/go.etcd.io/etcd/raft/node.go:258 +0x252

This indeed introduced in #10889. Now to find out whether the test is doing something silly.

tbg picture tbg on 19 Jul 2019

Still poking, but confirming that at least from the vantage point of raft there's a voter being removed:

../../bin/etcd-15895: {"level":"info","ts":"2019-07-19T13:02:09.719Z","caller":"raft/node.go:344","msg":"applying conf change {ID:0 Type:ConfChangeRemoveNode NodeID:14578408409545168728 Context:[] XXX_unrecognized:[]}, current config is voters=(14578408409545168728)"}

tbg picture tbg on 19 Jul 2019

Ok I got it, this shady behavior on the part of etcd. There's this code in restartAsStandaloneNode, a method, which if the name suggests correctly, basically wants to rewrite the config of a node and make it the only voter in it.

https://github.com/etcd-io/etcd/blob/905aefcd3d7f6be6593ac13c603a244898ded0f8/etcdserver/raft.go#L573-L580

Unfortunately that code removes before it adds, so if our node is B and the config is A, then it will first remove A and then add B, ending up with something empty in the middle. It needs to add everyone first, then remove. I'll send a patch.

tbg picture tbg on 19 Jul 2019

https://github.com/etcd-io/etcd/pull/10908

tbg picture tbg on 19 Jul 2019

Let's continue this on https://github.com/etcd-io/etcd/issues/10979

gyuho picture gyuho on 5 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gyuho picture gyuho  路  4Comments
invidian picture invidian  路  3Comments
el10savio picture el10savio  路  4Comments
gek0 picture gek0  路  3Comments
ghost picture ghost  路  3Comments