Dgraph: data losing

Created on 18 Apr 2020 · 11Comments · Source: dgraph-io/dgraph

What version of Dgraph are you using?

v1.2.2

Have you tried reproducing the issue with the latest release?

What is the hardware spec (RAM, OS)?

ubuntu 18.04 - CPU 4Core - 8GB Ram

Dgraph configurations

they registered as service in /etc/systemd/system/dg0.service and ...
for example

[Unit]
Description=my main dgraph zero server
After=network.target
StartLimitBurst=5
StartLimitIntervalSec=10

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/local/bin/dgraph zero --config /db/z0/config.json

[Install]
WantedBy=multi-user.target

1 Zero [
{
"my": "localhost:5080",
"replicas": 3,
"wal": "/db/z0/zw"
}
]
3 Alphas [
{
"my": "localhost:7080",
"zero": "localhost:5080",
"lru_mb": 2048,
"postings": "/db/a0/p",
"wal": "/db/a0/w"
},{
"my": "localhost:7079",
"zero": "localhost:5080",
"lru_mb": 2048,
"postings": "/db/a1/p",
"wal": "/db/a1/w",
"port_offset": -1
},{
"my": "localhost:7078",
"zero": "localhost:5080",
"lru_mb": 2048,
"postings": "/db/a2/p",
"wal": "/db/a2/w",
"port_offset": -2
}
]

Expected behaviour and actual result.

current Dgraph index 0x16993f
I have a really edgy and complicated data including 2000 exams and ~50000 Questions and 200000 choices. each node has a ~owner . in other word there is a person who owned all those nodes.
there are some other things he owed too.
i've got nothing when runing:

{
   data(func:uid(UserUid)) {
         owner @filter(type(Exam)) {
            uid
            name
         }
   }
}

bot i have this one

{
   data(func:uid(UserUid)) {
         owner @filter(uid(ExamUid)) {
            uid
            name
         }
   }
}

and this one

{
   data(func:type(Exam)) @filter(uid(ExamUid)) {
         uid
   }
}

It happened to me twice and i restore from backup each time.
i zipped the p&w and also export the database after crashing and zipped it to and here they are
the exported data is not complete. i mean don't contain the whole data. so i couldn't restore it.
export.zip
w.1.zip
the p.1.zip

Main Questions

fixing data is not important . the most important thing is that what should i do to prevent this happen again?

can Simultaneous requests cause this?

kinbug kinquestion statuaccepted

Source

H4midR

All 11 comments

I think the problem is in the way that you are creating the data. I also encountered some problems that confused me.

For example, in your query, you use the "owner" predicate with the first letter in lower case. However, in your dataset, it has no lower case predicate. And Dgraph is case sensitive. Also, there is no "tuid", maybe it was a typo.

When switching from "owner" to "Owner" I still didn't get data. I decided to appeal to expand using "expand (_all _) {expand (_all _) {expand (_all_)}}" to check the data structure.

What I found is that all data is there, but its type schema does not match what is in the dataset. Or the Type Schema doesn't reflects the structure of your dataset.

Also, you have repeated predicates with different cases. It is considered good practice to adopt a standard. e.g. camelCase or Pascal Case. So you avoid confusion.

Another thing, avoid zipping Dgraph folders. Always use Export for this. If you rely on the files, there may be compatibility issues in some cases unintentionally. It's good to avoid. But if you know what you're doing, okay tho.

Anyway, the problem is in the way you insert the data and how you are designing the structure. If you share these details I may be able to help.

MichelDiz on 19 Apr 2020

@MichelDiz
thanks brother for notations.
"tuid" was typing mistake.i updated it.
the types defined PascalCase and pridicates are cameCase
the structure was fine and working perfect before.

that export file is not correct due to exported after crash.
here is one old correct one.
but due to avoiding high load to database by programming mistake the types didn't defined completely, so the expand(_all_) wont work at all.
here is one good export.
the data are OK. and schema is correct.
export.OK.zip

i will prepare the full types definition for you and add it here as soon as possible.

i'm ready to get access to server to you and will be Grateful for your helping and appreciate for that.

H4midR on 19 Apr 2020

👍1

some of types are

type Person {
    xid
    name
    fallows
    owner
    buy
    used
    shared
}
type Exam {
    name
    component
}
type Question{
    xid
    name
    choice
}
type ExamSessionGroup {
    xid
    endDate
    startDate
    exam
    examSession
}
type ExamSession {
    xid
    corrected
    result
    answer
    question
    scoring
}

H4midR on 19 Apr 2020

2 Notes : that may cues this.
Max Open File was 1024 ;
*referring to official documentation it's better to be 65535 .
the server configuration is not enogph:

low Memory which cues some daily Out Of Memory Crash
*the recommended minimum is 8GB.

is it just it or other thing may inflect it too?
there are some bigger projects who are working fine with Dgraph which can be count this as one.

H4midR on 19 Apr 2020

About "Max Open File was 1024" this is standard on the OS. In the documentation, it is recommended that the user increase the limit https://dgraph.io/docs/deploy/#too-many-open-files

Regarding memory, it is good that you do load tests. Imagine scenarios and create a "load test" for these scenarios and see how your instances react. The recommendation in the minimum documentation is for minimum loads. Intense use and giant loads require more resources, so you have to adapt to the demand you need.

The Type seems better now.

Now, after the new dataset. I'm not sure what the issue is.
Are you saying that after a crash Dgraph has modified the dataset?
In your previous dataset you had an "Owner" predicate and in this new one is "owner". Did the crash change the predicate case? that's weird. In the previous dataset the data was there, but with different predicates.

With the new dataset provided, I got:

{
   data(func: uid(0x5c8e2)) {
    uid
    owner(first:1) @filter(type(Exam)) {
    uid
    name
  }
   }
}

{
  "data": {
    "data": [
      {
        "uid": "0x5c8e2",
        "owner": [
          {
            "uid": "0x509a7",
            "name": "آزمون آنلاین تستی عربی دهم رشته انسانی | درس 8: يا مَنْ فِی الْبِحارِ عَجائِبُهُ"
          }
        ]
      }
    ]
  }
}

So, there's no issue now right?

MichelDiz on 20 Apr 2020

I have a really edgy and complicated data including 2000 exams and ~50000 Questions and 200000 choices. each node has a ~owner . in other word there is a person who owned all those nodes.

A question. I didn't quite understand that part. Are you saying that a single user holds 252,000 objects? Sometimes the browser may crash with a certain amount of information on the screen. That was what happened?

MichelDiz on 20 Apr 2020

About "Max Open File was 1024" this is standard on the OS. In the documentation, it is recommended that the user increase the limit https://dgraph.io/docs/deploy/#too-many-open-files

Regarding memory, it is good that you do load tests. Imagine scenarios and create a "load test" for these scenarios and see how your instances react. The recommendation in the minimum documentation is for minimum loads. Intense use and giant loads require more resources, so you have to adapt to the demand you need.

The Type seems better now.

Now, after the new dataset. I'm not sure what the issue is.
Are you saying that after a crash Dgraph has modified the dataset?
In your previous dataset you had an "Owner" predicate and in this new one is "owner". Did the crash change the predicate case? that's weird. In the previous dataset the data was there, but with different predicates.

With the new dataset provided, I got:
{
   data(func: uid(0x5c8e2)) {
    uid
    owner(first:1) @filter(type(Exam)) {
    uid
    name
  }
   }
}
{
  "data": {
    "data": [
      {
        "uid": "0x5c8e2",
        "owner": [
          {
            "uid": "0x509a7",
            "name": "آزمون آنلاین تستی عربی دهم رشته انسانی | درس 8: يا مَنْ فِی الْبِحارِ عَجائِبُهُ"
          }
        ]
      }
    ]
  }
}
So, there's no issue now right?

yes , the export.OK.zip was exported a month ago it's ok and data are fine.
the Owner pridicate was a mistake from old preview versions. you can simply drop it. forget about the Owner .
the "owner" is correct and your result is fine .
my main question is what was the main reason of this happened?
the file attached to the first comment was exported after the crash and it wasn't complete, some predicates disappeared.

H4midR on 20 Apr 2020

I have a really edgy and complicated data including 2000 exams and ~50000 Questions and 200000 choices. each node has a ~owner . in other word there is a person who owned all those nodes.

A question. I didn't quite understand that part. Are you saying that a single user holds 252,000 objects? Sometimes the browser may crash with a certain amount of information on the screen. That was what happened?

the browser is not important , it's something usual that browser can't handle a canvas with too many nodes.
the go-client ( dgo ) couldn't get data from this query :

{
   data(func: uid(0x5c8e2)) {
    uid
    owner(first:1) @filter(type(Exam)) {
    uid
    name
  }
   }
}

the point is that this person who owned this 250000 nodes is the admin of website, other users owned things too. other users was fine while the admin lost his owned data.some of this users buied some objects from admin. so they could see that nodes and other part of website working well , but the admin who owned that 250000 nods couldn't see them.

H4midR on 20 Apr 2020

👀1

I done some tests with different server configurations .
1RAM - 1CPU

1Z - 1A
16RAM - 4CPU
1Z - 1A
1Z - 3A

test 1 :

adding Exam with Question Exist in bank;
*Each test has between one and 48,000 random questions.the average query from database was 17600 questions
* each exam took about 2.5 second for insertion.

test :

adding Questions and give its ownership to admin .
**5-8 questions per second inserted

the result was interesting and that error occurs again.
i thing the reason of it was the Out Of Memory Crash While Mutations.
it's like disk sector failure. the data exist but the path of getting them is damaged.
we have several crash in each test but none of them didn't damage the data.
but data damage in first configuration when the crash happend while Out Of Memory Crash.
here is logs.
i recorded some video if you are interested in.

View Log

...
es:18446744073709551615 expiryTs:1589755638 enabled:true >
I0420 06:07:19.498888   25203 node.go:182] Setting conf state to nodes:1
I0420 06:07:19.499218   25203 node.go:182] Setting conf state to nodes:1 nodes:2
I0420 06:07:19.500257   25203 node.go:182] Setting conf state to nodes:1 nodes:2 nodes:3
I0420 06:07:21.540806   25203 log.go:34] raft.node: 1 elected leader 2 at term 17
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x189bbf8, 0x16)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sysMap(0xc040000000, 0x4000000, 0x24a4298)
        /usr/local/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x248af20, 0x41e000, 0xa280e5, 0x7f8a989ff190)
        /usr/local/go/src/runtime/malloc.go:701 +0x1cd
runtime.(*mheap).grow(0x248af20, 0x20f, 0xffffffff)
        /usr/local/go/src/runtime/mheap.go:1255 +0xa3
runtime.(*mheap).allocSpanLocked(0x248af20, 0x20f, 0x24a42a8, 0x961805)
        /usr/local/go/src/runtime/mheap.go:1170 +0x266
runtime.(*mheap).alloc_m(0x248af20, 0x20f, 0x7f894c5f0101, 0x7f894c5fb5c8)
        /usr/local/go/src/runtime/mheap.go:1022 +0xc2
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:1093 +0x4c
runtime.(*mheap).alloc(0x248af20, 0x20f, 0x7f8a85010101, 0x7f894c5fb5c8)
        /usr/local/go/src/runtime/mheap.go:1092 +0x8a
runtime.largeAlloc(0x41ca8c, 0x990101, 0x7f894c5fb5c8)
        /usr/local/go/src/runtime/malloc.go:1138 +0x97
runtime.mallocgc.func1()
        /usr/local/go/src/runtime/malloc.go:1033 +0x46
runtime.systemstack(0x0)
        /usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1146

goroutine 693 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0249de758 sp=0xc0249de750 pc=0x9a1a10
runtime.mallocgc(0x41ca8c, 0x1699b60, 0x1, 0x0)
        /usr/local/go/src/runtime/malloc.go:1032 +0x895 fp=0xc0249de7f8 sp=0xc0249de758 pc=0x94ffa5
runtime.makeslice(0x1699b60, 0x41ca8c, 0x41ca8c, 0x9b0712)
        /usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc0249de828 sp=0xc0249de7f8 pc=0x98a3ec
bytes.makeSlice(0x41ca8c, 0x0, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:229 +0x77 fp=0xc0249de890 sp=0xc0249de828 pc=0xa42e37
bytes.(*Buffer).grow(0xc03f464780, 0xa, 0x1a58c01)
        /usr/local/go/src/bytes/buffer.go:142 +0x15b fp=0xc0249de8e0 sp=0xc0249de890 pc=0xa4277b
bytes.(*Buffer).WriteString(0xc03f464780, 0x173ea8a, 0xa, 0x7f8a98842f00, 0xc03f464780, 0x173ea01)
        /usr/local/go/src/bytes/buffer.go:184 +0xdd fp=0xc0249de910 sp=0xc0249de8e0 pc=0xa42b9d
io.WriteString(0x1a58ca0, 0xc03f464780, 0x173ea8a, 0xa, 0xc03f464780, 0x0, 0x1aa36c0)
        /usr/local/go/src/io/io.go:291 +0x74 fp=0xc0249de958 sp=0xc0249de910 pc=0xa0dd64
github.com/golang/protobuf/proto.(*textWriter).WriteString(0xc03bf35ce0, 0x173ea8a, 0xa, 0x1, 0x1, 0x5)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:83 +0xb7 fp=0xc0249de9a0 sp=0xc0249de958 pc=0xcd2977
github.com/golang/protobuf/proto.writeName(0xc03bf35ce0, 0xc023ba0e40, 0x185, 0x0)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:163 +0x46 fp=0xc0249de9e0 sp=0xc0249de9a0 pc=0xcd30b6
github.com/golang/protobuf/proto.(*TextMarshaler).writeStruct(0x22a0720, 0xc03bf35ce0, 0x18395c0, 0xc03f18dcc0, 0x199, 0x199, 0x1a58ca0)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:429 +0x7e2 fp=0xc0249dec78 sp=0xc0249de9e0 pc=0xcd4262
github.com/golang/protobuf/proto.(*TextMarshaler).writeAny(0x22a0720, 0xc03bf35ce0, 0x1852f60, 0xc03f3ee7a8, 0x196, 0xc023ba0b40, 0x196, 0x0)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:534 +0x236 fp=0xc0249ded30 sp=0xc0249dec78 pc=0xcd5706
github.com/golang/protobuf/proto.(*TextMarshaler).writeStruct(0x22a0720, 0xc03bf35ce0, 0x1832e40, 0xc03f463cc0, 0x199, 0x199, 0x7f8a989a0520)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:314 +0x149b fp=0xc0249defc8 sp=0xc0249ded30 pc=0xcd4f1b
github.com/golang/protobuf/proto.(*TextMarshaler).Marshal(0x22a0720, 0x1a58ca0, 0xc03f464780, 0x1a7c160, 0xc03f463cc0, 0x7f894c851fff, 0x400)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:809 +0x295 fp=0xc0249df068 sp=0xc0249defc8 pc=0xcd7805
github.com/golang/protobuf/proto.(*TextMarshaler).Text(0x22a0720, 0x1a7c160, 0xc03f463cc0, 0x101, 0x7f8a9898d6d0)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:821 +0x6e fp=0xc0249df0b8 sp=0xc0249df068 pc=0xcd7c4e
github.com/golang/protobuf/proto.CompactTextString(...)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/proto/text.go:843
github.com/dgraph-io/dgraph/protos/pb.(*Mutations).String(0xc03f463cc0, 0x1842fe0, 0xc03f463cc0)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:1895 +0x43 fp=0xc0249df0f0 sp=0xc0249df0b8 pc=0xe23f33
fmt.(*pp).handleMethods(0xc03c5861a0, 0x20300f00000076, 0x7f894c851f01)
        /usr/local/go/src/fmt/print.go:630 +0x302 fp=0xc0249df300 sp=0xc0249df0f0 pc=0xa27b22
fmt.(*pp).printArg(0xc03c5861a0, 0x1842fe0, 0xc03f463cc0, 0x20300f00000076)
        /usr/local/go/src/fmt/print.go:713 +0x206 fp=0xc0249df398 sp=0xc0249df300 pc=0xa280e6
fmt.(*pp).doPrintf(0xc03c5861a0, 0x189c1e8, 0x17, 0xc0249df6e8, 0x1, 0x1)
        /usr/local/go/src/fmt/print.go:1030 +0x15b fp=0xc0249df480 sp=0xc0249df398 pc=0xa2baab
fmt.Sprintf(0x189c1e8, 0x17, 0xc0249df6e8, 0x1, 0x1, 0xc03c58eae4, 0x10)
        /usr/local/go/src/fmt/print.go:219 +0x66 fp=0xc0249df4d8 sp=0xc0249df480 pc=0xa24986
go.opencensus.io/trace.(*Span).lazyPrintfInternal(0xc03a141380, 0x0, 0x0, 0x0, 0x189c1e8, 0x17, 0xc0249df6e8, 0x1, 0x1)
        /tmp/go/pkg/mod/[email protected]/trace/trace.go:419 +0x99 fp=0xc0249df5a8 sp=0xc0249df4d8 pc=0xe9b089
go.opencensus.io/trace.(*Span).Annotatef(...)
        /tmp/go/pkg/mod/[email protected]/trace/trace.go:464
github.com/dgraph-io/dgraph/edgraph.(*Server).doMutate(0x24a1bb0, 0x1a84a60, 0xc03c7620f0, 0xc03bf3b860, 0xc03d51dbc0, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:280 +0xb61 fp=0xc0249df7b8 sp=0xc0249df5a8 pc=0x14a2171
github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery(0x24a1bb0, 0x1a84a60, 0xc03c7620f0, 0xc03c57a070, 0x0, 0xc03d51dbc0, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:755 +0x518 fp=0xc0249df9d0 sp=0xc0249df7b8 pc=0x14a6428
github.com/dgraph-io/dgraph/edgraph.(*Server).Query(0x24a1bb0, 0x1a84a60, 0xc03a10ff50, 0xc03c57a070, 0x24a1bb0, 0xc03a10ff50, 0xc03b6f0a80)
        /tmp/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:669 +0x52 fp=0xc0249dfa20 sp=0xc0249df9d0 pc=0x14a5df2
github.com/dgraph-io/dgo/v2/protos/api._Dgraph_Query_Handler(0x17ead60, 0x24a1bb0, 0x1a84a60, 0xc03a10ff50, 0xc03bf3eae0, 0x0, 0x1a84a60, 0xc03a10ff50, 0xc03c5f2000, 0xb1c4c)
        /tmp/go/pkg/mod/github.com/dgraph-io/dgo/[email protected]/protos/api/api.pb.go:1594 +0x217 fp=0xc0249dfa90 sp=0xc0249dfa20 pc=0xe01187
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00039c420, 0x1a927c0, 0xc000743c80, 0xc03b738600, 0xc0000ebc80, 0x2293618, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:995 +0x460 fp=0xc0249dfe18 sp=0xc0249dfa90 pc=0xde3ca0
google.golang.org/grpc.(*Server).handleStream(0xc00039c420, 0x1a927c0, 0xc000743c80, 0xc03b738600, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1275 +0xd97 fp=0xc0249dff48 sp=0xc0249dfe18 pc=0xde7c07
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0285b1b40, 0xc00039c420, 0x1a927c0, 0xc000743c80, 0xc03b738600)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:710 +0xbb fp=0xc0249dffb8 sp=0xc0249dff48 pc=0xdf4b6b
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0249dffc0 sp=0xc0249dffb8 pc=0x9a3ae1
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:708 +0xa1

goroutine 1 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc000873c98)
        /usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc000873c90)
        /usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer()
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:503 +0x862
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run()
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:652 +0xda5
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1(0xc00039e000, 0xc0008e62a0, 0x0, 0x2)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:90 +0x66
github.com/spf13/cobra.(*Command).execute(0xc00039e000, 0xc0008e6260, 0x2, 0x2, 0xc00039e000, 0xc0008e6260)
        /tmp/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0x2296fa0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        /tmp/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute()
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:69 +0x80
main.main()
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/main.go:73 +0xb0

goroutine 6 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x2484c20)
        /tmp/go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
        /tmp/go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x26f

goroutine 10 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000b0aa0)
        /tmp/go/pkg/mod/[email protected]/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
        /tmp/go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57

goroutine 11 [syscall, 9 minutes]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 12 [chan receive]:
github.com/dgraph-io/dgraph/x.init.0.func1(0x1a849e0, 0xc00003a108)
        /tmp/go/src/github.com/dgraph-io/dgraph/x/metrics.go:212 +0xfd
created by github.com/dgraph-io/dgraph/x.init.0
        /tmp/go/src/github.com/dgraph-io/dgraph/x/metrics.go:208 +0x93

goroutine 21 [chan receive]:
main.main.func1()
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/main.go:46 +0x124
created by main.main
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/main.go:37 +0xab

goroutine 36 [select, 9 minutes]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc000325200)
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/policy.go:80 +0x129

goroutine 37 [select, 9 minutes]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc000074660)
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/cache.go:257 +0xc8
created by github.com/dgraph-io/ristretto.NewCache
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/cache.go:155 +0x1f3

goroutine 38 [select]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0000322c0, 0xc00000fe00)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:229 +0x2e6
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:78 +0xa6

goroutine 39 [select]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc000032380, 0xc00000fe00)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:229 +0x2e6
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:78 +0xa6

goroutine 40 [select]:
github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc000340800, 0xc00000fe60)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1085 +0x175
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:323 +0xa20

goroutine 796 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8000, 0xc00089c180)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:362 +0x228
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:343 +0x88

goroutine 797 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8000, 0xc00089c180)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:362 +0x228
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:343 +0x88

goroutine 798 [chan receive, 9 minutes]:
github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc000340800, 0xc00089c1a0, 0x0, 0x0)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:993 +0xc3
github.com/dgraph-io/badger/v2.Open.func4(0xc000010030)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:337 +0x37
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:336 +0x14b7

goroutine 801 [select]:
github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc000340800, 0xc00089c340)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:766 +0x418
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:371 +0xf9c

goroutine 800 [select, 9 minutes]:
github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0003409c8)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1790 +0x19e
created by github.com/dgraph-io/badger/v2.(*valueLog).open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1027 +0x22c

goroutine 850 [chan receive, 9 minutes]:
github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0003409c8, 0xc00089c360)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1693 +0x70
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:375 +0x11c4

goroutine 851 [select, 9 minutes]:
github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc0000eb500, 0xc00089c380)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/publisher.go:67 +0x17f
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:379 +0x107b

goroutine 852 [select]:
github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc0000eb5f0)
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/policy.go:96 +0xbe
created by github.com/dgraph-io/ristretto.newDefaultPolicy
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/policy.go:80 +0x129

goroutine 70 [select]:
github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0008a20c0)
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/cache.go:257 +0xc8
created by github.com/dgraph-io/ristretto.NewCache
        /tmp/go/pkg/mod/github.com/dgraph-io/[email protected]/cache.go:155 +0x1f3

goroutine 71 [select, 9 minutes]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0000325c0, 0xc0003583a0)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:229 +0x2e6
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:78 +0xa6

goroutine 72 [select, 9 minutes]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0000327c0, 0xc0003583a0)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:229 +0x2e6
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:78 +0xa6

goroutine 52 [select]:
github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc000341000, 0xc0001ee0c0)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1085 +0x175
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:323 +0xa20

goroutine 836 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8070, 0xc0008e6120)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:362 +0x228
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:343 +0x88

goroutine 837 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runWorker(0xc0000e8070, 0xc0008e6120)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:362 +0x228
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:343 +0x88

goroutine 838 [chan receive, 9 minutes]:
github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc000341000, 0xc0008e6140, 0x0, 0x0)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:993 +0xc3
github.com/dgraph-io/badger/v2.Open.func4(0xc000010270)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:337 +0x37
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:336 +0x14b7

goroutine 841 [select]:
github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc000341000, 0xc0008e62e0)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:766 +0x418
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:371 +0xf9c

goroutine 840 [select, 9 minutes]:
github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0003411c8)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1790 +0x19e
created by github.com/dgraph-io/badger/v2.(*valueLog).open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1027 +0x22c

goroutine 842 [chan receive, 9 minutes]:
github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0003411c8, 0xc0008e6300)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1693 +0x70
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:375 +0x11c4

goroutine 843 [select, 9 minutes]:
github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc00016d200, 0xc0008e6320)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/publisher.go:67 +0x17f
created by github.com/dgraph-io/badger/v2.Open
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:379 +0x107b

goroutine 844 [select]:
github.com/dgraph-io/dgraph/worker.(*ServerState).runVlogGC(0x2483f60, 0xc000341000)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:84 +0xfa
created by github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:194 +0x740

goroutine 845 [select]:
github.com/dgraph-io/dgraph/worker.(*ServerState).runVlogGC(0x2483f60, 0xc000340800)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:84 +0xfa
created by github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:195 +0x76e

goroutine 846 [chan receive]:
github.com/dgraph-io/dgraph/worker.(*ServerState).fillTimestampRequests(0x2483f60)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:228 +0x3ba
created by github.com/dgraph-io/dgraph/worker.InitServerState
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/server_state.go:59 +0xfb

goroutine 847 [select]:
github.com/dgraph-io/dgraph/posting.updateMemoryMetrics(0xc0008e63a0)
        /tmp/go/src/github.com/dgraph-io/dgraph/posting/lists.go:121 +0x16f
created by github.com/dgraph-io/dgraph/posting.Init
        /tmp/go/src/github.com/dgraph-io/dgraph/posting/lists.go:139 +0xd0

goroutine 848 [chan receive]:
github.com/dgraph-io/dgraph/worker.(*rateLimiter).bleed(0x2480ca0)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/proposal.go:65 +0x100
created by github.com/dgraph-io/dgraph/worker.Init
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/worker.go:58 +0xeb

goroutine 866 [chan receive, 9 minutes]:
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run.func3(0xc0001e6a20)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:627 +0x4d
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:625 +0xd01

goroutine 868 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f8a98996d68, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0000b2198, 0x72, 0x0, 0x0, 0x1888dfd)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0000b2180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc0000b2180, 0x37e11d600, 0x0, 0x0)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0001ee2a0, 0xc000161da8, 0xc000161db0, 0x18)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0001ee2a0, 0x18f0cc8, 0xc024838000, 0x1a91080, 0xc000130158)
        /usr/local/go/src/net/tcpsock.go:261 +0x47
google.golang.org/grpc.(*Server).Serve(0xc024838000, 0x1a7f2a0, 0xc0001ee2a0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:586 +0x22e
github.com/dgraph-io/dgraph/worker.RunServer(0x94fa01)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/worker.go:95 +0x2db
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:417 +0x52

goroutine 869 [IO wait]:
internal/poll.runtime_pollWait(0x7f8a98996e38, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002e6198, 0x72, 0x0, 0x0, 0x1888dfd)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0002e6180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc0002e6180, 0x37e11d600, 0x0, 0x0)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0008e6480, 0xc024415d28, 0xc024415d30, 0x18)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0008e6480, 0x18f0cc8, 0xc00039c420, 0x1a91080, 0xc0249ce430)
        /usr/local/go/src/net/tcpsock.go:261 +0x47
google.golang.org/grpc.(*Server).Serve(0xc00039c420, 0x1a7f2a0, 0xc0008e6480, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:586 +0x22e
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.serveGRPC(0x1a7f2a0, 0xc0008e6480, 0x0, 0xc000873c90)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:388 +0x31c
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:486 +0x68d

goroutine 870 [IO wait]:
internal/poll.runtime_pollWait(0x7f8a98996f08, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002e6118, 0x72, 0x0, 0x0, 0x1888dfd)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0002e6100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc0002e6100, 0xc000065d08, 0xc51704, 0xc0002ec0a0)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0008e6460, 0x5e9d3e38, 0xc000065d08, 0x9cca26)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0008e6460, 0xc000065d58, 0x18, 0xc000875e00, 0xc50c34)
        /usr/local/go/src/net/tcpsock.go:261 +0x47
net/http.(*Server).Serve(0xc0002ec000, 0x1a7f2a0, 0xc0008e6460, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:2896 +0x280
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.serveHTTP(0x1a7f2a0, 0xc0008e6460, 0x0, 0xc000873c90)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:406 +0x312
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:487 +0x6d0

goroutine 871 [chan receive, 9 minutes]:
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer.func1(0xc000873c90, 0x1a7f2a0, 0xc0008e6480, 0x1a7f2a0, 0xc0008e6460)
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:491 +0x7f
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
        /tmp/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:489 +0x71d

goroutine 54 [select, 9 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc024870000)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x12c
created by google.golang.org/grpc.newCCBalancerWrapper
        /tmp/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:106 +0x169

goroutine 55 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00035ad70, 0xc000359270, 0x5, 0x5, 0xc024875870, 0x9b036c, 0x18f1b70)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:185 +0xd3
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00035ad70, 0xc000359270, 0x5, 0x5, 0xc024875e98, 0xc023a11ea0, 0xc024875918)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:165 +0x195
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc024828570, 0xc000359270, 0x5, 0x5, 0xc024875940, 0xc024875940, 0xd4c2cc)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:488 +0x55
io.ReadAtLeast(0x1a5a3e0, 0xc024828570, 0xc000359270, 0x5, 0x5, 0x5, 0xc03bf6b300, 0xc024875a28, 0xa4b60a)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0248b0000, 0xc000359270, 0x5, 0x5, 0x1b084bcd, 0xc024875a08, 0x94d4ff)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:472 +0xc7
google.golang.org/grpc.(*parser).recvMsg(0xc000359260, 0x100000000, 0x1, 0xc024875a98, 0x945dc4, 0xc00016b130, 0xc03bf68660, 0x7f8a989ae001)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:508 +0x63
google.golang.org/grpc.recvAndDecompress(0xc000359260, 0xc0248b0000, 0x0, 0x0, 0x100000000, 0xc024875c58, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a0293982, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:639 +0x4d
google.golang.org/grpc.recv(0xc000359260, 0x7f8a9895e5f8, 0x24a1bb0, 0xc0248b0000, 0x0, 0x0, 0x1840c80, 0xc03bf5ed00, 0x100000000, 0xc024875c58, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:684 +0x9b
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0000b2300, 0x1840c80, 0xc03bf5ed00, 0xc024875c58, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:885 +0xed
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0000b2300, 0x20300e, 0x20300e)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:736 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc000394120, 0xc024875dc0, 0xc024875d90, 0x0, 0x68)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:590 +0x360
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000394120, 0x1840c80, 0xc03bf5ed00, 0x5e9d3e4b, 0x1b085acc)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:735 +0x103
github.com/dgraph-io/dgraph/protos/pb.(*raftHeartbeatClient).Recv(0xc0003a83b0, 0x83ad4411d8, 0x24842c0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4850 +0x62
github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat(0xc000096270, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:227 +0x1e5
github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc000096270)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:250 +0xce
created by github.com/dgraph-io/dgraph/conn.newPool
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:179 +0x402

goroutine 872 [chan receive, 9 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0000bc780)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1077 +0x6ac
created by google.golang.org/grpc.(*addrConn).connect
        /tmp/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:743 +0x120

goroutine 876 [runnable]:
internal/poll.runtime_pollWait(0x7f8a98996c98, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002e6418, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002e6400, 0xc024894000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0002e6400, 0xc024894000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001300b0, 0xc024894000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc0001e6ea0, 0xc0248ac038, 0x9, 0x9, 0x24859e0, 0x7f8a989af0c8, 0x0)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1a58c40, 0xc0001e6ea0, 0xc0248ac038, 0x9, 0x9, 0x9, 0xd1dd25, 0xc03b975f4c, 0xc024cf2e00)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0248ac038, 0x9, 0x9, 0x1a58c40, 0xc0001e6ea0, 0x0, 0xbf9f6d7200000000, 0x83be05252a, 0x24842c0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0248ac000, 0xc03b975f40, 0xc03b975f40, 0x0, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000188380)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1249 +0x196
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:289 +0xd75

goroutine 877 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00089e5f0, 0x1, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001e6f60, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000188380)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:333 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:331 +0xf12

goroutine 56 [select, 9 minutes]:
google.golang.org/grpc.newClientStream.func5(0xc000876700, 0xc000394120, 0x1a84a60, 0xc0248900f0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:319 +0xd7
created by google.golang.org/grpc.newClientStream
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:318 +0x9a5

goroutine 57 [select, 9 minutes]:
github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat.func1(0x1a849a0, 0xc023ab5ac0, 0xc000096270, 0xc00016b0c0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:218 +0xc7
created by github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:217 +0x1c3

goroutine 41 [select, 9 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc024974300)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x12c
created by google.golang.org/grpc.newCCBalancerWrapper
        /tmp/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:106 +0x169

goroutine 42 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0249ea000, 0xc0249d01b0, 0x5, 0x5, 0xc0249e3870, 0x9b036c, 0x18f1b70)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:185 +0xd3
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0249ea000, 0xc0249d01b0, 0x5, 0x5, 0xc0249e3e98, 0xc0249ec000, 0xc0249e3918)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:165 +0x195
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0249cc090, 0xc0249d01b0, 0x5, 0x5, 0xc0249e3940, 0xc0249e3940, 0xd4c2cc)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:488 +0x55
io.ReadAtLeast(0x1a5a3e0, 0xc0249cc090, 0xc0249d01b0, 0x5, 0x5, 0x5, 0xc03c550080, 0xc0249e3a28, 0xa4b60a)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0249e8000, 0xc0249d01b0, 0x5, 0x5, 0x1b9a5e61, 0xc0249e3a08, 0x94d4ff)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:472 +0xc7
google.golang.org/grpc.(*parser).recvMsg(0xc0249d01a0, 0x100000000, 0x1, 0xc0249e3a98, 0x945dc4, 0xc00006b8d0, 0xc03bef7f50, 0x7f8a989afe01)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:508 +0x63
google.golang.org/grpc.recvAndDecompress(0xc0249d01a0, 0xc0249e8000, 0x0, 0x0, 0x100000000, 0xc0249e3c58, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a0bb4b72, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:639 +0x4d
google.golang.org/grpc.recv(0xc0249d01a0, 0x7f8a9895e5f8, 0x24a1bb0, 0xc0249e8000, 0x0, 0x0, 0x1840c80, 0xc03bf15f80, 0x100000000, 0xc0249e3c58, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:684 +0x9b
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0002e6700, 0x1840c80, 0xc03bf15f80, 0xc0249e3c58, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:885 +0xed
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0002e6700, 0x20300e, 0x20300e)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:736 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc0249ac120, 0xc0249e3dc0, 0xc0249e3d90, 0x0, 0x68)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:590 +0x360
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0249ac120, 0x1840c80, 0xc03bf15f80, 0x5e9d3e4b, 0x1b9a747c)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:735 +0x103
github.com/dgraph-io/dgraph/protos/pb.(*raftHeartbeatClient).Recv(0xc0249d4040, 0x83add62b7a, 0x24842c0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4850 +0x62
github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat(0xc0249785b0, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:227 +0x1e5
github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc0249785b0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:250 +0xce
created by github.com/dgraph-io/dgraph/conn.newPool
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:179 +0x402

goroutine 878 [chan receive, 9 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0000bcf00)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1077 +0x6ac
created by google.golang.org/grpc.(*addrConn).connect
        /tmp/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:743 +0x120

goroutine 853 [runnable]:
internal/poll.runtime_pollWait(0x7f8a98996bc8, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002e6618, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002e6600, 0xc024990000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0002e6600, 0xc024990000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001180a8, 0xc024990000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc00039b2c0, 0xc0002ec118, 0x9, 0x9, 0xc000508000, 0x7f8a989ae040, 0x0)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1a58c40, 0xc00039b2c0, 0xc0002ec118, 0x9, 0x9, 0x9, 0xd1dd25, 0xc03bf6ca2c, 0xc0001f0e00)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0002ec118, 0x9, 0x9, 0x1a58c40, 0xc00039b2c0, 0x0, 0xbf9f6d7200000000, 0x83add6958a, 0x24842c0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002ec0e0, 0xc03bf6ca20, 0xc03bf6ca20, 0x0, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0249aa000)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1249 +0x196
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:289 +0xd75

goroutine 884 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00089ec30, 0xc0008e6a90, 0x5, 0x5, 0x30, 0x48, 0x18f1b70)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:185 +0xd3
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00089ec30, 0xc0008e6a90, 0x5, 0x5, 0x0, 0xc024832360, 0xc024bcd978)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:165 +0x195
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0248ae930, 0xc0008e6a90, 0x5, 0x5, 0x0, 0xc024bcd9a0, 0xd4c2cc)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:488 +0x55
io.ReadAtLeast(0x1a5a3e0, 0xc0248ae930, 0xc0008e6a90, 0x5, 0x5, 0x5, 0xc03be098c0, 0xc024bcda88, 0xa4b60a)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00019df00, 0xc0008e6a90, 0x5, 0x5, 0xc01d47beca, 0xc024bcda68, 0x94d4ff)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:472 +0xc7
google.golang.org/grpc.(*parser).recvMsg(0xc0008e6a80, 0x100000000, 0x1, 0xc024bcdaf8, 0x945dc4, 0xc00006b9a0, 0xc03bdfb2e0, 0x7f8a9899fe01)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:508 +0x63
google.golang.org/grpc.recvAndDecompress(0xc0008e6a80, 0xc00019df00, 0x0, 0x0, 0x100000000, 0xc024bcdcb8, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a268ab69, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:639 +0x4d
google.golang.org/grpc.recv(0xc0008e6a80, 0x7f8a9895e5f8, 0x24a1bb0, 0xc00019df00, 0x0, 0x0, 0x1849c60, 0xc03be10800, 0x100000000, 0xc024bcdcb8, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:684 +0x9b
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0002e6800, 0x1849c60, 0xc03be10800, 0xc024bcdcb8, 0xc024bcdd10, 0x9a05e0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:885 +0xed
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0002e6800, 0x20300e, 0x20300e)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:736 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc0249ac360, 0xc024bcde20, 0xc024bcddf0, 0x1, 0x68)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:590 +0x360
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0249ac360, 0x1849c60, 0xc03be10800, 0xc024bcdedc, 0x2)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:735 +0x103
github.com/dgraph-io/dgraph/protos/pb.(*zeroStreamMembershipClient).Recv(0xc00006ba20, 0xc024bcded8, 0x2, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:5116 +0x62
github.com/dgraph-io/dgraph/worker.(*groupi).receiveMembershipUpdates.func1(0xc000096270, 0x1a8e4a0, 0xc00006ba20, 0x1a849a0, 0xc024824500, 0xc00006b930, 0xc0001e7740)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:775 +0xf8
created by github.com/dgraph-io/dgraph/worker.(*groupi).receiveMembershipUpdates
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:771 +0x665

goroutine 854 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000b0a00, 0x1, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00039b380, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0249aa000)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:333 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:331 +0xf12

goroutine 43 [select, 9 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0249744c0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x12c
created by google.golang.org/grpc.newCCBalancerWrapper
        /tmp/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:106 +0x169

goroutine 44 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0249c2280, 0xc0249c42d0, 0x5, 0x5, 0xc0249e1870, 0x9b036c, 0x18f1b70)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:185 +0xd3
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0249c2280, 0xc0249c42d0, 0x5, 0x5, 0xc0249e1e98, 0xc0249b6080, 0xc0249e1918)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:165 +0x195
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0249bc360, 0xc0249c42d0, 0x5, 0x5, 0xc0249e1940, 0xc0249e1940, 0xd4c2cc)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:488 +0x55
io.ReadAtLeast(0x1a5a3e0, 0xc0249bc360, 0xc0249c42d0, 0x5, 0x5, 0x5, 0xc03bf6b340, 0xc0249e1a28, 0xa4b60a)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc024a1a100, 0xc0249c42d0, 0x5, 0x5, 0x1ba07018, 0xc0249e1a08, 0x94d4ff)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:472 +0xc7
google.golang.org/grpc.(*parser).recvMsg(0xc0249c42c0, 0x100000000, 0x1, 0xc0249e1a98, 0x945dc4, 0xc00016b210, 0xc03bf68670, 0x7f8a989ae001)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:508 +0x63
google.golang.org/grpc.recvAndDecompress(0xc0249c42c0, 0xc024a1a100, 0x0, 0x0, 0x100000000, 0xc0249e1c58, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0x3c8a0c15fa2, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:639 +0x4d
google.golang.org/grpc.recv(0xc0249c42c0, 0x7f8a9895e5f8, 0x24a1bb0, 0xc024a1a100, 0x0, 0x0, 0x1840c80, 0xc03bf5ed80, 0x100000000, 0xc0249e1c58, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:684 +0x9b
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0000b2480, 0x1840c80, 0xc03bf5ed80, 0xc0249e1c58, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:885 +0xed
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0000b2480, 0x20300e, 0x20300e)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:736 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc000394360, 0xc0249e1dc0, 0xc0249e1d90, 0x0, 0x68)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:590 +0x360
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000394360, 0x1840c80, 0xc03bf5ed80, 0x5e9d3e4b, 0x1ba0790e)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:735 +0x103
github.com/dgraph-io/dgraph/protos/pb.(*raftHeartbeatClient).Recv(0xc0249c0140, 0x83addc3015, 0x24842c0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4850 +0x62
github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat(0xc024978750, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:227 +0x1e5
github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc024978750)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:250 +0xce
created by github.com/dgraph-io/dgraph/conn.newPool
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:179 +0x402

goroutine 74 [chan receive, 9 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc00026a500)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1077 +0x6ac
created by google.golang.org/grpc.(*addrConn).connect
        /tmp/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:743 +0x120

goroutine 90 [select, 9 minutes]:
google.golang.org/grpc.newClientStream.func5(0xc00085a700, 0xc0249ac120, 0x1a84a60, 0xc0248ae4b0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:319 +0xd7
created by google.golang.org/grpc.newClientStream
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:318 +0x9a5

goroutine 91 [select, 9 minutes]:
github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat.func1(0x1a849a0, 0xc024824400, 0xc0249785b0, 0xc00006b870)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:218 +0xc7
created by github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:217 +0x1c3

goroutine 883 [select, 9 minutes]:
google.golang.org/grpc.newClientStream.func5(0xc000876700, 0xc0249ac360, 0x1a84a60, 0xc0248ae840)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:319 +0xd7
created by google.golang.org/grpc.newClientStream
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:318 +0x9a5

goroutine 77 [runnable]:
internal/poll.runtime_pollWait(0x7f8a98996af8, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc024a1c118, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc024a1c100, 0xc024a24000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc024a1c100, 0xc024a24000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc024a1e008, 0xc024a24000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc0249ca240, 0xc024a3c038, 0x9, 0x9, 0x20, 0x18, 0x17667a0)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1a58c40, 0xc0249ca240, 0xc024a3c038, 0x9, 0x9, 0x9, 0x100000000d1dd25, 0x0, 0x9)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc024a3c038, 0x9, 0x9, 0x1a58c40, 0xc0249ca240, 0x0, 0xc000000000, 0x950298, 0xc0249c21a0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc024a3c000, 0xc03ba87180, 0xc03ba87180, 0x0, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc024a42000)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1249 +0x196
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:289 +0xd75

goroutine 78 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0249c2190, 0x1, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0249ca3c0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc024a42000)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:333 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:331 +0xf12

goroutine 79 [select, 9 minutes]:
google.golang.org/grpc.newClientStream.func5(0xc00085aa80, 0xc000394360, 0x1a84a60, 0xc0248905d0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:319 +0xd7
created by google.golang.org/grpc.newClientStream
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:318 +0x9a5

goroutine 80 [select, 9 minutes]:
github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat.func1(0x1a849a0, 0xc023ab5c40, 0xc024978750, 0xc00016b1b0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:218 +0xc7
created by github.com/dgraph-io/dgraph/conn.(*Pool).listenToHeartbeat
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/pool.go:217 +0x1c3

goroutine 45 [select]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc024984430, 0x0)
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:229 +0x2e6
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
        /tmp/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/watermark.go:78 +0xa6

goroutine 1114 [select]:
github.com/dgraph-io/dgraph/worker.(*groupi).processOracleDeltaStream(0x2484640)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:976 +0x19a
created by github.com/dgraph-io/dgraph/worker.StartRaftNodes
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:157 +0x77a

goroutine 1112 [select]:
github.com/dgraph-io/dgraph/worker.(*groupi).sendMembershipUpdates(0x2484640)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:714 +0x1df
created by github.com/dgraph-io/dgraph/worker.StartRaftNodes
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:155 +0x732

goroutine 1110 [select]:
github.com/dgraph-io/dgraph/worker.(*node).checkpointAndClose(0xc02498e420, 0xc0249b8c00)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:707 +0x11d
created by github.com/dgraph-io/dgraph/worker.(*node).Run
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:790 +0x11e

goroutine 1109 [runnable]:
github.com/dgraph-io/dgraph/worker.(*node).Run(0xc02498e420)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:802 +0xcc5
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1474 +0x50a

goroutine 1108 [chan receive]:
github.com/dgraph-io/dgraph/conn.(*Node).BatchAndSendMessages(0xc024984360)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:347 +0x5c1
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1473 +0x4e5

goroutine 1107 [select]:
github.com/dgraph-io/dgraph/worker.(*node).processApplyCh(0xc02498e420)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:516 +0x218
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1472 +0x4bd

goroutine 1113 [select]:
github.com/dgraph-io/dgraph/worker.(*groupi).receiveMembershipUpdates(0x2484640)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:802 +0x200
created by github.com/dgraph-io/dgraph/worker.StartRaftNodes
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:156 +0x756

goroutine 1106 [select, 5 minutes]:
github.com/dgraph-io/dgraph/worker.(*node).processRollups(0xc02498e420)
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:430 +0x1b5
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
        /tmp/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1471 +0x498

goroutine 1101 [select]:
github.com/dgraph-io/dgraph/conn.(*Node).doSendMessage(0xc024984360, 0x2, 0xc024fe81e0, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:468 +0x4b1
github.com/dgraph-io/dgraph/conn.(*Node).streamMessages(0xc024984360, 0x2, 0xc024aeabe0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:413 +0x1b2
created by github.com/dgraph-io/dgraph/conn.(*Node).BatchAndSendMessages
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:386 +0x51a

goroutine 890 [runnable]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001f9e50, 0x1, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc024e2e4e0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00078be00)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:283 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:280 +0xfbe

goroutine 892 [runnable]:
internal/poll.runtime_pollWait(0x7f8a98996958, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc024972d18, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc024972d00, 0xc024f8c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc024972d00, 0xc024f8c000, 0x8000, 0x8000, 0x4, 0xc024b46d50, 0xc00077c600)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc02482a318, 0xc024f8c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc024b117a0, 0xc024f84118, 0x9, 0x9, 0xc024b46d00, 0xd488e2, 0xc00078bf01)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1a58c40, 0xc024b117a0, 0xc024f84118, 0x9, 0x9, 0x9, 0xc03b94b710, 0x0, 0x0)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc024f84118, 0x9, 0x9, 0x1a58c40, 0xc024b117a0, 0x0, 0x0, 0x5924fa4000, 0x5900000001)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc024f840e0, 0xc024fa4000, 0xc024fa4000, 0x0, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00078be00, 0xc024f47b60, 0x18f0d20)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:440 +0x9e
google.golang.org/grpc.(*Server).serveStreams(0xc024838000, 0x1a927c0, 0xc00078be00)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:706 +0xf3
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc024838000, 0x1a927c0, 0xc00078be00)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:668 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:667 +0x562

goroutine 894 [select]:
github.com/dgraph-io/dgraph/conn.(*RaftServer).Heartbeat(0x2482960, 0xc024824e80, 0x1a8e2c0, 0xc024f41260, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:298 +0x37f
github.com/dgraph-io/dgraph/protos/pb._Raft_Heartbeat_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024d32e40, 0xc0000b7ef8, 0xc024dd0200)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4942 +0x109
google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00078bc80, 0xc024dd0200, 0xc00016d9e0, 0x228f2e0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1199 +0xb2e
google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00078bc80, 0xc024dd0200, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc024f49430, 0xc024838000, 0x1a927c0, 0xc00078bc80, 0xc024dd0200)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:708 +0xa1

goroutine 1096 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc024f8a3c0, 0x1, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc024b11860, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00081ad80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:283 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:280 +0xfbe

goroutine 81 [select]:
go.etcd.io/etcd/raft.(*node).run(0xc0249ca780, 0xc024b40000)
        /tmp/go/pkg/mod/go.etcd.io/[email protected]/raft/node.go:342 +0x4c1
created by go.etcd.io/etcd/raft.RestartNode
        /tmp/go/pkg/mod/go.etcd.io/[email protected]/raft/node.go:246 +0x31b

goroutine 887 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001f9d60, 0x1, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc024e2e1e0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00078bc80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:283 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:280 +0xfbe

goroutine 6353 [IO wait]:
internal/poll.runtime_pollWait(0x7f8a989967b8, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc023e77718, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc023e77700, 0xc023bf4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc023e77700, 0xc023bf4000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0249ce430, 0xc023bf4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc025ed4000, 0xc0248ac3b8, 0x9, 0x9, 0x9cc4aa, 0x8000000000000000, 0x83a91da048)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1a58c40, 0xc025ed4000, 0xc0248ac3b8, 0x9, 0x9, 0x9, 0xd1dd25, 0xc023bccf18, 0xc024fd0e38)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0248ac3b8, 0x9, 0x9, 0x1a58c40, 0xc025ed4000, 0x0, 0x0, 0x7070e0910100402, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0248ac380, 0xc03bf6c980, 0xc03bf6c980, 0x0, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000743c80, 0xc02701a6f0, 0x18f0d20)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:440 +0x9e
google.golang.org/grpc.(*Server).serveStreams(0xc00039c420, 0x1a927c0, 0xc000743c80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:706 +0xf3
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00039c420, 0x1a927c0, 0xc000743c80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:668 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:667 +0x562

goroutine 6351 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc023bcceb0, 0x1, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc025eb2240, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000743c80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:283 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:280 +0xfbe

goroutine 1128 [select, 9 minutes]:
google.golang.org/grpc.newClientStream.func5(0xc00085aa80, 0xc02518c000, 0x1a84a60, 0xc025170870)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:319 +0xd7
created by google.golang.org/grpc.newClientStream
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:318 +0x9a5

goroutine 889 [IO wait]:
internal/poll.runtime_pollWait(0x7f8a98996a28, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc024972c98, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc024972c80, 0xc024f6c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc024972c80, 0xc024f6c000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc02482a310, 0xc024f6c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc024b11740, 0xc024f84038, 0x9, 0x9, 0xc024b45d10, 0x950298, 0x20)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1a58c40, 0xc024b11740, 0xc024f84038, 0x9, 0x9, 0x9, 0xc024b45dc8, 0xc024b45d70, 0xc024b45e38)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc024f84038, 0x9, 0x9, 0x1a58c40, 0xc024b11740, 0x0, 0x8300000000, 0xc03ba86f60, 0x20)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc024f84000, 0xc03ba86f60, 0xc03ba86f60, 0x0, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00078bc80, 0xc024f47a40, 0x18f0d20)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:440 +0x9e
google.golang.org/grpc.(*Server).serveStreams(0xc024838000, 0x1a927c0, 0xc00078bc80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:706 +0xf3
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc024838000, 0x1a927c0, 0xc00078bc80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:668 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:667 +0x562

goroutine 891 [select, 9 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00078be00)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:943 +0x218
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:289 +0xfe3

goroutine 888 [select, 9 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00078bc80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:943 +0x218
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:289 +0xfe3

goroutine 6352 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000743c80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:943 +0x218
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:289 +0xfe3

goroutine 1099 [select]:
github.com/dgraph-io/dgraph/conn.(*RaftServer).Heartbeat(0x2482960, 0xc024975000, 0x1a8e2c0, 0xc024aeaa60, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:298 +0x37f
github.com/dgraph-io/dgraph/protos/pb._Raft_Heartbeat_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024a77380, 0xc0000b7ef8, 0xc024ab6200)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4942 +0x109
google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00081ad80, 0xc024ab6200, 0xc00016d9e0, 0x228f2e0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1199 +0xb2e
google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00081ad80, 0xc024ab6200, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00084edc0, 0xc024838000, 0x1a927c0, 0xc00081ad80, 0xc024ab6200)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:708 +0xa1

goroutine 1097 [select, 9 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00081ad80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:943 +0x218
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:289 +0xfe3

goroutine 1098 [IO wait]:
internal/poll.runtime_pollWait(0x7f8a98996888, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002e7118, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002e7100, 0xc024fb0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0002e7100, 0xc024fb0000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000130158, 0xc024fb0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc024e2e960, 0xc0248ac118, 0x9, 0x9, 0xc024cecd10, 0x950298, 0x20)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1a58c40, 0xc024e2e960, 0xc0248ac118, 0x9, 0x9, 0x9, 0xc024cecdc8, 0xc024cecd70, 0xc024cece38)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0248ac118, 0x9, 0x9, 0x1a58c40, 0xc024e2e960, 0x0, 0x8300000000, 0xc03bf6ca80, 0x20)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0248ac0e0, 0xc03bf6ca80, 0xc03bf6ca80, 0x0, 0x0)
        /tmp/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00081ad80, 0xc024fa42d0, 0x18f0d20)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:440 +0x9e
google.golang.org/grpc.(*Server).serveStreams(0xc024838000, 0x1a927c0, 0xc00081ad80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:706 +0xf3
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc024838000, 0x1a927c0, 0xc00081ad80)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:668 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:667 +0x562

goroutine 1100 [runnable]:
github.com/dgraph-io/dgraph/conn.(*RaftServer).Heartbeat(0x2482960, 0xc0249751c0, 0x1a8e2c0, 0xc024aeab90, 0x0, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:298 +0x37f
github.com/dgraph-io/dgraph/protos/pb._Raft_Heartbeat_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024a77440, 0xc0000b7ef8, 0xc024ab6300)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4942 +0x109
google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024ab6300, 0xc00016d9e0, 0x228f2e0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1199 +0xb2e
google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024ab6300, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc024f49450, 0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024ab6300)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:708 +0xa1

goroutine 110 [runnable]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc024df8d20, 0xc024f6a770, 0x5, 0x5, 0xb700000000af8957, 0x0, 0x18f1b70)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:173 +0xc7
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc024df8d20, 0xc024f6a770, 0x5, 0x5, 0xc024ff9940, 0xc024f5e640, 0xc024ff9630)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:167 +0x21a
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc024de3f20, 0xc024f6a770, 0x5, 0x5, 0xc024ff9658, 0xc024ff9658, 0xd4d65c)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:488 +0x55
io.ReadAtLeast(0x1a5a3e0, 0xc024de3f20, 0xc024f6a770, 0x5, 0x5, 0x5, 0x11, 0x0, 0x0)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc024f54300, 0xc024f6a770, 0x5, 0x5, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:472 +0xc7
google.golang.org/grpc.(*parser).recvMsg(0xc024f6a760, 0x100000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:508 +0x63
google.golang.org/grpc.recvAndDecompress(0xc024f6a760, 0xc024f54300, 0x0, 0x0, 0x100000000, 0xc024ff99b8, 0x1a7fa60, 0x24a1bb0, 0x992d68, 0xc024ff9a98, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:639 +0x4d
google.golang.org/grpc.recv(0xc024f6a760, 0x7f8a9895e5f8, 0x24a1bb0, 0xc024f54300, 0x0, 0x0, 0x18289c0, 0xc03ebd61e0, 0x100000000, 0xc024ff99b8, ...)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:684 +0x9b
google.golang.org/grpc.(*serverStream).RecvMsg(0xc024f560c0, 0x18289c0, 0xc03ebd61e0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/stream.go:1474 +0x170
github.com/dgraph-io/dgraph/protos/pb.(*raftRaftMessageServer).Recv(0xc024f62800, 0x21, 0x30, 0x0)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4978 +0x62
github.com/dgraph-io/dgraph/conn.(*RaftServer).RaftMessage(0x2482960, 0x1a90d80, 0xc024f62800, 0x1a84c20, 0x2482960)
        /tmp/go/src/github.com/dgraph-io/dgraph/conn/raft_server.go:249 +0x206
github.com/dgraph-io/dgraph/protos/pb._Raft_RaftMessage_Handler(0x17818e0, 0x2482960, 0x1a8bfe0, 0xc024f560c0, 0xc0000b7ef8, 0xc024f54300)
        /tmp/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4959 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024f54300, 0xc00016d9e0, 0x228f300, 0x0, 0x0, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1199 +0xb2e
google.golang.org/grpc.(*Server).handleStream(0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024f54300, 0x0)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc024f49450, 0xc024838000, 0x1a927c0, 0xc00078be00, 0xc024f54300)
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /tmp/go/pkg/mod/google.golang.org/[email protected]/server.go:708 +0xa1

H4midR on 20 Apr 2020

👍1

after another test the server crashed again. how would you like to check it out?
here is some screenshots

12c0ff14-f4a0-4d15-b82f-08cdc579be96
a4d33706-de60-4a38-af9f-f2787c35ff01
deab5c3c-9ca7-4048-b7cd-70827c560ce3
dfe58405-81af-4a22-baef-38a588ad34ca

16G Ram , 4 CPU

1Zero - 3Alpha

"uid" filtering is fine while other filtering like type filtering and equality filtering "eq" are failed.

it sounds like a bug.

dgraph alpha logs

View Log

Apr 20 16:55:28 bahamaamooz dgraph[28766]: I0420 16:55:28.877248   28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 104.361534ms.
Apr 20 16:55:28 bahamaamooz dgraph[28766]: I0420 16:55:28.935575   28766 log.go:34] Rolling up Created batch of size: 1.4 MB in 55.47142ms.
Apr 20 16:55:28 bahamaamooz dgraph[28766]: I0420 16:55:28.993690   28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 57.210653ms.
Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.140797   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 49.984726ms.
Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.650206   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 9.4 MB, speed: 4.7 MB/sec
Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.812894   28766 log.go:34] Rolling up Created batch of size: 4.1 MB in 91.447791ms.
Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.812934   28766 log.go:34] Rolling up Sent 55240 keys
Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.915312   28766 draft.go:1102] Rolled up 55214 keys. Done
Apr 20 16:55:29 bahamaamooz dgraph[28766]: I0420 16:55:29.915786   28766 draft.go:445] List rollup at Ts 213334: OK.
Apr 20 16:55:33 bahamaamooz dgraph[28766]: W0420 16:55:33.125191   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 270ms. Breakdown: [{sync 270ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:55:35 bahamaamooz dgraph[28766]: W0420 16:55:35.843292   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 475ms. Breakdown: [{sync 475ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:55:51 bahamaamooz dgraph[28766]: W0420 16:55:51.615249   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 245ms. Breakdown: [{sync 244ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:55:52 bahamaamooz dgraph[28766]: W0420 16:55:52.006268   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 298ms. Breakdown: [{sync 298ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:55:52 bahamaamooz dgraph[28766]: W0420 16:55:52.309106   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 231ms. Breakdown: [{sync 231ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:56:07 bahamaamooz dgraph[28766]: W0420 16:56:07.653744   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 549ms. Breakdown: [{sync 548ms} {disk 1ms} {proposals 0s} {a
Apr 20 16:56:10 bahamaamooz dgraph[28766]: W0420 16:56:10.324320   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 253ms. Breakdown: [{sync 253ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:56:24 bahamaamooz dgraph[28766]: W0420 16:56:24.807072   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 276ms. Breakdown: [{sync 276ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:56:25 bahamaamooz dgraph[28766]: W0420 16:56:25.352025   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 247ms. Breakdown: [{sync 247ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:56:25 bahamaamooz dgraph[28766]: W0420 16:56:25.944496   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 281ms. Breakdown: [{sync 281ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:56:56 bahamaamooz dgraph[28766]: W0420 16:56:56.206866   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 274ms. Breakdown: [{sync 274ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:56:56 bahamaamooz dgraph[28766]: W0420 16:56:56.489422   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:56:56 bahamaamooz dgraph[28766]: W0420 16:56:56.813798   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 220ms. Breakdown: [{sync 220ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:57:43 bahamaamooz dgraph[28766]: W0420 16:57:43.675913   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 286ms. Breakdown: [{sync 286ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:58:27 bahamaamooz dgraph[28766]: I0420 16:58:27.759193   28766 draft.go:403] Creating snapshot at index: 217037. ReadTs: 225250.
Apr 20 16:58:34 bahamaamooz dgraph[28766]: W0420 16:58:34.350002   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.302s. Breakdown: [{sync 1.301s} {disk 0s} {proposals 0s} {
Apr 20 16:58:36 bahamaamooz dgraph[28766]: W0420 16:58:36.277534   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 765ms. Breakdown: [{sync 765ms} {disk 0s} {proposals 0s} {ad
Apr 20 16:58:37 bahamaamooz dgraph[28766]: W0420 16:58:37.840466   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 627ms. Breakdown: [{disk 627ms} {proposals 0s} {advance 0s}]
Apr 20 16:59:22 bahamaamooz dgraph[28766]: I0420 16:59:22.675589   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 16:59:22 bahamaamooz dgraph[28766]: I0420 16:59:22.676434   28766 log.go:34] Running for level: 0
Apr 20 16:59:23 bahamaamooz dgraph[28766]: I0420 16:59:23.109463   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 432.889606ms
Apr 20 16:59:23 bahamaamooz dgraph[28766]: I0420 16:59:23.109515   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 16:59:42 bahamaamooz dgraph[28766]: W0420 16:59:42.626777   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 202ms. Breakdown: [{sync 202ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:00:04 bahamaamooz dgraph[28766]: W0420 17:00:04.614443   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 339ms. Breakdown: [{sync 338ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:00:06 bahamaamooz dgraph[28766]: W0420 17:00:06.000347   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 313ms. Breakdown: [{sync 313ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:00:07 bahamaamooz dgraph[28766]: W0420 17:00:07.558387   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 273ms. Breakdown: [{sync 272ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:00:27 bahamaamooz dgraph[28766]: I0420 17:00:27.912439   28766 log.go:34] Rolling up Created batch of size: 110 kB in 27.74666ms.
Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.312017   28766 log.go:34] Rolling up Created batch of size: 526 kB in 40.101007ms.
Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.395887   28766 log.go:34] Rolling up Created batch of size: 298 kB in 54.284858ms.
Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.657942   28766 log.go:34] Rolling up Created batch of size: 867 kB in 46.594905ms.
Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.658346   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.8 MB, speed: 1.8 MB/sec
Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.724868   28766 log.go:34] Rolling up Created batch of size: 906 kB in 66.014821ms.
Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.817357   28766 log.go:34] Rolling up Created batch of size: 1.0 MB in 91.663612ms.
Apr 20 17:00:28 bahamaamooz dgraph[28766]: I0420 17:00:28.835354   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 17.724056ms.
Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.433018   28766 log.go:34] Rolling up Created batch of size: 3.9 MB in 31.095653ms.
Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.433771   28766 log.go:34] Rolling up Sent 32830 keys
Apr 20 17:00:29 bahamaamooz dgraph[28766]: W0420 17:00:29.705151   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 255ms. Breakdown: [{sync 255ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.705425   28766 draft.go:1102] Rolled up 32804 keys. Done
Apr 20 17:00:29 bahamaamooz dgraph[28766]: I0420 17:00:29.705448   28766 draft.go:445] List rollup at Ts 225250: OK.
Apr 20 17:01:26 bahamaamooz dgraph[28766]: W0420 17:01:26.562592   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 211ms. Breakdown: [{sync 211ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:02:27 bahamaamooz dgraph[28766]: I0420 17:02:27.782971   28766 draft.go:403] Creating snapshot at index: 228547. ReadTs: 236646.
Apr 20 17:02:30 bahamaamooz dgraph[28766]: W0420 17:02:30.878437   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 215ms. Breakdown: [{sync 182ms} {disk 33ms} {proposals 0s} {
Apr 20 17:03:53 bahamaamooz dgraph[28766]: W0420 17:03:53.478850   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 556ms. Breakdown: [{sync 555ms} {proposals 1ms} {disk 0s} {a
Apr 20 17:03:53 bahamaamooz dgraph[28766]: W0420 17:03:53.817084   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 313ms. Breakdown: [{sync 312ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:04:54 bahamaamooz dgraph[28766]: W0420 17:04:54.923402   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 524ms. Breakdown: [{sync 523ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:04:56 bahamaamooz dgraph[28766]: W0420 17:04:56.594668   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 208ms. Breakdown: [{sync 207ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:05:27 bahamaamooz dgraph[28766]: I0420 17:05:27.990930   28766 log.go:34] Rolling up Created batch of size: 105 kB in 47.529521ms.
Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.368632   28766 log.go:34] Rolling up Created batch of size: 530 kB in 41.622235ms.
Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.535526   28766 log.go:34] Rolling up Created batch of size: 285 kB in 68.817359ms.
Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.602977   28766 log.go:34] Rolling up Created batch of size: 854 kB in 66.942751ms.
Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.650452   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.8 MB, speed: 1.8 MB/sec
Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.766191   28766 log.go:34] Rolling up Created batch of size: 892 kB in 29.443225ms.
Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.860600   28766 log.go:34] Rolling up Created batch of size: 992 kB in 40.126009ms.
Apr 20 17:05:28 bahamaamooz dgraph[28766]: I0420 17:05:28.974255   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 8.313996ms.
Apr 20 17:05:29 bahamaamooz dgraph[28766]: I0420 17:05:29.620639   28766 log.go:34] Rolling up Created batch of size: 4.0 MB in 26.099804ms.
Apr 20 17:05:29 bahamaamooz dgraph[28766]: I0420 17:05:29.621298   28766 log.go:34] Rolling up Sent 31392 keys
Apr 20 17:05:31 bahamaamooz dgraph[28766]: I0420 17:05:31.501380   28766 draft.go:1102] Rolled up 31366 keys. Done
Apr 20 17:05:31 bahamaamooz dgraph[28766]: I0420 17:05:31.501442   28766 draft.go:445] List rollup at Ts 236646: OK.
Apr 20 17:06:27 bahamaamooz dgraph[28766]: I0420 17:06:27.770930   28766 draft.go:403] Creating snapshot at index: 240831. ReadTs: 248818.
Apr 20 17:06:35 bahamaamooz dgraph[28766]: W0420 17:06:35.599135   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 456ms. Breakdown: [{sync 455ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:06:35 bahamaamooz dgraph[28766]: W0420 17:06:35.886562   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 249ms. Breakdown: [{sync 248ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.109557   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.109635   28766 log.go:34] Running for level: 0
Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.792852   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 683.189212ms
Apr 20 17:07:34 bahamaamooz dgraph[28766]: I0420 17:07:34.792899   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:08:10 bahamaamooz dgraph[28766]: W0420 17:08:10.256750   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 202ms. Breakdown: [{sync 200ms} {disk 2ms} {proposals 0s} {a
Apr 20 17:08:10 bahamaamooz dgraph[28766]: W0420 17:08:10.974192   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 227ms. Breakdown: [{sync 226ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:08:28 bahamaamooz dgraph[28766]: W0420 17:08:28.993525   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 209ms. Breakdown: [{sync 209ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:09:28 bahamaamooz dgraph[28766]: W0420 17:09:28.260749   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 248ms. Breakdown: [{sync 247ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:10:27 bahamaamooz dgraph[28766]: I0420 17:10:27.799116   28766 draft.go:403] Creating snapshot at index: 252248. ReadTs: 260140.
Apr 20 17:10:27 bahamaamooz dgraph[28766]: I0420 17:10:27.860104   28766 log.go:34] Rolling up Created batch of size: 113 kB in 39.875248ms.
Apr 20 17:10:28 bahamaamooz dgraph[28766]: I0420 17:10:28.677161   28766 log.go:34] Rolling up Created batch of size: 561 kB in 37.342572ms.
Apr 20 17:10:28 bahamaamooz dgraph[28766]: I0420 17:10:28.677518   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 674 kB, speed: 674 kB/sec
Apr 20 17:10:28 bahamaamooz dgraph[28766]: I0420 17:10:28.902838   28766 log.go:34] Rolling up Created batch of size: 304 kB in 53.451758ms.
Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.082979   28766 log.go:34] Rolling up Created batch of size: 929 kB in 70.922104ms.
Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.281349   28766 log.go:34] Rolling up Created batch of size: 889 kB in 66.522093ms.
Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.376724   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 25.575976ms.
Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.432646   28766 log.go:34] Rolling up Created batch of size: 312 kB in 40.603µs.
Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.503263   28766 log.go:34] Rolling up Created batch of size: 1.0 MB in 57.82951ms.
Apr 20 17:10:29 bahamaamooz dgraph[28766]: I0420 17:10:29.650458   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 8.5 MB, speed: 4.3 MB/sec
Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.621580   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 52.862591ms.
Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.621645   28766 log.go:34] Rolling up Sent 33540 keys
Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.696088   28766 draft.go:1102] Rolled up 33514 keys. Done
Apr 20 17:10:30 bahamaamooz dgraph[28766]: I0420 17:10:30.696132   28766 draft.go:445] List rollup at Ts 248818: OK.
Apr 20 17:11:23 bahamaamooz dgraph[28766]: W0420 17:11:23.435541   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 318ms. Breakdown: [{sync 318ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:12:03 bahamaamooz dgraph[28766]: W0420 17:12:03.934698   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 225ms. Breakdown: [{sync 224ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:12:41 bahamaamooz dgraph[28766]: W0420 17:12:41.040913   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 248ms. Breakdown: [{sync 248ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:14:27 bahamaamooz dgraph[28766]: I0420 17:14:27.794983   28766 draft.go:403] Creating snapshot at index: 263009. ReadTs: 270790.
Apr 20 17:15:03 bahamaamooz dgraph[28766]: W0420 17:15:03.652858   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 785ms. Breakdown: [{sync 784ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:15:25 bahamaamooz dgraph[28766]: W0420 17:15:25.453327   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 328ms. Breakdown: [{sync 328ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:26 bahamaamooz dgraph[28766]: W0420 17:15:26.038692   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 268ms. Breakdown: [{sync 267ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:26 bahamaamooz dgraph[28766]: W0420 17:15:26.553953   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 207ms. Breakdown: [{sync 207ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:27 bahamaamooz dgraph[28766]: W0420 17:15:27.168443   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 326ms. Breakdown: [{sync 326ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:27 bahamaamooz dgraph[28766]: I0420 17:15:27.895821   28766 log.go:34] Rolling up Created batch of size: 203 kB in 54.381181ms.
Apr 20 17:15:28 bahamaamooz dgraph[28766]: I0420 17:15:28.480769   28766 log.go:34] Rolling up Created batch of size: 790 kB in 88.479115ms.
Apr 20 17:15:28 bahamaamooz dgraph[28766]: I0420 17:15:28.654812   28766 log.go:34] Rolling up Created batch of size: 549 kB in 105.288689ms.
Apr 20 17:15:28 bahamaamooz dgraph[28766]: I0420 17:15:28.654853   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.5 MB, speed: 1.5 MB/sec
Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.070099   28766 log.go:34] Rolling up Created batch of size: 1.3 MB in 136.848507ms.
Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.248852   28766 log.go:34] Rolling up Created batch of size: 6.0 MB in 177.10802ms.
Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.314059   28766 log.go:34] Rolling up Created batch of size: 2.0 MB in 63.865349ms.
Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.650261   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 11 MB, speed: 5.4 MB/sec
Apr 20 17:15:29 bahamaamooz dgraph[28766]: I0420 17:15:29.769228   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 68.021461ms.
Apr 20 17:15:30 bahamaamooz dgraph[28766]: I0420 17:15:30.135327   28766 log.go:34] Rolling up Created batch of size: 594 kB in 42.739573ms.
Apr 20 17:15:30 bahamaamooz dgraph[28766]: I0420 17:15:30.136304   28766 log.go:34] Rolling up Sent 59654 keys
Apr 20 17:15:33 bahamaamooz dgraph[28766]: I0420 17:15:33.831545   28766 draft.go:1102] Rolled up 59628 keys. Done
Apr 20 17:15:33 bahamaamooz dgraph[28766]: I0420 17:15:33.831595   28766 draft.go:445] List rollup at Ts 270790: OK.
Apr 20 17:15:35 bahamaamooz dgraph[28766]: I0420 17:15:35.109541   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:15:35 bahamaamooz dgraph[28766]: I0420 17:15:35.109615   28766 log.go:34] Running for level: 0
Apr 20 17:15:38 bahamaamooz dgraph[28766]: W0420 17:15:38.357770   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 590ms. Breakdown: [{sync 562ms} {disk 28ms} {proposals 0s} {
Apr 20 17:15:38 bahamaamooz dgraph[28766]: I0420 17:15:38.483341   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 3.373697571s
Apr 20 17:15:38 bahamaamooz dgraph[28766]: I0420 17:15:38.483443   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:15:40 bahamaamooz dgraph[28766]: W0420 17:15:40.722243   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.59s. Breakdown: [{sync 1.59s} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:44 bahamaamooz dgraph[28766]: W0420 17:15:44.948660   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 3.626s. Breakdown: [{sync 3.625s} {disk 0s} {proposals 0s} {
Apr 20 17:15:49 bahamaamooz dgraph[28766]: W0420 17:15:49.905504   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 247ms. Breakdown: [{disk 228ms} {sync 18ms} {proposals 0s} {
Apr 20 17:15:50 bahamaamooz dgraph[28766]: W0420 17:15:50.817021   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 325ms. Breakdown: [{sync 325ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:51 bahamaamooz dgraph[28766]: W0420 17:15:51.192417   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 219ms. Breakdown: [{sync 219ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:52 bahamaamooz dgraph[28766]: W0420 17:15:52.904968   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:15:57 bahamaamooz dgraph[28766]: W0420 17:15:57.296240   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 247ms. Breakdown: [{sync 247ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:16:00 bahamaamooz dgraph[28766]: W0420 17:16:00.381517   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:16:00 bahamaamooz dgraph[28766]: W0420 17:16:00.674039   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 262ms. Breakdown: [{sync 261ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:16:02 bahamaamooz dgraph[28766]: W0420 17:16:02.064656   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.241s. Breakdown: [{sync 1.24s} {disk 1ms} {proposals 0s} {
Apr 20 17:16:08 bahamaamooz dgraph[28766]: W0420 17:16:08.220373   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 220ms. Breakdown: [{sync 219ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:18:10 bahamaamooz dgraph[28766]: W0420 17:18:10.242891   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 214ms. Breakdown: [{sync 214ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:20:02 bahamaamooz dgraph[28766]: W0420 17:20:02.578438   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 301ms. Breakdown: [{sync 300ms} {advance 1ms} {disk 0s} {pro
Apr 20 17:20:09 bahamaamooz dgraph[28766]: W0420 17:20:09.110351   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 252ms. Breakdown: [{sync 252ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:20:27 bahamaamooz dgraph[28766]: I0420 17:20:27.757012   28766 draft.go:403] Creating snapshot at index: 274610. ReadTs: 282290.
Apr 20 17:20:41 bahamaamooz dgraph[28766]: W0420 17:20:41.352767   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 226ms. Breakdown: [{sync 222ms} {disk 4ms} {proposals 0s} {a
Apr 20 17:20:42 bahamaamooz dgraph[28766]: W0420 17:20:42.003781   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 271ms. Breakdown: [{sync 271ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:21:25 bahamaamooz dgraph[28766]: W0420 17:21:25.071562   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 203ms. Breakdown: [{sync 202ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:24:27 bahamaamooz dgraph[28766]: I0420 17:24:27.784905   28766 draft.go:403] Creating snapshot at index: 284970. ReadTs: 292556.
Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.109584   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.109754   28766 log.go:34] Running for level: 0
Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.860445   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 750.639156ms
Apr 20 17:25:14 bahamaamooz dgraph[28766]: I0420 17:25:14.860814   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:25:28 bahamaamooz dgraph[28766]: I0420 17:25:28.196467   28766 log.go:34] Rolling up Created batch of size: 201 kB in 38.274057ms.
Apr 20 17:25:28 bahamaamooz dgraph[28766]: I0420 17:25:28.708029   28766 log.go:34] Rolling up Created batch of size: 812 kB in 63.863736ms.
Apr 20 17:25:28 bahamaamooz dgraph[28766]: I0420 17:25:28.708083   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 1.0 MB, speed: 1.0 MB/sec
Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.001872   28766 log.go:34] Rolling up Created batch of size: 544 kB in 193.812676ms.
Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.239683   28766 log.go:34] Rolling up Created batch of size: 1.3 MB in 136.271068ms.
Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.367924   28766 log.go:34] Rolling up Created batch of size: 1.3 MB in 121.678506ms.
Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.597316   28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 24.135486ms.
Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.707355   28766 log.go:34] Rolling up Created batch of size: 1.5 MB in 107.384506ms.
Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.707466   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 10 MB, speed: 5.1 MB/sec
Apr 20 17:25:29 bahamaamooz dgraph[28766]: I0420 17:25:29.826113   28766 log.go:34] Rolling up Created batch of size: 1.1 MB in 36.396µs.
Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.243835   28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 147.521131ms.
Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.641936   28766 log.go:34] Rolling up Created batch of size: 623 kB in 15.062856ms.
Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.641980   28766 log.go:34] Rolling up Sent 59096 keys
Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.659444   28766 draft.go:1102] Rolled up 59070 keys. Done
Apr 20 17:25:30 bahamaamooz dgraph[28766]: I0420 17:25:30.659493   28766 draft.go:445] List rollup at Ts 292556: OK.
Apr 20 17:25:35 bahamaamooz dgraph[28766]: W0420 17:25:35.597698   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 409ms. Breakdown: [{sync 409ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:27:27 bahamaamooz dgraph[28766]: I0420 17:27:27.768234   28766 draft.go:403] Creating snapshot at index: 295286. ReadTs: 302778.
Apr 20 17:29:27 bahamaamooz dgraph[28766]: W0420 17:29:27.781013   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 377ms. Breakdown: [{sync 376ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:29:45 bahamaamooz dgraph[28766]: W0420 17:29:45.131899   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 452ms. Breakdown: [{sync 452ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:30:10 bahamaamooz dgraph[28766]: W0420 17:30:10.155710   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 239ms. Breakdown: [{sync 239ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:30:27 bahamaamooz dgraph[28766]: I0420 17:30:27.956105   28766 log.go:34] Rolling up Created batch of size: 95 kB in 15.298369ms.
Apr 20 17:30:28 bahamaamooz dgraph[28766]: I0420 17:30:28.581262   28766 log.go:34] Rolling up Created batch of size: 589 kB in 68.546857ms.
Apr 20 17:30:28 bahamaamooz dgraph[28766]: I0420 17:30:28.650365   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 683 kB, speed: 683 kB/sec
Apr 20 17:30:28 bahamaamooz dgraph[28766]: I0420 17:30:28.972709   28766 log.go:34] Rolling up Created batch of size: 256 kB in 75.623184ms.
Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.218469   28766 log.go:34] Rolling up Created batch of size: 847 kB in 115.691039ms.
Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.404551   28766 log.go:34] Rolling up Created batch of size: 970 kB in 107.183733ms.
Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.468659   28766 log.go:34] Rolling up Created batch of size: 882 kB in 63.274622ms.
Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.576570   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 35.345788ms.
Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.650639   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 7.9 MB, speed: 4.0 MB/sec
Apr 20 17:30:29 bahamaamooz dgraph[28766]: I0420 17:30:29.691596   28766 log.go:34] Rolling up Created batch of size: 1.1 MB in 52.47µs.
Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.244371   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 41.508643ms.
Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.543926   28766 log.go:34] Rolling up Created batch of size: 549 kB in 10.845674ms.
Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.543964   28766 log.go:34] Rolling up Sent 28201 keys
Apr 20 17:30:30 bahamaamooz dgraph[28766]: W0420 17:30:30.657917   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 410ms. Breakdown: [{sync 410ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.863283   28766 draft.go:1102] Rolled up 28173 keys. Done
Apr 20 17:30:30 bahamaamooz dgraph[28766]: I0420 17:30:30.863333   28766 draft.go:445] List rollup at Ts 302778: OK.
Apr 20 17:30:35 bahamaamooz dgraph[28766]: W0420 17:30:35.857710   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 377ms. Breakdown: [{sync 376ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:30:38 bahamaamooz dgraph[28766]: W0420 17:30:38.296406   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 224ms. Breakdown: [{sync 223ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:30:41 bahamaamooz dgraph[28766]: W0420 17:30:41.191843   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 364ms. Breakdown: [{sync 364ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:31:27 bahamaamooz dgraph[28766]: I0420 17:31:27.787463   28766 draft.go:403] Creating snapshot at index: 307075. ReadTs: 314474.
Apr 20 17:31:54 bahamaamooz dgraph[28766]: I0420 17:31:54.427110   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:31:54 bahamaamooz dgraph[28766]: I0420 17:31:54.427261   28766 log.go:34] Running for level: 0
Apr 20 17:32:03 bahamaamooz dgraph[28766]: I0420 17:32:03.547464   28766 log.go:34] LOG Compact 0->1, del 102 tables, add 1 tables, took 9.120163784s
Apr 20 17:32:03 bahamaamooz dgraph[28766]: I0420 17:32:03.548096   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:32:05 bahamaamooz dgraph[28766]: I0420 17:32:05.675812   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:32:05 bahamaamooz dgraph[28766]: I0420 17:32:05.676772   28766 log.go:34] Running for level: 0
Apr 20 17:32:06 bahamaamooz dgraph[28766]: I0420 17:32:06.413693   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 736.283404ms
Apr 20 17:32:06 bahamaamooz dgraph[28766]: I0420 17:32:06.413737   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:32:09 bahamaamooz dgraph[28766]: W0420 17:32:09.094520   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 205ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:33:29 bahamaamooz dgraph[28766]: W0420 17:33:29.744632   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 321ms. Breakdown: [{sync 320ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:35:27 bahamaamooz dgraph[28766]: I0420 17:35:27.785205   28766 draft.go:403] Creating snapshot at index: 318389. ReadTs: 325674.
Apr 20 17:35:28 bahamaamooz dgraph[28766]: I0420 17:35:28.652565   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 17:35:29 bahamaamooz dgraph[28766]: I0420 17:35:29.651975   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 17:35:30 bahamaamooz dgraph[28766]: I0420 17:35:30.062443   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 109.279337ms.
Apr 20 17:35:30 bahamaamooz dgraph[28766]: I0420 17:35:30.652010   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.4 MB, speed: 1.5 MB/sec
Apr 20 17:35:31 bahamaamooz dgraph[28766]: I0420 17:35:31.661646   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 112.126478ms.
Apr 20 17:35:31 bahamaamooz dgraph[28766]: I0420 17:35:31.661723   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.6 MB, speed: 2.1 MB/sec
Apr 20 17:35:32 bahamaamooz dgraph[28766]: I0420 17:35:32.011756   28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 7.012549ms.
Apr 20 17:35:32 bahamaamooz dgraph[28766]: I0420 17:35:32.651930   28766 log.go:34] Rolling up Time elapsed: 05s, bytes sent: 13 MB, speed: 2.6 MB/sec
Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.417130   28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 55.733975ms.
Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.417172   28766 log.go:34] Rolling up Sent 32242 keys
Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.456689   28766 draft.go:1102] Rolled up 32214 keys. Done
Apr 20 17:35:33 bahamaamooz dgraph[28766]: I0420 17:35:33.457124   28766 draft.go:445] List rollup at Ts 314474: OK.
Apr 20 17:38:29 bahamaamooz dgraph[28766]: W0420 17:38:29.268407   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 224ms. Breakdown: [{sync 222ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:38:36 bahamaamooz dgraph[28766]: W0420 17:38:36.095798   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 217ms. Breakdown: [{sync 216ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:39:04 bahamaamooz dgraph[28766]: W0420 17:39:04.533391   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 381ms. Breakdown: [{sync 380ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:39:16 bahamaamooz dgraph[28766]: W0420 17:39:16.848751   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 217ms. Breakdown: [{sync 217ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:39:21 bahamaamooz dgraph[28766]: W0420 17:39:21.441346   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 467ms. Breakdown: [{sync 466ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:39:31 bahamaamooz dgraph[28766]: W0420 17:39:31.890610   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 215ms. Breakdown: [{sync 214ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:40:27 bahamaamooz dgraph[28766]: I0420 17:40:27.809335   28766 draft.go:403] Creating snapshot at index: 330313. ReadTs: 337496.
Apr 20 17:40:28 bahamaamooz dgraph[28766]: I0420 17:40:28.650412   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 17:40:29 bahamaamooz dgraph[28766]: I0420 17:40:29.318832   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 159.928634ms.
Apr 20 17:40:29 bahamaamooz dgraph[28766]: I0420 17:40:29.650277   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.4 MB, speed: 2.2 MB/sec
Apr 20 17:40:30 bahamaamooz dgraph[28766]: I0420 17:40:30.605924   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 124.241092ms.
Apr 20 17:40:30 bahamaamooz dgraph[28766]: I0420 17:40:30.651682   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 8.6 MB, speed: 2.9 MB/sec
Apr 20 17:40:31 bahamaamooz dgraph[28766]: I0420 17:40:31.057692   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 8.31469ms.
Apr 20 17:40:31 bahamaamooz dgraph[28766]: I0420 17:40:31.650249   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.2 MB/sec
Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.181384   28766 log.go:34] Rolling up Created batch of size: 1.5 MB in 58.13928ms.
Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.181595   28766 log.go:34] Rolling up Sent 30868 keys
Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.222282   28766 draft.go:1102] Rolled up 30838 keys. Done
Apr 20 17:40:32 bahamaamooz dgraph[28766]: I0420 17:40:32.222766   28766 draft.go:445] List rollup at Ts 325674: OK.
Apr 20 17:41:07 bahamaamooz dgraph[28766]: I0420 17:41:07.675484   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:41:07 bahamaamooz dgraph[28766]: I0420 17:41:07.676361   28766 log.go:34] Running for level: 0
Apr 20 17:41:08 bahamaamooz dgraph[28766]: I0420 17:41:08.783031   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.106512061s
Apr 20 17:41:08 bahamaamooz dgraph[28766]: I0420 17:41:08.783081   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:41:42 bahamaamooz dgraph[28766]: W0420 17:41:42.419940   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 764ms. Breakdown: [{sync 763ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:43:27 bahamaamooz dgraph[28766]: I0420 17:43:27.767561   28766 draft.go:403] Creating snapshot at index: 341286. ReadTs: 348376.
Apr 20 17:45:28 bahamaamooz dgraph[28766]: I0420 17:45:28.650439   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 17:45:29 bahamaamooz dgraph[28766]: I0420 17:45:29.045354   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 282.454936ms.
Apr 20 17:45:29 bahamaamooz dgraph[28766]: I0420 17:45:29.650479   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.3 MB, speed: 2.1 MB/sec
Apr 20 17:45:30 bahamaamooz dgraph[28766]: I0420 17:45:30.081884   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 140.864727ms.
Apr 20 17:45:30 bahamaamooz dgraph[28766]: I0420 17:45:30.637112   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 72.212754ms.
Apr 20 17:45:30 bahamaamooz dgraph[28766]: I0420 17:45:30.650702   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 13 MB, speed: 4.3 MB/sec
Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.675493   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 120.445796ms.
Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.675547   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 17 MB, speed: 4.2 MB/sec
Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.683566   28766 log.go:34] Rolling up Created batch of size: 309 kB in 7.832957ms.
Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.683602   28766 log.go:34] Rolling up Sent 61620 keys
Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.805878   28766 draft.go:1102] Rolled up 61588 keys. Done
Apr 20 17:45:31 bahamaamooz dgraph[28766]: I0420 17:45:31.806360   28766 draft.go:445] List rollup at Ts 348376: OK.
Apr 20 17:47:27 bahamaamooz dgraph[28766]: I0420 17:47:27.774453   28766 draft.go:403] Creating snapshot at index: 352494. ReadTs: 359462.
Apr 20 17:48:30 bahamaamooz dgraph[28766]: I0420 17:48:30.682441   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:48:30 bahamaamooz dgraph[28766]: I0420 17:48:30.683396   28766 log.go:34] Running for level: 0
Apr 20 17:48:31 bahamaamooz dgraph[28766]: I0420 17:48:31.347651   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 664.033182ms
Apr 20 17:48:31 bahamaamooz dgraph[28766]: I0420 17:48:31.347765   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:49:55 bahamaamooz dgraph[28766]: W0420 17:49:55.426192   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 432ms. Breakdown: [{sync 432ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:50:28 bahamaamooz dgraph[28766]: I0420 17:50:28.650430   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 17:50:29 bahamaamooz dgraph[28766]: I0420 17:50:29.437822   28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 184.569795ms.
Apr 20 17:50:29 bahamaamooz dgraph[28766]: I0420 17:50:29.650874   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.5 MB, speed: 2.3 MB/sec
Apr 20 17:50:30 bahamaamooz dgraph[28766]: I0420 17:50:30.650455   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.5 MB, speed: 1.5 MB/sec
Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.019603   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 137.740531ms.
Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.354189   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 2.44292ms.
Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.650456   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec
Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.986989   28766 log.go:34] Rolling up Created batch of size: 1.1 MB in 53.488444ms.
Apr 20 17:50:31 bahamaamooz dgraph[28766]: I0420 17:50:31.987063   28766 log.go:34] Rolling up Sent 30565 keys
Apr 20 17:50:32 bahamaamooz dgraph[28766]: I0420 17:50:32.000591   28766 draft.go:1102] Rolled up 30531 keys. Done
Apr 20 17:50:32 bahamaamooz dgraph[28766]: I0420 17:50:32.000699   28766 draft.go:445] List rollup at Ts 359462: OK.
Apr 20 17:51:10 bahamaamooz dgraph[28766]: W0420 17:51:10.574433   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 214ms. Breakdown: [{sync 213ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:51:16 bahamaamooz dgraph[28766]: W0420 17:51:16.699467   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 303ms. Breakdown: [{sync 303ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:51:18 bahamaamooz dgraph[28766]: W0420 17:51:18.550492   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 335ms. Breakdown: [{sync 335ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:51:27 bahamaamooz dgraph[28766]: I0420 17:51:27.758368   28766 draft.go:403] Creating snapshot at index: 364846. ReadTs: 371710.
Apr 20 17:51:37 bahamaamooz dgraph[28766]: W0420 17:51:37.734522   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 554ms. Breakdown: [{sync 554ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:51:38 bahamaamooz dgraph[28766]: W0420 17:51:38.728562   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 345ms. Breakdown: [{sync 345ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:51:52 bahamaamooz dgraph[28766]: W0420 17:51:52.350879   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 614ms. Breakdown: [{sync 613ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:51:53 bahamaamooz dgraph[28766]: W0420 17:51:53.284614   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 604ms. Breakdown: [{sync 604ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:52:56 bahamaamooz dgraph[28766]: W0420 17:52:56.980791   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 210ms. Breakdown: [{sync 210ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:53:11 bahamaamooz dgraph[28766]: W0420 17:53:11.656884   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 217ms. Breakdown: [{sync 216ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:54:50 bahamaamooz dgraph[28766]: W0420 17:54:50.632289   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 240ms. Breakdown: [{sync 240ms} {disk 1ms} {proposals 0s} {a
Apr 20 17:54:57 bahamaamooz dgraph[28766]: W0420 17:54:57.595442   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 375ms. Breakdown: [{sync 375ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:55:27 bahamaamooz dgraph[28766]: I0420 17:55:27.863959   28766 draft.go:403] Creating snapshot at index: 375428. ReadTs: 382208.
Apr 20 17:55:28 bahamaamooz dgraph[28766]: I0420 17:55:28.650372   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 17:55:29 bahamaamooz dgraph[28766]: I0420 17:55:29.415184   28766 log.go:34] Rolling up Created batch of size: 4.7 MB in 147.820275ms.
Apr 20 17:55:29 bahamaamooz dgraph[28766]: I0420 17:55:29.651032   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.7 MB, speed: 2.3 MB/sec
Apr 20 17:55:30 bahamaamooz dgraph[28766]: I0420 17:55:30.596777   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 108.848499ms.
Apr 20 17:55:30 bahamaamooz dgraph[28766]: I0420 17:55:30.650416   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 8.9 MB, speed: 3.0 MB/sec
Apr 20 17:55:30 bahamaamooz dgraph[28766]: I0420 17:55:30.954348   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 848.458µs.
Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.650611   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec
Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.745629   28766 log.go:34] Rolling up Created batch of size: 1.2 MB in 45.890597ms.
Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.746086   28766 log.go:34] Rolling up Sent 33740 keys
Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.907607   28766 draft.go:1102] Rolled up 33706 keys. Done
Apr 20 17:55:31 bahamaamooz dgraph[28766]: I0420 17:55:31.907656   28766 draft.go:445] List rollup at Ts 371710: OK.
Apr 20 17:55:37 bahamaamooz dgraph[28766]: W0420 17:55:37.476764   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 225ms. Breakdown: [{sync 225ms} {disk 0s} {proposals 0s} {ad
Apr 20 17:56:27 bahamaamooz dgraph[28766]: I0420 17:56:27.109518   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 17:56:27 bahamaamooz dgraph[28766]: I0420 17:56:27.109590   28766 log.go:34] Running for level: 0
Apr 20 17:56:28 bahamaamooz dgraph[28766]: I0420 17:56:28.020378   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 910.757046ms
Apr 20 17:56:28 bahamaamooz dgraph[28766]: I0420 17:56:28.020428   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 17:59:27 bahamaamooz dgraph[28766]: I0420 17:59:27.776412   28766 draft.go:403] Creating snapshot at index: 389306. ReadTs: 395974.
Apr 20 18:00:28 bahamaamooz dgraph[28766]: I0420 18:00:28.650469   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:00:29 bahamaamooz dgraph[28766]: I0420 18:00:29.612647   28766 log.go:34] Rolling up Created batch of size: 4.5 MB in 223.948853ms.
Apr 20 18:00:29 bahamaamooz dgraph[28766]: I0420 18:00:29.650368   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.5 MB, speed: 2.3 MB/sec
Apr 20 18:00:30 bahamaamooz dgraph[28766]: I0420 18:00:30.650483   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.5 MB, speed: 1.5 MB/sec
Apr 20 18:00:30 bahamaamooz dgraph[28766]: I0420 18:00:30.968780   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 219.050328ms.
Apr 20 18:00:31 bahamaamooz dgraph[28766]: I0420 18:00:31.477868   28766 log.go:34] Rolling up Created batch of size: 4.6 MB in 37.609314ms.
Apr 20 18:00:31 bahamaamooz dgraph[28766]: I0420 18:00:31.650406   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 14 MB, speed: 3.4 MB/sec
Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.459630   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 128.021876ms.
Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.499964   28766 log.go:34] Rolling up Created batch of size: 329 kB in 14.666922ms.
Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.500007   28766 log.go:34] Rolling up Sent 65774 keys
Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.556360   28766 draft.go:1102] Rolled up 65740 keys. Done
Apr 20 18:00:32 bahamaamooz dgraph[28766]: I0420 18:00:32.556940   28766 draft.go:445] List rollup at Ts 395974: OK.
Apr 20 18:02:01 bahamaamooz dgraph[28766]: W0420 18:02:01.454977   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 374ms. Breakdown: [{sync 374ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:02:25 bahamaamooz dgraph[28766]: W0420 18:02:25.396453   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 204ms. Breakdown: [{sync 204ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:03:02 bahamaamooz dgraph[28766]: W0420 18:03:02.288886   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 338ms. Breakdown: [{sync 337ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:03:03 bahamaamooz dgraph[28766]: W0420 18:03:03.001275   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 554ms. Breakdown: [{sync 553ms} {disk 1ms} {proposals 0s} {a
Apr 20 18:03:07 bahamaamooz dgraph[28766]: W0420 18:03:07.618005   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 327ms. Breakdown: [{sync 327ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:03:27 bahamaamooz dgraph[28766]: I0420 18:03:27.727312   28766 draft.go:403] Creating snapshot at index: 399401. ReadTs: 405990.
Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.109524   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.109614   28766 log.go:34] Running for level: 0
Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.794071   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 684.427226ms
Apr 20 18:03:28 bahamaamooz dgraph[28766]: I0420 18:03:28.794126   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 18:05:28 bahamaamooz dgraph[28766]: I0420 18:05:28.653014   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:05:29 bahamaamooz dgraph[28766]: I0420 18:05:29.419442   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 122.518883ms.
Apr 20 18:05:29 bahamaamooz dgraph[28766]: I0420 18:05:29.656640   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.3 MB, speed: 2.1 MB/sec
Apr 20 18:05:30 bahamaamooz dgraph[28766]: I0420 18:05:30.653035   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.3 MB, speed: 1.4 MB/sec
Apr 20 18:05:31 bahamaamooz dgraph[28766]: I0420 18:05:31.177333   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 109.623607ms.
Apr 20 18:05:31 bahamaamooz dgraph[28766]: I0420 18:05:31.624712   28766 log.go:34] Rolling up Created batch of size: 4.6 MB in 9.0166ms.
Apr 20 18:05:31 bahamaamooz dgraph[28766]: I0420 18:05:31.653053   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec
Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.563344   28766 log.go:34] Rolling up Created batch of size: 1.6 MB in 52.749284ms.
Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.564131   28766 log.go:34] Rolling up Sent 27605 keys
Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.583824   28766 draft.go:1102] Rolled up 27571 keys. Done
Apr 20 18:05:32 bahamaamooz dgraph[28766]: I0420 18:05:32.583906   28766 draft.go:445] List rollup at Ts 405990: OK.
Apr 20 18:06:27 bahamaamooz dgraph[28766]: I0420 18:06:27.792530   28766 draft.go:403] Creating snapshot at index: 410167. ReadTs: 416672.
Apr 20 18:07:51 bahamaamooz dgraph[28766]: W0420 18:07:51.573044   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 209ms. Breakdown: [{sync 208ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:09:14 bahamaamooz dgraph[28766]: W0420 18:09:14.727818   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 371ms. Breakdown: [{sync 371ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:09:44 bahamaamooz dgraph[28766]: W0420 18:09:44.347542   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 393ms. Breakdown: [{sync 393ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:10:09 bahamaamooz dgraph[28766]: I0420 18:10:09.675581   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 18:10:09 bahamaamooz dgraph[28766]: I0420 18:10:09.676369   28766 log.go:34] Running for level: 0
Apr 20 18:10:10 bahamaamooz dgraph[28766]: I0420 18:10:10.693406   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.016823623s
Apr 20 18:10:10 bahamaamooz dgraph[28766]: I0420 18:10:10.693586   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 18:10:27 bahamaamooz dgraph[28766]: I0420 18:10:27.815062   28766 draft.go:403] Creating snapshot at index: 424051. ReadTs: 430432.
Apr 20 18:10:28 bahamaamooz dgraph[28766]: I0420 18:10:28.650160   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:10:29 bahamaamooz dgraph[28766]: I0420 18:10:29.546656   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 116.356416ms.
Apr 20 18:10:29 bahamaamooz dgraph[28766]: I0420 18:10:29.650221   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.4 MB, speed: 2.2 MB/sec
Apr 20 18:10:30 bahamaamooz dgraph[28766]: I0420 18:10:30.650186   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.4 MB, speed: 1.5 MB/sec
Apr 20 18:10:31 bahamaamooz dgraph[28766]: I0420 18:10:31.237963   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 97.218639ms.
Apr 20 18:10:31 bahamaamooz dgraph[28766]: I0420 18:10:31.511487   28766 log.go:34] Rolling up Created batch of size: 4.4 MB in 4.664048ms.
Apr 20 18:10:31 bahamaamooz dgraph[28766]: I0420 18:10:31.650164   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 13 MB, speed: 3.3 MB/sec
Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.467521   28766 log.go:34] Rolling up Created batch of size: 2.2 MB in 62.428891ms.
Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.467597   28766 log.go:34] Rolling up Sent 29462 keys
Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.709805   28766 draft.go:1102] Rolled up 29422 keys. Done
Apr 20 18:10:32 bahamaamooz dgraph[28766]: I0420 18:10:32.709890   28766 draft.go:445] List rollup at Ts 416672: OK.
Apr 20 18:10:33 bahamaamooz dgraph[28766]: W0420 18:10:33.868649   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 1.527s. Breakdown: [{disk 1.527s} {advance 1ms} {proposals 0
Apr 20 18:10:36 bahamaamooz dgraph[28766]: I0420 18:10:36.053683   28766 log.go:34] 1 [logterm: 2, index: 424185, vote: 1] cast MsgPreVote for 2 [logterm: 2, index: 424185] at term 2
Apr 20 18:10:37 bahamaamooz dgraph[28766]: I0420 18:10:37.047095   28766 log.go:34] 1 [logterm: 2, index: 424185, vote: 1] cast MsgPreVote for 3 [logterm: 2, index: 424185] at term 2
Apr 20 18:10:37 bahamaamooz dgraph[28766]: W0420 18:10:37.292070   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 3.404s. Breakdown: [{sync 3.402s} {disk 1ms} {proposals 0s}
Apr 20 18:11:04 bahamaamooz dgraph[28766]: W0420 18:11:04.180522   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 692ms. Breakdown: [{sync 692ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:11:34 bahamaamooz dgraph[28766]: W0420 18:11:34.772951   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 255ms. Breakdown: [{sync 255ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:11:35 bahamaamooz dgraph[28766]: W0420 18:11:35.627772   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 234ms. Breakdown: [{sync 234ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:11:36 bahamaamooz dgraph[28766]: W0420 18:11:35.953056   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 296ms. Breakdown: [{sync 295ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:12:41 bahamaamooz dgraph[28766]: W0420 18:12:41.595576   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 326ms. Breakdown: [{sync 325ms} {disk 1ms} {proposals 0s} {a
Apr 20 18:12:46 bahamaamooz dgraph[28766]: W0420 18:12:46.524123   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 501ms. Breakdown: [{sync 500ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:12:47 bahamaamooz dgraph[28766]: W0420 18:12:47.428564   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 446ms. Breakdown: [{sync 445ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:13:11 bahamaamooz dgraph[28766]: W0420 18:13:11.139660   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 345ms. Breakdown: [{sync 343ms} {disk 1ms} {advance 1ms} {pr
Apr 20 18:13:31 bahamaamooz dgraph[28766]: W0420 18:13:31.942161   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 283ms. Breakdown: [{sync 283ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:13:40 bahamaamooz dgraph[28766]: W0420 18:13:40.733562   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 297ms. Breakdown: [{sync 297ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:13:41 bahamaamooz dgraph[28766]: W0420 18:13:41.924924   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 207ms. Breakdown: [{sync 207ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:13:42 bahamaamooz dgraph[28766]: W0420 18:13:42.538701   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 243ms. Breakdown: [{sync 243ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:14:08 bahamaamooz dgraph[28766]: W0420 18:14:08.828918   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 252ms. Breakdown: [{sync 252ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:14:11 bahamaamooz dgraph[28766]: W0420 18:14:11.180321   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 215ms. Breakdown: [{sync 215ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:15:27 bahamaamooz dgraph[28766]: I0420 18:15:27.767331   28766 draft.go:403] Creating snapshot at index: 434521. ReadTs: 440824.
Apr 20 18:15:28 bahamaamooz dgraph[28766]: I0420 18:15:28.650629   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:15:29 bahamaamooz dgraph[28766]: I0420 18:15:29.654595   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 162.221311ms.
Apr 20 18:15:29 bahamaamooz dgraph[28766]: I0420 18:15:29.654635   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.3 MB, speed: 2.1 MB/sec
Apr 20 18:15:30 bahamaamooz dgraph[28766]: I0420 18:15:30.650663   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.3 MB, speed: 1.4 MB/sec
Apr 20 18:15:31 bahamaamooz dgraph[28766]: I0420 18:15:31.708641   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 300.451718ms.
Apr 20 18:15:31 bahamaamooz dgraph[28766]: I0420 18:15:31.708699   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.5 MB, speed: 2.1 MB/sec
Apr 20 18:15:32 bahamaamooz dgraph[28766]: I0420 18:15:32.182638   28766 log.go:34] Rolling up Created batch of size: 4.7 MB in 9.559719ms.
Apr 20 18:15:32 bahamaamooz dgraph[28766]: I0420 18:15:32.650591   28766 log.go:34] Rolling up Time elapsed: 05s, bytes sent: 13 MB, speed: 2.6 MB/sec
Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.442282   28766 log.go:34] Rolling up Created batch of size: 1.9 MB in 74.468065ms.
Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.442357   28766 log.go:34] Rolling up Sent 37804 keys
Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.475653   28766 draft.go:1102] Rolled up 37748 keys. Done
Apr 20 18:15:33 bahamaamooz dgraph[28766]: I0420 18:15:33.475745   28766 draft.go:445] List rollup at Ts 430432: OK.
Apr 20 18:17:37 bahamaamooz dgraph[28766]: W0420 18:17:37.018223   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 436ms. Breakdown: [{sync 435ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:18:09 bahamaamooz dgraph[28766]: I0420 18:18:09.675547   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 18:18:09 bahamaamooz dgraph[28766]: I0420 18:18:09.675631   28766 log.go:34] Running for level: 0
Apr 20 18:18:10 bahamaamooz dgraph[28766]: I0420 18:18:10.812278   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.136617977s
Apr 20 18:18:10 bahamaamooz dgraph[28766]: I0420 18:18:10.812464   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 18:18:27 bahamaamooz dgraph[28766]: I0420 18:18:27.793332   28766 draft.go:403] Creating snapshot at index: 445840. ReadTs: 452048.
Apr 20 18:18:48 bahamaamooz dgraph[28766]: W0420 18:18:48.796193   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 272ms. Breakdown: [{sync 271ms} {disk 1ms} {proposals 0s} {a
Apr 20 18:20:28 bahamaamooz dgraph[28766]: I0420 18:20:28.650235   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:20:29 bahamaamooz dgraph[28766]: I0420 18:20:29.355823   28766 log.go:34] Rolling up Created batch of size: 4.6 MB in 202.849543ms.
Apr 20 18:20:29 bahamaamooz dgraph[28766]: I0420 18:20:29.650247   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 4.6 MB, speed: 2.3 MB/sec
Apr 20 18:20:30 bahamaamooz dgraph[28766]: I0420 18:20:30.651730   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.6 MB, speed: 1.5 MB/sec
Apr 20 18:20:31 bahamaamooz dgraph[28766]: I0420 18:20:31.253274   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 200.436568ms.
Apr 20 18:20:31 bahamaamooz dgraph[28766]: I0420 18:20:31.661658   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.9 MB, speed: 2.2 MB/sec
Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.676519   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 91.651172ms.
Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.677558   28766 log.go:34] Rolling up Time elapsed: 05s, bytes sent: 13 MB, speed: 2.6 MB/sec
Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.782711   28766 log.go:34] Rolling up Created batch of size: 200 kB in 21.368785ms.
Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.782745   28766 log.go:34] Rolling up Sent 58750 keys
Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.789529   28766 draft.go:1102] Rolled up 58686 keys. Done
Apr 20 18:20:32 bahamaamooz dgraph[28766]: I0420 18:20:32.789978   28766 draft.go:445] List rollup at Ts 452048: OK.
Apr 20 18:21:27 bahamaamooz dgraph[28766]: I0420 18:21:27.777269   28766 draft.go:403] Creating snapshot at index: 456975. ReadTs: 463090.
Apr 20 18:21:59 bahamaamooz dgraph[28766]: W0420 18:21:59.664278   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 370ms. Breakdown: [{sync 370ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:24:23 bahamaamooz dgraph[28766]: I0420 18:24:23.675513   28766 log.go:34] Got compaction priority: {level:0 score:1 dropPrefix:[]}
Apr 20 18:24:23 bahamaamooz dgraph[28766]: I0420 18:24:23.675577   28766 log.go:34] Running for level: 0
Apr 20 18:24:24 bahamaamooz dgraph[28766]: I0420 18:24:24.769537   28766 log.go:34] LOG Compact 0->1, del 6 tables, add 1 tables, took 1.093931851s
Apr 20 18:24:24 bahamaamooz dgraph[28766]: I0420 18:24:24.770522   28766 log.go:34] Compaction for level: 0 DONE
Apr 20 18:24:27 bahamaamooz dgraph[28766]: I0420 18:24:27.809462   28766 draft.go:403] Creating snapshot at index: 467284. ReadTs: 473324.
Apr 20 18:25:07 bahamaamooz dgraph[28766]: W0420 18:25:07.389408   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 223ms. Breakdown: [{sync 223ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:25:09 bahamaamooz dgraph[28766]: W0420 18:25:09.059939   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 446ms. Breakdown: [{sync 445ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:25:28 bahamaamooz dgraph[28766]: I0420 18:25:28.650606   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:25:29 bahamaamooz dgraph[28766]: W0420 18:25:29.639356   28766 draft.go:1014] Raft.Ready took too long to process: Timer Total: 266ms. Breakdown: [{sync 266ms} {disk 0s} {proposals 0s} {ad
Apr 20 18:25:29 bahamaamooz dgraph[28766]: I0420 18:25:29.650630   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:25:30 bahamaamooz dgraph[28766]: I0420 18:25:30.067701   28766 log.go:34] Rolling up Created batch of size: 4.3 MB in 279.379256ms.
Apr 20 18:25:30 bahamaamooz dgraph[28766]: I0420 18:25:30.650923   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.3 MB, speed: 1.4 MB/sec
Apr 20 18:25:31 bahamaamooz dgraph[28766]: I0420 18:25:31.621366   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 228.702956ms.
Apr 20 18:25:31 bahamaamooz dgraph[28766]: I0420 18:25:31.650790   28766 log.go:34] Rolling up Time elapsed: 04s, bytes sent: 8.6 MB, speed: 2.1 MB/sec
Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.742783   28766 log.go:34] Rolling up Created batch of size: 2.7 MB in 103.084741ms.
Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.742835   28766 log.go:34] Rolling up Sent 57821 keys
Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.771294   28766 draft.go:1102] Rolled up 57757 keys. Done
Apr 20 18:25:32 bahamaamooz dgraph[28766]: I0420 18:25:32.771354   28766 draft.go:445] List rollup at Ts 473324: OK.
Apr 20 18:28:27 bahamaamooz dgraph[28766]: I0420 18:28:27.808939   28766 draft.go:403] Creating snapshot at index: 479238. ReadTs: 485157.
Apr 20 18:30:28 bahamaamooz dgraph[28766]: I0420 18:30:28.650650   28766 log.go:34] Rolling up Time elapsed: 01s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:30:29 bahamaamooz dgraph[28766]: I0420 18:30:29.650450   28766 log.go:34] Rolling up Time elapsed: 02s, bytes sent: 0 B, speed: 0 B/sec
Apr 20 18:30:30 bahamaamooz dgraph[28766]: I0420 18:30:30.225280   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 164.567164ms.
Apr 20 18:30:30 bahamaamooz dgraph[28766]: I0420 18:30:30.650342   28766 log.go:34] Rolling up Time elapsed: 03s, bytes sent: 4.2 MB, speed: 1.4 MB/sec
Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.432855   28766 log.go:34] Rolling up Created batch of size: 4.2 MB in 190.359427ms.
Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.443075   28766 log.go:34] Rolling up Created batch of size: 768 kB in 9.980586ms.
Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.443121   28766 log.go:34] Rolling up Sent 32654 keys
Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.617987   28766 draft.go:1102] Rolled up 32590 keys. Done
Apr 20 18:30:31 bahamaamooz dgraph[28766]: I0420 18:30:31.618044   28766 draft.go:445] List rollup at Ts 485157: OK.
Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.393971   28766 export.go:637] Got readonly ts from Zero: 486785
Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.394007   28766 export.go:641] Requesting export for groups: [1]
Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.394043   28766 export.go:389] Export requested at 486785.
Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.400568   28766 export.go:395] Running export for group 1 at timestamp 486785.
Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.400793   28766 export.go:416] Exporting data for group: 1 at /export/dgraph.r486785.u0420.1831/g01.rdf.gz
Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.401150   28766 export.go:427] Exporting schema for group: 1 at /export/dgraph.r486785.u0420.1831/g01.schema.gz
Apr 20 18:31:41 bahamaamooz dgraph[28766]: I0420 18:31:41.402635   28766 log.go:34] Export Created batch of size: 1.9 kB in 58.253µs.
Apr 20 18:31:42 bahamaamooz dgraph[28766]: I0420 18:31:42.508027   28766 log.go:34] Export Created batch of size: 4.4 MB in 393.898545ms.
Apr 20 18:31:42 bahamaamooz dgraph[28766]: I0420 18:31:42.508092   28766 log.go:34] Export Time elapsed: 01s, bytes sent: 4.4 MB, speed: 4.4 MB/sec
Apr 20 18:31:43 bahamaamooz dgraph[28766]: I0420 18:31:43.077085   28766 log.go:34] Export Created batch of size: 4.4 MB in 373.492215ms.
Apr 20 18:31:43 bahamaamooz dgraph[28766]: I0420 18:31:43.747006   28766 log.go:34] Export Created batch of size: 4.4 MB in 355.184183ms.
Apr 20 18:31:43 bahamaamooz dgraph[28766]: I0420 18:31:43.747061   28766 log.go:34] Export Time elapsed: 02s, bytes sent: 13 MB, speed: 6.6 MB/sec
Apr 20 18:31:44 bahamaamooz dgraph[28766]: I0420 18:31:44.383806   28766 log.go:34] Export Created batch of size: 4.4 MB in 423.703458ms.
Apr 20 18:31:44 bahamaamooz dgraph[28766]: I0420 18:31:44.402056   28766 log.go:34] Export Time elapsed: 03s, bytes sent: 18 MB, speed: 5.8 MB/sec
Apr 20 18:31:44 bahamaamooz dgraph[28766]: I0420 18:31:44.961854   28766 log.go:34] Export Created batch of size: 4.4 MB in 417.63053ms.
Apr 20 18:31:45 bahamaamooz dgraph[28766]: I0420 18:31:45.646537   28766 log.go:34] Export Created batch of size: 4.4 MB in 478.465043ms.
Apr 20 18:31:46 bahamaamooz dgraph[28766]: I0420 18:31:46.121838   28766 log.go:34] Export Created batch of size: 4.2 MB in 472.067314ms.
Apr 20 18:31:46 bahamaamooz dgraph[28766]: I0420 18:31:46.121894   28766 log.go:34] Export Time elapsed: 04s, bytes sent: 30 MB, speed: 7.6 MB/sec
Apr 20 18:31:47 bahamaamooz dgraph[28766]: I0420 18:31:47.175730   28766 log.go:34] Export Created batch of size: 8.5 MB in 1.048722542s.
Apr 20 18:31:49 bahamaamooz dgraph[28766]: I0420 18:31:49.047339   28766 log.go:34] Export Created batch of size: 14 MB in 1.861378622s.
Apr 20 18:31:51 bahamaamooz dgraph[28766]: I0420 18:31:51.943622   28766 log.go:34] Export Created batch of size: 17 MB in 2.872388501s.
Apr 20 18:31:58 bahamaamooz dgraph[28766]: I0420 18:31:58.180494   28766 log.go:34] Export Created batch of size: 22 MB in 6.208719501s.
Apr 20 18:31:58 bahamaamooz dgraph[28766]: I0420 18:31:58.181366   28766 log.go:34] Export Time elapsed: 16s, bytes sent: 92 MB, speed: 5.7 MB/sec
Apr 20 18:32:06 bahamaamooz dgraph[28766]: I0420 18:32:06.199603   28766 log.go:34] Export Created batch of size: 62 MB in 7.925219922s.
Apr 20 18:32:23 bahamaamooz dgraph[28766]: I0420 18:32:23.975419   28766 log.go:34] Export Created batch of size: 151 MB in 17.775674235s.
Apr 20 18:32:40 bahamaamooz dgraph[28766]: I0420 18:32:40.985126   28766 log.go:34] Export Created batch of size: 149 MB in 17.009504471s.
Apr 20 18:32:40 bahamaamooz dgraph[28766]: I0420 18:32:40.985176   28766 log.go:34] Export Time elapsed: 59s, bytes sent: 454 MB, speed: 7.7 MB/sec
Apr 20 18:32:58 bahamaamooz dgraph[28766]: I0420 18:32:58.388419   28766 log.go:34] Export Created batch of size: 149 MB in 17.403160347s.
Apr 20 18:32:58 bahamaamooz dgraph[28766]: I0420 18:32:58.389464   28766 log.go:34] Export Time elapsed: 01m16s, bytes sent: 603 MB, speed: 7.9 MB/sec
Apr 20 18:33:15 bahamaamooz dgraph[28766]: I0420 18:33:15.692125   28766 log.go:34] Export Created batch of size: 152 MB in 17.302413526s.
Apr 20 18:33:32 bahamaamooz dgraph[28766]: I0420 18:33:32.532131   28766 log.go:34] Export Created batch of size: 148 MB in 16.823994731s.
Apr 20 18:33:36 bahamaamooz dgraph[28766]: I0420 18:33:36.257165   28766 log.go:34] Export Created batch of size: 41 MB in 3.659248789s.
Apr 20 18:33:36 bahamaamooz dgraph[28766]: I0420 18:33:36.257221   28766 log.go:34] Export Time elapsed: 01m54s, bytes sent: 945 MB, speed: 8.3 MB/sec
Apr 20 18:33:36 bahamaamooz dgraph[28766]: I0420 18:33:36.257231   28766 log.go:34] Export Sent 2766963 keys
Apr 20 18:33:38 bahamaamooz dgraph[28766]: I0420 18:33:38.347383   28766 export.go:581] Export DONE for group 1 at timestamp 486785.
Apr 20 18:33:38 bahamaamooz dgraph[28766]: I0420 18:33:38.347441   28766 export.go:664] Export at readTs 486785 DONE
Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.650754   28766 draft.go:1194] Found 1 old transactions. Acting to abort them.
Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.668068   28766 draft.go:1155] TryAbort 1 txns with start ts. Error: <nil>
Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.668117   28766 draft.go:1178] TryAbort selectively proposing only aborted txns: txns:<start_ts:486784 >
Apr 20 18:34:27 bahamaamooz dgraph[28766]: I0420 18:34:27.678781   28766 draft.go:1197] Done abortOldTransactions for 1 txns. Error: <nil>
Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.507683   28766 export.go:637] Got readonly ts from Zero: 486787
Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.511182   28766 export.go:641] Requesting export for groups: [1]
Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.511512   28766 export.go:389] Export requested at 486787.
Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.518539   28766 export.go:395] Running export for group 1 at timestamp 486787.
Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.518988   28766 export.go:416] Exporting data for group: 1 at /export/dgraph.r486787.u0420.2037/g01.rdf.gz
Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.519572   28766 export.go:427] Exporting schema for group: 1 at /export/dgraph.r486787.u0420.2037/g01.schema.gz
Apr 20 20:37:58 bahamaamooz dgraph[28766]: I0420 20:37:58.524675   28766 log.go:34] Export Created batch of size: 1.9 kB in 103.882µs.
Apr 20 20:37:59 bahamaamooz dgraph[28766]: I0420 20:37:59.469617   28766 log.go:34] Export Created batch of size: 4.4 MB in 370.946991ms.
Apr 20 20:37:59 bahamaamooz dgraph[28766]: I0420 20:37:59.520637   28766 log.go:34] Export Time elapsed: 01s, bytes sent: 4.4 MB, speed: 4.4 MB/sec
Apr 20 20:37:59 bahamaamooz dgraph[28766]: I0420 20:37:59.883104   28766 log.go:34] Export Created batch of size: 4.4 MB in 297.129212ms.
Apr 20 20:38:00 bahamaamooz dgraph[28766]: I0420 20:38:00.480717   28766 log.go:34] Export Created batch of size: 4.4 MB in 301.697796ms.
Apr 20 20:38:00 bahamaamooz dgraph[28766]: I0420 20:38:00.520785   28766 log.go:34] Export Time elapsed: 02s, bytes sent: 13 MB, speed: 6.6 MB/sec
Apr 20 20:38:01 bahamaamooz dgraph[28766]: I0420 20:38:01.087791   28766 log.go:34] Export Created batch of size: 4.4 MB in 401.533857ms.
Apr 20 20:38:01 bahamaamooz dgraph[28766]: I0420 20:38:01.717108   28766 log.go:34] Export Created batch of size: 4.4 MB in 454.724942ms.
Apr 20 20:38:01 bahamaamooz dgraph[28766]: I0420 20:38:01.718066   28766 log.go:34] Export Time elapsed: 03s, bytes sent: 22 MB, speed: 7.3 MB/sec
Apr 20 20:38:02 bahamaamooz dgraph[28766]: I0420 20:38:02.429775   28766 log.go:34] Export Created batch of size: 4.4 MB in 444.008087ms.
Apr 20 20:38:03 bahamaamooz dgraph[28766]: I0420 20:38:03.276731   28766 log.go:34] Export Created batch of size: 8.5 MB in 843.888872ms.
Apr 20 20:38:04 bahamaamooz dgraph[28766]: I0420 20:38:04.537976   28766 log.go:34] Export Created batch of size: 9.5 MB in 1.254975667s.
Apr 20 20:38:06 bahamaamooz dgraph[28766]: I0420 20:38:06.236652   28766 log.go:34] Export Created batch of size: 13 MB in 1.683339894s.
Apr 20 20:38:06 bahamaamooz dgraph[28766]: I0420 20:38:06.236715   28766 log.go:34] Export Time elapsed: 07s, bytes sent: 57 MB, speed: 8.1 MB/sec
Apr 20 20:38:08 bahamaamooz dgraph[28766]: I0420 20:38:08.353316   28766 log.go:34] Export Created batch of size: 13 MB in 2.090577021s.
Apr 20 20:38:08 bahamaamooz dgraph[28766]: I0420 20:38:08.353361   28766 log.go:34] Export Time elapsed: 09s, bytes sent: 70 MB, speed: 7.8 MB/sec
Apr 20 20:38:13 bahamaamooz dgraph[28766]: I0420 20:38:13.832915   28766 log.go:34] Export Created batch of size: 17 MB in 5.464466068s.
Apr 20 20:38:13 bahamaamooz dgraph[28766]: I0420 20:38:13.832959   28766 log.go:34] Export Time elapsed: 15s, bytes sent: 87 MB, speed: 5.8 MB/sec
Apr 20 20:38:21 bahamaamooz dgraph[28766]: I0420 20:38:21.347762   28766 log.go:34] Export Created batch of size: 71 MB in 7.427539104s.
Apr 20 20:38:21 bahamaamooz dgraph[28766]: I0420 20:38:21.347823   28766 log.go:34] Export Time elapsed: 22s, bytes sent: 158 MB, speed: 7.2 MB/sec
Apr 20 20:38:37 bahamaamooz dgraph[28766]: I0420 20:38:37.374770   28766 log.go:34] Export Created batch of size: 151 MB in 16.026858075s.
Apr 20 20:38:37 bahamaamooz dgraph[28766]: I0420 20:38:37.374853   28766 log.go:34] Export Time elapsed: 38s, bytes sent: 310 MB, speed: 8.1 MB/sec
Apr 20 20:38:53 bahamaamooz dgraph[28766]: I0420 20:38:53.193042   28766 log.go:34] Export Created batch of size: 149 MB in 15.818086744s.
Apr 20 20:39:08 bahamaamooz dgraph[28766]: I0420 20:39:08.394121   28766 log.go:34] Export Created batch of size: 150 MB in 15.200937645s.
Apr 20 20:39:23 bahamaamooz dgraph[28766]: I0420 20:39:23.596759   28766 log.go:34] Export Created batch of size: 152 MB in 15.202505848s.
Apr 20 20:39:23 bahamaamooz dgraph[28766]: I0420 20:39:23.596842   28766 log.go:34] Export Time elapsed: 01m25s, bytes sent: 760 MB, speed: 8.9 MB/sec
Apr 20 20:39:38 bahamaamooz dgraph[28766]: I0420 20:39:38.889367   28766 log.go:34] Export Created batch of size: 148 MB in 15.266579314s.
Apr 20 20:39:41 bahamaamooz dgraph[28766]: I0420 20:39:41.785200   28766 log.go:34] Export Created batch of size: 37 MB in 2.852601845s.
Apr 20 20:39:41 bahamaamooz dgraph[28766]: I0420 20:39:41.785265   28766 log.go:34] Export Time elapsed: 01m43s, bytes sent: 945 MB, speed: 9.2 MB/sec
Apr 20 20:39:41 bahamaamooz dgraph[28766]: I0420 20:39:41.785288   28766 log.go:34] Export Sent 2766963 keys
Apr 20 20:39:42 bahamaamooz dgraph[28766]: I0420 20:39:42.003615   28766 export.go:581] Export DONE for group 1 at timestamp 486787.
Apr 20 20:39:42 bahamaamooz dgraph[28766]: I0420 20:39:42.003723   28766 export.go:664] Export at readTs 486787 DONE
lines 451-508/508 (END)

H4midR on 20 Apr 2020

👍1

Github issues have been deprecated.
This issue has been moved to discuss. You can follow the conversation there and also subscribe to updates by changing your notification preferences.

drawing