Dgraph: Alpha nodes got stuck in membership update

[andrewsmedina]

I'm using dgraph 1.0.14-rc2, with 3 alphas running on gke with a 8 cpus and 52GB memory for each alpha and 3 zeros with 2 cpus and 7.5gb of memory each zero.

I'm loading 1.1B of nodes splitted in chunks of 12k nodes using graph live. After 340k rdfs loaded, dgraph live could't insert more items (insert and reads outside live also does not works):

2019-04-09 01:50:40,134 [INFO] Starting Dgraph Live for lawsuit-bulk-174761725-174760629.rdf.gz                                                                        
I0409 01:50:40.171854    2769 init.go:88]                                                                                                                              

Dgraph version   : v1.0.13                                                                                                                                            
Commit SHA-1     : 691b3b35                                                                                                                                            
Commit timestamp : 2019-03-09 19:33:59 -0800                                                                                                                          
Branch           : HEAD                                                                                                                                                
Go version       : go1.11.5                                                                                                                                            

For Dgraph official documentation, visit https://docs.dgraph.io.                                                                                                      
For discussions about Dgraph     , visit https://discuss.dgraph.io.                                                                                                    
To say hi to the community       , visit https://dgraph.slack.com.                                                                                                    

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2018 Dgraph Labs, Inc.


2019/04/09 01:50:50 While trying to setup connection to Dgraph alpha. error: context deadline exceeded

I figure out that the load of alpha machines is to high, even when the cpu is low:

$ uptime                                                                                                  
 14:11:34 up 1 day, 14 min,  0 users,  load average: 51.66, 50.37, 50.20 

The alpha logs are stucked in No membership update for 10s

W0409 03:09:38.913842       1 groups.go:728] No membership update for 10s. Closing connection to Zero.
I0409 03:09:39.046952       1 groups.go:672] Got address of a Zero leader: dgraph-zero-2:5080
I0409 03:09:39.047161       1 groups.go:685] Starting a new membership stream receive from dgraph-zero-2:5080.
I0409 03:09:39.049432       1 groups.go:702] Received first state update from Zero: counter:2104743 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"dgraph-alpha-0:7080" last_update:1554661505 > > members:<key:2 value:<id:2 group_id:1 addr:"dgraph-alpha-1:7080" last_update:1554646775 > > members:<key:3 value:<id:3 group_id:1 addr:"dgraph-alpha-2:7080" last_update:1554698056 > > tablets:<key:"_predicate_" value:<group_id:1 predicate:"_predicate_" space:9239625119 > > tablets:<key:"apelido" value:<group_id:1 predicate:"apelido" > > tablets:<key:"dgraph.group.acl" value:<group_id:1 predicate:"dgraph.group.acl" space:39 > > tablets:<key:"dgraph.password" value:<group_id:1 predicate:"dgraph.password" space:37 > > tablets:<key:"dgraph.user.group" value:<group_id:1 predicate:"dgraph.user.group" space:43 > > tablets:<key:"dgraph.xid" value:<group_id:1 predicate:"dgraph.xid" space:36 > > tablets:<key:"lawsuit" value:<group_id:1 predicate:"lawsuit" space:1382663214 > > tablets:<key:"lawsuit_number" value:<group_id:1 predicate:"lawsuit_number" space:1625858758 > > tablets:<key:"name" value:<group_id:1 predicate:"name" space:3683039437 > > tablets:<key:"person" value:<group_id:1 predicate:"person" space:662342606 > > tablets:<key:"topic_id" value:<group_id:1 predicate:"topic_id" space:5527110520 > > snapshot_ts:252789 checksum:1119011364731594008 > > zeros:<key:1 value:<id:1 addr:"dgraph-zero-0:5080" > > zeros:<key:2 value:<id:2 addr:"dgraph-zero-1:5080" > > zeros:<key:3 value:<id:3 addr:"dgraph-zero-2:5080" leader:true > > maxLeaseId:20516620000 maxTxnTs:310000 maxRaftId:3 cid:"8bd76ea7-1b11-488c-8723-99383e774f74" 

[MichelDiz]

Hi Andrews,

What are the settings of the Liveloader you are using?

insert and reads outside live also does not works

I didn't get it, what is happening? and what are you doing and how are you doing it, outside Liveloader?

I would recommend that you use Bulkloader. It is perfect for big datasets. I have this repo https://github.com/MichelDiz/Dgraph-Bulk-Script with scripts working on k8s. Check if works for you.

BTW, We tried to focus Github issues for tracking bugs, features, and so on. If you need help you can open a topic in https://discuss.dgraph.io (Topics are preferable because it helps the community to understand use cases) or Slack. PS. If you have trouble getting into Slack. Send me your email and I can invite you.

[andrewsmedina]

Hi @MichaelJCompton thank you! I'm using the live because the bulk loader is only recommended for new clusters.

My live settings:

dgraph live -r rdffile -z dgraph_zero_uri -d config.dgraph_alpha_uri --conc 10 --batch 25000

I will move this discussion to the forum. :)

[MichelDiz]

Decrease the --batch size or let it default.

I will move this discussion to the forum. :)

Nice!

btw you've marked another person.

Cheers.

[MichelDiz]

Just an update on this.

I saw that you set almost 8GB of RAM to Zeros. Don't need tho. Dgraph Zero don't consume too much RAM. It's stays between 111MB and 214MB. It almost never pushes above 1GB. So, allocate these values ​​to Alphas xD

Forget what I said about Load Balancer and live load. u.u

If you are not using Loadbalance, it is recommended that you enter all Alphas exposed addresses (I mean, accessible) in the "-d" flag. (This flag will be renamed to -a --alpha soon)

e.g:

dgraph live -r rdffile -z dgraph_zero_uri -d "dgraph_alpha_uri:9080,dgraph_alpha_uri:9081,dgraph_alpha_uri:9082"

PS. No spaces between the uris