Project: Failure deploying MySQL in clustered mode

[JamesLaverack]

Following the MySQL Clustering Guide I'm attempting to deploy a multi-replica cluster.

Following the guide, I create the demo namespace and apply the following YAML:

apiVersion: kubedb.com/v1alpha1
kind: MySQL
metadata:
  name: my-group
  namespace: demo
spec:
  version: "5.7.25"
  replicas: 3
  topology:
    mode: GroupReplication
    group:
      name: "dc002fc3-c412-4d18-b1d4-66c1fbfbbc9b"
      baseServerID: 100
  storageType: Durable
  storage:
    storageClassName: "standard"
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  terminationPolicy: WipeOut

Expected Behavior

I'd expect this to deploy three pods, and come up as a MySQL cluster.

Actual Behavior

The second pod gets stuck in crash loop back off.

$ kubectl --namespace demo get pods 
NAME         READY   STATUS             RESTARTS   AGE
my-group-0   1/1     Running            0          52m
my-group-1   0/1     CrashLoopBackOff   19         51m

Looking at the logs for the two pods, I see the following errors:

$ kubectl --namespace demo logs --tail 5 my-group-0
2019-05-19T03:10:40.308864Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.0.1.8 refused. Address is not in the IP whitelist.'
2019-05-19T03:10:40.411158Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.0.1.8 refused. Address is not in the IP whitelist.'
2019-05-19T03:10:40.513019Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.0.1.8 refused. Address is not in the IP whitelist.'
2019-05-19T03:10:40.614700Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.0.1.8 refused. Address is not in the IP whitelist.'
2019-05-19T03:10:40.716439Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.0.1.8 refused. Address is not in the IP whitelist.'

$ kubectl --namespace demo logs --tail 5 my-group-1
2019-05-19T03:10:40.717623Z 0 [Note] Plugin group_replication reported: 'Error connecting using SSL 2000114 114.'
2019-05-19T03:10:40.718020Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error on opening a connection to my-group-0.my-group-gvr.demo:33060 on local port: 33060.'
2019-05-19T03:10:40.719267Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error connecting to all peers. Member join failed. Local port: 33060'
2019-05-19T03:10:40.726557Z 0 [Warning] Plugin group_replication reported: 'read failed'
2019-05-19T03:10:40.732262Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33060'

Checking, I do see that 10.0.1.8 is the assigned IP address for the my-group-1 pod.

$ kubectl --namespace demo --output json get pod my-group-1 | jq '.status.podIP'
"10.0.1.8"

Environment

I'm running in Google Kubernetes Engine.

Tool | Version
----- | ----------
KubeDB | 0.12.0
Kubernetes | v1.12.7-gke.10

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.79. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[tamalsaha]

We have identified the issue. The AUTOMATIC whitelist generated only works for single node cluster.

• https://dev.mysql.com/doc/refman/5.7/en/group-replication-options.html#sysvar_group_replication_ip_whitelist
• https://dev.mysql.com/doc/refman/5.7/en/group-replication-ip-address-whitelisting.html

resolution: set whitelist to pod_ip/16

• https://unix.stackexchange.com/a/201744/42136

# use hostname to detect pod ip inside the mysql docker image.
$ hostname -I
192.168.0.100 172.17.0.1 192.168.99.1 

$ myips=$(hostname -I)
$ first=${myips%% *}
$ echo $first
192.168.0.100

[JamesLaverack]

Thanks for the quick response @tamalsaha. Is there any workaround possible on the existing 0.12.0 KubeDB version, or is it simply a case of waiting for a new release that includes the changes in https://github.com/kubedb/mysql/pull/151?

[the-redback]

I have uploaded the new image with the same tag as it won't break the previous version.

@JamesLaverack , you can pull this image inside your nodes and see if it works.

[JamesLaverack]

@the-redback I'm seeing it working fine in a cluster now. All three pods now become ready and I see a Group Replication applier module successfully initialized! message in the logs.

Thanks for the fast turnaround. :+1:

[relengxing]

I think I have the same problem.
I use Google Kubernetes. 1.12.10-gke.17
KubeDB version is: 0.12.0
MySQL version is: 5.7.25

mysql-0   1/1     Running            0          7m51s
mysql-1   1/1     Running            0          7m5s
mysql-2   0/1     CrashLoopBackOff   5          6m9s

mysql-0 and mysql-1 in same node.

mysql-0 log:

2019-12-14T08:39:40.226238Z 59 [Note] Start binlog_dump to master_thread_id(59) slave_server(101), pos(, 4)
2019-12-14T08:39:40.323832Z 0 [Note] Plugin group_replication reported: 'The member with address mysql-1.mysql-gvr.demo:3306 was declared online within the replication group'
2019-12-14T08:40:40.226880Z 59 [Note] Aborted connection 59 to db: 'unconnected' user: 'repl' host: '10.9.12.241' (failed on flush_net())
2019-12-14T08:41:17.220671Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.10.3.2 refused. Address is not in the IP whitelist.'
2019-12-14T08:41:17.323264Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.10.3.2 refused. Address is not in the IP whitelist.'
2019-12-14T08:41:17.425174Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.10.3.2 refused. Address is not in the IP whitelist.'
2019-12-14T08:41:17.527228Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.10.3.2 refused. Address is not in the IP whitelist.'
2019-12-14T08:41:17.628872Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 10.10.3.2 refused. Address is not in the IP whitelist.'

mysql-2 log:

2019-12-14T08:41:18.032678Z 0 [Note] Plugin group_replication reported: 'Error connecting using SSL 2000114 114.'
2019-12-14T08:41:18.032767Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error on opening a connection to mysql-0.mysql-gvr.demo:33060 on local port: 33060.'
2019-12-14T08:41:18.038540Z 0 [Note] Plugin group_replication reported: 'Error connecting using SSL 2000114 114.'
2019-12-14T08:41:18.038592Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error on opening a connection to mysql-1.mysql-gvr.demo:33060 on local port: 33060.'
2019-12-14T08:41:18.135356Z 0 [Note] Plugin group_replication reported: 'Error connecting using SSL 2000114 114.'
2019-12-14T08:41:18.135443Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error on opening a connection to mysql-0.mysql-gvr.demo:33060 on local port: 33060.'
2019-12-14T08:41:18.139781Z 0 [Note] Plugin group_replication reported: 'Error connecting using SSL 2000114 114.'
2019-12-14T08:41:18.139838Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error on opening a connection to mysql-1.mysql-gvr.demo:33060 on local port: 33060.'
2019-12-14T08:41:18.140502Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error connecting to all peers. Member join failed. Local port: 33060'
2019-12-14T08:41:18.140734Z 0 [Warning] Plugin group_replication reported: 'read failed'
2019-12-14T08:41:18.149973Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33060'