Project: [MySQL/Cluster] Readiness Probe Fails
I'm trying to deploy sample mysql cluster but readiness probe fails with [Note] Access denied for user 'root'@'localhost' (using password: YES) on mysql container logs
this is my cr:
apiVersion: kubedb.com/v1alpha1
kind: MySQL
metadata:
name: my-group
namespace: demo
spec:
version: "5.7-v2"
replicas: 3
topology:
mode: GroupReplication
group:
name: "dc002fc3-c412-4d18-b1d4-66c1fbfbbc9b"
baseServerID: 100
storageType: Ephemeral
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
terminationPolicy: WipeOut
bug
ahmadalli
All 21 comments
Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.71. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
![issue-label-bot[bot] picture](https://avatars.githubusercontent.com/in/27079?v=4&s=40)
issue-label-bot[bot]
on 17 Jun 2019
👍2
@ahmadalli Can you give more details? For example, logs from database pods
shudipta
on 17 Jun 2019
I deployed the cr again and it deployed to my first 2 servers but I've faced to issue on the third server. here's the container log
2019/06/17 13:03:10 Determined Domain to be demo.svc.cluster.local
2019/06/17 13:03:10 Peer list updated
was []
now [my-group-0.my-group-gvr.demo.svc.cluster.local my-group-1.my-group-gvr.demo.svc.cluster.local my-group-2.my-group-gvr.demo.svc.cluster.local]
2019/06/17 13:03:10 execing: /on-start.sh with stdin: my-group-0.my-group-gvr.demo.svc.cluster.local
my-group-1.my-group-gvr.demo.svc.cluster.local
my-group-2.my-group-gvr.demo.svc.cluster.local
2019/06/17 13:03:10 [on-start.sh] [INFO] Reading standard input...
2019/06/17 13:03:11 [on-start.sh] [INFO] I am my-group-2.my-group-gvr.demo
2019/06/17 13:03:11 [on-start.sh] [INFO] Trying to start group with peers'my-group-0.my-group-gvr.demo my-group-1.my-group-gvr.demo my-group-2.my-group-gvr.demo'
2019/06/17 13:03:11 [on-start.sh] [INFO] Storing default mysqld config into /etc/mysql/my.cnf
2019/06/17 13:03:11 [on-start.sh] [INFO] Starting mysql server with 'docker-entrypoint.sh mysqld '...
MySQL Community Server 5.7.25 is already stopped.
2019/06/17 13:03:11 [on-start.sh] [INFO] The process id of mysqld is '78'
2019-06-17T13:03:11.843140Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-06-17T13:03:11.845196Z 0 [Note] mysqld (mysqld 5.7.25-log) starting as process 78 ...
2019-06-17T13:03:11.851040Z 0 [Note] InnoDB: PUNCH HOLE support available
2019-06-17T13:03:11.851066Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-06-17T13:03:11.851073Z 0 [Note] InnoDB: Uses event mutexes
2019-06-17T13:03:11.851245Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-06-17T13:03:11.851363Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-06-17T13:03:11.851467Z 0 [Note] InnoDB: Using Linux native AIO
2019-06-17T13:03:11.852060Z 0 [Note] InnoDB: Number of pools: 1
2019-06-17T13:03:11.852205Z 0 [Note] InnoDB: Using CPU crc32 instructions
2019-06-17T13:03:11.855158Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-06-17T13:03:11.868284Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-06-17T13:03:11.872270Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-06-17T13:03:11.900944Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2019-06-17T13:03:11.902822Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7195158
2019-06-17T13:03:11.902844Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 7195167
2019-06-17T13:03:11.902851Z 0 [Note] InnoDB: Database was not shutdown normally!
2019-06-17T13:03:11.902856Z 0 [Note] InnoDB: Starting crash recovery.
2019-06-17T13:03:12.076021Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-06-17T13:03:12.076068Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-06-17T13:03:12.076326Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-06-17T13:03:12.860042Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-06-17T13:03:12.861600Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2019-06-17T13:03:12.861617Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2019-06-17T13:03:12.862853Z 0 [Note] InnoDB: 5.7.25 started; log sequence number 7195167
2019-06-17T13:03:12.863126Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-06-17T13:03:12.863802Z 0 [Note] Plugin 'FEDERATED' is disabled.
2019-06-17T13:03:12.889525Z 0 [Warning] unknown variable 'loose-group_replication_bootstrap_group=OFF'
2019-06-17T13:03:12.889555Z 0 [Warning] unknown variable 'loose-group_replication_start_on_boot=OFF'
2019-06-17T13:03:12.889559Z 0 [Warning] unknown variable 'loose-group_replication_ssl_mode=REQUIRED'
2019-06-17T13:03:12.889563Z 0 [Warning] unknown variable 'loose-group_replication_recovery_use_ssl=1'
2019-06-17T13:03:12.889566Z 0 [Warning] unknown variable 'loose-group_replication_group_name=dc002fc3-c412-4d18-b1d4-66c1fbfbbc9b'
2019-06-17T13:03:12.889569Z 0 [Warning] unknown variable 'loose-group_replication_ip_whitelist=10.42.3.164/16'
2019-06-17T13:03:12.889573Z 0 [Warning] unknown variable 'loose-group_replication_group_seeds=my-group-0.my-group-gvr.demo:33060,my-group-1.my-group-gvr.demo:33060,my-group-2.my-group-gvr.demo:33060'
2019-06-17T13:03:12.889576Z 0 [Warning] unknown variable 'loose-group_replication_local_address=my-group-2.my-group-gvr.demo:33060'
2019-06-17T13:03:12.889673Z 0 [Note] Recovering after a crash using binlog
2019-06-17T13:03:12.889689Z 0 [Note] Starting crash recovery...
2019-06-17T13:03:12.889743Z 0 [Note] Crash recovery finished.
2019-06-17T13:03:12.893610Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190617 13:03:12
2019-06-17T13:03:13.187829Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2019-06-17T13:03:13.188133Z 0 [Warning] CA certificate ca.pem is self signed.
2019-06-17T13:03:13.191988Z 0 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
2019-06-17T13:03:13.192014Z 0 [Note] - '0.0.0.0' resolves to '0.0.0.0';
2019-06-17T13:03:13.192057Z 0 [Note] Server socket created on IP: '0.0.0.0'.
2019-06-17T13:03:13.214050Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2019-06-17T13:03:13.215794Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.215833Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.216043Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.216083Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.216092Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.216116Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.218132Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.218152Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-17T13:03:13.225107Z 0 [Note] Failed to start slave threads for channel ''
2019-06-17T13:03:13.239948Z 0 [Note] Event Scheduler: Loaded 0 events
2019-06-17T13:03:13.240230Z 0 [Note] mysqld: ready for connections.
Version: '5.7.25-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2019-06-17T13:03:42.452512Z 31 [Note] Access denied for user 'root'@'localhost' (using password: YES)
2019-06-17T13:03:43.087087Z 33 [Note] Access denied for user 'root'@'localhost' (using password: YES)
2019-06-17T13:03:47.466180Z 38 [Note] Access denied for user 'root'@'localhost' (using password: YES)
2019-06-17T13:03:48.074038Z 40 [Note] Access denied for user 'root'@'localhost' (using password: YES)
2019-06-17T13:03:52.456043Z 45 [Note] Access denied for user 'root'@'localhost' (using password: YES)
Describe your pod and see what events are written in it.
shudipta
on 18 Jun 2019
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 12m (x952 over 17h) kubelet, s2 Liveness probe failed:
Warning BackOff 2m36s (x3752 over 17h) kubelet, s2 Back-off restarting failed container
I've deleted pod and this is the new pod's events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 61s default-scheduler Successfully assigned demo/my-group-2 to s2
Normal Pulled 56s kubelet, s2 Container image "kubedb/busybox" already present on machine
Normal Created 55s kubelet, s2 Created container
Normal Started 54s kubelet, s2 Started container
Warning Unhealthy 12s (x2 over 18s) kubelet, s2 Readiness probe failed:
Warning Unhealthy 11s (x3 over 20s) kubelet, s2 Liveness probe failed:
Normal Pulled 8s (x2 over 53s) kubelet, s2 Container image "kubedb/mysql:5.7-v2" already present on machine
Normal Killing 8s kubelet, s2 Killing container with id docker://mysql:Container failed liveness probe.. Container will be killed and recreated.
Normal Created 7s (x2 over 52s) kubelet, s2 Created container
Normal Started 6s (x2 over 51s) kubelet, s2 Started container
Is the logs from container same as before?
And what is the output of the following command:
$ kubectl exec -it -n demo my-group-1 -- mysql -u root --password=<root_password> --host=localhost -e "select * from performance_schema.replication_group_members"
+---------------------------+--------------------------------------+------------------------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+------------------------------+-------------+--------------+
| group_replication_applier | 7b15184b-91b8-11e9-9ce0-faabc3810798 | my-group-0.my-group-gvr.demo | 3306 | ONLINE |
| group_replication_applier | 8f3d78a2-91b8-11e9-b160-3a1273166159 | my-group-1.my-group-gvr.demo | 3306 | ONLINE |
+---------------------------+--------------------------------------+------------------------------+-------------+--------------+
It means that in the 3rd Pod, the mysqld process couldn't start successfully. That's why group replication hasn't started.
If the mysqld process started successfully, then there must be a log saying "[INFO] All servers (my-group-0.my-group-gvr.demo my-group-1.my-group-gvr.demo my-group-2.my-group-gvr.demo) are ready".
We wait for some time to start the server. If timeout reaches, then an error log will be printed to stdout saying "[ERROR] Server my-group-2.my-group-gvr.demo start failed..."
shudipta
on 19 Jun 2019
The question is why the mysqld process can't start on that specific node? Is there any way I can figure it out?
ahmadalli
on 19 Jun 2019
How did you provision your cluster?
shudipta
on 20 Jun 2019
I'm using RKE on bare metal
ahmadalli
on 20 Jun 2019
What is the amount of memory you used there?
shudipta
on 20 Jun 2019
I have about 10 GB free memory on the server
ahmadalli
on 20 Jun 2019
I was able to generate your scenario by creating a cluster with insufficient memory where 3rd pod never succeeded to pass readiness probe.
shudipta
on 21 Jun 2019
I have 10 GB at the moment of pod creation. what's the minimum amount of memory required?
also here's the log of faulty pod (for the first initialization time. the other log was for after container restart).
2019/06/21 05:49:24 Determined Domain to be demo.svc.cluster.local
2019/06/21 05:49:24 Peer list updated
was []
now [my-group-0.my-group-gvr.demo.svc.cluster.local my-group-1.my-group-gvr.demo.svc.cluster.local my-group-2.my-group-gvr.demo.svc.cluster.local]
2019/06/21 05:49:24 execing: /on-start.sh with stdin: my-group-0.my-group-gvr.demo.svc.cluster.local
my-group-1.my-group-gvr.demo.svc.cluster.local
my-group-2.my-group-gvr.demo.svc.cluster.local
2019/06/21 05:49:24 [on-start.sh] [INFO] Reading standard input...
2019/06/21 05:49:24 [on-start.sh] [INFO] I am my-group-2.my-group-gvr.demo
2019/06/21 05:49:24 [on-start.sh] [INFO] Trying to start group with peers'my-group-0.my-group-gvr.demo my-group-1.my-group-gvr.demo my-group-2.my-group-gvr.demo'
2019/06/21 05:49:24 [on-start.sh] [INFO] Storing default mysqld config into /etc/mysql/my.cnf
2019/06/21 05:49:24 [on-start.sh] [INFO] Starting mysql server with 'docker-entrypoint.sh mysqld '...
MySQL Community Server 5.7.25 is already stopped.
2019/06/21 05:49:25 [on-start.sh] [INFO] The process id of mysqld is '79'
.Initializing database
2019-06-21T05:49:26.180230Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-06-21T05:49:32.531706Z 0 [Warning] InnoDB: New log files created, LSN=45790
2019-06-21T05:49:34.870323Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2019-06-21T05:49:35.909824Z 0 [Warning] unknown variable 'loose-group_replication_bootstrap_group=OFF'
2019-06-21T05:49:35.909859Z 0 [Warning] unknown variable 'loose-group_replication_start_on_boot=OFF'
2019-06-21T05:49:35.909866Z 0 [Warning] unknown variable 'loose-group_replication_ssl_mode=REQUIRED'
2019-06-21T05:49:35.909869Z 0 [Warning] unknown variable 'loose-group_replication_recovery_use_ssl=1'
2019-06-21T05:49:35.909873Z 0 [Warning] unknown variable 'loose-group_replication_group_name=dc002fc3-c412-4d18-b1d4-66c1fbfbbc9b'
2019-06-21T05:49:35.909876Z 0 [Warning] unknown variable 'loose-group_replication_ip_whitelist=10.42.3.117/16'
2019-06-21T05:49:35.909879Z 0 [Warning] unknown variable 'loose-group_replication_group_seeds=my-group-0.my-group-gvr.demo:33060,my-group-1.my-group-gvr.demo:33060,my-group-2.my-group-gvr.demo:33060'
2019-06-21T05:49:35.909882Z 0 [Warning] unknown variable 'loose-group_replication_local_address=my-group-2.my-group-gvr.demo:33060'
2019-06-21T05:49:36.084862Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 59dcc971-93e8-11e9-8763-3ef5e802d82f.
2019-06-21T05:49:36.294135Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2019-06-21T05:49:36.513969Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2019-06-21T05:49:50.370987Z 1 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:49:50.371035Z 1 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:49:50.371048Z 1 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:49:50.371072Z 1 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:49:50.371078Z 1 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:49:50.371128Z 1 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:49:50.371189Z 1 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:49:50.371206Z 1 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
.................................Database initialized
Initializing certificates
Generating a RSA private key
......+++++
....+++++
unable to write 'random state'
writing new private key to 'ca-key.pem'
-----
Generating a RSA private key
..+++++
......+++++
unable to write 'random state'
writing new private key to 'server-key.pem'
-----
Generating a RSA private key
.............+++++
.......................................................................................+++++
unable to write 'random state'
writing new private key to 'client-key.pem'
-----
..Certificates initialized
MySQL init process in progress...
2019-06-21T05:50:01.741855Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-06-21T05:50:01.744391Z 0 [Note] mysqld (mysqld 5.7.25-log) starting as process 294 ...
2019-06-21T05:50:01.749769Z 0 [Note] InnoDB: PUNCH HOLE support available
2019-06-21T05:50:01.749803Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-06-21T05:50:01.749809Z 0 [Note] InnoDB: Uses event mutexes
2019-06-21T05:50:01.749817Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-06-21T05:50:01.749825Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-06-21T05:50:01.749834Z 0 [Note] InnoDB: Using Linux native AIO
2019-06-21T05:50:01.750187Z 0 [Note] InnoDB: Number of pools: 1
2019-06-21T05:50:01.750365Z 0 [Note] InnoDB: Using CPU crc32 instructions
2019-06-21T05:50:01.753516Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-06-21T05:50:01.768689Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-06-21T05:50:01.772926Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-06-21T05:50:01.786218Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2019-06-21T05:50:01.825927Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-06-21T05:50:01.826018Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-06-21T05:50:02.183322Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-06-21T05:50:02.185374Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2019-06-21T05:50:02.185401Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2019-06-21T05:50:02.186421Z 0 [Note] InnoDB: 5.7.25 started; log sequence number 2525160
2019-06-21T05:50:02.186689Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-06-21T05:50:02.187316Z 0 [Note] Plugin 'FEDERATED' is disabled.
2019-06-21T05:50:02.190050Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190621 5:50:02
2019-06-21T05:50:02.193085Z 0 [Warning] unknown variable 'loose-group_replication_bootstrap_group=OFF'
2019-06-21T05:50:02.193109Z 0 [Warning] unknown variable 'loose-group_replication_start_on_boot=OFF'
2019-06-21T05:50:02.193116Z 0 [Warning] unknown variable 'loose-group_replication_ssl_mode=REQUIRED'
2019-06-21T05:50:02.193121Z 0 [Warning] unknown variable 'loose-group_replication_recovery_use_ssl=1'
2019-06-21T05:50:02.193126Z 0 [Warning] unknown variable 'loose-group_replication_group_name=dc002fc3-c412-4d18-b1d4-66c1fbfbbc9b'
2019-06-21T05:50:02.193131Z 0 [Warning] unknown variable 'loose-group_replication_ip_whitelist=10.42.3.117/16'
2019-06-21T05:50:02.193137Z 0 [Warning] unknown variable 'loose-group_replication_group_seeds=my-group-0.my-group-gvr.demo:33060,my-group-1.my-group-gvr.demo:33060,my-group-2.my-group-gvr.demo:33060'
2019-06-21T05:50:02.193142Z 0 [Warning] unknown variable 'loose-group_replication_local_address=my-group-2.my-group-gvr.demo:33060'
.MySQL init process in progress...
2019-06-21T05:50:02.548487Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2019-06-21T05:50:02.548957Z 0 [Warning] CA certificate ca.pem is self signed.
2019-06-21T05:50:02.651635Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2019-06-21T05:50:02.653527Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.653572Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.653586Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.653618Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.653641Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.653656Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.656208Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.656227Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-06-21T05:50:02.658573Z 0 [Note] Failed to start slave threads for channel ''
2019-06-21T05:50:02.670526Z 0 [Note] Event Scheduler: Loaded 0 events
2019-06-21T05:50:02.670815Z 0 [Note] mysqld: ready for connections.
Version: '5.7.25-log' socket: '/var/run/mysqld/mysqld.sock' port: 0 MySQL Community Server (GPL)
2019-06-21T05:50:07.046662Z 4 [Note] Access denied for user 'root'@'localhost' (using password: YES)
2019-06-21T05:50:07.297682Z 5 [Note] Access denied for user 'root'@'localhost' (using password: YES)
....
You can try again the CRD by using the following modification on a fresh cluster:
apiVersion: kubedb.com/v1alpha1
kind: MySQL
metadata:
name: my-group
namespace: demo
spec:
version: "5.7-v2"
replicas: 3
topology:
mode: GroupReplication
group:
name: "dc002fc3-c412-4d18-b1d4-66c1fbfbbc9b"
baseServerID: 100
storageType: Ephemeral
# storage:
# accessModes:
# - ReadWriteOnce
# resources:
# requests:
# storage: 1Gi
terminationPolicy: WipeOut
since it works on other nodes it seems that it's a problem with docker or kubelet on that specific node and not kubedb's issue. thank you for your help :)
ahmadalli
on 24 Jun 2019
I has resolved this problem by increase probe initialDelaySeconds.
Because I found that the failure pod have not finished replication initialization in default initialDelaySeconds(30s).
And after probe failed and restart the pod, the initialization already broken and failed again.
@ahmadalli @shudipta
xing0821
on 16 Oct 2019
apiVersion: kubedb.com/v1alpha1
kind: MySQL
metadata:
name: my-group
namespace: mysql-e2e
spec:
version: "5.7.25"
replicas: 3
topology:
mode: GroupReplication
storage:
storageClassName: local-storage
resources:
requests:
storage: 10Gi
terminationPolicy: WipeOut
podTemplate:
spec:
livenessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 15
exec:
command:
- "bash"
- "-c"
- 'mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h localhost -nsLNE -e "select member_state from performance_schema.replication_group_members where member_id=@@server_uuid;" 2>/dev/null | grep -v "*" | egrep -v "ERROR|OFFLINE"'
readinessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 15
exec:
command:
- "bash"
- "-c"
- 'mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h localhost -nsLNE -e "select member_state from performance_schema.replication_group_members where member_id=@@server_uuid;" 2>/dev/null | grep -v "*" | egrep -v "ERROR|OFFLINE"'
Great :+1
On Wed, Oct 16, 2019 at 9:58 AM xing0821 notifications@github.com wrote:
apiVersion: kubedb.com/v1alpha1
kind: MySQL
metadata:
name: my-group
namespace: mysql-e2e
spec:
version: "5.7.25"
replicas: 3
topology:
mode: GroupReplication
storage:
storageClassName: local-storage
resources:
requests:
storage: 10Gi
terminationPolicy: WipeOut
podTemplate:
spec:
livenessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 15
exec:
command:
- "bash"
- "-c"
- 'mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h localhost -nsLNE -e "select member_state from performance_schema.replication_group_members where member_id=@@server_uuid;" 2>/dev/null | grep -v "" | egrep -v "ERROR|OFFLINE"'
readinessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 15
exec:
command:
- "bash"
- "-c"
- 'mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h localhost -nsLNE -e "select member_state from performance_schema.replication_group_members where member_id=@@server_uuid;" 2>/dev/null | grep -v "" | egrep -v "ERROR|OFFLINE"'—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubedb/project/issues/552?email_source=notifications&email_token=AFR7JAGZMXJWZBBLVZ2DDKDQO2GPLA5CNFSM4HYVQPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBK6W4A#issuecomment-542501744,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFR7JAA2LLR5RPTJBQAOVOLQO2GPLANCNFSM4HYVQPRQ
.
shudipta
on 16 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
stephan2012
·
9Comments
iamliamc
·
9Comments
sherifkayad
·
3Comments
david-ecc
·
5Comments
yazfaz
·
4Comments
Most helpful comment
Issue-Label Bot is automatically applying the label
bugto this issue, with a confidence of 0.71. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!Links: app homepage, dashboard and code for this bot.