Charts: [mariadb-galera] Pods restart repeatedly due to readinessProbe failure--unable to login with 'root'@'localhost' user

Created on 6 Mar 2020 · 41Comments · Source: bitnami/charts

Which chart:

bitnami/mariadb-galera 0.8.2

Description

Pod restarts repeatedly due to readinessProbe failure. It appears that the rootUser might be getting ignored as a hostname is provided and the process is run with --skip-name-resolve argument.

Steps to reproduce the issue:

helm install galera --version 0.8.2 bitnami/mariadb-galera
Observe pod logs

Describe the results you received:

 00:03:20.56 INFO  ==> ** Starting MariaDB **
2020-03-06  0:03:20 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.4.12-MariaDB-log) starting as process 1 ...
/opt/bitnami/mariadb/sbin/mysqld: Can't create file '/opt/bitnami/mariadb/logs/mysqld.log' (errno: 13 "Permission denied")
2020-03-06  0:03:20 0 [Note] WSREP: Loading provider /opt/bitnami/mariadb/lib/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
wsrep loader: [INFO] wsrep_load(): loading provider library '/opt/bitnami/mariadb/lib/libgalera_smm.so'
wsrep loader: [INFO] wsrep_load(): Galera 4.3(r0) by Codership Oy <[email protected]> loaded successfully.
...
2020-03-05 23:38:39 0 [Note] Server socket created on IP: '0.0.0.0'.
2020-03-05 23:38:39 0 [Warning] 'user' entry 'root@galera-mariadb-galera-0' ignored in --skip-name-resolve mode.
2020-03-05 23:38:39 0 [Warning] 'user' entry '@galera-mariadb-galera-0' ignored in --skip-name-resolve mode.
2020-03-05 23:38:39 0 [Warning] 'proxies_priv' entry '@% root@galera-mariadb-galera-0' ignored in --skip-name-resolve mode.
...
2020-03-05 23:39:08 16 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-03-05 23:39:18 17 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-03-05 23:39:28 18 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-03-05 23:39:38 19 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-03-05 23:39:48 20 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-03-05 23:39:58 21 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
...
2020-03-05 23:40:59 0 [Note] InnoDB: Starting shutdown...
2020-03-05 23:40:59 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2020-03-05 23:40:59 0 [Note] InnoDB: Buffer pool(s) dump completed at 200305 23:40:59
2020-03-05 23:41:01 0 [Note] InnoDB: Shutdown completed; log sequence number 80969; transaction id 61
2020-03-05 23:41:01 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-03-05 23:41:01 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete

Describe the results you expected:

A working cluster. I'm having a hard time tracing out how these users were created. I extended the image and uncommented out # set -o xtrace in /entrypoint.sh, /run.sh and /setup.sh. I'm not seeing where the problem is. I'm assuming this is the part of the script in libmysqlgalera.sh actually being run:

mysql_ensure_root_user_exists() {
    local -r user="${1:?user is required}"
    local -r password="${2:-}"
    local -r auth_plugin="${3:-}"
    local auth_plugin_str=""

    if [[ -n "$auth_plugin" ]]; then
        auth_plugin_str="with $auth_plugin"
    fi

    debug "Configuring root user credentials"
    if [ "$DB_FLAVOR" == "mariadb" ]; then
        mysql_execute "mysql" "root" <<EOF
-- create root@localhost user for local admin access
-- create user 'root'@'localhost' $([ "$password" != "" ] && echo "identified by '$password'");
-- grant all on *.* to 'root'@'localhost' with grant option;
-- create admin user for remote access
create user '$user'@'%' $([ "$password" != "" ] && echo "identified $auth_plugin_str by '$password'");
grant all on *.* to '$user'@'%' with grant option;
flush privileges;
EOF
    else
        mysql_execute "mysql" "root" <<EOF
-- create admin user
create user '$user'@'%' $([ "$password" != "" ] && echo "identified by '$password'");
grant all on *.* to '$user'@'%' with grant option;
flush privileges;
EOF
    fi
}

I'm really confused since it looks like the root user is being created with '$user'@'%'. Which looks right to me. Any ideas of things I could try?

Additional information you deem important (e.g. issue happens only occasionally):

Issue seems persistent.

Version of Helm and Kubernetes:

Output of helm version:

version.BuildInfo{Version:"v3.1.1", GitCommit:"afe70585407b420d0097d07b21c47dc511525ac8", GitTreeState:"clean", GoVersion:"go1.13.8"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.7", GitCommit:"be3d344ed06bff7a4fc60656200a93c74f31f9a4", GitTreeState:"clean", BuildDate:"2020-02-11T19:24:46Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

Output of helm show chart rook-release/rook-ceph (PV/CSI provider):

apiVersion: v1
description: File, Block, and Object Storage Services for your Cloud-Native Environment
icon: https://rook.io/images/rook-logo.svg
name: rook-ceph
sources:
- https://github.com/rook/rook
version: v1.2.5

Output of helm show chart cilium/cilium (CNI):

apiVersion: v1
appVersion: 1.7.0
dependencies:
- condition: agent.enabled
  name: agent
  repository: ""
  version: 1.7.0
- condition: config.enabled
  name: config
  repository: ""
  version: 1.7.0
- condition: operator.enabled
  name: operator
  repository: ""
  version: 1.7.0
- condition: global.etcd.managed
  name: managed-etcd
  repository: ""
  version: 1.7.0
- condition: preflight.enabled
  name: preflight
  repository: ""
  version: 1.7.0
- condition: global.nodeinit.enabled
  name: nodeinit
  repository: ""
  version: 1.7.0
description: Helm chart for Cilium
name: cilium
sources:
- https://github.com/cilium/cilium
version: 1.7.0

Thank you!

on-hold

Source

chris-mccoy

All 41 comments

I should add I've provided a rootUser.password explicitly, and it still does this. I don't suspect the password is incorrect, it's just ignoring the whole user. Thank you.

chris-mccoy on 6 Mar 2020

I'm seeing the same problem here (rancher 2.3.5, rook cephfs 14.2.6, galera chart 0.8.2).
I had one try where it worked (but I removed that due to other changes), but I mainly have broken runs before and after my working attempt. I have no clue, why it worked one time.

Weirdly enough, the problem also happens with older versions. I tried 0.8, 0.7., 0.6, 0.5.

(And i'm using a values.yaml)

pcornelissen on 8 Mar 2020

Hi,

I am unable to reproduce the issue. I just executed the helm install command as you mentioned and did not find any authentication error. Could it be because of the password string?

galera-mariadb-galera-0   1/1     Running   0          13m
galera-mariadb-galera-1   1/1     Running   0          12m
galera-mariadb-galera-2   1/1     Running   0          12m

❯ stern galera
+ galera-mariadb-galera-0 › mariadb-galera
+ galera-mariadb-galera-2 › mariadb-galera
+ galera-mariadb-galera-1 › mariadb-galera
galera-mariadb-galera-0 mariadb-galera  08:37:01.23
galera-mariadb-galera-0 mariadb-galera  08:37:01.23 Welcome to the Bitnami mariadb-galera container
galera-mariadb-galera-0 mariadb-galera  08:37:01.24 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
galera-mariadb-galera-0 mariadb-galera  08:37:01.24 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
galera-mariadb-galera-0 mariadb-galera  08:37:01.24 Send us your feedback at [email protected]
galera-mariadb-galera-0 mariadb-galera  08:37:01.24
galera-mariadb-galera-0 mariadb-galera  08:37:01.24 INFO  ==> ** Starting MariaDB setup **
galera-mariadb-galera-0 mariadb-galera  08:37:01.30 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
galera-mariadb-galera-0 mariadb-galera  08:37:01.36 INFO  ==> Initializing mariadb database...
galera-mariadb-galera-0 mariadb-galera  08:37:04.19 INFO  ==> Starting mariadb in background
galera-mariadb-galera-0 mariadb-galera /libmysqlgalera.sh: line 545: DB_REPLICATION_MODE: unbound variable
galera-mariadb-galera-0 mariadb-galera  08:37:06.36 INFO  ==> Stopping mariadb
galera-mariadb-galera-0 mariadb-galera
galera-mariadb-galera-0 mariadb-galera  08:37:08.37 INFO  ==> ** MariaDB setup finished! **
galera-mariadb-galera-0 mariadb-galera  08:37:08.43 INFO  ==> ** Starting MariaDB **
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.4.12-MariaDB-log) starting as process 1 ...
galera-mariadb-galera-0 mariadb-galera /opt/bitnami/mariadb/sbin/mysqld: Can't create file '/opt/bitnami/mariadb/logs/mysqld.log' (errno: 13 "Permission denied")
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Loading provider /opt/bitnami/mariadb/lib/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
galera-mariadb-galera-0 mariadb-galera wsrep loader: [INFO] wsrep_load(): loading provider library '/opt/bitnami/mariadb/lib/libgalera_smm.so'
galera-mariadb-galera-0 mariadb-galera wsrep loader: [INFO] wsrep_load(): Galera 4.3(r0) by Codership Oy <[email protected]> loaded successfully.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: CRC-32C: using hardware acceleration.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Found saved state: 51bb9055-62aa-11ea-a237-33e10ea0b55a:12, safe_to_bootstrap: 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: GCache DEBUG: opened preamble:
galera-mariadb-galera-0 mariadb-galera Version: 2
galera-mariadb-galera-0 mariadb-galera UUID: 00000000-0000-0000-0000-000000000000
galera-mariadb-galera-0 mariadb-galera Seqno: 1 - 12
galera-mariadb-galera-0 mariadb-galera Offset: 1280
galera-mariadb-galera-0 mariadb-galera Synced: 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Passing config to GCS: base_dir = /bitnami/mariadb/data/; base_host = 10.32.0.9; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /bitnami/mariadb/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: ####### Assign initial position for certification: 51bb9055-62aa-11ea-a237-33e10ea0b55a:12, protocol version: -1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0 -> 51bb9055-62aa-11ea-a237-33e10ea0b55a:12
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Start replication
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Connecting with bootstrap option: 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Setting GCS initial position to 51bb9055-62aa-11ea-a237-33e10ea0b55a:12
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: protonet asio version 0
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Using CRC-32C for message checksums.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: backend: asio
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Warning] WSREP: access file(/bitnami/mariadb/data//gvwstate.dat) failed(No such file or directory)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: restore pc from disk failed
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: GMCast version 0
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: (544132af, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: (544132af, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: EVS version 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: gcomm: bootstrapping new group 'galera'
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: start_prim is enabled, turn off pc_recovery
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: EVS version upgrade 0 -> 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: PC protocol upgrade 0 -> 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Node 544132af state prim
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: view(view_id(PRIM,544132af,1) memb {
galera-mariadb-galera-0 mariadb-galera  544132af,0
galera-mariadb-galera-0 mariadb-galera } joined {
galera-mariadb-galera-0 mariadb-galera } left {
galera-mariadb-galera-0 mariadb-galera } partitioned {
galera-mariadb-galera-0 mariadb-galera })
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: save pc into disk
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: gcomm: connected
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Opened channel 'galera'
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5441d9de-62aa-11ea-bcff-07564b4a4e3c
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 5441d9de-62aa-11ea-bcff-07564b4a4e3c
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 5441d9de-62aa-11ea-bcff-07564b4a4e3c from 0 (galera-mariadb-galera-0)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Quorum results:
galera-mariadb-galera-0 mariadb-galera  version    = 6,
galera-mariadb-galera-0 mariadb-galera  component  = PRIMARY,
galera-mariadb-galera-0 mariadb-galera  conf_id    = 0,
galera-mariadb-galera-0 mariadb-galera  members    = 1/1 (joined/total),
galera-mariadb-galera-0 mariadb-galera  act_id     = 12,
galera-mariadb-galera-0 mariadb-galera  last_appl. = 12,
galera-mariadb-galera-0 mariadb-galera  protocols  = 2/10/4 (gcs/repl/appl),
galera-mariadb-galera-0 mariadb-galera  vote policy= 0,
galera-mariadb-galera-0 mariadb-galera  group UUID = 51bb9055-62aa-11ea-a237-33e10ea0b55a
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Flow-control interval: [16, 16]
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Restored state OPEN -> JOINED (13)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Member 0.0 (galera-mariadb-galera-0) synced with group.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 13)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Starting applier thread 1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: ####### processing CC 13, local, ordered
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: REPL Protocols: 10 (5, 3)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: ####### My UUID: 544132af-62aa-11ea-819e-f6a56520b43c
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Server galera-mariadb-galera-0 connected to cluster at position 51bb9055-62aa-11ea-a237-33e10ea0b55a:13 with ID 544132af-62aa-11ea-819e-f6a56520b43c
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Server status change disconnected -> connected
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: ####### ST not required
galera-mariadb-galera-1galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 2 [Note] WSREP: Starting rollbacker thread 2
galera-mariadb-galera-0 mariadb-galera  mariadb-galera  08:37:37.20
2020-03-10  8:37:08 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: ####### Adjusting cert position: -1 -> 13
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: ####### Setting monitor position to 13
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 13
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Min available from gcache for CC from group: 13
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: ================================================
galera-mariadb-galera-0 mariadb-galera View:
galera-mariadb-galera-0 mariadb-galera   id: 51bb9055-62aa-11ea-a237-33e10ea0b55a:13
galera-mariadb-galera-0 mariadb-galera   status: primary
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera  08:37:37.21 Welcome to the Bitnami mariadb-galera container
galera-mariadb-galera-1 galera-mariadb-galera-0 mariadb-galera   protocol_version: 4
galera-mariadb-galera-0 mariadb-galera  08:37:37.21 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
mariadb-galera   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera   final: no
galera-mariadb-galera-0 mariadb-galera   own_index: 0
galera-mariadb-galera-0 mariadb-galeramariadb-galera   members(1):
galera-mariadb-galera-0 mariadb-galera  08:37:37.21 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
galera-mariadb-galera-1 mariadb-galeramariadb-galera  08:37:37.21 Send us your feedback at [email protected]
  08:38:53.11
galera-mariadb-galera-1 mariadb-galera galera-mariadb-galera-2  08:37:37.21
mariadb-galera  08:38:53.11 Welcome to the Bitnami mariadb-galera container
galera-mariadb-galera-2 galera-mariadb-galera-1mariadb-galera   08:38:53.11 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
galera-mariadb-galera-2  mariadb-galera     0: 544132af-62aa-11ea-819e-f6a56520b43c, galera-mariadb-galera-0
 08:37:37.21 INFO  ==> ** Starting MariaDB setup **
galera-mariadb-galera-1 galera-mariadb-galera-0 mariadb-galera mariadb-galera  08:37:37.27 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
mariadb-galera =================================================
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera 08:38:53.11 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
galera-mariadb-galera-2  2020-03-10  8:37:08 1 [Note] WSREP: Server status change connected -> joiner
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera  08:37:37.33 INFO  ==> Initializing mariadb database...
galera-mariadb-galera-1 mariadb-galera  08:38:53.11 Send us your feedback at [email protected]
mariadb-galera  08:37:37.39 INFO  ==> ** MariaDB setup finished! **
galera-mariadb-galera-1 mariadb-galera
galera-mariadb-galera-1 mariadb-galera  08:37:37.45 INFO  ==> ** Starting MariaDB **
galera-mariadb-galera-1 galera-mariadb-galera-2mariadb-galera 2020-03-10  8:37:37 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.4.12-MariaDB-log) starting as process 1 ...
 galera-mariadb-galera-1 mariadb-galeramariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Server status change joiner -> initializing
mariadb-galera  08:38:53.12
galera-mariadb-galera-2  /opt/bitnami/mariadb/sbin/mysqld: Can't create file '/opt/bitnami/mariadb/logs/mysqld.log' (errno: 13 "Permission denied")
galera-mariadb-galera-1 mariadb-galera mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Loading provider /opt/bitnami/mariadb/lib/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
 08:38:53.12 INFO  ==> ** Starting MariaDB setup **
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galeramariadb-galera  08:39:33.20 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
galera-mariadb-galera-0 galera-mariadb-galera-2 mariadb-galera  wsrep loader: [INFO] wsrep_load(): loading provider library '/opt/bitnami/mariadb/lib/libgalera_smm.so'
galera-mariadb-galera-1 2020-03-10  8:37:08 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera mariadb-galera wsrep loader: [INFO] wsrep_load(): Galera 4.3(r0) by Codership Oy <[email protected]> loaded successfully.
galera-mariadb-galera-12020-03-10  8:37:08 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
galera-mariadb-galera-0 mariadb-galera mariadb-galera  mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Using Linux native AIO
galera-mariadb-galera-02020-03-10  8:37:37 0 [Note] WSREP: CRC-32C: using hardware acceleration.
  08:39:33.27 INFO  ==> Initializing mariadb database...
galera-mariadb-galera-2galera-mariadb-galera-1  mariadb-galera mariadb-galera 2020-03-10  8:37:37 0 [Warning] WSREP: Could not open state file for reading: '/bitnami/mariadb/data//grastate.dat'
 08:39:36.29 INFO  ==> Starting mariadb in background
galera-mariadb-galera-1 mariadb-galera mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2020-03-10  8:37:08 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera /libmysqlgalera.sh: line 545: DB_REPLICATION_MODE: unbound variable
galera-mariadb-galera-2 mariadb-galera  08:39:38.08 INFO  ==> Stopping mariadb
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: GCache DEBUG: opened preamble:
galera-mariadb-galera-1 mariadb-galera Version: 0
galera-mariadb-galera-1 mariadb-galera UUID: 00000000-0000-0000-0000-000000000000
galera-mariadb-galera-1 mariadb-galera Seqno: -1 - -1
galera-mariadb-galera-1 mariadb-galera Offset: -1
galera-mariadb-galera-1 mariadb-galera Synced: 0
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Passing config to GCS: base_dir = /bitnami/mariadb/data/; base_host = 10.32.0.10; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /bitnami/mariadb/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Start replication
galera-mariadb-galera-1 mariadb-galera  08:39:40.10 INFO  ==> ** MariaDB setup finished! **
galera-mariadb-galera-2 mariadb-galera
mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Connecting with bootstrap option: 0
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Uses event mutexes
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
galera-mariadb-galera-2 mariadb-galera  08:40:20.26 INFO  ==> ** Starting MariaDB **
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.4.12-MariaDB-log) starting as process 1 ...
galera-mariadb-galera-2galera-mariadb-galera-0 galera-mariadb-galera-1mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Number of pools: 1
 mariadb-galera /opt/bitnami/mariadb/sbin/mysqld: Can't create file '/opt/bitnami/mariadb/logs/mysqld.log' (errno: 13 "Permission denied")
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Loading provider /opt/bitnami/mariadb/lib/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
galera-mariadb-galera-0galera-mariadb-galera-2  mariadb-galera wsrep loader: [INFO] wsrep_load(): loading provider library '/opt/bitnami/mariadb/lib/libgalera_smm.so'
mariadb-galera galera-mariadb-galera-2 mariadb-galera wsrep loader: [INFO] wsrep_load(): Galera 4.3(r0) by Codership Oy <[email protected]> loaded successfully.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: CRC-32C: using hardware acceleration.
2020-03-10  8:37:08 0 [Note] InnoDB: Using SSE2 crc32 instructions
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Found saved state: ac647275-62aa-11ea-a17f-c2ca3d40169e:12, safe_to_bootstrap: 1
galera-mariadb-galera-2 galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: GCache DEBUG: opened preamble:
mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Completed initialization of buffer pool
galera-mariadb-galera-2 galera-mariadb-galera-0 mariadb-galera galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
galera-mariadb-galera-0 mariadb-galera Version: 2
mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
galera-mariadb-galera-02020-03-10  8:37:37 0 [Note] WSREP: protonet asio version 0
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: Using CRC-32C for message checksums.
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: backend: asio
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Warning] WSREP: access file(/bitnami/mariadb/data//gvwstate.dat) failed(No such file or directory)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: restore pc from disk failed
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: GMCast version 0
 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables
galera-mariadb-galera-0 mariadb-galera mariadb-galera mariadb-galera UUID: 00000000-0000-0000-0000-000000000000
galera-mariadb-galera-2 2020-03-10  8:37:37 0 [Note] WSREP: (658f0c29, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2020-03-10  8:37:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Waiting for purge to start
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: 10.4.12 started; log sequence number 88116; transaction id 46
galera-mariadb-galera-1 mariadb-galera galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] Plugin 'FEEDBACK' is disabled.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] InnoDB: Buffer pool(s) load completed at 200310  8:37:08
2020-03-10  8:37:37 0 [Note] WSREP: (658f0c29, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera mariadb-galera 2020-03-10  8:37:08 0 [Note] Server socket created on IP: '0.0.0.0'.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: EVS version 1
Seqno: 1 - 12
galera-mariadb-galera-2 mariadb-galera Offset: 1280
galera-mariadb-galera-2 mariadb-galera Synced: 1
galera-mariadb-galera-2 mariadb-galeramariadb-galera 2020-03-10  8:37:08 0 [Warning] 'proxies_priv' entry '@% root@galera-mariadb-galera-0' ignored in --skip-name-resolve mode.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Server initialized
 galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] WSREP: Server status change initializing -> initialized
2020-03-10  8:40:20 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Passing config to GCS: base_dir = /bitnami/mariadb/data/; base_host = 10.32.0.12; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /bitnami/mariadb/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-0 mariadb-galera galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: ####### Assign initial position for certification: ac647275-62aa-11ea-a17f-c2ca3d40169e:12, protocol version: -1
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: gcomm: connecting to group 'galera', peer 'galera-mariadb-galera-headless.default.svc.cluster.local:'
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: (658f0c29, 'tcp://0.0.0.0:4567') connection established to 544132af tcp://10.32.0.9:4567
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera2020-03-10  8:37:08 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 2020-03-10  8:37:37 0 [Note] WSREP: (658f0c29, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0 -> ac647275-62aa-11ea-a17f-c2ca3d40169e:12
galera-mariadb-galera-2 mariadb-galera galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: EVS version upgrade 0 -> 1
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Start replication
2020-03-10  8:37:38 0 [Note] WSREP: declaring 544132af at tcp://10.32.0.9:4567 stable
galera-mariadb-galera-2 mariadb-galera mariadb-galera galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
galera-mariadb-galera-0  2020-03-10  8:40:20 0 [Note] WSREP: Connecting with bootstrap option: 1
mariadb-galera galera-mariadb-galera-2 2020-03-10  8:37:38 0 [Note] WSREP: PC protocol upgrade 0 -> 1
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Setting GCS initial position to ac647275-62aa-11ea-a17f-c2ca3d40169e:12
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: protonet asio version 0
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:08 10 [Note] WSREP: Recovered cluster id 51bb9055-62aa-11ea-a237-33e10ea0b55a
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Server status change initialized -> joined
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-03-10  8:40:20 0 [Note] WSREP: Using CRC-32C for message checksums.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: backend: asio
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Warning] WSREP: access file(/bitnami/mariadb/data//gvwstate.dat) failed(No such file or directory)
galera-mariadb-galera-0 mariadb-galeramariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 11 [Note] WSREP: Starting applier thread 11
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 12 [Note] WSREP: Starting applier thread 12
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 13 [Note] WSREP: Starting applier thread 13
galera-mariadb-galera-0  mariadb-galeragalera-mariadb-galera-2  2020-03-10  8:37:08 0 [Note] Reading of all Master_info entries succeeded
2020-03-10  8:37:38 0 [Note] WSREP: Node 544132af state prim
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: view(view_id(PRIM,544132af,2) memb {
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: restore pc from disk failed
galera-mariadb-galera-2 mariadb-galera mariadb-galera 2020-03-10  8:37:08 0 [Note] Added new Master_info '' to hash table
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
galera-mariadb-galera-0 mariadb-galera  544132af,0
mariadb-galera Version: '10.4.12-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
2020-03-10  8:40:20 0 [Note] WSREP: GMCast version 0
galera-mariadb-galera-0 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: (c6a9d8ff, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: (c6a9d8ff, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: EVS version 1
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: gcomm: bootstrapping new group 'galera'
galera-mariadb-galera-2 galera-mariadb-galera-1mariadb-galera mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Server galera-mariadb-galera-0 synced with group
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: Server status change joined -> synced
2020-03-10  8:40:20 0 [Note] WSREP: start_prim is enabled, turn off pc_recovery
galera-mariadb-galera-2  mariadb-galera     658f0c29,0
galera-mariadb-galera-1 mariadb-galera } joined {
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera mariadb-galera } left {
mariadb-galera galera-mariadb-galera-1 mariadb-galera } partitioned {
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: EVS version upgrade 0 -> 1
galera-mariadb-galera-2})
 galera-mariadb-galera-1 2020-03-10  8:37:08 1 [Note] WSREP: Synchronized with group, ready for connections
mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: save pc into disk
mariadb-galeragalera-mariadb-galera-1 mariadb-galeragalera-mariadb-galera-0  2020-03-10  8:37:38 0 [Note] WSREP: gcomm: connected
galera-mariadb-galera-1  2020-03-10  8:40:20 0 [Note] WSREP: PC protocol upgrade 0 -> 1
mariadb-galera 2020-03-10  8:37:08 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Node c6a9d8ff state prim
galera-mariadb-galera-2 galera-mariadb-galera-0mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: view(view_id(PRIM,c6a9d8ff,1) memb {
galera-mariadb-galera-2 mariadb-galera  c6a9d8ff,0
galera-mariadb-galera-2 mariadb-galera } joined {
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
mariadb-galera galera-mariadb-galera-1 mariadb-galeramariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
 2020-03-10  8:37:37 0 [Note] WSREP: (544132af, 'tcp://0.0.0.0:4567') connection established to 658f0c29 tcp://10.32.0.10:4567
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Opened channel 'galera'
galera-mariadb-galera-1  mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:37 0 [Note] WSREP: (544132af, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: Starting rollbacker thread 1
galera-mariadb-galera-1galera-mariadb-galera-0 } left {
galera-mariadb-galera-2 mariadb-galera  2020-03-10  8:37:38 0 [Note] WSREP: declaring 658f0c29 at tcp://10.32.0.10:4567 stable
galera-mariadb-galera-0 mariadb-galera mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: Starting applier thread 2
2020-03-10  8:37:38 0 [Note] WSREP: Node 544132af state prim
mariadb-galera } partitioned {
galera-mariadb-galera-2 galera-mariadb-galera-0 mariadb-galera galera-mariadb-galera-1 mariadb-galeramariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 65e2bbf4-62aa-11ea-aa36-eff9c2c6286d
2020-03-10  8:37:38 0 [Note] WSREP: view(view_id(PRIM,544132af,2) memb {
 })
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 65e2bbf4-62aa-11ea-aa36-eff9c2c6286d from 0 (galera-mariadb-galera-0)
mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: save pc into disk
galera-mariadb-galera-0 galera-mariadb-galera-2 mariadb-galera  544132af,0
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 65e2bbf4-62aa-11ea-aa36-eff9c2c6286d from 1 (galera-mariadb-galera-1)
mariadb-galera mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: gcomm: connected
galera-mariadb-galera-2     658f0c29,0
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Quorum results:
mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
galera-mariadb-galera-2 mariadb-galera mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
} joined {
galera-mariadb-galera-0 galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera  version    = 6,
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Opened channel 'galera'
galera-mariadb-galera-2 mariadb-galera  component  = PRIMARY,
galera-mariadb-galera-1 mariadb-galera  conf_id    = 1,
galera-mariadb-galera-1 mariadb-galera  members    = 1/2 (joined/total),
galera-mariadb-galera-1 mariadb-galera  act_id     = 13,
mariadb-galera galera-mariadb-galera-1 2020-03-10  8:40:20 1 [Note] WSREP: Starting applier thread 1
mariadb-galera  last_appl. = 12,
galera-mariadb-galera-1 mariadb-galera mariadb-galera   protocols  = 2/10/4 (gcs/repl/appl),
galera-mariadb-galera-1 } left {
galera-mariadb-galera-0 mariadb-galera mariadb-galera   vote policy= 0,
galera-mariadb-galera-1 mariadb-galera galera-mariadb-galera-2 } partitioned {
mariadb-galera 2020-03-10  8:40:20 2 [Note] WSREP: Starting rollbacker thread 2
    group UUID = 51bb9055-62aa-11ea-a237-33e10ea0b55a
galera-mariadb-galera-1 galera-mariadb-galera-0 mariadb-galera })
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Flow-control interval: [23, 23]
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 14)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: save pc into disk
mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: ####### processing CC 14, local, ordered
galera-mariadb-galera-0 galera-mariadb-galera-2mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
galera-mariadb-galera-1galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 65e2bbf4-62aa-11ea-aa36-eff9c2c6286d
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 65e2bbf4-62aa-11ea-aa36-eff9c2c6286d
 galera-mariadb-galera-0  mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: REPL Protocols: 10 (5, 3)
mariadb-galera mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c6ab8cc1-62aa-11ea-a1e4-ae4f1315cbd6
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c6ab8cc1-62aa-11ea-a1e4-ae4f1315cbd6
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 65e2bbf4-62aa-11ea-aa36-eff9c2c6286d from 0 (galera-mariadb-galera-0)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: ####### My UUID: 658f0c29-62aa-11ea-b5f7-aa3b04fa812a
2020-03-10  8:37:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 65e2bbf4-62aa-11ea-aa36-eff9c2c6286d from 1 (galera-mariadb-galera-1)
galera-mariadb-galera-1mariadb-galera  mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: Server galera-mariadb-galera-1 connected to cluster at position 51bb9055-62aa-11ea-a237-33e10ea0b55a:14 with ID 658f0c29-62aa-11ea-b5f7-aa3b04fa812a
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: Server status change disconnected -> connected
2020-03-10  8:40:20 0 [Note] WSREP: STATE EXCHANGE: got state msg: c6ab8cc1-62aa-11ea-a1e4-ae4f1315cbd6 from 0 (galera-mariadb-galera-2)
galera-mariadb-galera-0 galera-mariadb-galera-2 mariadb-galera galera-mariadb-galera-1 2020-03-10  8:37:38 0 [Note] WSREP: Quorum results:
mariadb-galeramariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Quorum results:
 2020-03-10  8:37:38 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera galera-mariadb-galera-2  version    = 6,
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: State transfer required:
mariadb-galera galera-mariadb-galera-1  version    = 6,
mariadb-galera  Group state: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
galera-mariadb-galera-2 mariadb-galera galera-mariadb-galera-0 mariadb-galera   component  = PRIMARY,
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galeramariadb-galera    Local state: 00000000-0000-0000-0000-000000000000:-1
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: Server status change connected -> joiner
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    component  = PRIMARY,
galera-mariadb-galera-2 mariadb-galera  conf_id    = 0,
galera-mariadb-galera-2 mariadb-galera  members    = 1/1 (joined/total),
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.32.0.10' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --parent '1' --binlog 'mysql-bin' --mysqld-args --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --port=3306 --wsrep_cluster_name=galera --wsrep_node_name=galera-mariadb-galera-1 --wsrep_node_address=10.32.0.10 --wsrep_cluster_address=gcomm://galera-mariadb-galera-headless.default.svc.cluster.local --wsrep_sst_method=mariabackup --wsrep_sst_auth=mariabackup:B6HY8c27bY'
mariadb-galera  act_id     = 12,
    conf_id    = 1,
galera-mariadb-galera-0 mariadb-galera  members    = 1/2 (joined/total),
galera-mariadb-galera-0galera-mariadb-galera-1 mariadb-galera  mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Joiner monitor thread started to monitor
    act_id     = 13,
galera-mariadb-galera-1 mariadb-galera galera-mariadb-galera-0 mariadb-galera   last_appl. = 12,
galera-mariadb-galera-2galera-mariadb-galera-0 mariadb-galeraWSREP_SST: [INFO] Streaming with xbstream (20200310 08:37:38.887)
galera-mariadb-galera-1  mariadb-galera  mariadb-galeraWSREP_SST: [INFO] Using socat as streamer (20200310 08:37:38.890)
galera-mariadb-galera-1     protocols  = 2/10/4 (gcs/repl/appl),
    last_appl. = 12,
galera-mariadb-galera-2 mariadb-galera WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20200310 08:37:38.937)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: Prepared SST request: mariabackup|10.32.0.10:4444/xtrabackup_sst//1
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 14, STRv: 3
galera-mariadb-galera-1 mariadb-galera  vote policy= 0,
mariadb-galera  protocols  = 2/10/4 (gcs/repl/appl),
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: IST receiver addr using tcp://10.32.0.10:4568
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: Prepared IST receiver for 0-14, listening at: tcp://10.32.0.10:4568
galera-mariadb-galera-1 mariadb-galera mariadb-galera   vote policy= 0,
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: Member 1.0 (galera-mariadb-galera-1) requested state transfer from '*any*'. Selected 0.0 (galera-mariadb-galera-0)(SYNCED) as donor.
galera-mariadb-galera-1     group UUID = ac647275-62aa-11ea-a17f-c2ca3d40169e
mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 14)
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Flow-control interval: [16, 16]
galera-mariadb-galera-2 mariadb-galera galera-mariadb-galera-02020-03-10  8:37:39 2 [Note] WSREP: Requesting state transfer: success, donor: 0
 galera-mariadb-galera-1 mariadb-galera mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 2 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0 -> 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
    group UUID = 51bb9055-62aa-11ea-a237-33e10ea0b55a
galera-mariadb-galera-0 2020-03-10  8:40:20 0 [Note] WSREP: Restored state OPEN -> JOINED (13)
mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Flow-control interval: [23, 23]
galera-mariadb-galera-2 galera-mariadb-galera-0 mariadb-galera mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: ####### processing CC 14, local, ordered
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera WSREP_SST: [INFO] Proceeding with SST (20200310 08:37:39.799)
mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: REPL Protocols: 10 (5, 3)
galera-mariadb-galera-1 mariadb-galera galera-mariadb-galera-0 2020-03-10  8:40:20 0 [Note] WSREP: Member 0.0 (galera-mariadb-galera-2) synced with group.
mariadb-galera galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 13)
galera-mariadb-galera-22020-03-10  8:37:38 1 [Note] WSREP: ####### My UUID: 544132af-62aa-11ea-819e-f6a56520b43c
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: ####### ST not required
 galera-mariadb-galera-0 mariadb-galera mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: ####### processing CC 13, local, ordered
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20200310 08:37:39.798)
galera-mariadb-galera-2 2020-03-10  8:37:38 1 [Note] WSREP: Skipping cert index reset
galera-mariadb-galera-0 mariadb-galera galera-mariadb-galera-1 mariadb-galera WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20200310 08:37:39.803)
2020-03-10  8:40:20 1 [Note] WSREP: REPL Protocols: 10 (5, 3)
galera-mariadb-galera-2 galera-mariadb-galera-1mariadb-galera  mariadb-galera mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: ####### Adjusting cert position: 13 -> 14
2020-03-10  8:40:20 1 [Note] WSREP: ####### My UUID: c6a9d8ff-62aa-11ea-9809-da4af383911d
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: ####### Setting monitor position to 14
galera-mariadb-galera-0 mariadb-galeraWSREP_SST: [INFO] Waiting for SST streaming to complete! (20200310 08:37:39.819)
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:40 0 [Note] WSREP: (658f0c29, 'tcp://0.0.0.0:4567') turning message relay requesting off
mariadb-galera galera-mariadb-galera-1 2020-03-10  8:40:20 1 [Note] WSREP: Server galera-mariadb-galera-2 connected to cluster at position ac647275-62aa-11ea-a17f-c2ca3d40169e:13 with ID c6a9d8ff-62aa-11ea-9809-da4af383911d
mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: 0.0 (galera-mariadb-galera-0): State transfer to 1.0 (galera-mariadb-galera-1) complete.
galera-mariadb-galera-2 galera-mariadb-galera-1  2020-03-10  8:37:38 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 14
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: Min available from gcache for CC from group: 13
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: ================================================
mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: Member 0.0 (galera-mariadb-galera-0) synced with group.
mariadb-galeragalera-mariadb-galera-1  galera-mariadb-galera-0 2020-03-10  8:40:20 1 [Note] WSREP: Server status change disconnected -> connected
mariadb-galera View:
galera-mariadb-galera-2 galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mariadb-galera WSREP_SST: [INFO] Preparing the backup at /bitnami/mariadb/data//.sst (20200310 08:37:52.706)
galera-mariadb-galera-1 mariadb-galeramariadb-galera   id: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
 WSREP_SST: [INFO] Evaluating /opt/bitnami/mariadb//bin/mariabackup --innobackupex   $INNOEXTRA --apply-log $rebuildcmd ${DATA} &> /bitnami/mariadb/data//mariabackup.prepare.log (20200310 08:37:52.708)
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera WSREP_SST: [INFO] Moving the backup to /bitnami/mariadb/data/ (20200310 08:37:53.632)
galera-mariadb-galera-1 galera-mariadb-galera-0 mariadb-galera mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: ####### ST not required
  status: primary
galera-mariadb-galera-0 mariadb-galera   protocol_version: 4
galera-mariadb-galera-2 galera-mariadb-galera-0 mariadb-galera WSREP_SST: [INFO] Evaluating /opt/bitnami/mariadb//bin/mariabackup --innobackupex --defaults-file=/opt/bitnami/mariadb/conf/my.cnf      --move-back --force-non-empty-directories ${DATA} &> /bitnami/mariadb/data//mariabackup.move.log (20200310 08:37:53.635)
mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-2 mariadb-galera WSREP_SST: [INFO] Move successful, removing /bitnami/mariadb/data//.sst (20200310 08:37:53.649)
mariadb-galera   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera WSREP_SST: [INFO] Galera co-ords from recovery: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14 (20200310 08:37:53.654)
mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: ####### Adjusting cert position: -1 -> 13
galera-mariadb-galera-2 galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera mariadb-galera 2020-03-10  8:40:20 0 [Note] WSREP: Service thread queue flushed.
  final: no
galera-mariadb-galera-0 mariadb-galera WSREP_SST: [INFO] Total time on joiner: 0 seconds (20200310 08:37:53.657)
galera-mariadb-galera-1 mariadb-galera galera-mariadb-galera-2mariadb-galera WSREP_SST: [INFO] Removing the sst_in_progress file (20200310 08:37:53.660)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 3 [Note] WSREP: SST received
 galera-mariadb-galera-1 mariadb-galera   own_index: 0
2020-03-10  8:37:53 3 [Note] WSREP: Server status change joiner -> initializing
galera-mariadb-galera-1 mariadb-galeragalera-mariadb-galera-0 mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: ####### Setting monitor position to 13
 mariadb-galera   members(2):
galera-mariadb-galera-2 mariadb-galera galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 13
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: Min available from gcache for CC from group: 13
galera-mariadb-galera-2     0: 544132af-62aa-11ea-819e-f6a56520b43c, galera-mariadb-galera-0
2020-03-10  8:37:53 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera  1: 658f0c29-62aa-11ea-b5f7-aa3b04fa812a, galera-mariadb-galera-1
mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: ================================================
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:53 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
mariadb-galera galera-mariadb-galera-1 View:
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Using Linux native AIO
galera-mariadb-galera-1 mariadb-galeragalera-mariadb-galera-2 mariadb-galera   id: ac647275-62aa-11ea-a17f-c2ca3d40169e:13
mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
galera-mariadb-galera-1 mariadb-galera galera-mariadb-galera-2 mariadb-galera =================================================
   status: primary
galera-mariadb-galera-22020-03-10  8:37:53 0 [Note] InnoDB: Uses event mutexes
galera-mariadb-galera-1 mariadb-galera  2020-03-10  8:37:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
mariadb-galera   protocol_version: 4
galera-mariadb-galera-2 galera-mariadb-galera-0 mariadb-galera   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:38 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mariadb-galera   final: no
galera-mariadb-galera-2 galera-mariadb-galera-0 galera-mariadb-galera-1mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: Member 1.0 (galera-mariadb-galera-1) requested state transfer from '*any*'. Selected 0.0 (galera-mariadb-galera-0)(SYNCED) as donor.
 galera-mariadb-galera-0 mariadb-galera mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 14)
galera-mariadb-galera-0 2020-03-10  8:37:53 0 [Note] InnoDB: Number of pools: 1
mariadb-galera 2020-03-10  8:37:39 1 [Note] WSREP: Detected STR version: 1, req_len: 125, req: STRv1
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:39 1 [Note] WSREP: Cert index preload: 14 -> 14
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:39 1 [Note] WSREP: Server status change synced -> donor
galera-mariadb-galera-1 galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Using SSE2 crc32 instructions
mariadb-galera   own_index: 0
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:39 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:53 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera mariadb-galera   members(1):
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:39 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: Donor monitor thread started to monitor
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: async IST sender starting to serve tcp://10.32.0.10:4568 sending 14-14
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:39 0 [Note] WSREP: IST sender 14 -> 14
2020-03-10  8:37:53 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
galera-mariadb-galera-1 galera-mariadb-galera-0 mariadb-galeramariadb-galera mariadb-galera2020-03-10  8:37:39 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '10.32.0.10:4444/xtrabackup_sst//1' --socket '/opt/bitnami/mariadb/tmp/mysql.sock' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --gtid '51bb9055-62aa-11ea-a237-33e10ea0b55a:14' --gtid-domain-id '0' --binlog 'mysql-bin' --mysqld-args --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --port=3306 --wsrep_cluster_name=galera --wsrep_node_name=galera-mariadb-galera-0 --wsrep_node_address=10.32.0.9 --wsrep_cluster_address=gcomm:// --wsrep_sst_method=mariabackup --wsrep_sst_auth=mariabackup:B6HY8c27bY'
galera-mariadb-galera-0     0: c6a9d8ff-62aa-11ea-9809-da4af383911d, galera-mariadb-galera-2
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:39 1 [Note] WSREP: sst_donor_thread signaled with 0
galera-mariadb-galera-0 mariadb-galera mariadb-galera  2020-03-10  8:37:53 0 [Note] InnoDB: Completed initialization of buffer pool
=================================================
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: Server status change connected -> joiner
WSREP_SST: [INFO] Streaming with xbstream (20200310 08:37:39.752)
galera-mariadb-galera-2 mariadb-galera mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-03-10  8:37:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
galera-mariadb-galera-0 mariadb-galera WSREP_SST: [INFO] Using socat as streamer (20200310 08:37:39.755)
galera-mariadb-galera-2galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 134217728 bytes
 mariadb-galera 2020-03-10  8:40:20 1 [Note] WSREP: Server status change joiner -> initializing
galera-mariadb-galera-0 galera-mariadb-galera-1mariadb-galeragalera-mariadb-galera-2  mariadb-galera WSREP_SST: [INFO] Using /tmp/tmp.SQDIOHGLFN as innobackupex temporary directory (20200310 08:37:39.773)
galera-mariadb-galera-0  mariadb-galera mariadb-galera WSREP_SST: [INFO] Streaming GTID file before SST (20200310 08:37:39.780)
2020-03-10  8:37:53 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 134217728 bytes
galera-mariadb-galera-0 mariadb-galera WSREP_SST: [INFO] Evaluating mbstream -c ${INFO_FILE} | socat -u stdio TCP:10.32.0.10:4444; RC=( ${PIPESTATUS[@]} ) (20200310 08:37:39.784)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2020-03-10  8:40:20 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: New log files created, LSN=101426
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera galera-mariadb-galera-0 mariadb-galera mariadb-galera 2020-03-10  8:40:20 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
galera-mariadb-galera-2WSREP_SST: [INFO] Sleeping before data transfer for SST (20200310 08:37:39.791)
 galera-mariadb-galera-02020-03-10  8:37:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables
galera-mariadb-galera-1  mariadb-galera mariadb-galera2020-03-10  8:37:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: 10.4.12 started; log sequence number 101900; transaction id 73
galera-mariadb-galera-1 2020-03-10  8:40:20 0 [Note] InnoDB: Using Linux native AIO
mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galeramariadb-galeramariadb-galera 2020-03-10  8:37:53 0 [Note] Plugin 'FEEDBACK' is disabled.
 2020-03-10  8:37:41 0 [Note] WSREP: (544132af, 'tcp://0.0.0.0:4567') turning message relay requesting off
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] Recovering after a crash using mysql-bin
galera-mariadb-galera-0 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] Starting crash recovery...
galera-mariadb-galera-1 mariadb-galera mariadb-galera WSREP_SST: [INFO] Streaming the backup to joiner at 10.32.0.10 4444 (20200310 08:37:49.798)
2020-03-10  8:37:53 0 [Note] Crash recovery finished.
galera-mariadb-galera-0 galera-mariadb-galera-1  2020-03-10  8:40:20 0 [Note] InnoDB: Uses event mutexes
galera-mariadb-galera-2mariadb-galera 2020-03-10  8:37:53 0 [Note] InnoDB: Buffer pool(s) load completed at 200310  8:37:53
galera-mariadb-galera-1 mariadb-galera WSREP_SST: [INFO] Evaluating /opt/bitnami/mariadb//bin/mariabackup --innobackupex --defaults-file=/opt/bitnami/mariadb/conf/my.cnf     $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2> /bitnami/mariadb/data//mariabackup.backup.log | socat -u stdio TCP:10.32.0.10:4444; RC=( ${PIPESTATUS[@]} ) (20200310 08:37:49.800)
 galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:40:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
mariadb-galera galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] InnoDB: Number of pools: 1
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] InnoDB: Using SSE2 crc32 instructions
mariadb-galera 2020-03-10  8:37:53 0 [Note] Server socket created on IP: '0.0.0.0'.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Warning] 'proxies_priv' entry '@% root@galera-mariadb-galera-0' ignored in --skip-name-resolve mode.
2020-03-10  8:37:52 0 [Note] WSREP: SST sent: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: Server status change donor -> joined
galera-mariadb-galera-0 mariadb-galera mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: 0.0 (galera-mariadb-galera-0): State transfer to 1.0 (galera-mariadb-galera-1) complete.
galera-mariadb-galera-1 2020-03-10  8:40:20 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 14)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: Member 0.0 (galera-mariadb-galera-0) synced with group.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 14)
galera-mariadb-galera-0 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:20 0 [Note] InnoDB: Completed initialization of buffer pool
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Server initialized
mariadb-galera 2020-03-10  8:40:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
galera-mariadb-galera-1galera-mariadb-galera-2 mariadb-galera  2020-03-10  8:40:21 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Server status change initializing -> initialized
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:52 12 [Note] WSREP: Server galera-mariadb-galera-0 synced with group
mariadb-galera 2020-03-10  8:37:53 3 [Note] WSREP: Server status change initialized -> joined
galera-mariadb-galera-1 galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:53 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-2 galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 3 [Note] WSREP: Recovered position from storage: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:52 12 [Note] WSREP: Server status change joined -> synced
mariadb-galera 2020-03-10  8:37:53 3 [Note] WSREP: Recovered view from SST:
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 12 [Note] WSREP: Synchronized with group, ready for connections
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-0 mariadb-galera WSREP_SST: [INFO] Total time on donor: 0 seconds (20200310 08:37:52.644)
mariadb-galera 2020-03-10  8:40:21 0 [Note] InnoDB: Creating shared tablespace for temporary tables
galera-mariadb-galera-0 mariadb-galera WSREP_SST: [INFO] Cleaning up temporary directories (20200310 08:37:52.648)
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:52 0 [Note] WSREP: Donor monitor thread ended with total time 13 sec
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: async IST sender served
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: 1.0 (galera-mariadb-galera-1): State transfer from 0.0 (galera-mariadb-galera-0) complete.
galera-mariadb-galera-0 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Member 1.0 (galera-mariadb-galera-1) synced with group.
galera-mariadb-galera-1 mariadb-galera   id: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
galera-mariadb-galera-1 mariadb-galera   status: primary
galera-mariadb-galera-1 mariadb-galera   protocol_version: 4
galera-mariadb-galera-1 mariadb-galera   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
galera-mariadb-galera-1 mariadb-galera   final: no
galera-mariadb-galera-1 mariadb-galera   own_index: 1
galera-mariadb-galera-1 mariadb-galera   members(2):
galera-mariadb-galera-1 mariadb-galera  0: 544132af-62aa-11ea-819e-f6a56520b43c, galera-mariadb-galera-0
galera-mariadb-galera-1 mariadb-galera  1: 658f0c29-62aa-11ea-b5f7-aa3b04fa812a, galera-mariadb-galera-1
galera-mariadb-galera-1 mariadb-galera
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 11 [Note] WSREP: Recovered cluster id 51bb9055-62aa-11ea-a237-33e10ea0b55a
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 3 [Note] WSREP: SST received: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: Installed new state from SST: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Joiner monitor thread ended with total time 15 sec
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 12 [Note] WSREP: Starting applier thread 12
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 14 [Note] WSREP: Starting applier thread 14
galera-mariadb-galera-1 mariadb-galera mariadb-galera 2020-03-10  8:37:53 13 [Note] WSREP: Starting applier thread 13
2020-03-10  8:40:21 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: Cert. index preload up to 14
galera-mariadb-galera-1 galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: ####### IST applying starts with 15
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: ####### IST current seqno initialized to 14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Receiving IST...  0.0% (0/1 events) complete.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: IST preload starting at 14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: REPL Protocols: 10 (5, 3)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: ####### Assign initial position for certification: 51bb9055-62aa-11ea-a237-33e10ea0b55a:13, protocol version: 5
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: REPL Protocols: 10 (5, 3)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: ####### Adjusting cert position: 13 -> 14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Service thread queue flushed.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Lowest cert indnex boundary for CC from preload: 14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Min available from gcache for CC from preload: 14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Receiving IST...100.0% (1/1 events) complete.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: IST received: 51bb9055-62aa-11ea-a237-33e10ea0b55a:14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: Lowest cert indnex boundary for CC from group: 14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: Min available from gcache for CC from group: 14
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] Reading of all Master_info entries succeeded
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] Added new Master_info '' to hash table
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
galera-mariadb-galera-1 mariadb-galera Version: '10.4.12-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: 1.0 (galera-mariadb-galera-1): State transfer from 0.0 (galera-mariadb-galera-0) complete.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 14)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Member 1.0 (galera-mariadb-galera-1) synced with group.
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 14)
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: Server galera-mariadb-galera-1 synced with group
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: Server status change joined -> synced
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: Synchronized with group, ready for connections
galera-mariadb-galera-1 mariadb-galera 2020-03-10  8:37:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mariadb-galera 2020-03-10  8:40:21 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] InnoDB: Waiting for purge to start
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] InnoDB: 10.4.12 started; log sequence number 88116; transaction id 46
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] Plugin 'FEEDBACK' is disabled.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] Server socket created on IP: '0.0.0.0'.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Warning] 'proxies_priv' entry '@% root@galera-mariadb-galera-2' ignored in --skip-name-resolve mode.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] WSREP: Server initialized
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] WSREP: Server status change initializing -> initialized
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 10 [Note] WSREP: Recovered cluster id ac647275-62aa-11ea-a17f-c2ca3d40169e
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 11 [Note] WSREP: Starting applier thread 11
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 12 [Note] WSREP: Starting applier thread 12
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 13 [Note] WSREP: Starting applier thread 13
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] Reading of all Master_info entries succeeded
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] Added new Master_info '' to hash table
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: Server status change initialized -> joined
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
galera-mariadb-galera-2 mariadb-galera Version: '10.4.12-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 0 [Note] InnoDB: Buffer pool(s) load completed at 200310  8:40:21
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: Server galera-mariadb-galera-2 synced with group
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: Server status change joined -> synced
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: Synchronized with group, ready for connections
galera-mariadb-galera-2 mariadb-galera 2020-03-10  8:40:21 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

javsalgar on 10 Mar 2020

same issue

ubuntu18.04 kernel 4.15
ceph nautilus
helm 3.1.0
k8s 1.16.7
bitnami/mariadb-galera 0.8.5

cmd: helm install mariadb-galera bitnami/mariadb-galera --version 0.8.5

I use ceph rdb as default storageClass (test fine with other charts, such as bitnami/harbor)

by the way, if bitnami/harbor support None RWM volumes like ceph rdb, it will be very nice for helm upgrade, see: https://github.com/goharbor/harbor-helm/issues/224

goharbor/harbor-helm support this, https://github.com/goharbor/harbor-helm/pull/470

if I enable persistence, whether I set rootUser.password or not, even delete pvc, it failed with Access denied for user 'root'@'localhost' (using password: YES)

exec into docker container cannot login in either.

if I disable persistence, everything is fine.

xin053 on 18 Mar 2020

👍1

Kubernetes Version: v1.16.3-k3s.2

Tested on template version:
0.8.5
0.8.2
0.7.0

I use Longhorn storageClass and can confirm what @xin053 described: the Galera Cluster starts fine as long as persistance is disabled, but fails to initialize if enabled.

iben12 on 19 Mar 2020

Tried if I use rootUser.forcePassword and provide rootUser.password helps, but it doesn't.

iben12 on 19 Mar 2020

Hi,

So, if I undrestand correctly, the issue could be located in the persistence storage class then, right? Because in my case, I started it with persistence enabled and found no issues.

javsalgar on 20 Mar 2020

What storageClass you use @javsalgar? So far we experience the same behaviour on ceph and longhorn.

iben12 on 20 Mar 2020

I'm using Ceph RADOS block device myself.

$ k describe sc rook-ceph-block
Name:                  rook-ceph-block
IsDefaultClass:        Yes
Annotations:           storageclass.kubernetes.io/is-default-class=true
Provisioner:           rook-ceph-system.rbd.csi.ceph.com
Parameters:            clusterID=rook-ceph-system,csi.storage.k8s.io/fstype=xfs,csi.storage.k8s.io/node-stage-secret-name=rook-csi-rbd-node,csi.storage.k8s.io/node-stage-secret-namespace=rook-ceph-system,csi.storage.k8s.io/provisioner-secret-name=rook-csi-rbd-provisioner,csi.storage.k8s.io/provisioner-secret-namespace=rook-ceph-system,imageFeatures=layering,imageFormat=2,pool=rook-ceph-block-pool
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

One of the things I've noticed with PVs on Ceph RBD is they set POSIX permissions to 0777 on the directory by default, which a lot of software doesn't like. I wonder if the ignored in --skip-name-resolve mode error is a red herring, but instead file permissions are causing problems.

chris-mccoy on 20 Mar 2020

I'm using rook-ceph as well, but mariadb is the only software misbehaving on it until now

pcornelissen on 20 Mar 2020

Hi,

We test it with GKE, AKS, EKS, IKS and PKS default storage classes and we didn't find any issue in our daily testing. We have not tried with these storage classes that you mention so we would need some time in our side to test them.

javsalgar on 23 Mar 2020

@javsalgar Let me know if I can provide any help.

iben12 on 23 Mar 2020

Hi,

Do you have an easy way of setting a ceph filesystem in Kubernetes so we could reproduce? That would be very helpful

javsalgar on 24 Mar 2020

using rook-ceph is pretty easy https://rook.io/docs/rook/v1.2/ceph-quickstart.html
You can use a directory on the nodes for storage, you don't need a dedicated device. Especially not for testing

pcornelissen on 24 Mar 2020

Thanks for the information, I will create a task to investigate it. We cannot guarantee an ETA but we will update this ticket when we have more information.

javsalgar on 27 Mar 2020

I clean k8s cluster, and reboot all host, then reinstall k8s cluster with the same config, and mariadb-galera with pvc enabled works!

xin053 on 7 Apr 2020

Thanks for the information @xin053 :)

javsalgar on 9 Apr 2020

@xin053 With the same versions or was one of the images updated? (mariadb or the PVC provider) and which pvc type did you use?

pcornelissen on 9 Apr 2020

@pcornelissen with all the same version include mariadb-galera and PVC provider, I use ceph rbd deployed by ceph-deploy

I test at ubuntu18.04(kernel 4.15) and ubuntu16.04(kernel 4.20), all two master and one node with all components the same version(I deploy k8s by kubespray v2.12.5, and mariadb-galera chart version 0.8.5, ceph nautilus)

all works fine when I clean k8s cluster, and reboot all host, then reinstall k8s cluster. If I just reinstall k8s without reboot all host, mariadb-galera cannot works.

Maybe there is something wrong with network? By the way, I test calico and flannel network plugin with ipvs kube-proxy mode

xin053 on 10 Apr 2020

I am also using rook to provision ceph storage and I am also facing the same issue.

What's strange is that I tried it 5 times and once out of the 5 times it worked without a problem. This makes me think, that it has something to to with strange timing-related issues.

FYI: I am using HDDs for this storage class.

kubectl describe sc rook-ceph-block
Name:            rook-ceph-block
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"rook-ceph-block"},"parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/fstype":"xfs","csi.storage.k8s.io/node-stage-secret-name":"rook-ceph-csi","csi.storage.k8s.io/node-stage-secret-namespace":"rook-ceph","csi.storage.k8s.io/provisioner-secret-name":"rook-ceph-csi","csi.storage.k8s.io/provisioner-secret-namespace":"rook-ceph","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"provisioner":"rook-ceph.rbd.csi.ceph.com","reclaimPolicy":"Delete"}

Provisioner:           rook-ceph.rbd.csi.ceph.com
Parameters:            clusterID=rook-ceph,csi.storage.k8s.io/fstype=xfs,csi.storage.k8s.io/node-stage-secret-name=rook-ceph-csi,csi.storage.k8s.io/node-stage-secret-namespace=rook-ceph,csi.storage.k8s.io/provisioner-secret-name=rook-ceph-csi,csi.storage.k8s.io/provisioner-secret-namespace=rook-ceph,imageFeatures=layering,imageFormat=2,pool=replicapool
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>
root@k8s-m1:~#

p-andreas on 11 Apr 2020

I have the same problem with helm install on OKD 3.11
What I found, that if I install with persistence.enabled=false then everything works.
I have glusterfs as persistent storage.
And when I install with persistent storage then it seems that root password is not set in galera server, because I can connect without password:

$ mysql -u root
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 18
Server version: 10.4.12-MariaDB-log Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> \q
Bye

rsokolovs on 16 Apr 2020

👍1

Hi @rsokolovs. Could it be because the instance crashes first before setting the password?

javsalgar on 17 Apr 2020

That happens pretty early, so I don't think that the container gets terminated and I have not seen too early restarts in my containers

pcornelissen on 17 Apr 2020

I've the same issue with mysql chart.
It works fine if I let it generate a random password, but If I do specify a custom one in _root.password_ value, my POD never gets ready because of:

Access denied for user 'root'@'localhost' (using password: YES)

danparisi on 17 Apr 2020

Hi @rsokolovs. Could it be because the instance crashes first before setting the password?

Yes, it might be, because when I tested again with persistent storage I saw that first time when pod is started it get's restarted after few seconds.
Here is log from first start:

[38;5;6m [38;5;5m15:55:08.09 [0m
[38;5;6m [38;5;5m15:55:08.11 [0m[1mWelcome to the Bitnami mariadb-galera container[0m
[38;5;6m [38;5;5m15:55:08.11 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-mariadb-galera[0m
[38;5;6m [38;5;5m15:55:08.11 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-mariadb-galera/issues[0m
[38;5;6m [38;5;5m15:55:08.12 [0mSend us your feedback at [[email protected][0m
[38;5;6m [38;5;5m15:55:08.12 [0m
[38;5;6m [38;5;5m15:55:08.12 [0m[38;5;2mINFO [0m ==> ** Starting MariaDB setup **
[38;5;6m [38;5;5m15:55:08.27 [0m[38;5;2mINFO [0m ==> Validating settings in MYSQL_*/MARIADB_* env vars..
[38;5;6m [38;5;5m15:55:08.29 [0m[38;5;2mINFO [0m ==> Initializing mariadb database...
[38;5;6m [38;5;5m15:55:16.04 [0m[38;5;2mINFO [0m ==> Starting mariadb in background
[38;5;6m [38;5;5m15:55:17.60 [0m[38;5;2mINFO [0m ==> Stopping mariadb

And here is after it get's restarted:

[38;5;6m [38;5;5m15:55:22.66 [0m
[38;5;6m [38;5;5m15:55:22.68 [0m[1mWelcome to the Bitnami mariadb-galera container[0m
[38;5;6m [38;5;5m15:55:22.68 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-mariadb-galera[0m
[38;5;6m [38;5;5m15:55:22.69 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-mariadb-galera/issues[0m
[38;5;6m [38;5;5m15:55:22.69 [0mSend us your feedback at [[email protected][0m
[38;5;6m [38;5;5m15:55:22.70 [0m
[38;5;6m [38;5;5m15:55:22.70 [0m[38;5;2mINFO [0m ==> ** Starting MariaDB setup **
[38;5;6m [38;5;5m15:55:22.90 [0m[38;5;2mINFO [0m ==> Validating settings in MYSQL_*/MARIADB_* env vars..
[38;5;6m [38;5;5m15:55:22.93 [0m[38;5;2mINFO [0m ==> Initializing mariadb database...
[38;5;6m [38;5;5m15:55:22.98 [0m[38;5;2mINFO [0m ==> Persisted data detected. Restoring...
[38;5;6m [38;5;5m15:55:23.00 [0m[38;5;2mINFO [0m ==> ** MariaDB setup finished! **

[38;5;6m [38;5;5m15:55:23.15 [0m[38;5;2mINFO [0m ==> ** Starting MariaDB **

If I use persistence.enabled=false
Then here is logs without any restarts:

[38;5;6m [38;5;5m15:57:24.69 [0m
[38;5;6m [38;5;5m15:57:24.71 [0m[1mWelcome to the Bitnami mariadb-galera container[0m
[38;5;6m [38;5;5m15:57:24.72 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-mariadb-galera[0m
[38;5;6m [38;5;5m15:57:24.72 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-mariadb-galera/issues[0m
[38;5;6m [38;5;5m15:57:24.72 [0mSend us your feedback at [[email protected][0m
[38;5;6m [38;5;5m15:57:24.73 [0m
[38;5;6m [38;5;5m15:57:24.73 [0m[38;5;2mINFO [0m ==> ** Starting MariaDB setup **
[38;5;6m [38;5;5m15:57:24.90 [0m[38;5;2mINFO [0m ==> Validating settings in MYSQL_*/MARIADB_* env vars..
[38;5;6m [38;5;5m15:57:24.94 [0m[38;5;2mINFO [0m ==> Initializing mariadb database...
[38;5;6m [38;5;5m15:57:27.81 [0m[38;5;2mINFO [0m ==> Starting mariadb in background
[38;5;6m [38;5;5m15:57:29.22 [0m[38;5;2mINFO [0m ==> Stopping mariadb
[38;5;6m [38;5;5m15:57:31.24 [0m[38;5;2mINFO [0m ==> ** MariaDB setup finished! **

[38;5;6m [38;5;5m15:57:31.44 [0m[38;5;2mINFO [0m ==> ** Starting MariaDB **

rsokolovs on 18 Apr 2020

I definitely want to check these storage systems different from the default (we test most of the managed k8s solution storage systems). It would be good to know which command inside the contanier initialization makes it fail.

javsalgar on 21 Apr 2020

When I turn on debug: true in value.yaml, I notice that mariadb stop before set root password just as https://github.com/bitnami/charts/issues/2013#issuecomment-615895010 said. And I saw a ERROR after debug info Deleting all users to avoid issues with galera configuration..., the ERROR is WSREP has not yet prepared node for application use, so I look into libmysqlgalera.sh, I think the mariadb is not ready before execute

https://github.com/bitnami/bitnami-docker-mariadb-galera/blob/a541b21864117acd67124e9a7b4dbb26924747ff/10.4/debian-10/rootfs/libmysqlgalera.sh#L347

mysql_execute "mysql" <<EOF
DELETE FROM mysql.user WHERE user<>'mysql.sys';
EOF

so I sleep 10s before delete user

mysql_start_bg
debug "************Sleep 10s**********"
sleep 10s
debug "Deleting all users to avoid issues with galera configuration..."
mysql_execute "mysql" <<EOF
DELETE FROM mysql.user WHERE user<>'mysql.sys';
EOF

after build image, everything is fine with pvc enabled.

xin053 on 30 Apr 2020

I see, so it is a matter of the filesystem performance then. As it is slow, it is not ready when doing that step, right?

javsalgar on 30 Apr 2020

Seems plausible

pcornelissen on 30 Apr 2020

Ok, let me review the container code so we can find a solution for this kind of scenarios.

javsalgar on 1 May 2020

👍1

Hey,

I added support of extra environment variables to our charts. I also added an extra environment variable for sleeping while initialization.

https://github.com/bitnami/bitnami-docker-mariadb-galera#slow-filesystems

Please let us know if this solves the issue.

javsalgar on 5 May 2020

🚀1

Not sure if that's the same case, but I am also experiencing somewhat random inability to install MariaDB cluster. From my observations first cluster install almost always succeeds and second one almost always fails, but I was not yet able to identify the pattern. We are deploying MariaDB via AzureDevOps pipelines, which first create a dedicated namespace and DB secret in it, then run Helm install in next step. One thing I have noticed in a repeatable fashion is the following log messages sequence.

On a cluster, which will succeed the install and properly set root user password I do see the following sequence. Container is not restarted:

 06:35:05.71 Welcome to the Bitnami mariadb-galera container
 06:35:05.71 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
 06:35:05.71 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
 06:35:05.71 Send us your feedback at [email protected]
 06:35:05.71
 06:35:05.72 INFO  ==> ** Starting MariaDB setup **
 06:35:05.78 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
 06:35:05.92 INFO  ==> Initializing mariadb database...
 06:35:17.58 INFO  ==> Starting mariadb in background
**/libmysqlgalera.sh: line 545: DB_REPLICATION_MODE: unbound variable**
 06:35:20.66 INFO  ==> Stopping mariadb
 06:35:25.67 INFO  ==> ** MariaDB setup finished! **

Upadte: On a cluster which will fail the install there is no error message :) and container is restarted.

 06:49:37.92 Welcome to the Bitnami mariadb-galera container
 06:49:37.92 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
 06:49:37.92 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
 06:49:37.92 Send us your feedback at [email protected]
 06:49:37.92
 06:49:37.92 INFO  ==> ** Starting MariaDB setup **
 06:49:37.98 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
 06:49:38.67 INFO  ==> Initializing mariadb database...
 06:49:50.40 INFO  ==> Starting mariadb in background
 06:49:51.43 INFO  ==> Stopping mariadb
root@atl-ca2-m-fwxj:~# kubectl -n kube-keycloak logs -f oidc-db-0
 06:49:57.07
 06:49:57.07 Welcome to the Bitnami mariadb-galera container
 06:49:57.08 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
 06:49:57.08 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
 06:49:57.08 Send us your feedback at [email protected]
 06:49:57.08
 06:49:57.08 INFO  ==> ** Starting MariaDB setup **
 06:49:57.15 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
 06:49:57.38 INFO  ==> Initializing mariadb database...
 06:49:57.41 INFO  ==> Persisted data detected. Restoring...
 06:49:57.41 INFO  ==> ** MariaDB setup finished! **

elvinasp on 19 May 2020

@elvinasp try add this at value.yaml

extraEnvVars:
  - name: MARIADB_INIT_SLEEP_TIME
    value: "20"

https://github.com/bitnami/bitnami-docker-mariadb-galera#slow-filesystems

xin053 on 19 May 2020

👍1

@elvinasp try add this at value.yaml
extraEnvVars:
  - name: MARIADB_INIT_SLEEP_TIME
    value: "20"

Already did that. It seems 10 sec. was enough and install seems no longer failing. Thanks.

What is interesting this did not trigger until second MariaDB cluster was installed on an environment. But besides that there is already bunch of other things installed on main cluster.

elvinasp on 19 May 2020

Hi,

It would be great if you could share the values you used for the installations. This way we can easily reproduce.

javsalgar on 19 May 2020

Hi,

It would be great if you could share the values you used for the installations. This way we can easily reproduce.

Here is the updated values file. For the storage we are using Azure standard HDD storage, provisioned via Gluster FS. GlusterFS volume replication is 3. I have added replication username as per issue #1126 . Will try to run some tests without it, when I am satisfied with deployment stability. So far 5 runs in a row have not failed. :)

image:
  tag: 10.3.22-debian-10-r20
  # Uncomment for the More verbose debugging messages in MariaDB logs
  # debug: true

# Race condition fix for slower file systems
extraEnvVars:
  - name: MARIADB_INIT_SLEEP_TIME
    value: "10"

fullnameOverride: oidc-db
existingSecret: mariadb-galera
replicaCount: __([mariadb-replicas])__

secrityContext:
  enabled: true

db:
  user: oidc
  name: oidc_db

replication:
  user: oidc_replica

service:
  clusterIP: None

persistence:
  storageClass: standard-hdd
  size: __([oidc-db-size])__

resources:
  requests:
    memory: 256Mi
    cpu: 250m

podDisruptionBudget:
  create: true
  minAvailable: 1

metrics:
  enabled: false
  serviceMonitor:
    enabled: false

elvinasp on 19 May 2020

Please let us know if without that persistence type the issue appears or not

javsalgar on 20 May 2020

Please let us know if without that persistence type the issue appears or not
Not sure I understand the point of running Database cluster without persistence. When the pod dies, database is gone. Am I right? The whole point of DB is to persist the data.

elvinasp on 20 May 2020

Sorry, let me rephrase it. What I mean is if the issue appears when not using the Gluster FS type of persistence. I want to make sure that the issue appears when using that specific type of storage class.

javsalgar on 21 May 2020

I can confirm the following:

When not using persistence or using the local-path storage class the chart works out of the box.
Using longhorn storage class and adding the MARIADB_INIT_SLEEP_TIME extraVar with the value of 10 the chart works as expected.

Great work!

Environment: