Charts: [bitnami/mariadb] Protect SQL replica service from including replicas which are lagged

[addshore]

Which chart: bitnami/mariadb

Is your feature request related to a problem? Please describe.
When expanding the replica set you will often add a replica that is lagged behind the master and needs to catch up.
Currently new replicas get added to the replica service as soon as mysqld is running, meaning applications can end up querying old data.

Describe the solution you'd like
One of the below:

Some way to customize the ready check for pods in the replica node pool. So that I can use something like this:

SECONDS_BEHIND_MASTER=$(mysql -e "show slave status\G" -uroot -p$MARIADB_MASTER_ROOT_PASSWORD | grep "Seconds_Behind_Master" | awk '{print $2}')
if [ ${SECONDS_BEHIND_MASTER%.*} -gt 30 ]; then echo "lagged"; else echo "nolag"; fi

Or some other way to ensure that lagged nodes that are catching up are not included in the replica service.

Describe alternatives you've considered

There are probably many other alternatives that could work to some degree such as:

• Having a second replica set for catching nodes up, and then transferring them over into the main slave replica set? (requires messing with the chart)

[dani8art]

Hi @addshore, thank you for opening this issues, I see your point, however, how can we sure 30s would be a good time? maybe we should find something more accurate, shouldn't we?

[addshore]

If we went down the approach of a lag check we could have the lag amount configurable.
30s was only meant as an example.

An initial step towards this however would be to allow the check section of the resulting yaml to be altered using values.
However I think a slightly better default would also make sense, as currently if you scale up from 1 to 2, the second replica will by default be lagged and may take hours to catch up and should not be pooled.

I believe mysql also allows for some alternative ways to detect this sort of lag, without using seconds.

(Master_Log_file, Read_Master_Log_Pos): Coordinates in the master binary log indicating how far the slave I/O thread has read events from that log.

[dani8art]

I see, so the thing is to keep the liveness probe as it is now and change the readiness probe to a something like:

SECONDS_BEHIND_MASTER=$(mysql -e "show slave status\G" -uroot -p$MARIADB_MASTER_ROOT_PASSWORD | grep "Seconds_Behind_Master" | awk '{print $2}')
if [ ${SECONDS_BEHIND_MASTER%.*} -gt 30 ]; then echo "lagged"; else echo "nolag"; fi

Wouldn't this bring the new pods to an inconsistent status? crashloopbackoff or something similar? I would say no, right?

[addshore]

Looking at https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes the only thing the readiness probe will dictate is if the pod is pooled in the service or not.

But yes, keeping the liveness probe exactly as it is now.
And also having the level of readiness desired by the end user configurable via a variable (seconds).

[dani8art]

Yes, I think it could work, if not perfectly, at least better than before and more configurable than the other readiness probe, I like it.

If you think you a chance, you could create a PR adding this approach and we would be happy to handle it. Thank you for your great feedback!

[addshore]

Great! I'll try to make a PR this weekend!

[dani8art]

Great news! we will keep in touch!

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[dani8art]

Hi @addshore, Any updates?

[addshore]

So, I'm still looking at this and testing it in my production.
Ran into an issue today where the replication stopped on a replica but it kept thinking it was ready.
This seems to happen when Seconds_Behind_Master is NULL, so this code perhaps should also be looking for Slave_SQL_Running

I'll have a new PR soon and also rebase it to master

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[dani8art]

Cool!! thanks you so much for your effort here!

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.