Charts: [bitnami/postgres-ha] PgPool taking long to switch to new master

Created on 15 Jul 2020 · 24Comments · Source: bitnami/charts

Which chart:
I am using the postgres-ha chart in version 3.4.3

Describe the bug
When deleting the master-backend-node, pgpool sometimes takes about 30 seconds to switch to the newly promoted master-node.

To Reproduce
Steps to reproduce the behavior:

Use minikube (minikube version: v1.12.0) in a CentOS 8 VM.
Install helm and the tiller pod.
Deploy the chart using

helm install --set service.type=ClusterIP --set  postgresql.password=postgres --set pgpool.replicaCount=1 --set postgresql.repmgrReconnectAttempts=1 --set postgresql.repmgrConnectTimeout=1 --set postgresql.repmgrReconnectInterval=2 --set pgpoolImage.debug=true --name pqs-tester bitnami/postgresql-ha

Get the Cluster IP for pgpool:

kubectl get service

Connect to postgres from the VM using

psql -h <Cluster IP> -U postgres -d repmgr

The password is postgres

Query for the backend states:

select node_id, upstream_node_id, active, node_name, type from repmgr.nodes;

Delete the master backend node using

kubectl delete pod <name of master>

Try step 6 again.

A few times, the switching was performed almost immediately. But in most of the cases, the connection broke and the pgpool-pod gave the response:

psql: FATAL:  failed to create a backend 0 connection
DETAIL:  not executing failover because failover_on_backend_error is off

The log of pgpool was

2020-07-15 15:40:43: pid 3919: LOG:  get_query_result falied: status: -2
2020-07-15 15:40:43: pid 3919: CONTEXT:  while checking replication time lag
2020-07-15 15:40:52: pid 1: LOG:  child process with pid: 4051 exits with status 256
2020-07-15 15:40:52: pid 1: LOG:  fork a new child process with pid: 5844
2020-07-15 15:41:06: pid 1: LOG:  child process with pid: 4228 exits with status 256
2020-07-15 15:41:06: pid 1: LOG:  fork a new child process with pid: 5879
2020-07-15 15:41:13: pid 3919: LOG:  get_query_result falied: status: -2
2020-07-15 15:41:13: pid 3919: CONTEXT:  while checking replication time lag
2020-07-15 15:41:41: pid 1: LOG:  child process with pid: 4091 exits with status 256
2020-07-15 15:41:41: pid 1: LOG:  fork a new child process with pid: 5970
2020-07-15 15:41:42: pid 1: LOG:  child process with pid: 3962 exits with status 256
2020-07-15 15:41:42: pid 1: LOG:  fork a new child process with pid: 5984
2020-07-15 15:41:42: pid 1: LOG:  child process with pid: 4227 exits with status 256
2020-07-15 15:41:42: pid 1: LOG:  fork a new child process with pid: 5988
2020-07-15 15:41:43: pid 3919: LOG:  get_query_result falied: status: -2
2020-07-15 15:41:43: pid 3919: CONTEXT:  while checking replication time lag
2020-07-15 15:42:02: pid 1: LOG:  child process with pid: 3955 exits with status 256
2020-07-15 15:42:02: pid 1: LOG:  fork a new child process with pid: 6045
2020-07-15 15:42:09: pid 3849: FATAL:  unable to read data from DB node 1
2020-07-15 15:42:09: pid 3849: DETAIL:  EOF encountered with backend
2020-07-15 15:42:09: pid 1: LOG:  child process with pid: 3849 exits with status 256
2020-07-15 15:42:09: pid 1: LOG:  fork a new child process with pid: 6052
2020-07-15 15:42:10: pid 4046: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:10: pid 4046: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:10: pid 4046: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:10: pid 1: LOG:  child process with pid: 4046 exits with status 256
2020-07-15 15:42:10: pid 1: LOG:  fork a new child process with pid: 6053
2020-07-15 15:42:11: pid 5188: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:11: pid 5188: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:11: pid 5188: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:11: pid 1: LOG:  child process with pid: 5188 exits with status 256
2020-07-15 15:42:11: pid 1: LOG:  fork a new child process with pid: 6060
2020-07-15 15:42:12: pid 4050: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:12: pid 4050: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:12: pid 4050: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:12: pid 1: LOG:  child process with pid: 4050 exits with status 256
2020-07-15 15:42:12: pid 1: LOG:  fork a new child process with pid: 6074
2020-07-15 15:42:13: pid 3919: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:13: pid 3919: ERROR:  failed to make persistent db connection
2020-07-15 15:42:13: pid 3919: DETAIL:  connection to host:"pqs-tester-postgresql-ha-postgresql-1.pqs-tester-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-15 15:42:16: pid 4105: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:16: pid 4105: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:16: pid 4105: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:16: pid 1: LOG:  child process with pid: 4105 exits with status 256
2020-07-15 15:42:16: pid 1: LOG:  fork a new child process with pid: 6081
2020-07-15 15:42:16: pid 5686: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:16: pid 5686: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:16: pid 5686: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:16: pid 1: LOG:  child process with pid: 5686 exits with status 256
2020-07-15 15:42:16: pid 1: LOG:  fork a new child process with pid: 6082
2020-07-15 15:42:20: pid 5503: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:20: pid 5503: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:20: pid 5503: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:20: pid 1: LOG:  child process with pid: 5503 exits with status 256
2020-07-15 15:42:20: pid 1: LOG:  fork a new child process with pid: 6083
2020-07-15 15:42:21: pid 6083: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:21: pid 6083: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:21: pid 6083: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:21: pid 1: LOG:  child process with pid: 6083 exits with status 256
2020-07-15 15:42:21: pid 1: LOG:  fork a new child process with pid: 6090
2020-07-15 15:42:22: pid 5600: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:22: pid 5600: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:22: pid 5600: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:22: pid 1: LOG:  child process with pid: 5600 exits with status 256
2020-07-15 15:42:22: pid 1: LOG:  fork a new child process with pid: 6104
2020-07-15 15:42:24: pid 181: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:24: pid 181: ERROR:  failed to make persistent db connection
2020-07-15 15:42:24: pid 181: DETAIL:  connection to host:"pqs-tester-postgresql-ha-postgresql-1.pqs-tester-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-15 15:42:24: pid 181: LOG:  health check retrying on DB node: 1 (round:1)
2020-07-15 15:42:24: pid 6090: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:24: pid 6090: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:24: pid 6090: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:24: pid 1: LOG:  child process with pid: 6090 exits with status 256
2020-07-15 15:42:24: pid 1: LOG:  fork a new child process with pid: 6105
2020-07-15 15:42:26: pid 4053: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:26: pid 4053: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:26: pid 4053: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:26: pid 1: LOG:  child process with pid: 4053 exits with status 256
2020-07-15 15:42:26: pid 1: LOG:  fork a new child process with pid: 6112
2020-07-15 15:42:28: pid 6104: ERROR:  md5 authentication failed
2020-07-15 15:42:28: pid 6104: DETAIL:  password does not match
2020-07-15 15:42:29: pid 181: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:29: pid 181: ERROR:  failed to make persistent db connection
2020-07-15 15:42:29: pid 181: DETAIL:  connection to host:"pqs-tester-postgresql-ha-postgresql-1.pqs-tester-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-15 15:42:29: pid 181: LOG:  health check retrying on DB node: 1 (round:2)
2020-07-15 15:42:31: pid 5844: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:31: pid 5844: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:31: pid 5844: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:31: pid 1: LOG:  child process with pid: 5844 exits with status 256
2020-07-15 15:42:31: pid 1: LOG:  fork a new child process with pid: 6119
2020-07-15 15:42:32: pid 1: LOG:  child process with pid: 4049 exits with status 256
2020-07-15 15:42:32: pid 1: LOG:  fork a new child process with pid: 6133
2020-07-15 15:42:32: pid 6119: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:32: pid 6119: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:32: pid 6119: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:32: pid 1: LOG:  child process with pid: 6119 exits with status 256
2020-07-15 15:42:32: pid 1: LOG:  fork a new child process with pid: 6134
2020-07-15 15:42:34: pid 181: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:34: pid 181: ERROR:  failed to make persistent db connection
2020-07-15 15:42:34: pid 181: DETAIL:  connection to host:"pqs-tester-postgresql-ha-postgresql-1.pqs-tester-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-15 15:42:34: pid 181: LOG:  health check retrying on DB node: 1 (round:3)
2020-07-15 15:42:36: pid 6134: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:36: pid 6134: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:36: pid 6134: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:36: pid 1: LOG:  child process with pid: 6134 exits with status 256
2020-07-15 15:42:36: pid 1: LOG:  fork a new child process with pid: 6141
2020-07-15 15:42:39: pid 181: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:39: pid 181: ERROR:  failed to make persistent db connection
2020-07-15 15:42:39: pid 181: DETAIL:  connection to host:"pqs-tester-postgresql-ha-postgresql-1.pqs-tester-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-15 15:42:39: pid 181: LOG:  health check retrying on DB node: 1 (round:4)
2020-07-15 15:42:41: pid 4048: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:41: pid 4048: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:41: pid 4048: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:41: pid 1: LOG:  child process with pid: 4048 exits with status 256
2020-07-15 15:42:41: pid 1: LOG:  fork a new child process with pid: 6149
2020-07-15 15:42:42: pid 4009: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-15 15:42:42: pid 4009: FATAL:  failed to create a backend 1 connection
2020-07-15 15:42:42: pid 4009: DETAIL:  not executing failover because failover_on_backend_error is off
2020-07-15 15:42:42: pid 1: LOG:  child process with pid: 4009 exits with status 256
2020-07-15 15:42:42: pid 1: LOG:  fork a new child process with pid: 6163

Is there a timing problem or something? I was irritated by the 'get_query_result falied: status: -2' before the switching.
If I try this switching multiple times a few minutes in between, it works sometimes fast and most of the times not.
Expected behavior
This timeframe is too long in my opinion and I have seen it switching seemlesly (reconnecting the clients) a few times.
I would love to have this behaviour all the time.

Version of Helm and Kubernetes:
I am using kubernetes 1.18.3 and helm 2.16.9

Output of helm version:

Client: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Additional context
I also tried this on a bare metal kubernetes cluster. I had the same problems there.

Source

BenchmarkingBuffalo

👍1

Most helpful comment

To be honest, I dont understand why no one stumbled upon this before. Am I using this chart in a wrong way or did no one else get the situation of a failover? I really want to use this chart bcause it seems to be fitting my needs, but at the moment I am a little bit confused.

BenchmarkingBuffalo on 27 Jul 2020

👍2

All 24 comments

I figured out, what might be the problem:
The replication is not working correctly. When I insert data, it only shows up on the master and is not copied to the slave.
When I run

/opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show

on the master, it outputs

.....
WARNING: following issues were detected
  - node "pqs-tester-postgresql-ha-postgresql-1" (ID: 1001) is not attached to its upstream node "pqs-tester-postgresql-ha-postgresql-0" (ID: 1000)

I dont know how to fix this

BenchmarkingBuffalo on 16 Jul 2020

Hi,
Could you share the results of the following ?

postgres=# show pool_nodes;

postgres=# select * from pg_stat_replication;

About the node checking, there are two parameters that control this behavior: health_check_max_retries and health_check_period

You can change pgpool.conf values by these steps:>

Create the a file with the parameters (name doesn’t matter):

$ cat pgpool.conf
max_pool='333'

Create the configmap with that file:

$ kubectl create configmap testconfig --from-file=pgpool.conf
configmap/testconfig created

Install the chart by :
1) running

helm install mypg --set pgpool.configurationCM=testconfig airbus/postgresql-ha

Or 2. setting it in the values.yaml (for example in Rancher) and deploying via UI:

  configurationCM: testconfig

rafariossaa on 16 Jul 2020

Hi,
thank you for your response,
The output of show pool_nodes is

node_id |                                                   hostname                                                   | port | status | lb_weight |  role   | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
--------+--------------------------------------------------------------------------------------------------------------+------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
0       | pqs-tester-postgresql-ha-postgresql-0.pqs-tester-postgresql-ha-postgresql-headless.default.svc.cluster.local | 5432 | up     | 0.500000  | primary | 12         | true              | 0                 |                   |                        | 2020-07-16 13:23:38
1       | pqs-tester-postgresql-ha-postgresql-1.pqs-tester-postgresql-ha-postgresql-headless.default.svc.cluster.local | 5432 | up     | 0.500000  | standby | 2          | false             | 0                 |                   |                        | 2020-07-16 13:24:09

The output of select * from pg_stat_replication is

pid | usesysid | usename | application_name | client_addr | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state
----+----------+---------+------------------+-------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------
183 |    16384 | repmgr  | walreceiver      | 172.17.0.5  |                 |       58708 | 2020-07-16 13:25:27.955098+00 |              | streaming | 0/9000000 | 0/9000000 | 0/9000000 | 0/9000000  |           |           |            |             0 | async

I used your tips to tune the config a little:

health_check_max_retries='2'
health_check_period='1'

Now the failover happens much faster, which is good. The downside is, that I am now sure, that there is no working replication.
I created a table while pod ...-0 was master and inserted an entry. This was replicated to the standby. I could see the entry after I deleted the ...-0 pod.
Then I inserted an entry on the new master (pod ...-1) and this did not get replicated to the other, restarted pod.
Also the table pg_stat_replication was now empty.
When I deleted the ...-1 pod and pod ...-0 became master again, it overwrote the new entry I did while pod ...-1 was master.
Also the table pg_stat_replication had now one entry again.
So overall, the second entry got deleted.

BenchmarkingBuffalo on 16 Jul 2020

Hi,
That behavior is weird, I can not reproduce it.
Let's try a default deployment, connect to pgpool node and create a database and a table in the new database. Then check in the postgresql nodes if the new database and table have been replicated.
These are the steps I followed.

Install the chart with the default settings, use a new name for the deployment this is to avoid reusing the PVCs, in this case I used mypg :

$ helm install mypg bitnami/postgresql-ha

When all the pods are deployed and running, connect to pgpool node and create a database. These lines are copied from the exit of the previous command, so for example, the namespace will change for you.

$ export POSTGRES_PASSWORD=$(kubectl get secret --namespace rafael mypg-postgresql-ha-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)

$ kubectl run mypg-postgresql-ha-client --rm --tty -i --restart='Never' --namespace rafael --image bitnami/postgresql:11 --env="PGPASSWORD=$POSTGRES_PASSWORD"  \
        --command -- psql -h mypg-postgresql-ha-pgpool -p 5432 -U postgres -d postgres

Create a new database

...
postgres=# create database test1;
CREATE DATABASE
postgres=# ^D
...

Disconnect and connect to the new database

$ kubectl run mypg-postgresql-ha-client --rm --tty -i --restart='Never' --namespace pg-postgresql-ha-pgpool -p 5432 -U postgres -d test1
If you don't see a command prompt, try pressing enter.
psql (11.8)
Type "help" for help.

test1=#

Now, in other two terminal windows, I connected to node 0 and node 1 respectively, and connected to the new database in the local postgresql instance:

###Node 0

$ kubectl exec -it mypg-postgresql-ha-postgresql-0 bash

I have no name!@mypg-postgresql-ha-postgresql-0:/$ PGPASSWORD=$POSTGRES_PASSWORD psql -h localhost -p 5432 -U postgresql -d test1
psql (11.8)
Type "help" for help.

test1=#

###Node 1

$ kubectl exec -it mypg-postgresql-ha-postgresql-0 bash

I have no name!@mypg-postgresql-ha-postgresql-0:/$ PGPASSWORD=$POSTGRES_PASSWORD psql -h localhost -p 5432 -U postgresql -d test1
psql (11.8)
Type "help" for help.

test1=#

Now back to pgpool terminal window, lets create the new table:

test1=# create table t1 (c1 varchar(100));
CREATE TABLE
test1=#

Immediately, check in the terminals for Node 0 and Node if the new table exists:

### Node 0

test1=# \dt
        List of relations
 Schema | Name | Type  |  Owner   
--------+------+-------+----------
 public | t1   | table | postgres
(1 row)

### Node 1

test1=# \dt
        List of relations
 Schema | Name | Type  |  Owner   
--------+------+-------+----------
 public | t1   | table | postgres
(1 row)

rafariossaa on 17 Jul 2020

Hi,
the above steps work for me as well. But what happens in your setup, when you delete the primary pod?
In my setup the standby takes over and from there on, the replication is not working anymore.
In fact, the former master gets restarted twice and during the second restart, it clones the database from the new master.
But if you create a new table after the new standby is fully ready, the new table gets not replicated on it.
To reproduce:

Use your steps from above.
Delete the primary, for me this was:

kubectl delete pod mypg-postgresql-ha-postgresql-0

After the reconnection of pgpool to the new master start a new client:

kubectl run mypg-postgresql-ha-client --rm --tty -i --restart='Never' --namespace default --image bitnami/postgresql:11 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql -h mypg-postgresql-ha-pgpool -p 5432 -U postgres -d test1

Create a new table:

test1=# create table t2 (c1 varchar(100));
CREATE TABLE

Check the former master ist now back online as standby:

# kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
mypg-postgresql-ha-client                 1/1     Running   0          77s
mypg-postgresql-ha-pgpool-8465df9-49m7t   1/1     Running   0          7m18s
mypg-postgresql-ha-postgresql-0           1/1     Running   2          109s
mypg-postgresql-ha-postgresql-1           1/1     Running   0          7m4s

After step 4, the former master node gets restartet twice. After restarting, it clones the database from the new master. To check, run on the former master in psql

test1=# \dt
       List of relations
 Schema | Name | Type  |  Owner
--------+------+-------+----------
 public | t1   | table | postgres
 public | t2   | table | postgres

Create a new table on the client:

test1=# create table t3 (c1 varchar(100));
CREATE TABLE

Check presence in both nodes using

\dt

Output on new master:

        List of relations
 Schema | Name | Type  |  Owner
--------+------+-------+----------
 public | t1   | table | postgres
 public | t2   | table | postgres
 public | t3   | table | postgres

Output on new standby:

        List of relations
 Schema | Name | Type  |  Owner
--------+------+-------+----------
 public | t1   | table | postgres
 public | t2   | table | postgres

And the pg_stat_replication table is empty after the new master takes over.
Whe you now delete the new master, the table t3 will be lost.

BenchmarkingBuffalo on 17 Jul 2020

Hi,
Thanks for detailing it, I was able to reproduce it.

rafariossaa on 20 Jul 2020

Hi @rafariossaa,
can you please inform me in this thread, if you find the reason and/or a fix for this bug?
Thank you in advance.
Kind regards

BenchmarkingBuffalo on 22 Jul 2020

Hi,
We have just released v.3.4.9 of the chart. Can you give it a try ?

rafariossaa on 23 Jul 2020

Hi,
I tried the new release and it still does not work.
Here the logs of the new standby after switching:

←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.40 ←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.40 ←[0m←[1mWelcome to the Bitnami postgresql-repmgr container←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.41 ←[0mSubscribe to project updates by watching ←[1mhttps://github.com/bitnami/bitnami-docker-postgresql-repmgr←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.41 ←[0mSubmit issues and feature requests at ←[1mhttps://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.41 ←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.43 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting PostgreSQL with Replication Manager setup **
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.44 ←[0m←[38;5;2mINFO ←[0m ==> Validating settings in REPMGR_* env vars...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.44 ←[0m←[38;5;2mINFO ←[0m ==> Validating settings in POSTGRESQL_* env vars..
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.45 ←[0m←[38;5;2mINFO ←[0m ==> Querying all partner nodes for common upstream node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.49 ←[0m←[38;5;2mINFO ←[0m ==> Auto-detected primary node: 'mypg-postgresql-ha-postgresql-1.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.50 ←[0m←[38;5;2mINFO ←[0m ==> Preparing PostgreSQL configuration...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.50 ←[0m←[38;5;2mINFO ←[0m ==> postgresql.conf file not detected. Generating it...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.56 ←[0m←[38;5;2mINFO ←[0m ==> Preparing repmgr configuration...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.57 ←[0m←[38;5;2mINFO ←[0m ==> Initializing Repmgr...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.57 ←[0m←[38;5;2mINFO ←[0m ==> Waiting for primary node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:19.58 ←[0m←[38;5;2mINFO ←[0m ==> Cloning data from primary node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.04 ←[0m←[38;5;2mINFO ←[0m ==> Initializing PostgreSQL database...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.05 ←[0m←[38;5;2mINFO ←[0m ==> Custom configuration /opt/bitnami/postgresql/conf/postgresql.conf detected
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.05 ←[0m←[38;5;2mINFO ←[0m ==> Custom configuration /opt/bitnami/postgresql/conf/pg_hba.conf detected
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.07 ←[0m←[38;5;2mINFO ←[0m ==> Deploying PostgreSQL with persisted data...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.08 ←[0m←[38;5;2mINFO ←[0m ==> Configuring replication parameters
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.09 ←[0m←[38;5;2mINFO ←[0m ==> Configuring fsync
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.11 ←[0m←[38;5;2mINFO ←[0m ==> Setting up streaming replication slave...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:22.12 ←[0m←[38;5;2mINFO ←[0m ==> Starting PostgreSQL in background...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:23.05 ←[0m←[38;5;2mINFO ←[0m ==> Unregistering standby node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:23.09 ←[0m←[38;5;2mINFO ←[0m ==> Registering Standby node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:23.13 ←[0m←[38;5;2mINFO ←[0m ==> Stopping PostgreSQL...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:24.16 ←[0m←[38;5;2mINFO ←[0m ==> ** PostgreSQL with Replication Manager setup finished! **

←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:24.19 ←[0m←[38;5;2mINFO ←[0m ==> Starting PostgreSQL in background...
←[38;5;6mpostgresql-repmgr ←[38;5;5m09:21:24.32 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting repmgrd **
[2020-07-27 09:21:24] [NOTICE] repmgrd (repmgrd 5.1.0) starting up
INFO:  set_repmgrd_pid(): provided pidfile is /opt/bitnami/repmgr/tmp/repmgr.pid
[2020-07-27 09:21:24] [NOTICE] starting monitoring of node "mypg-postgresql-ha-postgresql-0" (ID: 1000)

And here the logs of the new master:

←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.37 ←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.37 ←[0m←[1mWelcome to the Bitnami postgresql-repmgr container←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.38 ←[0mSubscribe to project updates by watching ←[1mhttps://github.com/bitnami/bitnami-docker-postgresql-repmgr←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.38 ←[0mSubmit issues and feature requests at ←[1mhttps://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.38 ←[0m
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.39 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting PostgreSQL with Replication Manager setup **
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.41 ←[0m←[38;5;2mINFO ←[0m ==> Validating settings in REPMGR_* env vars...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.41 ←[0m←[38;5;2mINFO ←[0m ==> Validating settings in POSTGRESQL_* env vars..
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.42 ←[0m←[38;5;2mINFO ←[0m ==> Querying all partner nodes for common upstream node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.45 ←[0m←[38;5;2mINFO ←[0m ==> Preparing PostgreSQL configuration...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.46 ←[0m←[38;5;2mINFO ←[0m ==> postgresql.conf file not detected. Generating it...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.51 ←[0m←[38;5;2mINFO ←[0m ==> Preparing repmgr configuration...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.51 ←[0m←[38;5;2mINFO ←[0m ==> Initializing Repmgr...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:51:39.52 ←[0m←[38;5;2mINFO ←[0m ==> Waiting for primary node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:09.56 ←[0m←[38;5;2mINFO ←[0m ==> Cloning data from primary node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.52 ←[0m←[38;5;2mINFO ←[0m ==> Initializing PostgreSQL database...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.53 ←[0m←[38;5;2mINFO ←[0m ==> Custom configuration /opt/bitnami/postgresql/conf/postgresql.conf detected
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.53 ←[0m←[38;5;2mINFO ←[0m ==> Custom configuration /opt/bitnami/postgresql/conf/pg_hba.conf detected
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.55 ←[0m←[38;5;2mINFO ←[0m ==> Deploying PostgreSQL with persisted data...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.56 ←[0m←[38;5;2mINFO ←[0m ==> Configuring replication parameters
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.58 ←[0m←[38;5;2mINFO ←[0m ==> Configuring fsync
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.59 ←[0m←[38;5;2mINFO ←[0m ==> Setting up streaming replication slave...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:11.62 ←[0m←[38;5;2mINFO ←[0m ==> Starting PostgreSQL in background...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:12.44 ←[0m←[38;5;2mINFO ←[0m ==> Unregistering standby node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:12.46 ←[0m←[38;5;2mINFO ←[0m ==> Registering Standby node...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:12.52 ←[0m←[38;5;2mINFO ←[0m ==> Stopping PostgreSQL...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:13.52 ←[0m←[38;5;2mINFO ←[0m ==> ** PostgreSQL with Replication Manager setup finished! **

←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:13.55 ←[0m←[38;5;2mINFO ←[0m ==> Starting PostgreSQL in background...
←[38;5;6mpostgresql-repmgr ←[38;5;5m08:52:13.67 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting repmgrd **
[2020-07-27 08:52:13] [NOTICE] repmgrd (repmgrd 5.1.0) starting up
INFO:  set_repmgrd_pid(): provided pidfile is /opt/bitnami/repmgr/tmp/repmgr.pid
[2020-07-27 08:52:13] [NOTICE] starting monitoring of node "mypg-postgresql-ha-postgresql-1" (ID: 1001)
[2020-07-27 09:19:39] [WARNING] unable to ping "user=repmgr password=DFBShmi5xt host=mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5"
[2020-07-27 09:19:39] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-07-27 09:19:39] [WARNING] unable to connect to upstream node "mypg-postgresql-ha-postgresql-0" (ID: 1000)
[2020-07-27 09:19:39] [WARNING] unable to ping "user=repmgr password=DFBShmi5xt connect_timeout=5 dbname=repmgr host=mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr"
[2020-07-27 09:19:39] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-07-27 09:19:44] [WARNING] unable to ping "user=repmgr password=DFBShmi5xt connect_timeout=5 dbname=repmgr host=mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr"
[2020-07-27 09:19:44] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-07-27 09:19:49] [WARNING] unable to ping "user=repmgr password=DFBShmi5xt connect_timeout=5 dbname=repmgr host=mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr"
[2020-07-27 09:19:49] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-07-27 09:19:49] [WARNING] unable to reconnect to node 1000 after 3 attempts
[2020-07-27 09:19:49] [NOTICE] this node is the only available candidate and will now promote itself
NOTICE: using provided configuration file "/opt/bitnami/repmgr/conf/repmgr.conf"
DEBUG: connecting to: "user=repmgr password=DFBShmi5xt connect_timeout=5 dbname=repmgr host=mypg-postgresql-ha-postgresql-1.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr"
DEBUG: set_config():
  SET synchronous_commit TO 'local'
INFO: connected to standby, checking its state
DEBUG: get_recovery_type(): SELECT pg_catalog.pg_is_in_recovery()
DEBUG: get_node_record():
  SELECT n.node_id, n.type, n.upstream_node_id, n.node_name,  n.conninfo, n.repluser, n.slot_name, n.location, n.priority, n.active, n.config_file, '' AS upstream_node_name, NULL AS attached   FROM repmgr.nodes n  WHERE n.node_id = 1001
DEBUG: get_replication_info():
 SELECT ts,         in_recovery,         last_wal_receive_lsn,         last_wal_replay_lsn,         last_xact_replay_timestamp,         CASE WHEN (last_wal_receive_lsn = last_wal_replay_lsn)           THEN 0::INT         ELSE           CASE WHEN last_xact_replay_timestamp IS NULL             THEN 0::INT           ELSE             EXTRACT(epoch FROM (pg_catalog.clock_timestamp() - last_xact_replay_timestamp))::INT           END         END AS replication_lag_time,         last_wal_receive_lsn >= last_wal_replay_lsn AS receiving_streamed_wal,         wal_replay_paused,         upstream_last_seen,         upstream_node_id    FROM (  SELECT CURRENT_TIMESTAMP AS ts,         pg_catalog.pg_is_in_recovery() AS in_recovery,         pg_catalog.pg_last_xact_replay_timestamp() AS last_xact_replay_timestamp,         COALESCE(pg_catalog.pg_last_wal_receive_lsn(), '0/0'::PG_LSN) AS last_wal_receive_lsn,         COALESCE(pg_catalog.pg_last_wal_replay_lsn(),  '0/0'::PG_LSN) AS last_wal_replay_lsn,         CASE WHEN pg_catalog.pg_is_in_recovery() IS FALSE           THEN FALSE           ELSE pg_catalog.pg_is_wal_replay_paused()         END AS wal_replay_paused,         CASE WHEN pg_catalog.pg_is_in_recovery() IS FALSE           THEN -1           ELSE repmgr.get_upstream_last_seen()         END AS upstream_last_seen,         CASE WHEN pg_catalog.pg_is_in_recovery() IS FALSE           THEN -1           ELSE repmgr.get_upstream_node_id()         END AS upstream_node_id           ) q
INFO: searching for primary node
DEBUG: get_primary_connection():
  SELECT node_id, conninfo,          CASE WHEN type = 'primary' THEN 1 ELSE 2 END AS type_priority         FROM repmgr.nodes    WHERE active IS TRUE      AND type != 'witness' ORDER BY active DESC, type_priority, priority, node_id
INFO: checking if node 1000 is primary
DEBUG: connecting to: "user=repmgr password=DFBShmi5xt connect_timeout=5 dbname=repmgr host=mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr"
ERROR: connection to database failed
DETAIL:
could not translate host name "mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local" to address: Name or service not known

DETAIL: attempted to connect using:
  user=repmgr password=DFBShmi5xt connect_timeout=5 dbname=repmgr host=mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr
INFO: checking if node 1001 is primary
DEBUG: connecting to: "user=repmgr password=DFBShmi5xt connect_timeout=5 dbname=repmgr host=mypg-postgresql-ha-postgresql-1.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr"
DEBUG: set_config():
  SET synchronous_commit TO 'local'
DEBUG: get_recovery_type(): SELECT pg_catalog.pg_is_in_recovery()
DEBUG: get_node_replication_stats():
 SELECT pg_catalog.current_setting('max_wal_senders')::INT AS max_wal_senders,         (SELECT pg_catalog.count(*) FROM pg_catalog.pg_stat_replication) AS attached_wal_receivers,         current_setting('max_replication_slots')::INT AS max_replication_slots,         (SELECT pg_catalog.count(*) FROM pg_catalog.pg_replication_slots WHERE slot_type='physical') AS total_replication_slots,         (SELECT pg_catalog.count(*) FROM pg_catalog.pg_replication_slots WHERE active IS TRUE AND slot_type='physical')  AS active_replication_slots,         (SELECT pg_catalog.count(*) FROM pg_catalog.pg_replication_slots WHERE active IS FALSE AND slot_type='physical') AS inactive_replication_slots,         pg_catalog.pg_is_in_recovery() AS in_recovery
DEBUG: get_active_sibling_node_records():
  SELECT n.node_id, n.type, n.upstream_node_id, n.node_name,  n.conninfo, n.repluser, n.slot_name, n.location, n.priority, n.active, n.config_file, '' AS upstream_node_name, NULL AS attached     FROM repmgr.nodes n    WHERE n.upstream_node_id = 1000      AND n.node_id != 1001      AND n.active IS TRUE ORDER BY n.node_id
DEBUG: clear_node_info_list() - closing open connections
DEBUG: clear_node_info_list() - unlinking
DEBUG: get_node_record():
  SELECT n.node_id, n.type, n.upstream_node_id, n.node_name,  n.conninfo, n.repluser, n.slot_name, n.location, n.priority, n.active, n.config_file, '' AS upstream_node_name, NULL AS attached   FROM repmgr.nodes n  WHERE n.node_id = 1001
NOTICE: promoting standby to primary
DETAIL: promoting server "mypg-postgresql-ha-postgresql-1" (ID: 1001) using "/opt/bitnami/postgresql/bin/pg_ctl  -w -D '/bitnami/postgresql/data' promote"
NOTICE: waiting up to 60 seconds (parameter "promote_check_timeout") for promotion to complete
DEBUG: get_recovery_type(): SELECT pg_catalog.pg_is_in_recovery()
INFO: standby promoted to primary after 0 second(s)
DEBUG: setting node 1001 as primary and marking existing primary as failed
DEBUG: begin_transaction()
DEBUG: commit_transaction()
NOTICE: STANDBY PROMOTE successful
DETAIL: server "mypg-postgresql-ha-postgresql-1" (ID: 1001) was successfully promoted to primary
DEBUG: _create_event(): event is "standby_promote" for node 1001
DEBUG: get_recovery_type(): SELECT pg_catalog.pg_is_in_recovery()
DEBUG: _create_event():
   INSERT INTO repmgr.events (              node_id,              event,              successful,              details             )       VALUES ($1, $2, $3, $4)    RETURNING event_timestamp
DEBUG: _create_event(): Event timestamp is "2020-07-27 09:19:49.814485+00"
DEBUG: _create_event(): command is '/opt/bitnami/repmgr/events/router.sh %n %e %s "%t" "%d"'
INFO: executing notification command for event "standby_promote"
DETAIL: command is:
  /opt/bitnami/repmgr/events/router.sh 1001 standby_promote 1 "2020-07-27 09:19:49.814485+00" "server \"mypg-postgresql-ha-postgresql-1\" (ID: 1001) was successfully promoted to primary"
DEBUG: clear_node_info_list() - closing open connections
DEBUG: clear_node_info_list() - unlinking
[2020-07-27 09:19:49] [NOTICE] monitoring cluster primary "mypg-postgresql-ha-postgresql-1" (ID: 1001)
[2020-07-27 09:20:56] [NOTICE] new standby "mypg-postgresql-ha-postgresql-0" (ID: 1000) has connected

The pg_stat_replication table is still empty after the switching.
The log of the pgpool-pod is

←[38;5;6mpgpool ←[38;5;5m09:20:20.48 ←[0m
←[38;5;6mpgpool ←[38;5;5m09:20:20.48 ←[0m←[1mWelcome to the Bitnami pgpool container←[0m
←[38;5;6mpgpool ←[38;5;5m09:20:20.48 ←[0mSubscribe to project updates by watching ←[1mhttps://github.com/bitnami/bitnami-docker-pgpool←[0m
←[38;5;6mpgpool ←[38;5;5m09:20:20.48 ←[0mSubmit issues and feature requests at ←[1mhttps://github.com/bitnami/bitnami-docker-pgpool/issues←[0m
←[38;5;6mpgpool ←[38;5;5m09:20:20.48 ←[0m
←[38;5;6mpgpool ←[38;5;5m09:20:20.49 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting Pgpool-II setup **
←[38;5;6mpgpool ←[38;5;5m09:20:20.51 ←[0m←[38;5;2mINFO ←[0m ==> Validating settings in PGPOOL_* env vars...
←[38;5;6mpgpool ←[38;5;5m09:20:20.53 ←[0m←[38;5;2mINFO ←[0m ==> Initializing Pgpool-II...
←[38;5;6mpgpool ←[38;5;5m09:20:20.53 ←[0m←[38;5;2mINFO ←[0m ==> Generating pg_hba.conf file...
←[38;5;6mpgpool ←[38;5;5m09:20:20.53 ←[0m←[38;5;2mINFO ←[0m ==> Generating pgpool.conf file...
←[38;5;6mpgpool ←[38;5;5m09:20:20.64 ←[0m←[38;5;2mINFO ←[0m ==> Generating password file for local authentication...
←[38;5;6mpgpool ←[38;5;5m09:20:20.65 ←[0m←[38;5;2mINFO ←[0m ==> Generating password file for pgpool admin user...
←[38;5;6mpgpool ←[38;5;5m09:20:20.65 ←[0m←[38;5;2mINFO ←[0m ==> ** Pgpool-II setup finished! **

←[38;5;6mpgpool ←[38;5;5m09:20:20.67 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting Pgpool-II **
2020-07-27 09:20:20: pid 1: LOG:  Backend status file /opt/bitnami/pgpool/logs/pgpool_status does not exist
2020-07-27 09:20:20: pid 1: LOG:  memory cache initialized
2020-07-27 09:20:20: pid 1: DETAIL:  memcache blocks :64
2020-07-27 09:20:20: pid 1: LOG:  pool_discard_oid_maps: discarded memqcache oid maps
2020-07-27 09:20:20: pid 1: LOG:  Setting up socket for 0.0.0.0:5432
2020-07-27 09:20:20: pid 1: LOG:  Setting up socket for :::5432
2020-07-27 09:20:20: pid 1: LOG:  find_primary_node_repeatedly: waiting for finding a primary node
2020-07-27 09:20:20: pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:20: pid 1: ERROR:  failed to make persistent db connection
2020-07-27 09:20:20: pid 1: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:20: pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2020-07-27 09:20:20: pid 1: LOG:  find_primary_node: primary node is 1
2020-07-27 09:20:20: pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:20: pid 148: ERROR:  failed to make persistent db connection
2020-07-27 09:20:20: pid 148: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:20: pid 148: LOG:  health check retrying on DB node: 0 (round:1)
2020-07-27 09:20:20: pid 146: LOG:  PCP process: 146 started
2020-07-27 09:20:20: pid 147: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:20: pid 147: ERROR:  failed to make persistent db connection
2020-07-27 09:20:20: pid 147: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:20: pid 1: LOG:  pgpool-II successfully started. version 4.1.2 (karasukiboshi)
2020-07-27 09:20:20: pid 1: LOG:  node status[0]: 0
2020-07-27 09:20:20: pid 1: LOG:  node status[1]: 1
2020-07-27 09:20:25: pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:25: pid 148: ERROR:  failed to make persistent db connection
2020-07-27 09:20:25: pid 148: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:25: pid 148: LOG:  health check retrying on DB node: 0 (round:2)
2020-07-27 09:20:28: pid 128: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:28: pid 128: LOG:  failed to create a backend 0 connection
2020-07-27 09:20:28: pid 128: DETAIL:  skip this backend because because failover_on_backend_error is off and we are in streaming replication mode and node is standby node
2020-07-27 09:20:28: pid 128: LOG:  master node 0 is down. Update master node to 1
2020-07-27 09:20:28: pid 1: WARNING:  child process with pid: 128 was terminated by segmentation fault
2020-07-27 09:20:28: pid 1: LOG:  fork a new child process with pid: 157
2020-07-27 09:20:28: pid 157: LOG:  failover or failback event detected
2020-07-27 09:20:28: pid 157: DETAIL:  restarting myself
2020-07-27 09:20:28: pid 1: LOG:  child process with pid: 157 exits with status 256
2020-07-27 09:20:28: pid 1: LOG:  fork a new child process with pid: 158
2020-07-27 09:20:30: pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:30: pid 148: ERROR:  failed to make persistent db connection
2020-07-27 09:20:30: pid 148: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:30: pid 148: LOG:  health check retrying on DB node: 0 (round:3)
2020-07-27 09:20:32: pid 115: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:32: pid 115: LOG:  failed to create a backend 0 connection
2020-07-27 09:20:32: pid 115: DETAIL:  skip this backend because because failover_on_backend_error is off and we are in streaming replication mode and node is standby node
2020-07-27 09:20:32: pid 1: WARNING:  child process with pid: 115 was terminated by segmentation fault
2020-07-27 09:20:32: pid 1: LOG:  fork a new child process with pid: 165
2020-07-27 09:20:32: pid 165: LOG:  failover or failback event detected
2020-07-27 09:20:32: pid 165: DETAIL:  restarting myself
2020-07-27 09:20:32: pid 1: LOG:  child process with pid: 165 exits with status 256
2020-07-27 09:20:32: pid 1: LOG:  fork a new child process with pid: 166
2020-07-27 09:20:35: pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:35: pid 148: ERROR:  failed to make persistent db connection
2020-07-27 09:20:35: pid 148: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:35: pid 148: LOG:  health check retrying on DB node: 0 (round:4)
2020-07-27 09:20:38: pid 116: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:38: pid 116: LOG:  failed to create a backend 0 connection
2020-07-27 09:20:38: pid 116: DETAIL:  skip this backend because because failover_on_backend_error is off and we are in streaming replication mode and node is standby node
2020-07-27 09:20:38: pid 1: WARNING:  child process with pid: 116 was terminated by segmentation fault
2020-07-27 09:20:38: pid 1: LOG:  fork a new child process with pid: 174
2020-07-27 09:20:38: pid 174: LOG:  failover or failback event detected
2020-07-27 09:20:38: pid 174: DETAIL:  restarting myself
2020-07-27 09:20:38: pid 1: LOG:  child process with pid: 174 exits with status 256
2020-07-27 09:20:38: pid 1: LOG:  fork a new child process with pid: 175
2020-07-27 09:20:40: pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:40: pid 148: ERROR:  failed to make persistent db connection
2020-07-27 09:20:40: pid 148: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:40: pid 148: LOG:  health check retrying on DB node: 0 (round:5)
2020-07-27 09:20:42: pid 175: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:42: pid 175: LOG:  failed to create a backend 0 connection
2020-07-27 09:20:42: pid 175: DETAIL:  skip this backend because because failover_on_backend_error is off and we are in streaming replication mode and node is standby node
2020-07-27 09:20:43: pid 175: LOG:  pool_reuse_block: blockid: 0
2020-07-27 09:20:43: pid 175: CONTEXT:  while searching system catalog, When relcache is missed
2020-07-27 09:20:43: pid 175: LOG:  failover or failback event detected
2020-07-27 09:20:43: pid 175: DETAIL:  restarting myself
2020-07-27 09:20:43: pid 1: LOG:  child process with pid: 175 exits with status 256
2020-07-27 09:20:43: pid 1: LOG:  fork a new child process with pid: 182
2020-07-27 09:20:45: pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2020-07-27 09:20:45: pid 148: ERROR:  failed to make persistent db connection
2020-07-27 09:20:45: pid 148: DETAIL:  connection to host:"mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432" failed
2020-07-27 09:20:45: pid 148: LOG:  health check failed on node 0 (timeout:0)
2020-07-27 09:20:45: pid 148: LOG:  received degenerate backend request for node_id: 0 from pid [148]
2020-07-27 09:20:45: pid 1: LOG:  Pgpool-II parent process has received failover request
2020-07-27 09:20:45: pid 1: LOG:  starting degeneration. shutdown host mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local(5432)
2020-07-27 09:20:45: pid 1: LOG:  Do not restart children because we are switching over node id 0 host: mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port: 5432 and we are in streaming replication mode
2020-07-27 09:20:45: pid 1: LOG:  execute command: echo ">>> Failover - that will initialize new primary node search!"
>>> Failover - that will initialize new primary node search!
2020-07-27 09:20:45: pid 1: LOG:  failover: set new primary node: 1
2020-07-27 09:20:45: pid 1: LOG:  failover: set new master node: 1
failover done. shutdown host mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local(5432)2020-07-27 09:20:45: pid 1: LOG:  failover done. shutdown host mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local(5432)
2020-07-27 09:20:45: pid 147: ERROR:  Failed to check replication time lag
2020-07-27 09:20:45: pid 147: DETAIL:  No persistent db connection for the node 0
2020-07-27 09:20:45: pid 147: HINT:  check sr_check_user and sr_check_password
2020-07-27 09:20:45: pid 147: CONTEXT:  while checking replication time lag
2020-07-27 09:20:45: pid 147: LOG:  worker process received restart request
2020-07-27 09:20:46: pid 146: LOG:  restart request received in pcp child process
2020-07-27 09:20:46: pid 1: LOG:  PCP child 146 exits with status 0 in failover()
2020-07-27 09:20:46: pid 1: LOG:  fork a new PCP child pid 184 in failover()
2020-07-27 09:20:46: pid 1: LOG:  worker child process with pid: 147 exits with status 256
2020-07-27 09:20:46: pid 184: LOG:  PCP process: 184 started
2020-07-27 09:20:46: pid 1: LOG:  fork a new worker child process with pid: 185
2020-07-27 09:20:48: pid 182: LOG:  failover or failback event detected
2020-07-27 09:20:48: pid 182: DETAIL:  restarting myself
2020-07-27 09:20:48: pid 166: LOG:  selecting backend connection
2020-07-27 09:20:48: pid 166: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:20:48: pid 1: LOG:  child process with pid: 182 exits with status 256
2020-07-27 09:20:48: pid 1: LOG:  fork a new child process with pid: 192
2020-07-27 09:20:57: pid 134: LOG:  failover or failback event detected
2020-07-27 09:20:57: pid 134: DETAIL:  restarting myself
2020-07-27 09:20:57: pid 1: LOG:  child process with pid: 134 exits with status 256
2020-07-27 09:20:57: pid 1: LOG:  fork a new child process with pid: 207
2020-07-27 09:21:13: pid 158: LOG:  selecting backend connection
2020-07-27 09:21:13: pid 158: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:21:18: pid 135: LOG:  failover or failback event detected
2020-07-27 09:21:18: pid 135: DETAIL:  restarting myself
2020-07-27 09:21:18: pid 1: LOG:  child process with pid: 135 exits with status 256
2020-07-27 09:21:18: pid 1: LOG:  fork a new child process with pid: 298
2020-07-27 09:21:48: pid 133: LOG:  failover or failback event detected
2020-07-27 09:21:48: pid 133: DETAIL:  restarting myself
2020-07-27 09:21:48: pid 1: LOG:  child process with pid: 133 exits with status 256
2020-07-27 09:21:48: pid 1: LOG:  fork a new child process with pid: 433
2020-07-27 09:22:00: pid 184: LOG:  forked new pcp worker, pid=508 socket=7
2020-07-27 09:22:00: pid 508: LOG:  received failback request for node_id: 0 from pid [508]
2020-07-27 09:22:00: pid 1: LOG:  Pgpool-II parent process has received failover request
2020-07-27 09:22:00: pid 1: LOG:  starting fail back. reconnect host mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local(5432)
2020-07-27 09:22:00: pid 1: LOG:  Node 1 is not down (status: 2)
2020-07-27 09:22:00: pid 184: LOG:  PCP process with pid: 508 exit with SUCCESS.
2020-07-27 09:22:00: pid 184: LOG:  PCP process with pid: 508 exits with status 0
2020-07-27 09:22:00: pid 1: LOG:  Do not restart children because we are failing back node id 0 host: mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local port: 5432 and we are in streaming replication mode and not all backends were down
2020-07-27 09:22:00: pid 1: LOG:  find_primary_node_repeatedly: waiting for finding a primary node
2020-07-27 09:22:00: pid 1: LOG:  find_primary_node: standby node is 0
2020-07-27 09:22:00: pid 1: LOG:  find_primary_node: primary node is 1
2020-07-27 09:22:00: pid 1: LOG:  failover: set new primary node: 1
2020-07-27 09:22:00: pid 1: LOG:  failover: set new master node: 0
2020-07-27 09:22:00: pid 1: LOG:  failback done. reconnect host mypg-postgresql-ha-postgresql-0.mypg-postgresql-ha-postgresql-headless.default.svc.cluster.local(5432)
2020-07-27 09:22:00: pid 185: LOG:  worker process received restart request
2020-07-27 09:22:01: pid 184: LOG:  restart request received in pcp child process
2020-07-27 09:22:01: pid 1: LOG:  PCP child 184 exits with status 0 in failover()
2020-07-27 09:22:01: pid 1: LOG:  fork a new PCP child pid 517 in failover()
2020-07-27 09:22:01: pid 1: LOG:  worker child process with pid: 185 exits with status 256
2020-07-27 09:22:01: pid 517: LOG:  PCP process: 517 started
2020-07-27 09:22:01: pid 1: LOG:  fork a new worker child process with pid: 518
2020-07-27 09:22:01: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:22:01: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:22:02: pid 166: LOG:  failover or failback event detected
2020-07-27 09:22:02: pid 166: DETAIL:  restarting myself
2020-07-27 09:22:02: pid 207: LOG:  selecting backend connection
2020-07-27 09:22:02: pid 207: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:22:02: pid 1: LOG:  child process with pid: 166 exits with status 256
2020-07-27 09:22:02: pid 1: LOG:  fork a new child process with pid: 526
2020-07-27 09:22:08: pid 158: LOG:  selecting backend connection
2020-07-27 09:22:08: pid 158: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:22:12: pid 192: LOG:  selecting backend connection
2020-07-27 09:22:12: pid 192: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:22:18: pid 433: LOG:  selecting backend connection
2020-07-27 09:22:18: pid 433: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:22:20: pid 130: LOG:  failover or failback event detected
2020-07-27 09:22:20: pid 130: DETAIL:  restarting myself
2020-07-27 09:22:20: pid 1: LOG:  child process with pid: 130 exits with status 256
2020-07-27 09:22:20: pid 1: LOG:  fork a new child process with pid: 578
2020-07-27 09:22:27: pid 125: LOG:  failover or failback event detected
2020-07-27 09:22:27: pid 125: DETAIL:  restarting myself
2020-07-27 09:22:27: pid 1: LOG:  child process with pid: 125 exits with status 256
2020-07-27 09:22:27: pid 1: LOG:  fork a new child process with pid: 594
2020-07-27 09:22:31: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:22:31: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:22:48: pid 117: LOG:  failover or failback event detected
2020-07-27 09:22:48: pid 117: DETAIL:  restarting myself
2020-07-27 09:22:48: pid 1: LOG:  child process with pid: 117 exits with status 256
2020-07-27 09:22:48: pid 1: LOG:  fork a new child process with pid: 662
2020-07-27 09:22:48: pid 298: LOG:  failover or failback event detected
2020-07-27 09:22:48: pid 298: DETAIL:  restarting myself
2020-07-27 09:22:48: pid 1: LOG:  child process with pid: 298 exits with status 256
2020-07-27 09:22:48: pid 1: LOG:  fork a new child process with pid: 663
2020-07-27 09:22:50: pid 118: LOG:  selecting backend connection
2020-07-27 09:22:50: pid 118: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:22:58: pid 145: LOG:  failover or failback event detected
2020-07-27 09:22:58: pid 145: DETAIL:  restarting myself
2020-07-27 09:22:58: pid 1: LOG:  child process with pid: 145 exits with status 256
2020-07-27 09:22:58: pid 1: LOG:  fork a new child process with pid: 699
2020-07-27 09:23:01: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:23:01: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:23:31: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:23:31: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:24:01: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:24:01: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:24:31: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:24:31: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:24:40: pid 121: LOG:  failover or failback event detected
2020-07-27 09:24:40: pid 121: DETAIL:  restarting myself
2020-07-27 09:24:40: pid 1: LOG:  child process with pid: 121 exits with status 256
2020-07-27 09:24:40: pid 1: LOG:  fork a new child process with pid: 1027
2020-07-27 09:25:01: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:25:01: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:25:12: pid 144: LOG:  selecting backend connection
2020-07-27 09:25:12: pid 144: DETAIL:  failover or failback event detected, discarding existing connections
2020-07-27 09:25:20: pid 114: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 114: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 120: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 120: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 136: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 136: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 123: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 123: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 124: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 124: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 132: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 132: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 127: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 127: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 142: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 142: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 143: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 143: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 137: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 137: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 140: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 140: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 114 exits with status 256
2020-07-27 09:25:20: pid 129: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 129: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 139: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 139: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 122: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 122: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 138: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 138: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 131: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 131: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 126: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 126: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 141: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 141: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1151
2020-07-27 09:25:20: pid 119: LOG:  failover or failback event detected
2020-07-27 09:25:20: pid 119: DETAIL:  restarting myself
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 123 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1152
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 124 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1153
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 132 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1154
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 136 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1155
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 140 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1156
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 142 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1157
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 141 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1158
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 120 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1159
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 127 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1160
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 122 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1161
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 119 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1162
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 138 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1163
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 126 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1164
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 129 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1165
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 131 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1166
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 137 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1167
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 139 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1168
2020-07-27 09:25:20: pid 1: LOG:  child process with pid: 143 exits with status 256
2020-07-27 09:25:20: pid 1: LOG:  fork a new child process with pid: 1169
2020-07-27 09:25:31: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:25:31: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:26:01: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:26:01: pid 518: CONTEXT:  while checking replication time lag
2020-07-27 09:26:31: pid 518: LOG:  get_query_result falied: status: -2
2020-07-27 09:26:31: pid 518: CONTEXT:  while checking replication time lag

Thanks for your help

BenchmarkingBuffalo on 27 Jul 2020

👍2

Hi @BenchmarkingBuffalo

Just to let you know that I am also actively investigating this. Should I find any information, I will keep post it here so as to keep you updated.

Thanks!

joancafom on 28 Jul 2020

@joancafom I am running into the exact same issue as described here. pg_stat_replication is empty on the new master and that makes pgpool fail as it queries the replication time tag (the get_query_result failure is from this). https://git.postgresql.org/gitweb/?p=pgpool2.git;a=blob;f=src/streaming_replication/pool_worker_child.c;hb=d7c7578148f0026bbe439f8920c62cb809206466#l392

Is there any ssh connectivity needed between the pods for repmgr commands to work ?

anandgia on 31 Jul 2020

I also replicated the bug, and besides the potential data loss, the fact that pgpool fails during the new master creation makes this not HA for pgpool clients.

antonin07130 on 3 Aug 2020

Thanks for your patience, we are actively working internally to reproduce and solve this issue in the next 2-3 weeks. We do not have a large team and part of it is on vacation this month.

We are aware this issue is important for keeping the HA for the master node, we will keep you posted with our findings. Any help with more info that could help to figure out this issue or possible PRs to the PostgreSQL container or the Chart will be welcome.

beltran-rubo on 5 Aug 2020

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

stale[bot] on 21 Aug 2020

This issue is not stale, we saw the same two days ago on latest chart and image versions.

dominik-bln on 22 Aug 2020

👍1

Hi,
There was a release of postgresql-repmgr container that should fix this issue. This image is used in v3.5.6 of postgresql-ha chart.

Could you give it a try and see if this solves the issue for you ?

rafariossaa on 24 Aug 2020

Hi,
I used the steps described above and I was not able to reproduce the problem anymore.
I am going to close this ticket now. If someone else still runs into this problem, he or she can comment and I will reopen the issue.
Thank you very much for fixing this.
Best regards,
The Buffalo

BenchmarkingBuffalo on 24 Aug 2020

Hi @BenchmarkingBuffalo ,
Thanks for checking the issue is solved. Please, don't hesitate to come back if you have any issue.

rafariossaa on 25 Aug 2020

There was a release of postgresql-repmgr container that should fix this issue. This image is used in v3.5.6 of postgresql-ha chart.

Could you give it a try and see if this solves the issue for you ?

Ah ok, then we probably missed this by a bit. I'll try to find some time to test this as well.

Does anyone know the underlying cause? The history of the bitnami-docker-postgresql-repmgr repository doesn't really tell much.

dominik-bln on 26 Aug 2020

Hi,
Thanks for asking, can you check this commit ? The change was in the file `librepmgr.sh'.
Our CI/CD produces daily releases and sometime is difficult to see what has changed.

rafariossaa on 26 Aug 2020

Hello,

There is an error appeared with the following scenario: after deleting and restarting POD from master node, replication stopped to work. And after restarting the deleted POD, both of nodes became primary ones.
Log of module ha-pgpool:

2020-09-29 07:13:08: pid 313: CONTEXT:  while checking replication time lag
2020-09-29 07:13:09: pid 1: LOG:  child process with pid: 12066 exits with status 256
2020-09-29 07:13:09: pid 1: LOG:  fork a new child process with pid: 13588
2020-09-29 07:13:11: pid 1: LOG:  child process with pid: 11287 exits with status 256
2020-09-29 07:13:11: pid 1: LOG:  fork a new child process with pid: 13595
2020-09-29 07:13:16: pid 1: LOG:  child process with pid: 12647 exits with status 256
2020-09-29 07:13:16: pid 1: LOG:  fork a new child process with pid: 13603
2020-09-29 07:13:19: pid 1: LOG:  child process with pid: 11681 exits with status 256
2020-09-29 07:13:19: pid 1: LOG:  fork a new child process with pid: 13621
2020-09-29 07:13:31: pid 1: LOG:  child process with pid: 12083 exits with status 256
2020-09-29 07:13:31: pid 1: LOG:  fork a new child process with pid: 13660
2020-09-29 07:13:38: pid 313: LOG:  get_query_result falied: status: -2
2020-09-29 07:13:38: pid 313: CONTEXT:  while checking replication time lag
2020-09-29 07:14:08: pid 313: LOG:  get_query_result falied: status: -2
2020-09-29 07:14:08: pid 313: CONTEXT:  while checking replication time lag
2020-09-29 07:14:09: pid 1: LOG:  child process with pid: 12316 exits with status 256
2020-09-29 07:14:09: pid 1: LOG:  fork a new child process with pid: 13776
2020-09-29 07:14:31: pid 1: LOG:  child process with pid: 10639 exits with status 256
2020-09-29 07:14:31: pid 1: LOG:  fork a new child process with pid: 13847
2020-09-29 07:14:38: pid 313: LOG:  get_query_result falied: status: -2
2020-09-29 07:14:38: pid 313: CONTEXT:  while checking replication time lag
2020-09-29 07:14:49: pid 1: LOG:  child process with pid: 11573 exits with status 256
2020-09-29 07:14:49: pid 1: LOG:  fork a new child process with pid: 13904

Result of cluster show:

I have no name!@postgres-postgresql-ha-postgresql-1:/$ /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 07:19:38.29
postgresql-repmgr 07:19:38.29 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 07:19:38.29 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 07:19:38.30 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 07:19:38.30

 ID   | Name                                | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string                                                                                                            
------+-------------------------------------+---------+-----------+----------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1000 | postgres-postgresql-ha-postgresql-0 | primary | ! running |          | default  | 100      | 1        | user=repmgr password=dg1WI52b4j host=postgres-postgresql-ha-postgresql-0.postgres-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
 1001 | postgres-postgresql-ha-postgresql-1 | primary | * running |          | default  | 100      | 2        | user=repmgr password=dg1WI52b4j host=postgres-postgresql-ha-postgresql-1.postgres-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5

WARNING: following issues were detected
  - node "postgres-postgresql-ha-postgresql-0" (ID: 1000) is running but the repmgr node record is inactive

I have no name!@postgres-postgresql-ha-postgresql-0:/$ /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 07:19:13.45
postgresql-repmgr 07:19:13.45 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 07:19:13.47 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 07:19:13.47 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 07:19:13.48

 ID   | Name                                | Role    | Status               | Upstream | Location | Priority | Timeline | Connection string                                                                                                 
------+-------------------------------------+---------+----------------------+----------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1000 | postgres-postgresql-ha-postgresql-0 | primary | * running            |          | default  | 100      | 1        | user=repmgr password=dg1WI52b4j host=postgres-postgresql-ha-postgresql-0.postgres-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
 1001 | postgres-postgresql-ha-postgresql-1 | standby | ! running as primary |          | default  | 100      | 2        | user=repmgr password=dg1WI52b4j host=postgres-postgresql-ha-postgresql-1.postgres-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5

WARNING: following issues were detected
  - node "postgres-postgresql-ha-postgresql-1" (ID: 1001) is registered as standby but running as primary

VyacheslavSemin on 29 Sep 2020

Hi @VyacheslavSemin,
Could you please open a new issue since it seems to be different from this one, please?
You can add that information to the new issue and how we can reproduce it.
Regards.

miguelaeh on 29 Sep 2020

Hi @VyacheslavSemin,
Could you please open a new issue since it seems to be different from this one, please?
You can add that information to the new issue and how we can reproduce it.
Regards.

I opened a new issue #3806

VyacheslavSemin on 29 Sep 2020

Was this page helpful?

0 / 5 - 0 ratings