patroni and etcd restart after configuration changes

[Tekchanddagar]

Hello Team,
Currently we are using single node etcd in our patroni. Tomorrow morning we are going to made it 3 etcd node cluster. But we don't want loose the data in postgresql.

I have few question so i can make changes easily :
1 First of all we need to create 3 etcd node, we want to use existing etcd node also then we need to create 2 more etcd node. Now we need to remove the data directory from existing etcd node or it will join the cluster and need to restart etcd service or reload will work on all 3 etcd node?

1. Now we will make change in patroni.yml file also From

     host: etcd:2379

To

etcd:
      hosts: etcd1:2379,etcd2:2379,etcd3:2379

Now we need to restart the patroni or reload will work?

Please help me so we can do it with minimum downtime or no downtime.

Thanks.

[vitabaks]

Do not delete data!
etcd cluster scales very easily and without downtime. It doesn’t need to stop services.

Please read the documentation before working in a production environment:
https://etcd.io/docs/v3.3.12/op-guide/

Specifically for your task - "Add a new member":
https://etcd.io/docs/v3.3.12/op-guide/runtime-configuration/#add-a-new-member

Add a second node to the etcd cluster, make sure that the cluster is in a healthy state (after adding the node), add a third node, check the status of the etcd cluster.
Next, add a list of all nodes to the configuration file on each server of the patroni cluster, and perform a reload of patroni service.

Please test in your test environment before using in a production.

[Tekchanddagar]

@vitabaks,

Thank you for your response.

I am little bit confuse with doc as it not specify which command to run on which node means which command will be run over existing etcd node and which command will be run over new node which we need to add cluster.

First Part:

etcdctl member add infra3 --peer-urls=http://10.0.1.13:2380

above command will be run over existing etcd node with my new etcd node values like name and IP.

But where we need to run below command?

$ export ETCD_NAME="infra3"
$ export ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"
$ export ETCD_INITIAL_CLUSTER_STATE=existing
$ etcd --listen-client-urls http://10.0.1.13:2379 --advertise-client-urls http://10.0.1.13:2379 --listen-peer-urls http://10.0.1.13:2380 --initial-advertise-peer-urls http://10.0.1.13:2380 --data-dir %data_dir%

I will replace the name and IP's.
One more question, does these command add the config in /etc/default/etcd file on new node?

The configuration on my existing etcd node is:

ETCD_LISTEN_PEER_URLS="http://192.168.56.3:2380,http://127.0.0.1:7001"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379, http://192.168.56.3:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.56.3:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.56.3:2380,"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.56.3:2379"
ETCD_INITIAL_CLUSTER_TOKEN="cluster1"
ETCD_INITIAL_CLUSTER_STATE="new"

If possible can you please give some example config for all 3 node.
Please help me because we need to complete this activity by 11 am IST and its prod environment.

Currently i am testing in my test environment.

Thanks.

[Tekchanddagar]

@vitabaks,

Can you please help me?

Thanks.

[Tekchanddagar]

@vitabaks,
I have tried to follow the doc but no success.
We postponed this activity for tomorrow morning i.e 10/11/2019 at 9 AM IST.

But i tried different approach but its have downtime and little bit lengthy process.
Below are the steps which i need to follow:

1. My current working single etcd node have below configuration:

ETCD_LISTEN_PEER_URLS="http://192.168.56.3:2380,http://127.0.0.1:7001"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379, http://192.168.56.3:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.56.3:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.56.3:2380,"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.56.3:2379"
ETCD_INITIAL_CLUSTER_TOKEN="cluster1"
ETCD_INITIAL_CLUSTER_STATE="new"

i need to replace the above config with below config over all 3 node with small changes like hostname and IP:

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.56.3:2380,etcd2=http://192.168.56.4:2380,etcd3=http://192.168.56.5:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="cluster1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.56.4:2380"
ETCD_DATA_DIR="/var/lib/etcd/default"
ETCD_LISTEN_PEER_URLS="http://192.168.56.4:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.56.4:2379,http://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.56.4:2379"
ETCD_NAME="etcd2"

1. Now i need to remove the data from /var/lib/etcd/default/member/snap and /var/lib/etcd/default/member/wal file from current working etcd node.
2. Now need to start etcd service on all 3 etcd node and one node will work as leader where i start etcd service first but from here my application will be down.
3. Now need to specify all etcd node in patroni.yml file on master and slave node.
4. Need to restart patroni service on patroni nodes.

Please check my etcd file config is it okay and we can go with it?

Confirmation on above config or new approach guidance will be very helpful.

Thanks.

[vitabaks]

But i tried different approach but its have downtime and little bit lengthy process.

It is not right. You completely destroy your current cluster etcd, and create a new one with three nodes. This approach will require downtime.

You just need to join the new etcd nodes into your current cluster.
You specified ETCD_INITIAL_CLUSTER_STATE = "new" for the new node, but you must specify "existing".

1. On the node etcd1 run this command:
   etcdctl member add etcd2 http://192.168.56.4:2380

etcdctl will inform the cluster of the new member and print out the variables (ETCD_NAME, ETCD_INITIAL_CLUSTER, ETCD_INITIAL_CLUSTER_STATE) that must be specified in the etcd configuration file on the new node.

1. Copy the configuration file from etcd1 on etcd2 node, replace the ip address, replace the variables (that you received from etcdctl after adding the node to the cluster).
2. Run the etcd process on the new node (etcd2)

3. Check the status of the etcd cluster.

4. If the cluster is healthy, repeat the steps for node etcd3.

5. Check the status of the etcd cluster.

6. Update the configuration on patroni servers, and reload the patroni service.

But, first test on your test environment.

[vitabaks]

Also note the parameters ETCD_ELECTION_TIMEOUT and ETCD_HEARTBEAT_INTERVAL. The default value is low. See tuning guide.

There are quite a lot of things to consider if you want to create a really robust cluster.
If you deploy your cluster manually, especially if you have not had experience with this yet, there are very high risks of making a mistake in configuring your cluster, because of which it may not work as stable as you expect from it ...

I am developing fully automated solution that will deploy a PostgreSQL high availability cluster (based on "Patroni" and "DCS(etcd)") on dedicated physical servers or virtual machines.
I recommend that you see on this tool. Perhaps it will be useful to you!

[Tekchanddagar]

@vitabaks,
Thank you for your response.

1. On the node etcd1 run this command:
   http://192.168.56.4:2380
   I run the above command on etcd1 but getting error:

root@etcd1:~# etcdctl member add etcd2 --peer-urls=http://192.168.56.4:2380
Incorrect Usage: flag provided but not defined: -peer-urls

NAME:
   etcdctl member add - add a new member to the etcd cluster

USAGE:
   etcdctl member add <name> <peerURL>
flag provided but not defined: -peer-urls

When i run same command after removing --peer-urls= parts its working and giving output

ETCD_NAME="etcd2"
ETCD_INITIAL_CLUSTER="etcd2=http://192.168.56.4:2380,etcd1=http://192.168.56.3:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Then i copy the etcd1 configuration on etcd2 and update the variable like below:

ETCD_NAME="etcd2"
ETCD_INITIAL_CLUSTER="etcd2=192.168.56.4:2380,etcd1=http://192.168.56.3:2380"
ETCD_LISTEN_PEER_URLS="http://192.168.56.4:2380,http://127.0.0.1:7001"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379, http://192.168.56.4:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.56.4:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.56.4:2380,"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.56.4:2379"
ETCD_INITIAL_CLUSTER_TOKEN="cluster1"
ETCD_INITIAL_CLUSTER_STATE="existing"

and start the etcd service on etcd2 node and check the status of cluster health on etcd1:

root@etcd1:~# etcdctl cluster-health
member 1f5bd8ecacec540c is unreachable: no available published client urls
member 74c91bce4b94fc8e is unhealthy: got unhealthy result from http://192.168.56.3:2379
cluster is unhealthy

Getting below error log on etcd2 node:

Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://192.168.56.4:2379
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/default
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.56.4:2380
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd1=http://192.168.56.4:2380,
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=cluster1
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://127.0.0.1:2379, http://192.168.56.4:2379
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://192.168.56.4:2380,http://127.0.0.1:7001
Nov  9 15:15:03 etcd2 etcd[4378]: recognized and used environment variable ETCD_NAME=etcd2
Nov  9 15:15:03 etcd2 etcd[4378]: etcd Version: 3.2.17
Nov  9 15:15:03 etcd2 etcd[4378]: Git SHA: Not provided (use ./build instead of go build)
Nov  9 15:15:03 etcd2 etcd[4378]: Go Version: go1.10
Nov  9 15:15:03 etcd2 etcd[4378]: Go OS/Arch: linux/amd64
Nov  9 15:15:03 etcd2 etcd[4378]: setting maximum number of CPUs to 1, total number of available CPUs is 1
Nov  9 15:15:03 etcd2 etcd[4378]: the server is already initialized as member before, starting as etcd member...
Nov  9 15:15:03 etcd2 etcd[4378]: listening for peers on http://127.0.0.1:7001
Nov  9 15:15:03 etcd2 etcd[4378]: listening for peers on http://192.168.56.4:2380
Nov  9 15:15:03 etcd2 etcd[4378]: listening for client requests on 127.0.0.1:2379
Nov  9 15:15:03 etcd2 etcd[4378]: listening for client requests on 192.168.56.4:2379
Nov  9 15:15:03 etcd2 etcd[4378]: couldn't find local name "etcd2" in the initial cluster configuration
Nov  9 15:15:03 etcd2 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Nov  9 15:15:03 etcd2 systemd[1]: etcd.service: Failed with result 'exit-code'.
Nov  9 15:15:03 etcd2 systemd[1]: Failed to start etcd - highly-available key value store.

Please help.

[vitabaks]

Are you sure you did not start the service with the parameter ETCD_INITIAL_CLUSTER_STATE = "new" before that?
If so, you must first clear the etcd directory on the node being added (etcd2).

[vitabaks]

ETCD_INITIAL_CLUSTER="etcd2=192.168.56.4:2380,etcd1=http://192.168.56.3:2380"

ETCD_INITIAL_CLUSTER="etcd2=http://192.168.56.4:2380,etcd1=http://192.168.56.3:2380"

http:// and you indicated ETCD_INITIAL_CLUSTER twice.

Try this configuration:

ETCD_NAME="etcd2"
ETCD_INITIAL_CLUSTER="etcd2=http://192.168.56.4:2380,etcd1=http://192.168.56.3:2380"
ETCD_LISTEN_PEER_URLS="http://192.168.56.4:2380,http://127.0.0.1:7001"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379, http://192.168.56.4:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.56.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.56.4:2379"
ETCD_INITIAL_CLUSTER_TOKEN="cluster1"
ETCD_INITIAL_CLUSTER_STATE="existing"

[vitabaks]

Here is another configuration example:

postgres@edb-pgnode02:~$ cat /etc/etcd/etcd.conf

ETCD_NAME="edb-pgnode02"
ETCD_LISTEN_CLIENT_URLS="http://172.30.52.26:2379,http://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://172.30.52.26:2379"
ETCD_LISTEN_PEER_URLS="http://172.30.52.26:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.30.52.26:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcdCluster"
ETCD_INITIAL_CLUSTER="edb-pgnode01=http://172.30.52.25:2380,edb-pgnode02=http://172.30.52.26:2380,edb-pgnode03=http://172.30.52.27:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_ELECTION_TIMEOUT="5000"
ETCD_HEARTBEAT_INTERVAL="1000"

[Tekchanddagar]

@vitabaks,

Thank you and take a bow.

ETCD_NAME="etcd2"
ETCD_INITIAL_CLUSTER="etcd2=http://192.168.56.4:2380,etcd1=http://192.168.56.3:2380"
ETCD_LISTEN_PEER_URLS="http://192.168.56.4:2380,http://127.0.0.1:7001"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379, http://192.168.56.4:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.56.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.56.4:2379"
ETCD_INITIAL_CLUSTER_TOKEN="cluster1"
ETCD_INITIAL_CLUSTER_STATE="existing"

its work like charm. Now all 3 node joined the cluster.

Now i made the changes in patroni.yml file and try to reload the patroni service using below command and getting error

root@slave1:/etc/default# patronictl reload tcd  slave1
Error: 'Can not find suitable configuration of distributed configuration store\nAvailable implementations: etcd'

tcd is my cluster name. Error is pointing something is missing in my dcs config part.
Below is the some config part of my patroni.yml:

root@slave1:/etc/default# cat /etc/patroni.yml
scope: tcd
namespace: /db/
name: slave1

restapi:
    listen: 192.168.56.9:8008
    connect_address: 192.168.56.9:8008

etcd:
    hosts: 192.168.56.3:2379,192.168.56.4:2379,192.168.56.5:2379
bootstrap:
    dcs:
        ttl: 30
        loop_wait: 10
        retry_timeout: 10
        maximum_lag_on_failover: 1048576
        postgresql:
            use_pg_rewind: true

What are the other parameter we should define in dcs? Because when slave node lag then it not sync with master and we us reinit for base backup then both node sync again.

Please suggest we will restart our patroni service if required after making changes in dcs part.
We need to use below variable and setting all 3 etcd node or over 2 node which join the cluster later:

ETCD_ELECTION_TIMEOUT="5000"
ETCD_HEARTBEAT_INTERVAL="1000"

Please guide it is last obstacle in my project

[Tekchanddagar]

@vitabaks, Thanks a lot for your helping hands.
Able to reload the patroni service using below command:

patronictl -c /etc/patroni.yml reload tcd slave1

If you see any wrong config in my patroni.yml file in my earlier post please point me in right direction so i can made the changes.

Thanks.

[vitabaks]

Because when slave node lag then it not sync with master and we us reinit for base backup then both node sync again.

This information is not enough to give you recommendations. If a similar problem occurs again, I suggest you open a new issue and describe the problem in detail, and attach logs.

Please suggest we will restart our patroni service if required after making changes in dcs part.
We need to use below variable and setting all 3 etcd node or over 2 node which join the cluster later:

There is no need to restart the patroni service.

If you see any wrong config in my patroni.yml file

I do not see a problem in what you showed.

Now all 3 node joined the cluster.

Good! As I understand it, everything is working for you, and your main problem has been solved.