Etcd setup¶

In our solutions, we use etcd distributed configuration store. Refresh your knowledge about etcd.

etcd relies heavily on Raft consensus heartbeats. If the system clocks drift by more than a fraction of a second, it causes problems. Sync them using NTP.

Install etcd¶

Use etcd under /opt/axdb/axdb-etcd/ (when <axdb-dir> is /opt/axdb/) on all etcd nodes: node1, node2 and node3.

Create a dedicated system user for the etcd background process on every node:

$ sudo groupadd --system etcd
$ sudo useradd --system --gid etcd --home-dir /var/lib/etcd --shell /sbin/nologin etcd
$ sudo mkdir -p /etc/etcd /var/lib/etcd
$ sudo chown -R etcd:etcd /etc/etcd /var/lib/etcd

This file allows systemd to start, stop, restart, and manage the etcd service. This includes handling dependencies, monitoring the service, and ensuring it runs as expected.

/etc/systemd/system/etcd.service

[Unit]
After=network.target
Description=etcd - highly-available key value store

[Service]
LimitNOFILE=65536
Restart=on-failure
Type=notify
ExecStart=/opt/axdb/axdb-etcd/bin/etcd --config-file /etc/etcd/etcd.conf.yaml
User=etcd
Group=etcd

[Install]
WantedBy=multi-user.target

Configure etcd¶

To get started with etcd cluster, you need to bootstrap it. This means setting up the initial configuration and starting the etcd nodes so they can form a cluster. There are the following bootstrapping mechanisms:

Static in the case when the IP addresses of the cluster nodes are known
Discovery service - for cases when the IP addresses of the cluster are not known ahead of time.

Since we know the IP addresses of the nodes, we will use the static method. For using the discovery service, please refer to the etcd documentation .

We will configure and start all etcd nodes in parallel.

Modify the configuration file¶

Create the etcd configuration file on every node. You can edit the sample configuration file /etc/etcd/etcd.conf.yaml or create your own one. Replace the node names and IP addresses with the actual names and IP addresses of your nodes. Make sure this file is owned by etcd user and group.

node1node2node3

/etc/etcd/etcd.conf.yaml

name: 'node1'
initial-cluster-token: PostgreSQL_HA_Cluster_1
initial-cluster-state: new
initial-cluster: node1=http://192.168.3.201:2380,node2=http://192.168.3.202:2380,node3=http://192.168.3.203:2380
data-dir: /var/lib/etcd
initial-advertise-peer-urls: http://192.168.3.201:2380 
listen-peer-urls: http://192.168.3.201:2380
advertise-client-urls: http://192.168.3.201:2379
listen-client-urls: http://192.168.3.201:2379

/etc/etcd/etcd.conf.yaml

name: 'node2'
initial-cluster-token: PostgreSQL_HA_Cluster_1
initial-cluster-state: new
initial-cluster: node1=http://192.168.3.201:2380,node2=http://192.168.3.202:2380,node3=http://192.168.3.203:2380
data-dir: /var/lib/etcd
initial-advertise-peer-urls: http://192.168.3.202:2380 
listen-peer-urls: http://192.168.3.202:2380
advertise-client-urls: http://192.168.3.202:2379
listen-client-urls: http://192.168.3.202:2379

/etc/etcd/etcd.conf.yaml

name: 'node3'
initial-cluster-token: PostgreSQL_HA_Cluster_1
initial-cluster-state: new
initial-cluster: node1=http://192.168.3.201:2380,node2=http://192.168.3.202:2380,node3=http://192.168.3.203:2380
data-dir: /var/lib/etcd
initial-advertise-peer-urls: http://192.168.3.203:2380 
listen-peer-urls: http://192.168.3.203:2380
advertise-client-urls: http://192.168.3.203:2379
listen-client-urls: http://192.168.3.203:2379

Enable and start the etcd service on all nodes:
```
$ sudo systemctl daemon-reload
$ sudo systemctl enable etcd
$ sudo systemctl start etcd
$ sudo systemctl status etcd
```
During the node start, etcd searches for other cluster nodes defined in the configuration. If the other nodes are not yet running, the start may fail by a quorum timeout. This is expected behavior. Try starting all nodes again at the same time for the etcd cluster to be created.
Check the etcd cluster members. Use etcdctl for this purpose. Ensure that etcdctl interacts with etcd using API version 3 and knows which nodes, or endpoints, to communicate with. For this, we will define the required information as environment variables. Run the following commands on one of the nodes:
```
export ETCDCTL_API=3
HOST_1=192.168.3.201
HOST_2=192.168.3.202
HOST_3=192.168.3.203
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
```

Now, list the cluster members and output the result as a table as follows:

$ sudo /opt/axdb/axdb-etcd/bin/etcdctl --endpoints=$ENDPOINTS -w table member list

Sample output

+------------------+---------+-------+---------------------------+---------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS        |        CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-------+---------------------------+---------------------------+------------+
| 4788684035f976d3 | started | node2 | http://192.168.3.202:2380 | http://192.168.3.202:2379 |      false |
| 67684e355c833ffa | started | node3 | http://192.168.3.203:2380 | http://192.168.3.203:2379 |      false |
| 9d2e318af9306c67 | started | node1 | http://192.168.3.201:2380 | http://192.168.3.201:2379 |      false |
+------------------+---------+-------+---------------------------+---------------------------+------------+

To check what node is currently the leader, use the following command

$ sudo /opt/axdb/axdb-etcd/bin/etcdctl --endpoints=$ENDPOINTS -w table endpoint status

Sample output

+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.3.201:2379 | 9d2e318af9306c67 |  3.5.30 |   20 kB |      true |      false |         2 |         10 |                 10 |        |
| 192.168.3.202:2379 | 4788684035f976d3 |  3.5.30 |   20 kB |     false |      false |         2 |         10 |                 10 |        |
| 192.168.3.203:2379 | 67684e355c833ffa |  3.5.30 |   20 kB |     false |      false |         2 |         10 |                 10 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

To check the health of the etcd nodes, use the following command

$ sudo /opt/axdb/axdb-etcd/bin/etcdctl --endpoints=$ENDPOINTS -w table endpoint health

Sample output

+--------------------+--------+------------+-------+
|      ENDPOINT      | HEALTH |    TOOK    | ERROR |
+--------------------+--------+------------+-------+
| 192.168.3.201:2379 |   true | 2.208412ms |       |
| 192.168.3.202:2379 |   true | 2.783703ms |       |
| 192.168.3.203:2379 |   true | 48.64194ms |       |
+--------------------+--------+------------+-------+

Next steps¶

Patroni setup