SRE文档

深浅模式

1.配置文件集群 ​

1.1下载 ​

https://github.com/etcd-io/etcd/releases/download/v3.4.24/etcd-v3.4.24-linux-amd64.tar.gz

1.2配置 ​

  • 三节点信息
主机名IP地址
sht-sgmhadoopdn-01172.16.101.58
sht-sgmhadoopdn-02172.16.101.59
sht-sgmhadoopdn-03172.16.101.60
  • 各节点添加hosts文件解析
bash
# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.101.58    sht-sgmhadoopdn-01
172.16.101.59    sht-sgmhadoopdn-02
172.16.101.60    sht-sgmhadoopdn-03
# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.101.58    sht-sgmhadoopdn-01
172.16.101.59    sht-sgmhadoopdn-02
172.16.101.60    sht-sgmhadoopdn-03
  • 创建运行etcd的用户和组
bash
# groupadd -r dba
# useradd -r -g dba -G root tnuser
# groupadd -r dba
# useradd -r -g dba -G root tnuser
  • 下载解压etcd
bash
wget https://github.com/etcd-io/etcd/releases/download/v3.4.24/etcd-v3.4.24-linux-amd64.tar.gz

#mkdir data dir
mkdir /data/etcd/data
wget https://github.com/etcd-io/etcd/releases/download/v3.4.24/etcd-v3.4.24-linux-amd64.tar.gz

#mkdir data dir
mkdir /data/etcd/data
  • 新建etcd配置文件etcd.conf，各节点配置文件内容如下
bash
#sht-sgmhadoopdn-01

# cat /usr/local/etcd/etcd.conf 
ETCD_NAME="sht-sgmhadoopdn-01"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-01:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-01:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380"
ETCD_ENABLE_V2="true"
#sht-sgmhadoopdn-01

# cat /usr/local/etcd/etcd.conf 
ETCD_NAME="sht-sgmhadoopdn-01"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-01:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-01:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380"
ETCD_ENABLE_V2="true"
bash
#sht-sgmhadoopdn-02

# cat /usr/local/etcd/etcd.conf 
ETCD_NAME="sht-sgmhadoopdn-02"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-02:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-02:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380"
#sht-sgmhadoopdn-02

# cat /usr/local/etcd/etcd.conf 
ETCD_NAME="sht-sgmhadoopdn-02"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-02:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-02:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380"
bash
#sht-sgmhadoopdn-03

# cat /usr/local/etcd/etcd.conf 
ETCD_NAME="sht-sgmhadoopdn-03"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-02:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-03:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380"
#sht-sgmhadoopdn-03

# cat /usr/local/etcd/etcd.conf 
ETCD_NAME="sht-sgmhadoopdn-03"
ETCD_DATA_DIR="/data/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-02:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-03:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380"
  • etcd配置系统服务，各节点的etcd.service文件内容如下
bash
# cat /usr/lib/systemd/system/etcd.service 
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=tnuser
Type=notify
EnvironmentFile=/usr/local/etcd/etcd.conf
WorkingDirectory=/usr/local/etcd
ExecStart=/usr/local/etcd/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
# cat /usr/lib/systemd/system/etcd.service 
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=tnuser
Type=notify
EnvironmentFile=/usr/local/etcd/etcd.conf
WorkingDirectory=/usr/local/etcd
ExecStart=/usr/local/etcd/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 各节点启动etcd服务
# systemctl daemon-reload
# systemctl start etcd
# systemctl daemon-reload
# systemctl start etcd
  • 查看集群信息
bash
# etcdctl endpoint status --endpoints http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379 --write-out=table
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|            ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://sht-sgmhadoopdn-01:2379 | 44d8bc3300880bcd |   3.4.1 |   16 kB |      true |      false |         8 |         11 |                 11 |        |
| http://sht-sgmhadoopdn-02:2379 | e9136c1ad1754783 |   3.4.1 |   16 kB |     false |      false |         8 |         11 |                 11 |        |
| http://sht-sgmhadoopdn-03:2379 | d446fbe3296eb85a |   3.4.1 |   16 kB |     false |      false |         8 |         11 |                 11 |        |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

# etcdctl endpoint health --endpoints http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379 --write-out=table
+--------------------------------+--------+------------+-------+
|            ENDPOINT            | HEALTH |    TOOK    | ERROR |
+--------------------------------+--------+------------+-------+
| http://sht-sgmhadoopdn-02:2379 |   true |  5.23352ms |       |
| http://sht-sgmhadoopdn-01:2379 |   true | 5.832925ms |       |
| http://sht-sgmhadoopdn-03:2379 |   true | 6.431018ms |       |
+--------------------------------+--------+------------+-------+


# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
# etcdctl endpoint status --endpoints http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379 --write-out=table
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|            ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://sht-sgmhadoopdn-01:2379 | 44d8bc3300880bcd |   3.4.1 |   16 kB |      true |      false |         8 |         11 |                 11 |        |
| http://sht-sgmhadoopdn-02:2379 | e9136c1ad1754783 |   3.4.1 |   16 kB |     false |      false |         8 |         11 |                 11 |        |
| http://sht-sgmhadoopdn-03:2379 | d446fbe3296eb85a |   3.4.1 |   16 kB |     false |      false |         8 |         11 |                 11 |        |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

# etcdctl endpoint health --endpoints http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379 --write-out=table
+--------------------------------+--------+------------+-------+
|            ENDPOINT            | HEALTH |    TOOK    | ERROR |
+--------------------------------+--------+------------+-------+
| http://sht-sgmhadoopdn-02:2379 |   true |  5.23352ms |       |
| http://sht-sgmhadoopdn-01:2379 |   true | 5.832925ms |       |
| http://sht-sgmhadoopdn-03:2379 |   true | 6.431018ms |       |
+--------------------------------+--------+------------+-------+


# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+

修改各节点etcd.conf配置文件如下参数

原

ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_STATE="new"

修改为

ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_INITIAL_CLUSTER_STATE="existing"

重启各节点

# systemctl stop etcd
# systemctl start etcd
# systemctl stop etcd
# systemctl start etcd

注：

该修改步骤不是必须的，只是为了使后续进行节点增加时候保持各配置文件一致性，该参数只是在添加新节点进入集群中才会起作用，重启现有集群会忽略该参数

bash
#把下载好的二进制文件直接cp 到/usr/bin目录下

[root@hkbcwbeta data]# tree etcd_data
etcd_data
├── etcd.conf
├── etcds
│   └── member
│       └── snap
│           └── db
└── wal

[root@hkbcwbeta etcd_data]# cat etcd.conf
#[Member]
ETCD_NAME=k8s3
ETCD_DATA_DIR="/data/etcd_data"
ETCD_WAL_DIR="/data/etcd_data/wal"
ETCD_AUTO_COMPACTION_RETENTION="1"
ETCD_AUTO_COMPACTION_MODE="periodic"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_SNAPSHOT_COUNT="50000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"
ETCD_MAX_REQUEST_BYTES="10485760"

ETCD_LISTEN_PEER_URLS="http://192.168.17.134:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.17.134:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.17.134:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.17.134:2379"
ETCD_INITIAL_CLUSTER="k8s1=http://192.168.17.132:2380,k8s2=http://192.168.17.133:2380,k8s3=http://192.168.17.134:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_ENABLE_V2="true"
#把下载好的二进制文件直接cp 到/usr/bin目录下

[root@hkbcwbeta data]# tree etcd_data
etcd_data
├── etcd.conf
├── etcds
│   └── member
│       └── snap
│           └── db
└── wal

[root@hkbcwbeta etcd_data]# cat etcd.conf
#[Member]
ETCD_NAME=k8s3
ETCD_DATA_DIR="/data/etcd_data"
ETCD_WAL_DIR="/data/etcd_data/wal"
ETCD_AUTO_COMPACTION_RETENTION="1"
ETCD_AUTO_COMPACTION_MODE="periodic"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_SNAPSHOT_COUNT="50000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"
ETCD_MAX_REQUEST_BYTES="10485760"

ETCD_LISTEN_PEER_URLS="http://192.168.17.134:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.17.134:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.17.134:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.17.134:2379"
ETCD_INITIAL_CLUSTER="k8s1=http://192.168.17.132:2380,k8s2=http://192.168.17.133:2380,k8s3=http://192.168.17.134:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_ENABLE_V2="true"

启动 ​

bash
[root@hkbcwbeta etcd_data]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=etcd
Type=simple
WorkingDirectory=/data/etcd_data/etcds/
EnvironmentFile=-/data/etcd_data/etcd.conf
ExecStart=/usr/bin/etcd --auto-compaction-retention '1' --max-request-bytes '33554432' --quota-backend-bytes '8589934592'
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
[root@hkbcwbeta etcd_data]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=etcd
Type=simple
WorkingDirectory=/data/etcd_data/etcds/
EnvironmentFile=-/data/etcd_data/etcd.conf
ExecStart=/usr/bin/etcd --auto-compaction-retention '1' --max-request-bytes '33554432' --quota-backend-bytes '8589934592'
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
bash
 #集群方式启动
 [Unit]
    Description=Etcd Server
    After=network.target
    After=network-online.target
    Wants=network-online.target
    [Service]
    Type=notify
    WorkingDirectory=/var/lib/etcd/
    EnvironmentFile=-/etc/etcd/etcd.conf
    User=etcd
    # set GOMAXPROCS to number of processors
    ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \
    --name=\"${ETCD_NAME}\" \
    --data-dir=\"${ETCD_DATA_DIR}\" \
    --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
    --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
    --initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
    --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \
    --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" \
    --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
    Restart=on-failure
    LimitNOFILE=65536
    [Install]
    WantedBy=multi-user.target
 #集群方式启动
 [Unit]
    Description=Etcd Server
    After=network.target
    After=network-online.target
    Wants=network-online.target
    [Service]
    Type=notify
    WorkingDirectory=/var/lib/etcd/
    EnvironmentFile=-/etc/etcd/etcd.conf
    User=etcd
    # set GOMAXPROCS to number of processors
    ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \
    --name=\"${ETCD_NAME}\" \
    --data-dir=\"${ETCD_DATA_DIR}\" \
    --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
    --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
    --initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
    --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \
    --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" \
    --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
    Restart=on-failure
    LimitNOFILE=65536
    [Install]
    WantedBy=multi-user.target

注意几点

1、--listen-client-urls用于监听客户端消息,必须设置为真实ip地址，如果机器为云主机，可以设置为云主机的私有ip地址或0.0.0.0(代表监听所有地址),不能设置为公网ip地址，因为云主机的公网ip是虚拟的，如果设置为公网ip会报“bind: cannot assign requested address”的错

2、--listen-peer-urls用于监听其他member发送过来的消息，跟listen-client-urls一样，必须设置为真实ip地址,如果机器为云主机,不能设置为公网ip

3、--initial-advertise-peer-urls用于监听其他member同步信号，该地址其他member必须能直接访问，所以如果是云主机该地址必须设置为云主机的公网ip地址

4、--initial-cluster群集列表，该列表中的值必须跟各个member的initial-advertise-peer-urls值一样

2.命令行方式集群 ​

bash
$ ETCD1=http://10.0.0.1
$ ETCD2=http://10.0.0.2
$ ETCD2=http://10.0.0.3

# 节点1为4核4GB的机器
$ etcd --name etcd1 \
--initial-advertise-peer-urls $ETCD1:2380  \
--listen-peer-urls $ETCD1:2380  \
--listen-client-urls $ETCD1:2379,http://127.0.0.1:2379 \
--advertise-client-urls $ETCD1:2379  \
--initial-cluster-token etcd-cluster \
--initial-cluster etcd1=$ETCD1:2380,etcd2=$ETCD2:2380,etcd3=$ETCD3:2380 \
--auto-compaction-retention=1 \
--quota-backend-bytes=$((4*1024*1024*1024)) \
--initial-cluster-state new

# 节点2为4核4GB的机器
$ etcd  --name etcd2 \
--initial-advertise-peer-urls $ETCD2:2380 \
--listen-peer-urls $ETCD2:2380 \
--listen-client-urls $ETCD2:2379,http://127.0.0.1:2379 \
--advertise-client-urls $ETCD2:2379 \
--initial-cluster-token etcd-cluster \
--initial-cluster etcd1=$ETCD1:2380,etcd2=$ETCD2:2380,etcd3=$ETCD3:2380 \
--auto-compaction-retention=1 \
--quota-backend-bytes=$((4*1024*1024*1024)) \
--initial-cluster-state new

# 节点3为4核4GB的机器
$ etcd  --name etcd3 \
--initial-advertise-peer-urls $ETCD3:2380 \
--listen-peer-urls $ETCD3:2380 \
--listen-client-urls $ETCD3:2379,http://127.0.0.1:2379 \
--advertise-client-urls $ETCD3:2379 \
--initial-cluster-token etcd-cluster \
--initial-cluster etcd1=$ETCD1:2380,etcd2=$ETCD2:2380,etcd3=$ETCD3:2380 \
--auto-compaction-retention=1 \
--quota-backend-bytes=$((4*1024*1024*1024)) \
--initial-cluster-state new

# 检查集群是否正确启动
$ etcdctl --endpoints=$ETCD1:2379 member list
400938f1eaf1d0ed: name=etcd3 peerURLs=http://10.0.0.1:2380 clientURLs=http://10.0.0.1:2379 isLeader=false
bd0ff97b33ac1165: name=etcd2 peerURLs=http://10.0.0.2:2380 clientURLs=http://10.0.0.2:2379 isLeader=false
be62429afec4445f: name=etcd1 peerURLs=http://10.0.0.3:2380 clientURLs=http://10.0.0.3:2379 isLeader=true
$ ETCD1=http://10.0.0.1
$ ETCD2=http://10.0.0.2
$ ETCD2=http://10.0.0.3

# 节点1为4核4GB的机器
$ etcd --name etcd1 \
--initial-advertise-peer-urls $ETCD1:2380  \
--listen-peer-urls $ETCD1:2380  \
--listen-client-urls $ETCD1:2379,http://127.0.0.1:2379 \
--advertise-client-urls $ETCD1:2379  \
--initial-cluster-token etcd-cluster \
--initial-cluster etcd1=$ETCD1:2380,etcd2=$ETCD2:2380,etcd3=$ETCD3:2380 \
--auto-compaction-retention=1 \
--quota-backend-bytes=$((4*1024*1024*1024)) \
--initial-cluster-state new

# 节点2为4核4GB的机器
$ etcd  --name etcd2 \
--initial-advertise-peer-urls $ETCD2:2380 \
--listen-peer-urls $ETCD2:2380 \
--listen-client-urls $ETCD2:2379,http://127.0.0.1:2379 \
--advertise-client-urls $ETCD2:2379 \
--initial-cluster-token etcd-cluster \
--initial-cluster etcd1=$ETCD1:2380,etcd2=$ETCD2:2380,etcd3=$ETCD3:2380 \
--auto-compaction-retention=1 \
--quota-backend-bytes=$((4*1024*1024*1024)) \
--initial-cluster-state new

# 节点3为4核4GB的机器
$ etcd  --name etcd3 \
--initial-advertise-peer-urls $ETCD3:2380 \
--listen-peer-urls $ETCD3:2380 \
--listen-client-urls $ETCD3:2379,http://127.0.0.1:2379 \
--advertise-client-urls $ETCD3:2379 \
--initial-cluster-token etcd-cluster \
--initial-cluster etcd1=$ETCD1:2380,etcd2=$ETCD2:2380,etcd3=$ETCD3:2380 \
--auto-compaction-retention=1 \
--quota-backend-bytes=$((4*1024*1024*1024)) \
--initial-cluster-state new

# 检查集群是否正确启动
$ etcdctl --endpoints=$ETCD1:2379 member list
400938f1eaf1d0ed: name=etcd3 peerURLs=http://10.0.0.1:2380 clientURLs=http://10.0.0.1:2379 isLeader=false
bd0ff97b33ac1165: name=etcd2 peerURLs=http://10.0.0.2:2380 clientURLs=http://10.0.0.2:2379 isLeader=false
be62429afec4445f: name=etcd1 peerURLs=http://10.0.0.3:2380 clientURLs=http://10.0.0.3:2379 isLeader=true

2.1 查看集群状态 ​

bash
etcdctl --endpoint=http://10...:2380 cluster-health（version：3.3.9）

#这个的一台一台查看
[root@k8s3 etcd_data]# etcdctl endpoint status --write-out=table

+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 467cafa3ad8836df |  3.4.13 |   20 kB |     false |      false |       247 |          9 |                  9 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

#或者
[root@k8s3 etcd_data]# etcdctl  --endpoints=http://192.168.17.134:2380,http://192.168.17.133:2380,http://192.168.17.132:2380 endpoint status --write-out=table

+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://192.168.17.134:2380 | 467cafa3ad8836df |  3.4.13 |   20 kB |     false |      false |       247 |          9 |                  9 |        |
| http://192.168.17.133:2380 | c9a1488472af4d3f |  3.4.13 |   20 kB |     false |      false |       247 |          9 |                  9 |        |
| http://192.168.17.132:2380 | e73ab1a36f7d200a |  3.4.13 |   20 kB |      true |      false |       247 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
etcdctl --endpoint=http://10...:2380 cluster-health（version：3.3.9）

#这个的一台一台查看
[root@k8s3 etcd_data]# etcdctl endpoint status --write-out=table

+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 467cafa3ad8836df |  3.4.13 |   20 kB |     false |      false |       247 |          9 |                  9 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

#或者
[root@k8s3 etcd_data]# etcdctl  --endpoints=http://192.168.17.134:2380,http://192.168.17.133:2380,http://192.168.17.132:2380 endpoint status --write-out=table

+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://192.168.17.134:2380 | 467cafa3ad8836df |  3.4.13 |   20 kB |     false |      false |       247 |          9 |                  9 |        |
| http://192.168.17.133:2380 | c9a1488472af4d3f |  3.4.13 |   20 kB |     false |      false |       247 |          9 |                  9 |        |
| http://192.168.17.132:2380 | e73ab1a36f7d200a |  3.4.13 |   20 kB |      true |      false |       247 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

2.2 查看集群健康性 ​

[root@k8s3 etcd_data]# etcdctl  --endpoints=http://192.168.17.134:2380,http://192.168.17.133:2380,http://192.168.17.132:2380 endpoint health --write-out=table
+----------------------------+--------+------------+-------+
|          ENDPOINT          | HEALTH |    TOOK    | ERROR |
+----------------------------+--------+------------+-------+
| http://192.168.17.132:2380 |   true | 4.246671ms |       |
| http://192.168.17.134:2380 |   true | 4.186323ms |       |
| http://192.168.17.133:2380 |   true | 5.224277ms |       |
+----------------------------+--------+------------+-------+

#或者--write-out=tabl   更换成 -w table
[root@k8s3 etcd_data]# etcdctl  --endpoints=http://192.168.17.134:2380,http://192.168.17.133:2380,http://192.168.17.132:2380 endpoint health --write-out=table
+----------------------------+--------+------------+-------+
|          ENDPOINT          | HEALTH |    TOOK    | ERROR |
+----------------------------+--------+------------+-------+
| http://192.168.17.132:2380 |   true | 4.246671ms |       |
| http://192.168.17.134:2380 |   true | 4.186323ms |       |
| http://192.168.17.133:2380 |   true | 5.224277ms |       |
+----------------------------+--------+------------+-------+

#或者--write-out=tabl   更换成 -w table

2.3 查看集群成员 ​

etcdctl  --endpoints=http://192.168.17.134:2380,http://192.168.17.133:2380,http://192.168.17.132:2380 member list --write-out=table
etcdctl  --endpoints=http://192.168.17.134:2380,http://192.168.17.133:2380,http://192.168.17.132:2380 member list --write-out=table

2.4 集群添加新节点 ​

查看当前集群节点信息 ​

bash
# etcdctl member list --write-out=table

+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
# etcdctl member list --write-out=table

+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+

添加新节点sht-sgmhadoopdn-04(172.16.101.66)

  • 添加集群节点对应hosts文件解析
bash
# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.101.58    sht-sgmhadoopdn-01
172.16.101.59    sht-sgmhadoopdn-02
172.16.101.60    sht-sgmhadoopdn-03
172.16.101.66    sht-sgmhadoopdn-04
# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.101.58    sht-sgmhadoopdn-01
172.16.101.59    sht-sgmhadoopdn-02
172.16.101.60    sht-sgmhadoopdn-03
172.16.101.66    sht-sgmhadoopdn-04

在现有集群接点添加新节点 ​

bash
# etcdctl member add sht-sgmhadoopdn-04 --peer-urls="http://sht-sgmhadoopdn-04:2380"
Member 7796493c3943f891 added to cluster 69bef0b9ccf44365

ETCD_NAME="sht-sgmhadoopdn-04"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-04=http://sht-sgmhadoopdn-04:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-04:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
# etcdctl member add sht-sgmhadoopdn-04 --peer-urls="http://sht-sgmhadoopdn-04:2380"
Member 7796493c3943f891 added to cluster 69bef0b9ccf44365

ETCD_NAME="sht-sgmhadoopdn-04"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-04=http://sht-sgmhadoopdn-04:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-04:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

新节点的etcd配置文件必须包括以上输出内容

  • 查看当前集群信息
bash
# etcdctl member list --write-out=table
+------------------+-----------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        |  STATUS   |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+-----------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd |   started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| 7796493c3943f891 | unstarted |                    | http://sht-sgmhadoopdn-04:2380 |                                                     |      false |
| d446fbe3296eb85a |   started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 |   started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+-----------+--------------------+--------------------------------+-----------------------------------------------------+------------+
# etcdctl member list --write-out=table
+------------------+-----------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        |  STATUS   |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+-----------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd |   started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| 7796493c3943f891 | unstarted |                    | http://sht-sgmhadoopdn-04:2380 |                                                     |      false |
| d446fbe3296eb85a |   started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 |   started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+-----------+--------------------+--------------------------------+-----------------------------------------------------+------------+

新节点配置etcd ​

  • etcd系统服务文件
bash
# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=tnuser
Type=notify
EnvironmentFile=/usr/local/etcd/etcd.conf
WorkingDirectory=/usr/local/etcd
ExecStart=/usr/local/etcd/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=tnuser
Type=notify
EnvironmentFile=/usr/local/etcd/etcd.conf
WorkingDirectory=/usr/local/etcd
ExecStart=/usr/local/etcd/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • etcd配置文件
bash
# cat /usr/local/etcd/etcd.conf
ETCD_NAME="sht-sgmhadoopdn-04"
ETCD_DATA_DIR="/usr/local/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-01:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-04:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380,sht-sgmhadoopdn-04=http://sht-sgmhadoopdn-04:2380"
ETCD_ENABLE_V2="true"
# cat /usr/local/etcd/etcd.conf
ETCD_NAME="sht-sgmhadoopdn-04"
ETCD_DATA_DIR="/usr/local/etcd/data"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://sht-sgmhadoopdn-01:2379,http://10.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sht-sgmhadoopdn-04:2380"
ETCD_INITIAL_CLUSTER="sht-sgmhadoopdn-01=http://sht-sgmhadoopdn-01:2380,sht-sgmhadoopdn-02=http://sht-sgmhadoopdn-02:2380,sht-sgmhadoopdn-03=http://sht-sgmhadoopdn-03:2380,sht-sgmhadoopdn-04=http://sht-sgmhadoopdn-04:2380"
ETCD_ENABLE_V2="true"
  • 启动新节点
bash
# systemctl start etcd
# systemctl start etcd
  • 再次查看集群状态
bash
# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| 7796493c3943f891 | started | sht-sgmhadoopdn-04 | http://sht-sgmhadoopdn-04:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| 7796493c3943f891 | started | sht-sgmhadoopdn-04 | http://sht-sgmhadoopdn-04:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+

将各节点etcd.conf配置文件的变量ETCD_INITIAL_CLUSTER添加新节点信息，然后依次重启

2.5 etcd集群移除节点 ​

查看当前集群信息 ​

bash
# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| 7796493c3943f891 | started | sht-sgmhadoopdn-04 | http://sht-sgmhadoopdn-04:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+

# etcdctl endpoint status --endpoints=http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379,http://sht-sgmhadoopdn-04:2379 --write-out=table
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|            ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://sht-sgmhadoopdn-01:2379 | 44d8bc3300880bcd |   3.4.1 |   16 kB |     false |      false |        17 |         38 |                 38 |        |
| http://sht-sgmhadoopdn-02:2379 | e9136c1ad1754783 |   3.4.1 |   16 kB |     false |      false |        17 |         38 |                 38 |        |
| http://sht-sgmhadoopdn-03:2379 | d446fbe3296eb85a |   3.4.1 |   16 kB |     false |      false |        17 |         38 |                 38 |        |
| http://sht-sgmhadoopdn-04:2379 | 7796493c3943f891 |   3.4.1 |   20 kB |      true |      false |        17 |         38 |                 38 |        |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

# etcdctl endpoint health --endpoints=http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379,http://sht-sgmhadoopdn-04:2379 --write-out=table
+--------------------------------+--------+------------+-------+
|            ENDPOINT            | HEALTH |    TOOK    | ERROR |
+--------------------------------+--------+------------+-------+
| http://sht-sgmhadoopdn-04:2379 |   true | 5.329583ms |       |
| http://sht-sgmhadoopdn-02:2379 |   true | 5.943424ms |       |
| http://sht-sgmhadoopdn-03:2379 |   true | 4.715634ms |       |
| http://sht-sgmhadoopdn-01:2379 |   true | 6.611197ms |       |
+--------------------------------+--------+------------+-------+
# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| 7796493c3943f891 | started | sht-sgmhadoopdn-04 | http://sht-sgmhadoopdn-04:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+

# etcdctl endpoint status --endpoints=http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379,http://sht-sgmhadoopdn-04:2379 --write-out=table
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|            ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://sht-sgmhadoopdn-01:2379 | 44d8bc3300880bcd |   3.4.1 |   16 kB |     false |      false |        17 |         38 |                 38 |        |
| http://sht-sgmhadoopdn-02:2379 | e9136c1ad1754783 |   3.4.1 |   16 kB |     false |      false |        17 |         38 |                 38 |        |
| http://sht-sgmhadoopdn-03:2379 | d446fbe3296eb85a |   3.4.1 |   16 kB |     false |      false |        17 |         38 |                 38 |        |
| http://sht-sgmhadoopdn-04:2379 | 7796493c3943f891 |   3.4.1 |   20 kB |      true |      false |        17 |         38 |                 38 |        |
+--------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

# etcdctl endpoint health --endpoints=http://sht-sgmhadoopdn-01:2379,http://sht-sgmhadoopdn-02:2379,http://sht-sgmhadoopdn-03:2379,http://sht-sgmhadoopdn-04:2379 --write-out=table
+--------------------------------+--------+------------+-------+
|            ENDPOINT            | HEALTH |    TOOK    | ERROR |
+--------------------------------+--------+------------+-------+
| http://sht-sgmhadoopdn-04:2379 |   true | 5.329583ms |       |
| http://sht-sgmhadoopdn-02:2379 |   true | 5.943424ms |       |
| http://sht-sgmhadoopdn-03:2379 |   true | 4.715634ms |       |
| http://sht-sgmhadoopdn-01:2379 |   true | 6.611197ms |       |
+--------------------------------+--------+------------+-------+
  • 将节点sht-sgmhadoopdn-04移除
bash
# etcdctl member remove 7796493c3943f891
Member 7796493c3943f891 removed from cluster 69bef0b9ccf44365

# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
# etcdctl member remove 7796493c3943f891
Member 7796493c3943f891 removed from cluster 69bef0b9ccf44365

# etcdctl member list --write-out=table
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
|        ID        | STATUS  |        NAME        |           PEER ADDRS           |                    CLIENT ADDRS                     | IS LEARNER |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+
| 44d8bc3300880bcd | started | sht-sgmhadoopdn-01 | http://sht-sgmhadoopdn-01:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-01:2379 |      false |
| d446fbe3296eb85a | started | sht-sgmhadoopdn-03 | http://sht-sgmhadoopdn-03:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
| e9136c1ad1754783 | started | sht-sgmhadoopdn-02 | http://sht-sgmhadoopdn-02:2380 | http://10.0.0.1:2379,http://sht-sgmhadoopdn-02:2379 |      false |
+------------------+---------+--------------------+--------------------------------+-----------------------------------------------------+------------+

修改配置文件etcd.conf，修改参数ETCD_INITIAL_CLUSTER并移除节点信息，重启etcd服务

3. 操作 ​

  • 写
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem put /chj/3 api3
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem put /chj/3 api3
  • 读
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem get /chj/3
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem get /chj/3
  • 查看所有key
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem get / --prefix --keys-only
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem get / --prefix --keys-only
  • 删除所有key
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem del "" --prefix
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem del "" --prefix
  • 查看集群成员
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem member list --write-out=table
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem member list --write-out=table
  • 查看集群健康状态
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem endpoint health
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem endpoint health
  • 查看节点状态
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem endpoint status --write-out=table
ETCDCTL_API=3 etcdctl --endpoints="https://192.168.9.87:2379,https://192.168.9.88:2379,https://192.168.9.89:2379" --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --cacert=/etc/etcd/ssl/ca.pem endpoint status --write-out=table

4.Etcd集群必须是奇数节点？ ​

etcd官方推荐3、5、7个节点，虽然raft算法也是半数以上投票才能有 leader，但奇数只是推荐，其实偶数也是可以的。如 2、4、8个节点。分情况说明：

  • 1 个节点：就是单实例，没有集群概念，不做讨论；
  • 2 个节点：是集群，但没人会这么配，这里说点废话：双节点的etcd能启动，启动时也能有主，可以正常提供服务，但是一台挂掉之后，就选不出主了，因为他只能拿到1票，剩下的那台也无法提供服务，也就是双节点无容错能力，不要使用

发现偶数节点虽然多了一台机器，但是容错能力是一样的，也就是说，你可以设置偶数节点，但没增加什么能力，还浪费了一台机器。同时etcd 是通过复制数据给所有节点来达到一致性，因此偶数的多一台机器增加不了性能，反而会拉低写入速度

5.Etcd集群机器越多越好？ ​

etcd 集群是一个 Raft Group，没有 shared。所以它的极限有两部分，一是单机的容量限制，内存和磁盘；二是网络开销，每次 Raft 操作需要所有节点参与，每一次写操作需要集群中大多数节点将日志落盘成功后，Leader 节点才能修改内部状态机，并将结果返回给客户端。因此节点越多性能越低，所以扩展很多 etcd 节点是没有意义的，一般是 3、5、7， 7 个也足够了

6.Etcd有脑裂问题？ ​

python
The majority side becomes the available cluster and the minority side is unavailable; there is no “split-brain” in etcd.
The majority side becomes the available cluster and the minority side is unavailable; there is no “split-brain” in etcd.

7.ssl证书方式搭建 ​

4.集群形式 ​

  • 主要有三种形式

1)静态发现: 预先已知 Etcd 集群中有哪些节点，在启动时直接指定好Etcd的各个node节点地址

2)Etcd动态发现: 通过已有的Etcd集群作为数据交互点，然后在扩展新的集群时实现通过已有集群进行服务发现的机制

3)DNS动态发现: 通过DNS查询方式获取其他节点地址信息

集群服务中的通信一般包括两种场景：

对外提供服务的通信，发生在集群外部的客户端和集群某个节点之间，etcd默认端口为2379，例如 etcdctl 就属于客户端

集群内部的通信，发生在集群内部的任意两个节点之间，etcd的默认端口为2380
集群服务中的通信一般包括两种场景：

对外提供服务的通信，发生在集群外部的客户端和集群某个节点之间，etcd默认端口为2379，例如 etcdctl 就属于客户端

集群内部的通信，发生在集群内部的任意两个节点之间，etcd的默认端口为2380