SRE文档1.单节点快照备份
1.备份
bash
DATE=$(date +%Y-%m-%d)
BIND_IP=`/sbin/ifconfig ${ETH0}| grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1' | head -1`
[root@localhost ~]#etcdctl --endpoints localhost:2379 snapshot save $BIND_IP-snapshot-$DATE.db
{"level":"info","ts":1623915233.1032534,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"uu.db.part"}
{"level":"info","ts":"2021-06-17T03:33:53.105-0400","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1623915233.1055887,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"localhost:2379"}
{"level":"info","ts":"2021-06-17T03:33:53.108-0400","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1623915233.1104572,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"localhost:2379","size":"20 kB","took":0.007136312}
{"level":"info","ts":1623915233.1106055,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"uu.db"}
Snapshot saved at uu.dbDATE=$(date +%Y-%m-%d)
BIND_IP=`/sbin/ifconfig ${ETH0}| grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1' | head -1`
[root@localhost ~]#etcdctl --endpoints localhost:2379 snapshot save $BIND_IP-snapshot-$DATE.db
{"level":"info","ts":1623915233.1032534,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"uu.db.part"}
{"level":"info","ts":"2021-06-17T03:33:53.105-0400","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1623915233.1055887,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"localhost:2379"}
{"level":"info","ts":"2021-06-17T03:33:53.108-0400","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1623915233.1104572,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"localhost:2379","size":"20 kB","took":0.007136312}
{"level":"info","ts":1623915233.1106055,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"uu.db"}
Snapshot saved at uu.db- 查看快照信息
[root@localhost ~]# etcdctl snapshot status uu.db --write-out=table
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 6d1803a9 | 110 | 146 | 37 kB |
+----------+----------+------------+------------+[root@localhost ~]# etcdctl snapshot status uu.db --write-out=table
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 6d1803a9 | 110 | 146 | 37 kB |
+----------+----------+------------+------------+2.恢复
bash
删除当前etcd的data目录，否则会报错目录已存在
操作：
export ETCDCTL_API=3
cd /data/etcd_data/
[root@localhost etcd_data]# etcdctl snapshot restore /root/192.168.122.247-snapshot-2021-06-17.db --data-dir=./data/etcd_data/
{"level":"info","ts":1623915905.37449,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/root/192.168.122.247-snapshot-2021-06-17.db","wal-dir":"data/etcd_data/member/wal","data-dir":"./data/etcd_data/","snap-dir":"data/etcd_data/member/snap"}
{"level":"info","ts":1623915905.382676,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"0","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":1623915905.391657,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/root/192.168.122.247-snapshot-2021-06-17.db","wal-dir":"data/etcd_data/member/wal","data-dir":"./data/etcd_data/","snap-dir":"data/etcd_data/member/snap"}
#恢复完成后需要将从备份重新生成的data目录的属主和属组修改为etcd系统服务中的对应属主和属组
[root@localhost etcd_data]# ls
data default.etcd etcd.conf etcd.yml etcds member wal
[root@localhost etcd_data]# ls data/
etcd_data
此时会在/data/etcd/ 下新建立 data目录，因此会改变存储路径，此时需要进行数据目录的修改
mv default.etcd default.etcd_bak
mv /data/etcd/data/etcd/default.etcd /data/etcd/
rm -rf /data/etcd/data
重启etcd
# systemctl stop etcd
# systemctl start etcd删除当前etcd的data目录，否则会报错目录已存在
操作：
export ETCDCTL_API=3
cd /data/etcd_data/
[root@localhost etcd_data]# etcdctl snapshot restore /root/192.168.122.247-snapshot-2021-06-17.db --data-dir=./data/etcd_data/
{"level":"info","ts":1623915905.37449,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/root/192.168.122.247-snapshot-2021-06-17.db","wal-dir":"data/etcd_data/member/wal","data-dir":"./data/etcd_data/","snap-dir":"data/etcd_data/member/snap"}
{"level":"info","ts":1623915905.382676,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"0","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":1623915905.391657,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/root/192.168.122.247-snapshot-2021-06-17.db","wal-dir":"data/etcd_data/member/wal","data-dir":"./data/etcd_data/","snap-dir":"data/etcd_data/member/snap"}
#恢复完成后需要将从备份重新生成的data目录的属主和属组修改为etcd系统服务中的对应属主和属组
[root@localhost etcd_data]# ls
data default.etcd etcd.conf etcd.yml etcds member wal
[root@localhost etcd_data]# ls data/
etcd_data
此时会在/data/etcd/ 下新建立 data目录，因此会改变存储路径，此时需要进行数据目录的修改
mv default.etcd default.etcd_bak
mv /data/etcd/data/etcd/default.etcd /data/etcd/
rm -rf /data/etcd/data
重启etcd
# systemctl stop etcd
# systemctl start etcd3. 2380端口备份
ETCDCTL_API=3 etcdctl --endpoints=http://192.168.122.249:2380 snapshot save snapshot.db
#恢复
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db
#重启故障etcd,修改已存在集群
initial-cluster-state为existingETCDCTL_API=3 etcdctl --endpoints=http://192.168.122.249:2380 snapshot save snapshot.db
#恢复
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db
#重启故障etcd,修改已存在集群
initial-cluster-state为existing2.集群快照备份
1.备份
bash
#!/bin/bash
DATE=$(date +%Y-%m-%d-%H)
BIND_IP=`/sbin/ifconfig ${ETH0}| grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1' | head -1`
backup_path="/data/aw-etcdbak/"
cd $backup_path
export ETCDCTL_API=3
ENDPOINTS='10.16.1.110:12379,10.16.1.111:12379,10.16.1.112:12379'
etcdctl --endpoints=$ENDPOINTS snapshot save $BIND_IP-snapshot-$DATE.db
find ./ -type f -name "*.db" -mtime +7 |xargs rm -f
#查看备份快照信息
[root@localhost ~]# etcdctl --write-out=table snapshot status 192.168.122.247-snapshot-2021-06-17.db
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| cefed78f | 0 | 3 | 20 kB |
+----------+----------+------------+------------+#!/bin/bash
DATE=$(date +%Y-%m-%d-%H)
BIND_IP=`/sbin/ifconfig ${ETH0}| grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1' | head -1`
backup_path="/data/aw-etcdbak/"
cd $backup_path
export ETCDCTL_API=3
ENDPOINTS='10.16.1.110:12379,10.16.1.111:12379,10.16.1.112:12379'
etcdctl --endpoints=$ENDPOINTS snapshot save $BIND_IP-snapshot-$DATE.db
find ./ -type f -name "*.db" -mtime +7 |xargs rm -f
#查看备份快照信息
[root@localhost ~]# etcdctl --write-out=table snapshot status 192.168.122.247-snapshot-2021-06-17.db
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| cefed78f | 0 | 3 | 20 kB |
+----------+----------+------------+------------+2.数据恢复
bash
etcd 获取备份数据172.24.119.41-snapshot-2021-06-17.db 后，分别将改备份数据分发至三个节点
分别停止etcd 三个节点
systemctl stop etcd
确认停掉三个节点后依次执行恢复操作
node1 恢复操作
mv /data/etcd/node1.etcd /data/etcd/node1.etcd_bak
cd /data/etcd/
etcdctl --data-dir=/var/lib/etcd snapshot restore /data/172.24.119.41-snapshot-2019-09-24.db --name node1 --initial-cluster node1=http://172.25.102.10:2380,node2=http://172.25.102.39:2380,node3=http://172.25.102.17:2380 --initial-advertise-peer-urls http://172.25.102.10:2380
node2 恢复操作
mv /data/etcd/node2.etcd /data/etcd/node2.etcd_bak
cd /data/etcd/
etcdctl --data-dir=/var/lib/etcd snapshot restore /data/172.24.119.41-snapshot-2019-09-24.db --name node2 --initial-cluster node1=http://172.25.102.10:2380,node2=http://172.25.102.39:2380,node3=http://172.25.102.17:2380 --initial-advertise-peer-urls http://172.25.102.39:2380
node3 恢复操作
mv /data/etcd/node3.etcd /data/etcd/node3.etcd_bak
cd /data/etcd/
etcdctl --data-dir=/var/lib/etcd snapshot restore /data/172.24.119.41-snapshot-2019-09-24.db --name node3 --initial-cluster node1=http://172.25.102.10:2380,node2=http://172.25.102.39:2380,node3=http://172.25.102.17:2380 --initial-advertise-peer-urls http://172.25.102.17:2380
node1 node2 node3 依次启动 etcd
systemctl start etcd
etcdctl get /ad/media 查看恢复的数据已正常etcd 获取备份数据172.24.119.41-snapshot-2021-06-17.db 后，分别将改备份数据分发至三个节点
分别停止etcd 三个节点
systemctl stop etcd
确认停掉三个节点后依次执行恢复操作
node1 恢复操作
mv /data/etcd/node1.etcd /data/etcd/node1.etcd_bak
cd /data/etcd/
etcdctl --data-dir=/var/lib/etcd snapshot restore /data/172.24.119.41-snapshot-2019-09-24.db --name node1 --initial-cluster node1=http://172.25.102.10:2380,node2=http://172.25.102.39:2380,node3=http://172.25.102.17:2380 --initial-advertise-peer-urls http://172.25.102.10:2380
node2 恢复操作
mv /data/etcd/node2.etcd /data/etcd/node2.etcd_bak
cd /data/etcd/
etcdctl --data-dir=/var/lib/etcd snapshot restore /data/172.24.119.41-snapshot-2019-09-24.db --name node2 --initial-cluster node1=http://172.25.102.10:2380,node2=http://172.25.102.39:2380,node3=http://172.25.102.17:2380 --initial-advertise-peer-urls http://172.25.102.39:2380
node3 恢复操作
mv /data/etcd/node3.etcd /data/etcd/node3.etcd_bak
cd /data/etcd/
etcdctl --data-dir=/var/lib/etcd snapshot restore /data/172.24.119.41-snapshot-2019-09-24.db --name node3 --initial-cluster node1=http://172.25.102.10:2380,node2=http://172.25.102.39:2380,node3=http://172.25.102.17:2380 --initial-advertise-peer-urls http://172.25.102.17:2380
node1 node2 node3 依次启动 etcd
systemctl start etcd
etcdctl get /ad/media 查看恢复的数据已正常3.备份目录
#v3.3
etcdctl --ca-file=ca.pem --cert-file=etcd.pem --key-file=etcd-key.pem backup --data-dir /var/lib/etcd --backup-dir /tmp/etcd
--data-dir：指明数据目录的位置
--backup-dir：指明备份的位置#v3.3
etcdctl --ca-file=ca.pem --cert-file=etcd.pem --key-file=etcd-key.pem backup --data-dir /var/lib/etcd --backup-dir /tmp/etcd
--data-dir：指明数据目录的位置
--backup-dir：指明备份的位置