1. 节点notready时pod快速驱逐
1.1 场景
节点notready后大概5分钟才能重新调度pod,生产环境中高并发的场景下,一个副本5分钟无法提供服务,肯定会将请求压力转到其他副本,容易造成堵塞,严重的会阻断服务。如何缩短这个时间?
1.2 解决方式
k8s中有个准入控制器:DefaultTolerationSeconds
。
此准入控制器基于 k8s-apiserver 的输入参数 default-not-ready-toleration-seconds
和 default-unreachable-toleration-seconds
为 Pod 设置默认的容忍度,以容忍 notready:NoExecute 和 unreachable:NoExecute 污点 (如果 Pod 尚未容忍 node.kubernetes.io/not-ready:NoExecute 和 node.kubernetes.io/unreachable:NoExecute 污点的话)。
default-not-ready-toleration-seconds
和 default-unreachable-toleration-seconds
的默认值是 5 分钟
2. 测试
2.1 修改参数之前
bash
[root@k8s-master daemonset]# kubectl get pod -n test -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-854d5f75db-f9h89 1/1 Running 0 2m30s 10.244.169.143 k8s-node2 <none> <none>
[root@k8s-master daemonset]# kubectl get pod -n test -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-854d5f75db-f9h89 1/1 Running 0 2m30s 10.244.169.143 k8s-node2 <none> <none>
停止k8s-node2的kubelet,使node notready
bash
systemctl stop kubelet
systemctl stop kubelet
- 查看节点
bash
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 29d v1.20.13
k8s-node1 Ready <none> 29d v1.20.13
k8s-node2 NotReady <none> 29d v1.20.13
k8s-node3 Ready,SchedulingDisabled <none> 29d v1.20.13
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 29d v1.20.13
k8s-node1 Ready <none> 29d v1.20.13
k8s-node2 NotReady <none> 29d v1.20.13
k8s-node3 Ready,SchedulingDisabled <none> 29d v1.20.13
- 再次查看Pod
bash
[root@k8s-master daemonset]# kubectl get pod -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-854d5f75db-f9h89 1/1 Terminating 0 50m 10.244.169.143 k8s-node2 <none> <none>
nginx-854d5f75db-r29rb 1/1 Running 0 3m54s 10.244.36.116 k8s-node1 <none> <none>
[root@k8s-master daemonset]# kubectl get pod -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-854d5f75db-f9h89 1/1 Terminating 0 50m 10.244.169.143 k8s-node2 <none> <none>
nginx-854d5f75db-r29rb 1/1 Running 0 3m54s 10.244.36.116 k8s-node1 <none> <none>
2.1.1 判断节点notready5分钟后pod被调度
- 查看node的污点创建时间
bash
[root@k8s-master ~]# kubectl get node k8s-node2 -o custom-columns=Name:.metadata.name,Taints:.spec.taints
Name Taints
k8s-node2 [map[effect:NoSchedule key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:52Z] map[effect:NoExecute key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:57Z]]
## effect=NoExcute污点添加时间是27分57秒
[root@k8s-master ~]# kubectl get node k8s-node2 -o custom-columns=Name:.metadata.name,Taints:.spec.taints
Name Taints
k8s-node2 [map[effect:NoSchedule key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:52Z] map[effect:NoExecute key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:57Z]]
## effect=NoExcute污点添加时间是27分57秒
- 查看pod创建时间
bash
[root@k8s-master daemonset]# kubectl describe pod nginx-854d5f75db-r29rb -n test
Name: nginx-854d5f75db-r29rb
Namespace: test
Priority: 0
Node: k8s-node1/192.168.0.38
Start Time: Thu, 09 Mar 2023 17:32:57 +0800 # pod启动时间是32分57秒
Labels: k8s.kuboard.cn/layer=gateway
k8s.kuboard.cn/name=nginx
pod-template-hash=854d5f75db
Annotations: cni.projectcalico.org/containerID: d0a0cd39f52fa3dfb65a6a0b710e78dd44b8a9c31780885a28ec56aa7a0cd07c
cni.projectcalico.org/podIP: 10.244.36.116/32
cni.projectcalico.org/podIPs: 10.244.36.116/32
Status: Running
IP: 10.244.36.116
。。。。。。
[root@k8s-master daemonset]# kubectl describe pod nginx-854d5f75db-r29rb -n test
Name: nginx-854d5f75db-r29rb
Namespace: test
Priority: 0
Node: k8s-node1/192.168.0.38
Start Time: Thu, 09 Mar 2023 17:32:57 +0800 # pod启动时间是32分57秒
Labels: k8s.kuboard.cn/layer=gateway
k8s.kuboard.cn/name=nginx
pod-template-hash=854d5f75db
Annotations: cni.projectcalico.org/containerID: d0a0cd39f52fa3dfb65a6a0b710e78dd44b8a9c31780885a28ec56aa7a0cd07c
cni.projectcalico.org/podIP: 10.244.36.116/32
cni.projectcalico.org/podIPs: 10.244.36.116/32
Status: Running
IP: 10.244.36.116
。。。。。。
pod创建时间 - 污点添加时间=5分钟
2.2 修改参数
bash
[root@k8s-master daemonset]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
##在command末尾添加
- --default-not-ready-toleration-seconds=30
- --default-unreachable-toleration-seconds=30
[root@k8s-master daemonset]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
##在command末尾添加
- --default-not-ready-toleration-seconds=30
- --default-unreachable-toleration-seconds=30
之后,kube-apiserver会自动重启
查看时间,根据2.1.1中的方式查看