Skip to content

1. 节点notready时pod快速驱逐

1.1 场景

节点notready后大概5分钟才能重新调度pod,生产环境中高并发的场景下,一个副本5分钟无法提供服务,肯定会将请求压力转到其他副本,容易造成堵塞,严重的会阻断服务。如何缩短这个时间?

1.2 解决方式

k8s中有个准入控制器:DefaultTolerationSeconds

此准入控制器基于 k8s-apiserver 的输入参数 default-not-ready-toleration-secondsdefault-unreachable-toleration-seconds 为 Pod 设置默认的容忍度,以容忍 notready:NoExecute 和 unreachable:NoExecute 污点 (如果 Pod 尚未容忍 node.kubernetes.io/not-ready:NoExecute 和 node.kubernetes.io/unreachable:NoExecute 污点的话)。

default-not-ready-toleration-secondsdefault-unreachable-toleration-seconds 的默认值是 5 分钟

2. 测试

2.1 修改参数之前

bash
[root@k8s-master daemonset]# kubectl get pod -n test -o wide -w
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
nginx-854d5f75db-f9h89   1/1     Running   0          2m30s   10.244.169.143   k8s-node2   <none>           <none>
[root@k8s-master daemonset]# kubectl get pod -n test -o wide -w
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
nginx-854d5f75db-f9h89   1/1     Running   0          2m30s   10.244.169.143   k8s-node2   <none>           <none>

停止k8s-node2的kubelet,使node notready

bash
systemctl stop kubelet
systemctl stop kubelet
  • 查看节点
bash
[root@k8s-master ~]# kubectl get node
NAME         STATUS                     ROLES                  AGE   VERSION
k8s-master   Ready                      control-plane,master   29d   v1.20.13
k8s-node1    Ready                      <none>                 29d   v1.20.13
k8s-node2    NotReady                   <none>                 29d   v1.20.13
k8s-node3    Ready,SchedulingDisabled   <none>                 29d   v1.20.13
[root@k8s-master ~]# kubectl get node
NAME         STATUS                     ROLES                  AGE   VERSION
k8s-master   Ready                      control-plane,master   29d   v1.20.13
k8s-node1    Ready                      <none>                 29d   v1.20.13
k8s-node2    NotReady                   <none>                 29d   v1.20.13
k8s-node3    Ready,SchedulingDisabled   <none>                 29d   v1.20.13
  • 再次查看Pod
bash
[root@k8s-master daemonset]# kubectl get pod -n test -o wide
NAME                     READY   STATUS        RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
nginx-854d5f75db-f9h89   1/1     Terminating   0          50m     10.244.169.143   k8s-node2   <none>           <none>
nginx-854d5f75db-r29rb   1/1     Running       0          3m54s   10.244.36.116    k8s-node1   <none>           <none>
[root@k8s-master daemonset]# kubectl get pod -n test -o wide
NAME                     READY   STATUS        RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
nginx-854d5f75db-f9h89   1/1     Terminating   0          50m     10.244.169.143   k8s-node2   <none>           <none>
nginx-854d5f75db-r29rb   1/1     Running       0          3m54s   10.244.36.116    k8s-node1   <none>           <none>

2.1.1 判断节点notready5分钟后pod被调度

  • 查看node的污点创建时间
bash
[root@k8s-master ~]# kubectl get node k8s-node2 -o custom-columns=Name:.metadata.name,Taints:.spec.taints
Name        Taints
k8s-node2   [map[effect:NoSchedule key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:52Z] map[effect:NoExecute key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:57Z]]
## effect=NoExcute污点添加时间是27分57秒
[root@k8s-master ~]# kubectl get node k8s-node2 -o custom-columns=Name:.metadata.name,Taints:.spec.taints
Name        Taints
k8s-node2   [map[effect:NoSchedule key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:52Z] map[effect:NoExecute key:node.kubernetes.io/unreachable timeAdded:2023-03-09T09:27:57Z]]
## effect=NoExcute污点添加时间是27分57秒
  • 查看pod创建时间
bash
[root@k8s-master daemonset]# kubectl describe  pod nginx-854d5f75db-r29rb -n test
Name:         nginx-854d5f75db-r29rb
Namespace:    test
Priority:     0
Node:         k8s-node1/192.168.0.38
Start Time:   Thu, 09 Mar 2023 17:32:57 +0800        # pod启动时间是32分57秒
Labels:       k8s.kuboard.cn/layer=gateway
              k8s.kuboard.cn/name=nginx
              pod-template-hash=854d5f75db
Annotations:  cni.projectcalico.org/containerID: d0a0cd39f52fa3dfb65a6a0b710e78dd44b8a9c31780885a28ec56aa7a0cd07c
              cni.projectcalico.org/podIP: 10.244.36.116/32
              cni.projectcalico.org/podIPs: 10.244.36.116/32
Status:       Running
IP:           10.244.36.116
。。。。。。
[root@k8s-master daemonset]# kubectl describe  pod nginx-854d5f75db-r29rb -n test
Name:         nginx-854d5f75db-r29rb
Namespace:    test
Priority:     0
Node:         k8s-node1/192.168.0.38
Start Time:   Thu, 09 Mar 2023 17:32:57 +0800        # pod启动时间是32分57秒
Labels:       k8s.kuboard.cn/layer=gateway
              k8s.kuboard.cn/name=nginx
              pod-template-hash=854d5f75db
Annotations:  cni.projectcalico.org/containerID: d0a0cd39f52fa3dfb65a6a0b710e78dd44b8a9c31780885a28ec56aa7a0cd07c
              cni.projectcalico.org/podIP: 10.244.36.116/32
              cni.projectcalico.org/podIPs: 10.244.36.116/32
Status:       Running
IP:           10.244.36.116
。。。。。。

pod创建时间 - 污点添加时间=5分钟

2.2 修改参数

bash
[root@k8s-master daemonset]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
##在command末尾添加
    - --default-not-ready-toleration-seconds=30
    - --default-unreachable-toleration-seconds=30
[root@k8s-master daemonset]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
##在command末尾添加
    - --default-not-ready-toleration-seconds=30
    - --default-unreachable-toleration-seconds=30

之后,kube-apiserver会自动重启

查看时间,根据2.1.1中的方式查看