SRE文档

深浅模式

1. k8s+containerd部署方式 ​

通常使用 kubeadm 部署和二进制方式部署，他们之间的区别：

  • kubeadm 方式部署，组件容器化部署，只有 kubelet 没有被容器化
  • 二进制方式部署，传统的守护进程（systemd）管理服务 systemctl
bash
目前生产环境部署kubernetes集群主要由两种方式:
	- kubeadm:
		kubeadm是一个K8S部署工具，提供kubeadm init和kubeadm join，用于快速部署kubernetes集群。
	- 二进制部署:
		从GitHub下载发行版的二进制包，手动部署每个组件，组成kubernetes集群。
		
		
除了上述介绍的两种方式部署外，还有其他部署方式的途径:
	- yum: 
		已废弃，目前支持的最新版本为2017年发行的1.5.2版本。
	- minikube:
		适合开发环境，能够快速在Windows或者Linux构建K8S集群。
		参考链接:
			https://minikube.sigs.k8s.io/docs/
	- rancher:
		基于K8S改进发行了轻量级K8S，让K3S孕育而生。
		参考链接:
			https://www.rancher.com/
	- KubeSphere:
		青云科技基于开源KubeSphere快速部署K8S集群。
		参考链接:
			https://kubesphere.com.cn
	- kuboard:
		也是对k8s进行二次开发的产品，新增了很多独有的功能。
		参考链接: 
			https://kuboard.cn/
    - kubeasz:
        使用ansible部署，扩容，缩容kubernetes集群，安装步骤官方文档已经非常详细了。
        参考链接: 
            https://github.com/easzlab/kubeasz/
			
	- 第三方云厂商:
		比如aws，阿里云，腾讯云，京东云等云厂商均有K8S的相关SAAS产品。

	- 更多的第三方部署工具:
		参考链接:
           https://landscape.cncf.io/
目前生产环境部署kubernetes集群主要由两种方式:
	- kubeadm:
		kubeadm是一个K8S部署工具，提供kubeadm init和kubeadm join，用于快速部署kubernetes集群。
	- 二进制部署:
		从GitHub下载发行版的二进制包，手动部署每个组件，组成kubernetes集群。
		
		
除了上述介绍的两种方式部署外，还有其他部署方式的途径:
	- yum: 
		已废弃，目前支持的最新版本为2017年发行的1.5.2版本。
	- minikube:
		适合开发环境，能够快速在Windows或者Linux构建K8S集群。
		参考链接:
			https://minikube.sigs.k8s.io/docs/
	- rancher:
		基于K8S改进发行了轻量级K8S，让K3S孕育而生。
		参考链接:
			https://www.rancher.com/
	- KubeSphere:
		青云科技基于开源KubeSphere快速部署K8S集群。
		参考链接:
			https://kubesphere.com.cn
	- kuboard:
		也是对k8s进行二次开发的产品，新增了很多独有的功能。
		参考链接: 
			https://kuboard.cn/
    - kubeasz:
        使用ansible部署，扩容，缩容kubernetes集群，安装步骤官方文档已经非常详细了。
        参考链接: 
            https://github.com/easzlab/kubeasz/
			
	- 第三方云厂商:
		比如aws，阿里云，腾讯云，京东云等云厂商均有K8S的相关SAAS产品。

	- 更多的第三方部署工具:
		参考链接:
           https://landscape.cncf.io/

2. k8s部署规划 ​

2.1 集群模式 ​

1. nginx ​

2. haproxy+keepalived ​

1.所有节点(k8s-master)安装keepalived和haproxy ​

bash
yum -y install keepalived haproxy
yum -y install keepalived haproxy

2.所有节点(k8s-master)配置haproxy，配置文件各个节点相同 ​

bash
(1)备份配置文件
cp /etc/haproxy/haproxy.cfg{,`date +%F`}


(2)所有节点的配置文件内容相同
cat > /etc/haproxy/haproxy.cfg <<'EOF'
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01   10.0.0.201:6443  check
  server k8s-master02   10.0.0.202:6443  check
  server k8s-master03   10.0.0.203:6443  check
EOF
(1)备份配置文件
cp /etc/haproxy/haproxy.cfg{,`date +%F`}


(2)所有节点的配置文件内容相同
cat > /etc/haproxy/haproxy.cfg <<'EOF'
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01   10.0.0.201:6443  check
  server k8s-master02   10.0.0.202:6443  check
  server k8s-master03   10.0.0.203:6443  check
EOF

3.所有节点(k8s-master)配置keepalived，配置文件各节点不同 ​

bash
(1)备份配置文件
cp /etc/keepalived/keepalived.conf{,`date +%F`}

(2)"k8s-master01"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 10.0.0.201
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        10.0.0.250
    }
    track_script {
       chk_apiserver
    }
}
EOF


(3)"k8s-master02"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 10.0.0.202
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        10.0.0.250
    }
    track_script {
       chk_apiserver
    }
}
EOF

(4)"k8s-master03"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 10.0.0.203
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        10.0.0.250
    }
    track_script {
       chk_apiserver
    }
}
EOF
(1)备份配置文件
cp /etc/keepalived/keepalived.conf{,`date +%F`}

(2)"k8s-master01"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 10.0.0.201
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        10.0.0.250
    }
    track_script {
       chk_apiserver
    }
}
EOF


(3)"k8s-master02"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 10.0.0.202
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        10.0.0.250
    }
    track_script {
       chk_apiserver
    }
}
EOF

(4)"k8s-master03"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 10.0.0.203
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        10.0.0.250
    }
    track_script {
       chk_apiserver
    }
}
EOF

4.所有节点(k8s-master)配置KeepAlived健康检查文件 ​

bash
(1)创建检查脚本
cat > /etc/keepalived/check_apiserver.sh <<'EOF'
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
EOF


(2)添加执行权限
chmod +x /etc/keepalived/check_apiserver.sh
(1)创建检查脚本
cat > /etc/keepalived/check_apiserver.sh <<'EOF'
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
EOF


(2)添加执行权限
chmod +x /etc/keepalived/check_apiserver.sh

❌ 注意

1.我们通过KeepAlived虚拟出来一个VIP，VIP会配置到一个master节点上面，它会通过haproxy暴露的16443的端口反向代理到我们的三个master节点上面，所以我们可以通过VIP的地址加上16443访问到我们的API server;

2.健康检查会检查haproxy的状态，三次失败就会将KeepAlived停掉，停掉之后KeepAlived会跳到其他的节点;

  • 启动服务
bash
1.启动harproxy
systemctl daemon-reload
systemctl enable --now haproxy

2.启动keepalived
systemctl enable --now keepalived

2.查看VIP
ip a
1.启动harproxy
systemctl daemon-reload
systemctl enable --now haproxy

2.启动keepalived
systemctl enable --now keepalived

2.查看VIP
ip a

3.云厂商用nlb,elb ​

如果在云上安装K8S则无安装高可用组件了，毕竟公有云大部分都是不支持keepalived的，可以直接使用云产品，比如阿里的"SLB"，腾讯的"ELB"等SAAS产品;

推荐使用ELB，SLB有回环的问题，也就是SLB代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题;

2.2 资源 ​

生产环境中，建议使用小版本大于5的Kubernetes版本，比如1.19.5以后的才可用于生产环境

vm 环境，规划如下：

k8s 集群角色ip 地址hostname 主机名称资源规格操作系统安装组件
masterkube-master-012x4g/60gRocklinux9.4containerd，nerdctl，etcd，apiserver，controller-manager，scheduler，kubectl，kubelet，kube-proxy，calico
work-nodekube-node-012x4g/60gRocklinux9.4containerd，etcd,nerdctl，kubelet，kube-proxy，calico
work-nodekube-node-022x4g/60gRocklinux9.4containerd，etcd,nerdctl，kubelet，kube-proxy，calico

etcd节省资源,在master,work进行安装,线上环境单独安装

2.3 Ip分配 ​

主机名ip地址
slb10.103.236.236
kubeadm-master10.103.236.150
kubeadm-node0110.103.236.151
kubeadm-node0210.103.236.152
pod网段172.16.0.0/12
service网段192.168.0.0/16
host网段10.103.236.0/12

2.4 下载 ​

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md

bash
#查看最新版本
curl -Ls https://dl.k8s.io/release/stable.txt
#查看最新版本
curl -Ls https://dl.k8s.io/release/stable.txt
软件名字版本
kubernetes-serverv1.29.7
containerdv1.7.22
etcdv3.5.12
cfsslv1.6.5
cfssljsonv1.6.5
bash
wget https://dl.k8s.io/v1.29.7/kubernetes-server-linux-amd64.tar.gz

wget https://github.com/containerd/containerd/releases/download/v1.7.22/cri-containerd-cni-1.7.22-linux-amd64.tar.gz

wget https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_amd64 -O /usr/local/bin/cfssl

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_amd64 -O /usr/local/bin/cfssljson
wget https://dl.k8s.io/v1.29.7/kubernetes-server-linux-amd64.tar.gz

wget https://github.com/containerd/containerd/releases/download/v1.7.22/cri-containerd-cni-1.7.22-linux-amd64.tar.gz

wget https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_amd64 -O /usr/local/bin/cfssl

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_amd64 -O /usr/local/bin/cfssljson

3. 部署 ​

3.1 containerd部署-所有节点 ​

bash
#解压
tar zxvf cri-containerd-cni-1.7.22-linux-amd64.tar.gz -C /
#解压
tar zxvf cri-containerd-cni-1.7.22-linux-amd64.tar.gz -C /
  • 生成containerd配置文件
bash
mkdir /etc/containerd/

containerd config default | tee /etc/containerd/config.toml

vim config.toml
...
SystemdCgroup = false #修改为true
...


再修改/etc/containerd/config.toml中的
[plugins."io.containerd.grpc.v1.cri"]
  ...
  # sandbox_image = "k8s.gcr.io/pause:3.6"
  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"	#这里一定要注意，要根据下载到本地 pause镜像的版本来进行修改，否则初始化会过不去。

再修改,默认是io.containerd.runc.v2,否则crictl 无法连接containerd
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runtime.v1.linux
mkdir /etc/containerd/

containerd config default | tee /etc/containerd/config.toml

vim config.toml
...
SystemdCgroup = false #修改为true
...


再修改/etc/containerd/config.toml中的
[plugins."io.containerd.grpc.v1.cri"]
  ...
  # sandbox_image = "k8s.gcr.io/pause:3.6"
  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"	#这里一定要注意，要根据下载到本地 pause镜像的版本来进行修改，否则初始化会过不去。

再修改,默认是io.containerd.runc.v2,否则crictl 无法连接containerd
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runtime.v1.linux

或者用sed修改

bash
sed -ri -e 's/(.*SystemdCgroup = ).*/\1true/' -e 's@(.*sandbox_image = ).*@\1\"registry.aliyuncs.com/google_containers/pause:3.9\"@' /etc/containerd/config.toml

# 命令直接修改， SystemdCgroup ： false -> true
sed -i '/SystemdCgroup/s/false/true/' /etc/containerd/config.toml

#镜像加速，也可以不修改
sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml
sed -ri -e 's/(.*SystemdCgroup = ).*/\1true/' -e 's@(.*sandbox_image = ).*@\1\"registry.aliyuncs.com/google_containers/pause:3.9\"@' /etc/containerd/config.toml

# 命令直接修改， SystemdCgroup ： false -> true
sed -i '/SystemdCgroup/s/false/true/' /etc/containerd/config.toml

#镜像加速，也可以不修改
sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml
  • 启动containerd服务
bash
# 重新加载Unit file
systemctl daemon-reload

#开机启动,并启动服务
systemctl enable --now containerd
# 重新加载Unit file
systemctl daemon-reload

#开机启动,并启动服务
systemctl enable --now containerd
  • 节点配置crictl客户端连接的运行时位置
bash
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

❌ 注意

由于containerd,桥接默认启动分配的网络是10.4.0.0/24,按需进行修改

  • 验证crictl连通性
bash
crictl -r unix:///var/run/containerd/containerd.sock info

或者
[root@kube-master-01 ~]# crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.7.22
RuntimeApiVersion:  v1
crictl -r unix:///var/run/containerd/containerd.sock info

或者
[root@kube-master-01 ~]# crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.7.22
RuntimeApiVersion:  v1

如果没有这个工具,下载方式

VERSION="1.29.0"

wget https://github.com/kubernetes-sigs/cri-tools/releases/download/${VERSION}/crictl-${VERSION}-linux-amd64.tar.gz

3.2 部署buildkit-所有节点 ​

  • 解压
bash
tar zxvf nerdctl-full-1.7.7-linux-amd64.tar.gz -C /usr/local/
tar zxvf nerdctl-full-1.7.7-linux-amd64.tar.gz -C /usr/local/
  • 配置systemd文件
bash
cp  /usr/local/lib/systemd/system/buildkit.service /etc/systemd/system/buildkitd.service
cp  /usr/local/lib/systemd/system/buildkit.service /etc/systemd/system/buildkitd.service
  • 配置内核参数,否则nerdctl执行命令会提示警告
bash
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf

sysctl -p
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf

sysctl -p
  • 启动服务
# 重新加载Unit file
systemctl daemon-reload

#开机启动,并启动服务
systemctl enable --now buildkit
# 重新加载Unit file
systemctl daemon-reload

#开机启动,并启动服务
systemctl enable --now buildkit

3.3 部署nerdctl-所有节点 ​

看3.2环节

  • 验证
bash
[root@kube-node-01 ~]# nerdctl info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: v1.7.22
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-427.33.1.el9_4.x86_64
 Operating System: Rocky Linux 9.4 (Blue Onyx)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.543GiB
 Name: kube-node-01
 ID: c8f2114c-e5a8-4e14-a8f2-af9f6ca67ba5
[root@kube-node-01 ~]# nerdctl info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: v1.7.22
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-427.33.1.el9_4.x86_64
 Operating System: Rocky Linux 9.4 (Blue Onyx)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.543GiB
 Name: kube-node-01
 ID: c8f2114c-e5a8-4e14-a8f2-af9f6ca67ba5

3.4 部署证书-在master操作 ​

bash
cp cfssl_1.6.5_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.5_linux_amd64 /usr/local/bin/cfssljson
#添加权限
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
cp cfssl_1.6.5_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.5_linux_amd64 /usr/local/bin/cfssljson
#添加权限
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

3.5 部署etcd ​

3.5.1 安装 ​

bash
tar -xf etcd-v3.5.12-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.7-linux-amd64/etcd{,ctl}

#验证
[root@kube-master-01 init_pack]# etcdctl version
etcdctl version: 3.5.12
API version: 3.5
tar -xf etcd-v3.5.12-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.7-linux-amd64/etcd{,ctl}

#验证
[root@kube-master-01 init_pack]# etcdctl version
etcdctl version: 3.5.12
API version: 3.5
  • 部署其它节点
bash
MasterNodes='kube-node-01 kube-node-02'

for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
MasterNodes='kube-node-01 kube-node-02'

for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done

3.5.2 配置证书 ​

  • 创建证书目录和配置目录,需要在部署etcd所在节点进行创建
bash
mkdir -p  /etc/etcd/ssl/
mkdir -p /etc/etcd/cfg
mkdir -p /var/lib/etcd
mkdir -p /var/log/etcd
mkdir -p  /etc/etcd/ssl/
mkdir -p /etc/etcd/cfg
mkdir -p /var/lib/etcd
mkdir -p /var/log/etcd
  • 配置CA证书,只在master01上面执行
bash
cd /etc/etcd/ssl/

cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF


cat > etcd-ca-csr.json  << EOF 
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

[root@kube-master-01 ssl]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca
2024/09/19 15:29:52 [INFO] generating a new CA key and certificate from CSR
2024/09/19 15:29:52 [INFO] generate received request
2024/09/19 15:29:52 [INFO] received CSR
2024/09/19 15:29:52 [INFO] generating key: rsa-2048
2024/09/19 15:29:52 [INFO] encoded CSR
2024/09/19 15:29:52 [INFO] signed certificate with serial number 120213013390660577809730590415784165837334607123
cd /etc/etcd/ssl/

cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF


cat > etcd-ca-csr.json  << EOF 
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

[root@kube-master-01 ssl]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca
2024/09/19 15:29:52 [INFO] generating a new CA key and certificate from CSR
2024/09/19 15:29:52 [INFO] generate received request
2024/09/19 15:29:52 [INFO] received CSR
2024/09/19 15:29:52 [INFO] generating key: rsa-2048
2024/09/19 15:29:52 [INFO] encoded CSR
2024/09/19 15:29:52 [INFO] signed certificate with serial number 120213013390660577809730590415784165837334607123
  • 自签CA签发Etcd HTTPS证书

创建证书申请文件

文件hosts字段中IP为所有etcd节点的集群内部通信IP，不要漏了！为了方便后期扩容可以多写几个ip预留扩容

bash
cat > etcd-csr.json << EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
 "hosts": [
    "127.0.0.1",
    "kube-master-01",
    "kube-node-02",
    "kube-node-01",
    "10.103.236.150",
    "10.103.236.151",
    "10.103.236.152"
    ],

  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF


#生成证书
cfssl gencert -ca=/etc/etcd/ssl/etcd-ca.pem  -ca-key=/etc/etcd/ssl/etcd-ca-key.pem  -config=ca-config.json -profile=kubernetes  etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

#查看
ls -thl etcd*
-rw-r--r-- 1 root root 1017  9月 19 15:44 etcd.csr
-rw-r--r-- 1 root root  296  9月 19 15:35 etcd-csr.json
-rw------- 1 root root 1675  9月 19 15:44 etcd-key.pem
-rw-r--r-- 1 root root 1346  9月 19 15:44 etcd.pem
cat > etcd-csr.json << EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
 "hosts": [
    "127.0.0.1",
    "kube-master-01",
    "kube-node-02",
    "kube-node-01",
    "10.103.236.150",
    "10.103.236.151",
    "10.103.236.152"
    ],

  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF


#生成证书
cfssl gencert -ca=/etc/etcd/ssl/etcd-ca.pem  -ca-key=/etc/etcd/ssl/etcd-ca-key.pem  -config=ca-config.json -profile=kubernetes  etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

#查看
ls -thl etcd*
-rw-r--r-- 1 root root 1017  9月 19 15:44 etcd.csr
-rw-r--r-- 1 root root  296  9月 19 15:35 etcd-csr.json
-rw------- 1 root root 1675  9月 19 15:44 etcd-key.pem
-rw-r--r-- 1 root root 1346  9月 19 15:44 etcd.pem

3.5.3 配置etcd文件 ​

在master-01上执行，然后把生成的文件拷贝到其他etcd集群主机,修改name(主机名字)和ip

bash
cat > /etc/etcd/cfg/etcd.yml << EOF
name: 'kube-master-01'
data-dir: /var/lib/etcd
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 8589934592  # 设置为 8GB 或根据需要调整
listen-peer-urls: 'https://10.103.236.150:2380'
listen-client-urls: 'https://10.103.236.150:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
initial-advertise-peer-urls: 'https://10.103.236.150:2380'
advertise-client-urls: 'https://10.103.236.150:2379'
initial-cluster: 'kube-master-01=https://10.103.236.150:2380,kube-node-01=https://10.103.236.151:2380,kube-node-02=https://10.103.236.152:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: false  # 推荐关闭
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
log-outputs: ["/var/log/etcd/etcd.log"]
force-new-cluster: false
EOF
cat > /etc/etcd/cfg/etcd.yml << EOF
name: 'kube-master-01'
data-dir: /var/lib/etcd
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 8589934592  # 设置为 8GB 或根据需要调整
listen-peer-urls: 'https://10.103.236.150:2380'
listen-client-urls: 'https://10.103.236.150:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
initial-advertise-peer-urls: 'https://10.103.236.150:2380'
advertise-client-urls: 'https://10.103.236.150:2379'
initial-cluster: 'kube-master-01=https://10.103.236.150:2380,kube-node-01=https://10.103.236.151:2380,kube-node-02=https://10.103.236.152:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: false  # 推荐关闭
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
log-outputs: ["/var/log/etcd/etcd.log"]
force-new-cluster: false
EOF
  • 复制ssl证书和配置文件到其他etcd所在节点
bash
[root@kube-master-01 ssl]# Master='kube-node-02 kube-node-01'

for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done


#复制配置文件
[root@kube-master-01 cfg]# cd /etc/etcd/cfg/

[root@kube-master-01 cfg]# scp etcd.yml kube-node-01:/etc/etcd/cfg/

[root@kube-master-01 cfg]# scp etcd.yml kube-node-02:/etc/etcd/cfg/
[root@kube-master-01 ssl]# Master='kube-node-02 kube-node-01'

for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done


#复制配置文件
[root@kube-master-01 cfg]# cd /etc/etcd/cfg/

[root@kube-master-01 cfg]# scp etcd.yml kube-node-01:/etc/etcd/cfg/

[root@kube-master-01 cfg]# scp etcd.yml kube-node-02:/etc/etcd/cfg/
  • 修改name和ip
vim /etc/etcd/cfg/etcd.yml
vim /etc/etcd/cfg/etcd.yml

3.5.4 配置unit文件 ​

bash
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/v3.5.12/
After=network.target
Wants=network-online.target

[Service]
Type=notify
Delegate=yes
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/cfg/etcd.yml
Restart=always
RestartSec=10
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

EOF
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/v3.5.12/
After=network.target
Wants=network-online.target

[Service]
Type=notify
Delegate=yes
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/cfg/etcd.yml
Restart=always
RestartSec=10
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

EOF
  • 启动服务
bash
systemctl daemon-reload
systemctl enable --now etcd
systemctl daemon-reload
systemctl enable --now etcd
  • 验证
bash
export ETCDCTL_API=3
etcdctl --endpoints="10.103.236.150:2379,10.103.236.151:2379,10.103.236.152:2379" --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem  endpoint status --write-out=table
export ETCDCTL_API=3
etcdctl --endpoints="10.103.236.150:2379,10.103.236.151:2379,10.103.236.152:2379" --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem  endpoint status --write-out=table

3.6 部署kubernetes组件 ​

二进制安装,master节点默认是没有污点, kubeadm安装则有污点

3.6.1 master节点-所有master节点 ​

bash
#解压
tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

#验证
[root@kube-master-01 init_pack]# kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server localhost:8080 was refused - did you specify the right host or port?
#解压
tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

#验证
[root@kube-master-01 init_pack]# kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server localhost:8080 was refused - did you specify the right host or port?

如果有其它master节点

bash
MasterNodes='kube-master-02 kube-master-03'
for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; done
MasterNodes='kube-master-02 kube-master-03'
for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; done

1.0 生成k8s所需证书 ​

bash
#所有节点操作
mkdir -p /etc/kubernetes/pki
#所有节点操作
mkdir -p /etc/kubernetes/pki
1.生成ca ​
  • 创建ca-csr文件

自签证书颁发机构（CA）,只在master其中一个节点执行

bash
#master01 节点生成 k8s 证书, 写入生成证书所需的配置文件
cd /etc/kubernetes/pki

cat > ca-csr.json   << EOF 
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

#Kubernetes 使用证书中的 'subject' 的通用名称（Common Name）字段（例如，"/CN=kubernetes"）来 确定用户名， Kubernetes使用证书中的 'subject' 的单位名称 (Organization Name) 字段（例如，"/O=system:masters"）来确定用户组
#master01 节点生成 k8s 证书, 写入生成证书所需的配置文件
cd /etc/kubernetes/pki

cat > ca-csr.json   << EOF 
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

#Kubernetes 使用证书中的 'subject' 的通用名称（Common Name）字段（例如，"/CN=kubernetes"）来 确定用户名， Kubernetes使用证书中的 'subject' 的单位名称 (Organization Name) 字段（例如，"/O=system:masters"）来确定用户组
  • 生成证书
bash
[root@kube-master-01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
[root@kube-master-01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
2.签发apiserver ​
  • 创建apiserver-csr

hosts字段中IP为所有集群成员的ip集群内部ip，一个都不能少！为了方便后期扩容可以多写几个预留的IP或者domain

bash
cat > apiserver-csr.json << EOF 
{
  "CN": "kube-apiserver",
    "hosts": [
        "127.0.0.1",
        "10.0.0.1",
        "10.255.0.1",
        "192.168.0.1",
        "10.103.236.150",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "10.103.236.151",
        "10.103.236.152"
    ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

# signing：表示该证书可用于签名其它证书；生成的 .pem 证书文件中 CA=TRUE
# expiry：默认的证书到期时间为50年,876000h为100年
# profiles：包含了server auth和client auth，所以可以签发三种不同类型证书；expiry 证书有效期，默认50年，可以定义多个profile
# kubernetes：kubernetes这个名字可以自定义，后续指定profiles时要一致，这就是一个单个的profile
# server auth：表示client可以用该 CA 对server提供的证书进行验证
# client auth：表示server可以用该CA对client提供的证书进行验证
cat > apiserver-csr.json << EOF 
{
  "CN": "kube-apiserver",
    "hosts": [
        "127.0.0.1",
        "10.0.0.1",
        "10.255.0.1",
        "192.168.0.1",
        "10.103.236.150",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "10.103.236.151",
        "10.103.236.152"
    ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

# signing：表示该证书可用于签名其它证书；生成的 .pem 证书文件中 CA=TRUE
# expiry：默认的证书到期时间为50年,876000h为100年
# profiles：包含了server auth和client auth，所以可以签发三种不同类型证书；expiry 证书有效期，默认50年，可以定义多个profile
# kubernetes：kubernetes这个名字可以自定义，后续指定profiles时要一致，这就是一个单个的profile
# server auth：表示client可以用该 CA 对server提供的证书进行验证
# client auth：表示server可以用该CA对client提供的证书进行验证

❌ 注意

"192.168.0.1"为咱们的svc网段的第一个地址，您需要根据自己的场景稍作修改。

"10.103.236.158"是负载均衡器的VIP地址。如果没有请用eth0地址

"kubernetes,...,kubernetes.default.svc.cluster.loca"对应的是apiServer解析的A记录。

"10.103.236.151,...,10.103.236.152"对应的是K8S集群的地址。这里也可以多预留几个,后期扩容

  • 生成证书
bash
[root@kube-master-01 pki]# cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem  -config=ca-config.json -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver
[root@kube-master-01 pki]# cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem  -config=ca-config.json -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver
3.生成 apiserver 聚合证书 ​
bash
cat > front-proxy-ca-csr.json  << EOF 
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  },
  "ca": {
    "expiry": "876000h"
  }
}
EOF

#生成
cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

cat > front-proxy-client-csr.json  << EOF 
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF

cfssl gencert  \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem   \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   \
-config=ca-config.json   \
-profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
cat > front-proxy-ca-csr.json  << EOF 
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  },
  "ca": {
    "expiry": "876000h"
  }
}
EOF

#生成
cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

cat > front-proxy-client-csr.json  << EOF 
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF

cfssl gencert  \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem   \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   \
-config=ca-config.json   \
-profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
4.签发controller-manage ​

❌ 注意

注意：节点hosts ip根据所需设置即可。hosts 列表包含所有 kube-controller-manager 节点 IP； CN 为 system:kube- controller-manager

O 为 system:kube-controller-manager， kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限

bash
cat > manager-csr.json << EOF 
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

#生成
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
cat > manager-csr.json << EOF 
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

#生成
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
  • 配置上下文
bash
#设置一个集群
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://10.103.236.150:6443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

#设置一个用户项
kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

#设置一个上下文环境
kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
    --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

#使用默认的上下文    
kubectl config use-context system:kube-controller-manager@kubernetes \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
#设置一个集群
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://10.103.236.150:6443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

#设置一个用户项
kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

#设置一个上下文环境
kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
    --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

#使用默认的上下文    
kubectl config use-context system:kube-controller-manager@kubernetes \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
5.签发kube-scheduler ​

❌ 注意

注意：节点hostsip根据所需设置即可。hosts 列表包含所有 kube-scheduler 节点 IP；

CN 为 system:kube-scheduler、

O 为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限

bash
cat > scheduler-csr.json << EOF 
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

#生成
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
cat > scheduler-csr.json << EOF 
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

#生成
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
  • 配置上下文
bash
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://10.103.236.150:6443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
     --client-certificate=/etc/kubernetes/pki/scheduler.pem \
     --client-key=/etc/kubernetes/pki/scheduler-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
   
kubectl config set-context system:kube-scheduler@kubernetes \
     --cluster=kubernetes \
     --user=system:kube-scheduler \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
kubectl config use-context system:kube-scheduler@kubernetes \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://10.103.236.150:6443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
     --client-certificate=/etc/kubernetes/pki/scheduler.pem \
     --client-key=/etc/kubernetes/pki/scheduler-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
   
kubectl config set-context system:kube-scheduler@kubernetes \
     --cluster=kubernetes \
     --user=system:kube-scheduler \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
kubectl config use-context system:kube-scheduler@kubernetes \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
6.签发admin ​
bash
# 生成 admin 的证书配置
cat > admin-csr.json << EOF 
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

#生成
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
# 生成 admin 的证书配置
cat > admin-csr.json << EOF 
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

#生成
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

❌ 注意

O: system:masters：kube-apiserver 收到使用该证书的客户端请求后，为请求添加组（Group）认证标识 system:masters；

预定义的 ClusterRoleBinding cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予操作集群所需的最高权限；

"hosts": [""],该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空

  • 配置上下文
bash
kubectl config set-cluster kubernetes     \
  --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  --embed-certs=true     \
  --server=https://10.103.236.150:6443     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig
  
kubectl config set-credentials kubernetes-admin  \
  --client-certificate=/etc/kubernetes/pki/admin.pem     \
  --client-key=/etc/kubernetes/pki/admin-key.pem     \
  --embed-certs=true     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig
  
kubectl config set-context kubernetes-admin@kubernetes    \
  --cluster=kubernetes     \
  --user=kubernetes-admin     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig
  
kubectl config use-context kubernetes-admin@kubernetes  --kubeconfig=/etc/kubernetes/admin.kubeconfig
kubectl config set-cluster kubernetes     \
  --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  --embed-certs=true     \
  --server=https://10.103.236.150:6443     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig
  
kubectl config set-credentials kubernetes-admin  \
  --client-certificate=/etc/kubernetes/pki/admin.pem     \
  --client-key=/etc/kubernetes/pki/admin-key.pem     \
  --embed-certs=true     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig
  
kubectl config set-context kubernetes-admin@kubernetes    \
  --cluster=kubernetes     \
  --user=kubernetes-admin     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig
  
kubectl config use-context kubernetes-admin@kubernetes  --kubeconfig=/etc/kubernetes/admin.kubeconfig

❌ 注意

--certificate-authority：验证 kube-apiserver 证书的根证书；

--client-certificate、--client-key：刚生成的 admin 证书和私钥，与 kube-apiserver https 通信时使用；

--embed-certs=true：将 ca.pem 和 admin.pem 证书内容嵌入到生成的 kubectl.kubeconfig 文件中(否则，写入的是证书文件路径，后续拷贝 kubeconfig 到其它机器时，还需要单独拷贝证书文件，不方便。)；

--server：指定 kube-apiserver 的地址;

7.签发kube-proxy ​
bash
# 创建 kube-proxy 证书
cat > kube-proxy-csr.json  << EOF 
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-proxy",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy
# 创建 kube-proxy 证书
cat > kube-proxy-csr.json  << EOF 
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-proxy",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy
  • 配置上下文

10.103.236.150 根据环境进行修改,如果有高可用,则这里修改成高可用ip地址

bash
#设置一个集群
kubectl config set-cluster kubernetes     \
  --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  --embed-certs=true     \
  --server=https://10.103.236.150:6443     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
 
#设置一个用户项
kubectl config set-credentials kube-proxy  \
  --client-certificate=/etc/kubernetes/pki/kube-proxy.pem     \
  --client-key=/etc/kubernetes/pki/kube-proxy-key.pem     \
  --embed-certs=true     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

#设置一个上下文环境
kubectl config set-context kube-proxy@kubernetes    \
  --cluster=kubernetes     \
  --user=kube-proxy     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

#使用默认的上下文
kubectl config use-context kube-proxy@kubernetes  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
#设置一个集群
kubectl config set-cluster kubernetes     \
  --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  --embed-certs=true     \
  --server=https://10.103.236.150:6443     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
 
#设置一个用户项
kubectl config set-credentials kube-proxy  \
  --client-certificate=/etc/kubernetes/pki/kube-proxy.pem     \
  --client-key=/etc/kubernetes/pki/kube-proxy-key.pem     \
  --embed-certs=true     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

#设置一个上下文环境
kubectl config set-context kube-proxy@kubernetes    \
  --cluster=kubernetes     \
  --user=kube-proxy     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

#使用默认的上下文
kubectl config use-context kube-proxy@kubernetes  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

查看证书有效时间

bash
find /etc/kubernetes/pki -type f -name "*.pem" ! -name "*key*" -exec sh -c 'echo "File: {}"; openssl x509 -noout -dates -in "{}"' \;
find /etc/kubernetes/pki -type f -name "*.pem" ! -name "*key*" -exec sh -c 'echo "File: {}"; openssl x509 -noout -dates -in "{}"' \;
8.创建ServiceAccount ​
bash
#ServiceAccount是k8s一种认证方式，创建ServiceAccount的时候会创建一个与之绑定的secret，这个secret会生成一个token
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

#基于sa.key创建sa.pub
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub
#ServiceAccount是k8s一种认证方式，创建ServiceAccount的时候会创建一个与之绑定的secret，这个secret会生成一个token
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

#基于sa.key创建sa.pub
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

分发到其它master节点上面

# 将证书发送到其他 master 节点
# 其他节点创建目录
mkdir  /etc/kubernetes/pki/ -p

for NODE in k8s-master02 k8s-master03; do  for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do  scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done;  for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do  scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done
# 将证书发送到其他 master 节点
# 其他节点创建目录
mkdir  /etc/kubernetes/pki/ -p

for NODE in k8s-master02 k8s-master03; do  for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do  scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done;  for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do  scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done

1.1 kube-Apiserver ​

所有Master节点创建kube-apiserver service

❌ 注意

本文档使用的k8s service网段为192.168.0.0/16，该网段不能和宿主机的网段、Pod网段的重复，请按需修改

  • 所有节点执行
bash
mkdir -p /etc/kubernetes/manifests/ /var/lib/kubelet /var/log/kubernetes /etc/kubernetes/cfg
mkdir -p /etc/kubernetes/manifests/ /var/lib/kubelet /var/log/kubernetes /etc/kubernetes/cfg
  • 创建token.csv文件
bash
cd /etc/kubernetes/cfg/
cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
cd /etc/kubernetes/cfg/
cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
1.创建api-server的配置文件 ​
bash
cat > /etc/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logging-format=json \\
--v=2 \\
--etcd-servers=https://10.103.236.150:2379,https://10.103.236.151:2379,https://10.103.236.152:2379 \\
--bind-address=0.0.0.0 \\
--secure-port=6443 \\
--advertise-address=10.103.236.150 \\
--allow-privileged=true \\
--service-cluster-ip-range=192.168.0.0/16 \\
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--service-node-port-range=30000-61000 \\
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
--client-ca-file=/etc/kubernetes/pki/ca.pem \\
--service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \\
--etcd-certfile=/etc/etcd/ssl/etcd.pem \\
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
--requestheader-allowed-names=aggregator \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/var/log/kubernetes/k8s-audit.log"
EOF
cat > /etc/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logging-format=json \\
--v=2 \\
--etcd-servers=https://10.103.236.150:2379,https://10.103.236.151:2379,https://10.103.236.152:2379 \\
--bind-address=0.0.0.0 \\
--secure-port=6443 \\
--advertise-address=10.103.236.150 \\
--allow-privileged=true \\
--service-cluster-ip-range=192.168.0.0/16 \\
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--service-node-port-range=30000-61000 \\
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
--client-ca-file=/etc/kubernetes/pki/ca.pem \\
--service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \\
--etcd-certfile=/etc/etcd/ssl/etcd.pem \\
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
--requestheader-allowed-names=aggregator \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/var/log/kubernetes/k8s-audit.log"
EOF
2.配置unit文件 ​
bash
cat >/usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
EOF
cat >/usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
EOF
3.启动 ​
bash
systemctl daemon-reload && systemctl enable --now kube-apiserver && systemctl status kube-apiserver
systemctl daemon-reload && systemctl enable --now kube-apiserver && systemctl status kube-apiserver
bash
#查看
[root@kube-master-01 ~]# curl -k https://10.103.236.150:6443/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}
#查看
[root@kube-master-01 ~]# curl -k https://10.103.236.150:6443/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

1.2 kube-controller-manager ​

官档: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

1.创建kube-controller-manager配置文件 ​
bash
cat >/etc/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS=" \
  --bind-address=0.0.0.0 \
  --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
  --service-cluster-ip-range=192.168.0.0/16 \
  --cluster-cidr=172.16.0.0/12 \
  --cluster-name=kubernetes \
  --allocate-node-cidrs=true \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-sync-period=10s \
  --use-service-account-credentials=true \
  --logging-format=json \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --root-ca-file=/etc/kubernetes/pki/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
  --cluster-signing-duration=438000h0m0s  \
  --node-monitor-grace-period=40s \
  --node-monitor-period=5s \
  --allocate-node-cidrs=true \
  --node-cidr-mask-size=24 \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --v=2"
EOF
cat >/etc/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS=" \
  --bind-address=0.0.0.0 \
  --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
  --service-cluster-ip-range=192.168.0.0/16 \
  --cluster-cidr=172.16.0.0/12 \
  --cluster-name=kubernetes \
  --allocate-node-cidrs=true \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-sync-period=10s \
  --use-service-account-credentials=true \
  --logging-format=json \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --root-ca-file=/etc/kubernetes/pki/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
  --cluster-signing-duration=438000h0m0s  \
  --node-monitor-grace-period=40s \
  --node-monitor-period=5s \
  --allocate-node-cidrs=true \
  --node-cidr-mask-size=24 \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --v=2"
EOF

service-cluster-ip-range ----> service 网段

cluster-cidr -->Pod网段

2.创建unit文件 ​
bash
cat >/usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
cat >/usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
3.启动 ​
bash
systemctl daemon-reload ;systemctl --now enable kube-controller-manager;systemctl status kube-controller-manager.service
systemctl daemon-reload ;systemctl --now enable kube-controller-manager;systemctl status kube-controller-manager.service

1.3 kube-scheduler ​

1.创建 kube-scheduler 配置文件 ​
bash
cat >/etc/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logging-format=json \\
--v=2 \\
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig \\
--authentication-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \\
--authorization-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \\
--leader-elect \\
--bind-address=0.0.0.0"
EOF
cat >/etc/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logging-format=json \\
--v=2 \\
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig \\
--authentication-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \\
--authorization-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \\
--leader-elect \\
--bind-address=0.0.0.0"
EOF
2.创建unit文件 ​
bash
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
3.启动 ​
bash
systemctl daemon-reload ;systemctl --now enable kube-scheduler;systemctl status kube-scheduler
systemctl daemon-reload ;systemctl --now enable kube-scheduler;systemctl status kube-scheduler
  • 查看状态

如果这里执行kubectl命令日提示8080,则是没有配置kubeconfig文件

bash
# 查看集群状态
[root@kube-master-01 kubernetes]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok
etcd-0               Healthy   ok
scheduler            Healthy   ok

[root@kube-master-01 kubernetes]# kubectl cluster-info
Kubernetes control plane is running at https://10.103.236.150:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
# 查看集群状态
[root@kube-master-01 kubernetes]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok
etcd-0               Healthy   ok
scheduler            Healthy   ok

[root@kube-master-01 kubernetes]# kubectl cluster-info
Kubernetes control plane is running at https://10.103.236.150:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

1.4 kube-proxy ​

1.创建 kube-proxy 配置文件 ​
bash
cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF
cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF
2.配置 unit文件 ​
bash
# 所有 k8s 节点添加 kube-proxy 的 service 文件
cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy.yaml \\
  --cluster-cidr=172.16.0.0/12 \\
  --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target

EOF
# 所有 k8s 节点添加 kube-proxy 的 service 文件
cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy.yaml \\
  --cluster-cidr=172.16.0.0/12 \\
  --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target

EOF
3.启动 ​
bash
systemctl daemon-reload

systemctl enable --now kube-proxy.service

systemctl status kube-proxy.service
systemctl daemon-reload

systemctl enable --now kube-proxy.service

systemctl status kube-proxy.service

1.5 kubelet ​

1.创建配置文件 ​
bash
cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 192.168.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF
cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 192.168.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF
2.配置unit文件 ​
bash
cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \\
    --cert-dir=/etc/kubernete/pki \\
    --client-ca-file=/etc/kubernetes/pki/ca.pem \\
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
    --config=/etc/kubernetes/kubelet-conf.yml \\
    --container-runtime-endpoint=unix:///run/containerd/containerd.sock \\
    --node-labels=node.kubernetes.io/node= \\
    --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \\
    --cert-dir=/etc/kubernete/pki \\
    --client-ca-file=/etc/kubernetes/pki/ca.pem \\
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
    --config=/etc/kubernetes/kubelet-conf.yml \\
    --container-runtime-endpoint=unix:///run/containerd/containerd.sock \\
    --node-labels=node.kubernetes.io/node= \\
    --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
3.启动 ​
bash
systemctl daemon-reload

systemctl enable --now kubelet.service

systemctl status kubelet.service
systemctl daemon-reload

systemctl enable --now kubelet.service

systemctl status kubelet.service

1.6 TLS Bootstrapping ​

https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/

1.介绍 ​

在一个 Kubernetes 集群中，Worker 节点上的组件（kubelet 和 kube-proxy）需要与 Kubernetes 控制平面组件通信，尤其是 kube-apiserver。 为了通信的安全性， 需要使用到节点上的客户端 TLS 证书。

但是客户端很多，又很难有通用的 TSL 证书直接使用，如果每一次加节点都需要重新生成证书，那维护将变得非常麻烦。

为了简化这一过程，从 1.4 版本开始，Kubernetes 引入了一个证书请求和签名 API

采用 TLS bootstrapping 生成证书的大致简化流程如下：

  • 管理员通过 apiserver 生成一个 bootstrap token 并将它写入到 kubeconfig 文件中。
  • Kubelet 通过 --bootstrap-kubeconfig 启动参数指定 kubeconfig 文件，然后调用 apiserver 的 API 接口，生成自己所需的服务器和客户端证书。
  • 证书生成后，Kubelet 采用生成的证书和 apiserver 进行通信，并删除本地的 kubeconfig 文件，避免 bootstrap token 泄漏。

想要启动该功能，只需要在 apiserver 中启动参数中添加 --enable-bootstrap-token-auth，并创建一个 Kubelet 访问的 bootstrap token secret 即可。

https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/bootstrap-tokens/

2.部署 ​
bash
# 在 master01 上配置
	- "--server"只想的是负载均衡器的IP地址，由负载均衡器对master节点进行反向代理
	- "--token"也可以自定义，但也要同时修改"bootstrap"的Secret的"token-id"和"token-secret"对应值

#设置集群
kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     --server=https://10.103.236.150:6443     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

#创建用户
kubectl config set-credentials tls-bootstrap-token-user     \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

#将集群和用户进行绑定
kubectl config set-context tls-bootstrap-token-user@kubernetes     \
--cluster=kubernetes     \
--user=tls-bootstrap-token-user     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

#配置默认的上下文
kubectl config use-context tls-bootstrap-token-user@kubernetes     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
# 在 master01 上配置
	- "--server"只想的是负载均衡器的IP地址，由负载均衡器对master节点进行反向代理
	- "--token"也可以自定义，但也要同时修改"bootstrap"的Secret的"token-id"和"token-secret"对应值

#设置集群
kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     --server=https://10.103.236.150:6443     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

#创建用户
kubectl config set-credentials tls-bootstrap-token-user     \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

#将集群和用户进行绑定
kubectl config set-context tls-bootstrap-token-user@kubernetes     \
--cluster=kubernetes     \
--user=tls-bootstrap-token-user     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

#配置默认的上下文
kubectl config use-context tls-bootstrap-token-user@kubernetes     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
bash
cat > bootstrap.secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c
  token-secret: 2e4d610cf3e7426e
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-certificate-rotation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver
EOF
cat > bootstrap.secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c
  token-secret: 2e4d610cf3e7426e
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-certificate-rotation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver
EOF
  • 创建
bash
kubectl create -f bootstrap.secret.yaml
kubectl create -f bootstrap.secret.yaml

3.6.2 node节点 ​

1.部署 ​

bash
WorkNodes='kube-node-01 kube-node-02'
for NODE in $WorkNodes; do echo $NODE; scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/; done
WorkNodes='kube-node-01 kube-node-02'
for NODE in $WorkNodes; do echo $NODE; scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/; done
1.复制证书 ​
bash
# 在 master01 上将证书复制到 node 节点
cd /etc/kubernetes/
 
for NODE in kube-node-01 kube-node-02; do ssh $NODE mkdir -p /etc/kubernetes/{pki,manifests}; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done
# 在 master01 上将证书复制到 node 节点
cd /etc/kubernetes/
 
for NODE in kube-node-01 kube-node-02; do ssh $NODE mkdir -p /etc/kubernetes/{pki,manifests}; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done

2.kubelet ​

1.创建配置文件 ​
bash
cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 192.168.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

EOF
cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 192.168.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

EOF
2.配置unit文件 ​
bash
cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \\
    --cert-dir=/etc/kubernete/pki \\
    --client-ca-file=/etc/kubernetes/pki/ca.pem \\
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
    --config=/etc/kubernetes/kubelet-conf.yml \\
    --container-runtime-endpoint=unix:///run/containerd/containerd.sock \\
    --node-labels=node.kubernetes.io/node= \\
    --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \\
    --cert-dir=/etc/kubernete/pki \\
    --client-ca-file=/etc/kubernetes/pki/ca.pem \\
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
    --config=/etc/kubernetes/kubelet-conf.yml \\
    --container-runtime-endpoint=unix:///run/containerd/containerd.sock \\
    --node-labels=node.kubernetes.io/node= \\
    --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
3.启动 ​
bash
systemctl daemon-reload

systemctl enable --now kubelet.service

systemctl status kubelet.service
systemctl daemon-reload

systemctl enable --now kubelet.service

systemctl status kubelet.service

3.kube-proxy ​

1.创建 kube-proxy 配置文件 ​
bash
# 所有 k8s节点添加 kube-proxy 的配置
cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
cgroupDriver: systemd
clusterDNS:
- 192.168.0.10
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
# 所有 k8s节点添加 kube-proxy 的配置
cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
cgroupDriver: systemd
clusterDNS:
- 192.168.0.10
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
2.配置 unit文件 ​
bash
# 所有 k8s 节点添加 kube-proxy 的 service 文件
cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy.yaml \\
  --cluster-cidr=172.16.0.0/12 \\
  --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target

EOF
# 所有 k8s 节点添加 kube-proxy 的 service 文件
cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy.yaml \\
  --cluster-cidr=172.16.0.0/12 \\
  --v=2
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target

EOF
3.启动 ​
bash
systemctl daemon-reload

systemctl enable --now kube-proxy.service

systemctl status kube-proxy.service
systemctl daemon-reload

systemctl enable --now kube-proxy.service

systemctl status kube-proxy.service

3.6.3检查集群健康性 ​

bash
kubectl get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   ok


kubectl get --raw='/readyz?verbose'
...
readyz check passed

kubectl get --raw='/livez?verbose'
...
livez check passed
kubectl get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   ok


kubectl get --raw='/readyz?verbose'
...
readyz check passed

kubectl get --raw='/livez?verbose'
...
livez check passed

3.6.4 部署网络插件-master节点 ​

  • 版本兼容性
calico版本kubernetes版本
v3.18v1.20、1.19、1.18
v3.19v1.21、v1.20、1.19
v3.20v1.21、v1.20、1.19
v3.21v1.22、v1.21、v1.20
v3.22v1.23、v1.22、v1.21
v3.23v1.23、v1.22、v1.21
v3.24v1.25、v1.24、v1.23、v1.22
v3.26v1.28、v1.27、v1.26、v1.25、v1.24
v3.27v1.29、v1.28、v1.27
v3.28v1.30、v1.29、v1.28、v1.27

1.calico ​

bash
wget https://github.com/projectcalico/calico/blob/master/manifests/calico-typha.yaml

或者
wget https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml
wget https://github.com/projectcalico/calico/blob/master/manifests/calico-typha.yaml

或者
wget https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml
bash
#修改POD网段
 - name: CALICO_IPV4POOL_CIDR
   value: "172.16.0.0/12"

# 修改为 BGP 模式
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "Always"  #改成Off
#修改POD网段
 - name: CALICO_IPV4POOL_CIDR
   value: "172.16.0.0/12"

# 修改为 BGP 模式
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "Always"  #改成Off
bash
#或者通过sed修改
sed -i 's/# - name: CALICO_IPV4POOL_CIDR/- name: CALICO_IPV4POOL_CIDR/g' calico.yaml
sed -i 's/#   value: "192.168.0.0\/16"/  value: "172.16.0.0\/12"/g' calico.yaml
sed -i 's/"type": "calico-ipam"/"type": "calico-ipam",\n              "assign_ipv4": "true"/g' calico.yaml
sed -i "s#docker.io/calico/#m.daocloud.io/docker.io/calico/#g" calico.yaml
#或者通过sed修改
sed -i 's/# - name: CALICO_IPV4POOL_CIDR/- name: CALICO_IPV4POOL_CIDR/g' calico.yaml
sed -i 's/#   value: "192.168.0.0\/16"/  value: "172.16.0.0\/12"/g' calico.yaml
sed -i 's/"type": "calico-ipam"/"type": "calico-ipam",\n              "assign_ipv4": "true"/g' calico.yaml
sed -i "s#docker.io/calico/#m.daocloud.io/docker.io/calico/#g" calico.yaml
  • 执行
bash
kubectl apply -f calico.yaml
kubectl apply -f calico.yaml
  • 查看效果
bash
[root@kube-master-01 init_pack]# kubectl get pod -A -owide
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE   IP               NODE             NOMINATED NODE   READINESS GATES
kube-node-02     <none>           <none>
kube-system   calico-kube-controllers-bd77b565b-tv5jt   1/1     Running   0          34m   10.88.0.2        kube-node-01     <none>           <none>
kube-system   calico-node-2nc7p                         1/1     Running   0          34m   10.103.236.152   kube-node-02     <none>           <none>
kube-system   calico-node-8qw5w                         1/1     Running   0          34m   10.103.236.151   kube-node-01     <none>           <none>
kube-system   calico-node-sn9xm                         1/1     Running   0          34m   10.103.236.150   kube-master-01   <none>           <none>
[root@kube-master-01 init_pack]# kubectl get pod -A -owide
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE   IP               NODE             NOMINATED NODE   READINESS GATES
kube-node-02     <none>           <none>
kube-system   calico-kube-controllers-bd77b565b-tv5jt   1/1     Running   0          34m   10.88.0.2        kube-node-01     <none>           <none>
kube-system   calico-node-2nc7p                         1/1     Running   0          34m   10.103.236.152   kube-node-02     <none>           <none>
kube-system   calico-node-8qw5w                         1/1     Running   0          34m   10.103.236.151   kube-node-01     <none>           <none>
kube-system   calico-node-sn9xm                         1/1     Running   0          34m   10.103.236.150   kube-master-01   <none>           <none>
calicoctl客户端工具 ​

下载地址：https://github.com/projectcalico/calico

bash
#创建配置文件
mkdir /etc/calico -p
cat >/etc/calico/calicoctl.cfg <<EOF
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/root/.kube/config"
EOF


#验证
[root@kube-master-01 init_pack]# calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+----------------+-------------------+-------+----------+-------------+
| 10.103.236.151 | node-to-node mesh | up    | 02:11:45 | Established |
| 10.103.236.152 | node-to-node mesh | up    | 02:11:48 | Established |
+----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
#创建配置文件
mkdir /etc/calico -p
cat >/etc/calico/calicoctl.cfg <<EOF
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/root/.kube/config"
EOF


#验证
[root@kube-master-01 init_pack]# calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+----------------+-------------------+-------+----------+-------------+
| 10.103.236.151 | node-to-node mesh | up    | 02:11:45 | Established |
| 10.103.236.152 | node-to-node mesh | up    | 02:11:48 | Established |
+----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

2.coredns ​

官方文档，https://coredns.io/plugins/ready/

CoreDNS用于集群内部Service名称解析

bash
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed

#修改corefile.yaml
  Corefile: |
    .:53 {
        log
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa { #修改为
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf #修改为本机的resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }#去掉这个地方的后缀
#修改文件，参考/etc/kubernetes/kubelet-config.yml 配置文件中的clusterIP
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.10 #根据自己需求进行修改
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed

#修改corefile.yaml
  Corefile: |
    .:53 {
        log
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa { #修改为
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf #修改为本机的resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }#去掉这个地方的后缀
#修改文件，参考/etc/kubernetes/kubelet-config.yml 配置文件中的clusterIP
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.10 #根据自己需求进行修改

3.6.5部署 Metrics Server ​

各版本对应关系

Metrics Serverkubernetes
v0.7.2v1.29.8
v0.7.1v1.29.2
v0.6.3v1.23.17
v0.6.0v1.22.2
bash
#下载最新
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
#下载最新
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
  • 修改配置
bash
# 修改配置
vim components.yaml

---
# 1
			- args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
        - --requestheader-username-headers=X-Remote-User
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-extra-headers-prefix=X-Remote-Extra-


# 2
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
        - name: ca-ssl
          mountPath: /etc/kubernetes/pki

# 3
      volumes:
      - emptyDir: {}
        name: tmp-dir
      - name: ca-ssl
        hostPath:
          path: /etc/kubernetes/pki
---
# 修改配置
vim components.yaml

---
# 1
			- args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
        - --requestheader-username-headers=X-Remote-User
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-extra-headers-prefix=X-Remote-Extra-


# 2
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
        - name: ca-ssl
          mountPath: /etc/kubernetes/pki

# 3
      volumes:
      - emptyDir: {}
        name: tmp-dir
      - name: ca-ssl
        hostPath:
          path: /etc/kubernetes/pki
---
  • 执行
bash
kubectl apply -f components.yaml
kubectl apply -f components.yaml
  • 查看效果
[root@kube-master-01 init_pack]# kubectl top node
NAME             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
kube-master-01   263m         13%    1487Mi          39%
kube-node-01     212m         10%    593Mi           15%
kube-node-02     162m         8%     566Mi           14%
[root@kube-master-01 init_pack]# kubectl top node
NAME             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
kube-master-01   263m         13%    1487Mi          39%
kube-node-01     212m         10%    593Mi           15%
kube-node-02     162m         8%     566Mi           14%

3.6.6 验证集群 ​

集群结果：

  1. Pod必须能解析Service
  2. Pod必须能解析跨namespace的Service
  3. 每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53
  4. Pod和Pod之前要能通

a) 同namespace能通信

b) 跨namespace能通信

c) 跨机器能通信

1.创建 ​

bash
kubectl create deployment nginx --image=nginx --replicas=3

#创建负载均衡
kubectl expose deployment nginx --port=8080 --target-port=80

#查看pod，默认命名空间是defalut
[root@kube-master-01 CoreDNS]# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7854ff8877-8z9jj   1/1     Running   0          118s
nginx-7854ff8877-985nf   1/1     Running   0          118s
nginx-7854ff8877-9sszv   1/1     Running   0          118s

#查看service
[root@kube-master-01 CoreDNS]#  kubectl get service
NAME         TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   192.168.0.1       <none>        443/TCP    151m
nginx        ClusterIP   192.168.212.179   <none>        8080/TCP   119s

#验证svc
[root@kube-master-01 CoreDNS]# curl -I 192.168.212.179:8080
HTTP/1.1 200 OK
Server: nginx/1.27.1
Date: Sun, 22 Sep 2024 05:25:47 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Mon, 12 Aug 2024 14:21:01 GMT
Connection: keep-alive
ETag: "66ba1a4d-267"
Accept-Ranges: bytes
kubectl create deployment nginx --image=nginx --replicas=3

#创建负载均衡
kubectl expose deployment nginx --port=8080 --target-port=80

#查看pod，默认命名空间是defalut
[root@kube-master-01 CoreDNS]# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7854ff8877-8z9jj   1/1     Running   0          118s
nginx-7854ff8877-985nf   1/1     Running   0          118s
nginx-7854ff8877-9sszv   1/1     Running   0          118s

#查看service
[root@kube-master-01 CoreDNS]#  kubectl get service
NAME         TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   192.168.0.1       <none>        443/TCP    151m
nginx        ClusterIP   192.168.212.179   <none>        8080/TCP   119s

#验证svc
[root@kube-master-01 CoreDNS]# curl -I 192.168.212.179:8080
HTTP/1.1 200 OK
Server: nginx/1.27.1
Date: Sun, 22 Sep 2024 05:25:47 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Mon, 12 Aug 2024 14:21:01 GMT
Connection: keep-alive
ETag: "66ba1a4d-267"
Accept-Ranges: bytes
yaml
#或者

cat <<EOF | kubectl create -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.1
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30080
  type: NodePort
EOF

# 查看
kubectl get pod
kubectl get svc
#或者

cat <<EOF | kubectl create -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.1
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30080
  type: NodePort
EOF

# 查看
kubectl get pod
kubectl get svc

2.检查ip连通性 ​

bash
[root@kube-master-01 CoreDNS]# kubectl get pod -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP              NODE             NOMINATED NODE   READINESS GATES
nginx-7854ff8877-8z9jj   1/1     Running   0          4m35s   172.18.10.65    kube-node-01     <none>           <none>
nginx-7854ff8877-985nf   1/1     Running   0          4m35s   172.25.241.2    kube-node-02     <none>           <none>
nginx-7854ff8877-9sszv   1/1     Running   0          4m35s   172.18.201.68   kube-master-01   <none>           <none>

#在每个节点上ping其他pod节点上的ip
kubectl exec -ti busybox -- sh
/#ping  pod_ip
[root@kube-master-01 CoreDNS]# kubectl get pod -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP              NODE             NOMINATED NODE   READINESS GATES
nginx-7854ff8877-8z9jj   1/1     Running   0          4m35s   172.18.10.65    kube-node-01     <none>           <none>
nginx-7854ff8877-985nf   1/1     Running   0          4m35s   172.25.241.2    kube-node-02     <none>           <none>
nginx-7854ff8877-9sszv   1/1     Running   0          4m35s   172.18.201.68   kube-master-01   <none>           <none>

#在每个节点上ping其他pod节点上的ip
kubectl exec -ti busybox -- sh
/#ping  pod_ip

3.检查service可达性 ​

bash
#在每个节点上访问服务

curl service-ip:port
#在每个节点上访问服务

curl service-ip:port

4.检查dns ​

bash
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF
bash
#进入pod容器中去
[root@kube-master-01 CoreDNS]# kubectl exec -it nginx-7854ff8877-9sszv /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-7854ff8877-9sszv:/#



#查看dns解析
root@nginx-7854ff8877-9sszv:/#  cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 192.168.0.10
options ndots:5

[root@kube-master-01 CoreDNS]# kubectl get service -n kube-system
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   192.168.0.10   <none>        53/UDP,53/TCP,9153/TCP   24m


root@nginx-7854ff8877-9sszv:/# curl -I nginx.default.svc.cluster.local:8080
#进入pod容器中去
[root@kube-master-01 CoreDNS]# kubectl exec -it nginx-7854ff8877-9sszv /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-7854ff8877-9sszv:/#



#查看dns解析
root@nginx-7854ff8877-9sszv:/#  cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 192.168.0.10
options ndots:5

[root@kube-master-01 CoreDNS]# kubectl get service -n kube-system
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   192.168.0.10   <none>        53/UDP,53/TCP,9153/TCP   24m


root@nginx-7854ff8877-9sszv:/# curl -I nginx.default.svc.cluster.local:8080
  • 用pod解析默认命名空间中的kubernetes
bash
kubectl get svc
#NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes    ClusterIP   192.168.0.1      <none>        443/TCP   32d

[root@kube-master-01 ~]# kubectl exec  busybox -n default -- nslookup kubernetes
Server:    192.168.0.10
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 192.168.0.1 kubernetes.default.svc.cluster.local
kubectl get svc
#NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes    ClusterIP   192.168.0.1      <none>        443/TCP   32d

[root@kube-master-01 ~]# kubectl exec  busybox -n default -- nslookup kubernetes
Server:    192.168.0.10
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 192.168.0.1 kubernetes.default.svc.cluster.local

  • 测试跨命名空间是否可以解析 ​

bash
[root@kube-master-01 ~]# kubectl exec  busybox -n default -- nslookup kube-dns.kube-system
Server:    192.168.0.10
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local
[root@kube-master-01 ~]# kubectl exec  busybox -n default -- nslookup kube-dns.kube-system
Server:    192.168.0.10
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

3.7 安装命令补全 ​

官当

bash
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
bash
alias ectl='etcdhelper  --endpoints="10.103.236.150:2379,10.103.236.151:2379,10.103.236.152:2379" --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem'

# 查看当前的 schedule 主节点
ectl get /registry/leases/kube-system/kube-scheduler

# 查看当前的 controllermanager 主节点
ectl get /registry/leases/kube-system/kube-controller-manager



cat >> /etc/profile << EOF
# ETCD 变量
export ETCDCTL_API=3
alias etcdctl='etcdctl --endpoints="10.103.236.150:2379,10.103.236.151:2379,10.103.236.152:2379" --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem'
EOF
alias ectl='etcdhelper  --endpoints="10.103.236.150:2379,10.103.236.151:2379,10.103.236.152:2379" --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem'

# 查看当前的 schedule 主节点
ectl get /registry/leases/kube-system/kube-scheduler

# 查看当前的 controllermanager 主节点
ectl get /registry/leases/kube-system/kube-controller-manager



cat >> /etc/profile << EOF
# ETCD 变量
export ETCDCTL_API=3
alias etcdctl='etcdctl --endpoints="10.103.236.150:2379,10.103.236.151:2379,10.103.236.152:2379" --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem'
EOF

读取 etcd 的方式：https://github.com/openshift/origin/tree/master/tools/etcdhelper

3.8 修改k8s数据目录 ​

3.9 FAQ ​

[ERROR] Get "https://192.168.0.1:443/api?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1,10.103.236.150,10.103.236.151,10.103.236.152, not 192.168.0.1

报错原因: 使用"192.168.0.1"作为svc地址，和证书预定义的svc的IP地址不匹配导致的错误。