SRE文档

深浅模式

基于centos7.x

1.下载地址 ​

libseccomp-2.5.1 ​

2.安装libseccomp ​

注释：

在containerd前，需要优先升级libseccomp。

在CentOS 7中yum下载libseccomp的版本是2.3的，版本不满足最新 Containerd 的需求，需要下载2.4以上的

2.1卸载 ​

bash
[root@ecs-65685 ~]# rpm -qa | grep libseccomp
libseccomp-2.3.1-4.el7.x86_64
[root@ecs-65685 ~]# 
[root@ecs-65685 ~]# rpm -e libseccomp-2.3.1-4.el7.x86_64 --nodeps
[root@ecs-65685 ~]# rpm -qa | grep libseccomp
libseccomp-2.3.1-4.el7.x86_64
[root@ecs-65685 ~]# 
[root@ecs-65685 ~]# rpm -e libseccomp-2.3.1-4.el7.x86_64 --nodeps

2.2下载 ​

bash
wget http://rpmfind.net/linux/centos/8-stream/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm


root@k8s-master01 sbin]# rpm -ivh libseccomp-2.5.1-1.el8.x86_64.rpm 
警告：libseccomp-2.5.1-1.el8.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID 8483c65d: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:libseccomp-2.5.1-1.el8           ################################# [100%]

#查看新版本
[root@k8s-master01 sbin]# rpm -qa | grep libseccomp
libseccomp-2.5.1-1.el8.x86_64

#验证runc
[root@k8s-master01 sbin]# ./runc -version
runc version 1.1.9
commit: v1.1.9-0-gccaecfcb
spec: 1.0.2-dev
go: go1.20.8
libseccomp: 2.5.1
wget http://rpmfind.net/linux/centos/8-stream/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm


root@k8s-master01 sbin]# rpm -ivh libseccomp-2.5.1-1.el8.x86_64.rpm 
警告：libseccomp-2.5.1-1.el8.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID 8483c65d: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:libseccomp-2.5.1-1.el8           ################################# [100%]

#查看新版本
[root@k8s-master01 sbin]# rpm -qa | grep libseccomp
libseccomp-2.5.1-1.el8.x86_64

#验证runc
[root@k8s-master01 sbin]# ./runc -version
runc version 1.1.9
commit: v1.1.9-0-gccaecfcb
spec: 1.0.2-dev
go: go1.20.8
libseccomp: 2.5.1

CentOS 8.2 默认2.5.1，可以不用更新

注意：使用场景在 Kubernetes 1.24+版本中的Containerd会用到

2.3不升级libseccomp ​

由于二进制包中提供的runC默认需要系统中安装seccomp支持，需要单独安装，且不同版本runC对seccomp版本要求一致，所以建议单独下载runC 二进制包进行安装，里面包含了seccomp模块支持

bash
wget https://github.com/opencontainers/runc/releases/download/v1.1.9/runc.amd64
[root@k8s-master01 tmp]#mv runc.amd64 runc && chmod +x runc

[root@k8s-master01 tmp]# ./runc  -version
runc version 1.1.9
commit: v1.1.9-0-gccaecfcb
spec: 1.0.2-dev
go: go1.20.3
libseccomp: 2.5.4
wget https://github.com/opencontainers/runc/releases/download/v1.1.9/runc.amd64
[root@k8s-master01 tmp]#mv runc.amd64 runc && chmod +x runc

[root@k8s-master01 tmp]# ./runc  -version
runc version 1.1.9
commit: v1.1.9-0-gccaecfcb
spec: 1.0.2-dev
go: go1.20.3
libseccomp: 2.5.4

3. 安装Containerd ​

https://github.com/containerd/containerd/blob/main/docs/getting-started.md

3.1 下载 ​

bash
wget https://github.com/containerd/containerd/releases/download/v1.7.7/cri-containerd-cni-1.7.7-linux-amd64.tar.gz
wget https://github.com/containerd/containerd/releases/download/v1.7.7/cri-containerd-cni-1.7.7-linux-amd64.tar.gz

3.2 二进制安装 ​

Containerd有两种安装包：

  • 第一种是containerd-xxx,这种包用于单机测试没问题，不包含runC，需要提前安装。
  • 第二种是cri-containerd-cni-xxxx，包含runc和k8s里的所需要的相关文件。k8s集群里需要用到此包。虽然包含runC，但是依赖系统中的seccomp（安全计算模式，是一种限制容器调用系统资源的模式。），推荐这个模式安装
bash
#解压
[root@other test]#  tar zxvf cri-containerd-cni-1.7.7-linux-amd64.tar.gz -C /
[root@other test]# tree -L 3 etc opt usr/
etc
├── cni
│   └── net.d
│       └── 10-containerd-net.conflist
├── crictl.yaml
└── systemd
    └── system
        └── containerd.service
opt
├── cni
│   └── bin
│       ├── bandwidth
│       ├── bridge
│       ├── dhcp
│       ├── dummy
│       ├── firewall
│       ├── host-device
│       ├── host-local
│       ├── ipvlan
│       ├── loopback
│       ├── macvlan
│       ├── portmap
│       ├── ptp
│       ├── sbr
│       ├── static
│       ├── tuning
│       ├── vlan
│       └── vrf
└── containerd
    └── cluster
        ├── gce
        └── version
usr/
└── local
    ├── bin
    │   ├── containerd
    │   ├── containerd-shim
    │   ├── containerd-shim-runc-v1
    │   ├── containerd-shim-runc-v2
    │   ├── containerd-stress
    │   ├── crictl
    │   ├── critest
    │   ├── ctd-decoder
    │   └── ctr
    └── sbin
        └── runc

#把上面的复制到相关路径
#解压
[root@other test]#  tar zxvf cri-containerd-cni-1.7.7-linux-amd64.tar.gz -C /
[root@other test]# tree -L 3 etc opt usr/
etc
├── cni
│   └── net.d
│       └── 10-containerd-net.conflist
├── crictl.yaml
└── systemd
    └── system
        └── containerd.service
opt
├── cni
│   └── bin
│       ├── bandwidth
│       ├── bridge
│       ├── dhcp
│       ├── dummy
│       ├── firewall
│       ├── host-device
│       ├── host-local
│       ├── ipvlan
│       ├── loopback
│       ├── macvlan
│       ├── portmap
│       ├── ptp
│       ├── sbr
│       ├── static
│       ├── tuning
│       ├── vlan
│       └── vrf
└── containerd
    └── cluster
        ├── gce
        └── version
usr/
└── local
    ├── bin
    │   ├── containerd
    │   ├── containerd-shim
    │   ├── containerd-shim-runc-v1
    │   ├── containerd-shim-runc-v2
    │   ├── containerd-stress
    │   ├── crictl
    │   ├── critest
    │   ├── ctd-decoder
    │   └── ctr
    └── sbin
        └── runc

#把上面的复制到相关路径
  • 配置文件
bash
[root@other sbin]# mkdir /etc/containerd

#生成默认配置
containerd config default > /etc/containerd/config.toml

#修改，单机或者k8s都适用的配置
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = -999

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7" #配置了沙箱镜像
    stats_collect_period = 10
    systemd_cgroup = true
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.runtimes.runc.options]
        BinaryName = ""
        CriuImagePath = ""
        CriuPath = ""
        CriuWorkPath = ""
        IoGid = 0
        IoUid = 0
        NoNewKeyring = false
        NoPivotRoot = false
        Root = ""
        ShimCgroup = ""
        SystemdCgroup = true
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/kube/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = "/etc/cni/net.d/10-default.conf"
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = [
            "https://docker.mirrors.ustc.edu.cn",
            "http://hub-mirror.c.163.com"
          ]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn"
          ]
        [plugins.cri.registry.mirrors."k8s.gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn/google-containers/"
          ]
        [plugins.cri.registry.mirrors."quay.io"]
          endpoint = [
            "https://quay.mirrors.ustc.edu.cn"
          ]
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"
[root@other sbin]# mkdir /etc/containerd

#生成默认配置
containerd config default > /etc/containerd/config.toml

#修改，单机或者k8s都适用的配置
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = -999

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7" #配置了沙箱镜像
    stats_collect_period = 10
    systemd_cgroup = true
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.runtimes.runc.options]
        BinaryName = ""
        CriuImagePath = ""
        CriuPath = ""
        CriuWorkPath = ""
        IoGid = 0
        IoUid = 0
        NoNewKeyring = false
        NoPivotRoot = false
        Root = ""
        ShimCgroup = ""
        SystemdCgroup = true
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/kube/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = "/etc/cni/net.d/10-default.conf"
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = [
            "https://docker.mirrors.ustc.edu.cn",
            "http://hub-mirror.c.163.com"
          ]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn"
          ]
        [plugins.cri.registry.mirrors."k8s.gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn/google-containers/"
          ]
        [plugins.cri.registry.mirrors."quay.io"]
          endpoint = [
            "https://quay.mirrors.ustc.edu.cn"
          ]
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

❌ 注意

这个配置文件是给crictl和kubelet使用，ctr是不可以用这个配置文件的，ctr 不使用 CRI，因此它不读取plugins."io.containerd.grpc.v1.cri"配置

  • 配置systemd
bash
cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF
cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF
  • 启动
bash
#重新加载配置
systemctl daemon-reload

#启动
systemctl enable --now containerd.service ; systemctl status containerd.service

[root@other containerd]# systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since 四 2023-10-26 10:59:58 CST; 6s ago
     Docs: https://containerd.io
  Process: 1767 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 1779 (containerd)
    Tasks: 8
   Memory: 15.9M
   CGroup: /system.slice/containerd.service
           └─1779 /usr/local/bin/containerd
#重新加载配置
systemctl daemon-reload

#启动
systemctl enable --now containerd.service ; systemctl status containerd.service

[root@other containerd]# systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since 四 2023-10-26 10:59:58 CST; 6s ago
     Docs: https://containerd.io
  Process: 1767 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 1779 (containerd)
    Tasks: 8
   Memory: 15.9M
   CGroup: /system.slice/containerd.service
           └─1779 /usr/local/bin/containerd
  • 查看版本号
bash
[root@kube-master-01 containers]# ctr version
Client:
  Version:  v1.7.7
  Revision: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
  Go version: go1.22.7

Server:
  Version:  v1.7.7
  Revision: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
  UUID: e99af880-4220-489a-b0a0-6ae2fb786877
[root@kube-master-01 containers]# ctr version
Client:
  Version:  v1.7.7
  Revision: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
  Go version: go1.22.7

Server:
  Version:  v1.7.7
  Revision: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
  UUID: e99af880-4220-489a-b0a0-6ae2fb786877

https://blog.csdn.net/ChaITSimpleLove/article/details/125031074

https://blog.huweihuang.com/

4.container客户端工具 ​

客户端工具有两种，分别是crictl和nerdctl

推荐使用nerdctl，使用效果与docker命令的语法一致

github下载链接：https://github.com/containerd/nerdctl/releases

4.1 crictl ​

crictl 是 CRI 兼容的容器运行时命令行接口，和containerd无关，由Kubernetes提供，可以使用它来检查和调试 k8s 节点上的容器运行时和应用程序。

下载地址：https://github.com/kubernetes-sigs/cri-tools/releases

bash
# 下载,根据k8s版本进行安装
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.27.1/crictl-v1.27.1-linux-amd64.tar.gz

# 解压
tar -zxvf crictl-v1.27.1-linux-amd64.tar.gz -C /usr/local/bin

# 配置
cat > /etc/crictl.yaml << EOF
runtime-endpoint: "unix:///run/containerd/containerd.sock"
image-endpoint: "unix:///run/containerd/containerd.sock"
timeout: 0
debug: false
pull-image-on-create: false
disable-pull-on-run: false
EOF

# 验证
crictl version
# 下载,根据k8s版本进行安装
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.27.1/crictl-v1.27.1-linux-amd64.tar.gz

# 解压
tar -zxvf crictl-v1.27.1-linux-amd64.tar.gz -C /usr/local/bin

# 配置
cat > /etc/crictl.yaml << EOF
runtime-endpoint: "unix:///run/containerd/containerd.sock"
image-endpoint: "unix:///run/containerd/containerd.sock"
timeout: 0
debug: false
pull-image-on-create: false
disable-pull-on-run: false
EOF

# 验证
crictl version

4.2 nerdctl ​

containerd虽然可直接提供给终端用户直接使用，也提供了命令行工具(ctr)，但并不是很友好，所以nerdctl应运而生，它也是containerd的命令行工具，支持docker cli关于容器生命周期管理的所有命令，并且支持docker compose (nerdctl compose up).

下载地址：https://github.com/containerd/nerdctl/releases

  • 精简 (nerdctl--linux-amd64.tar.gz): 只包含nerdctl
  • 完整 (nerdctl-full--linux-amd64.tar.gz): 包含 containerd, runc, and CNI等依赖

nerdctl 的目标并不是单纯地复制 docker 的功能，它还实现了很多 docker 不具备的功能，例如延迟拉取镜像（lazy-pulling）、镜像加密（imgcrypt）等

bash
#二进制包,如果没有安装 containerd，则可以下载 nerdctl-full-<VERSION>-linux-amd64.tar.gz 包进行安装
wget https://github.com/containerd/nerdctl/releases/download/v1.6.2/nerdctl-1.6.2-linux-amd64.tar.gz

#解压cp
tar xvf nerdctl-1.6.2-linux-amd64.tar.gz

cp nerdctl /usr/local/bin/

#验证
nerdctl version
#二进制包,如果没有安装 containerd，则可以下载 nerdctl-full-<VERSION>-linux-amd64.tar.gz 包进行安装
wget https://github.com/containerd/nerdctl/releases/download/v1.6.2/nerdctl-1.6.2-linux-amd64.tar.gz

#解压cp
tar xvf nerdctl-1.6.2-linux-amd64.tar.gz

cp nerdctl /usr/local/bin/

#验证
nerdctl version