SRE文档

深浅模式

官当

1. 安装MinIO Tenant ​

用于在多租户环境中实现数据隔离、资源管理和安全性。

租户架构

1.1 创建目录 ​

对 MiniO 来说，使用 Local 类型的 pv 进行数据的持久化的存储。

在每个节点创建不同的目录，用于 local 类型的本地存储目录

bash
 mkdir -p /data/minio/{pv1,pv2,pv3,pv4}
 mkdir -p /data/minio/{pv1,pv2,pv3,pv4}

1.2 创建 StorageClass ​

cat 1.minio-local-storage.yaml

yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: minio-local-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: minio-local-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

❌ 注意

注意：volumeBindingMode: WaitForFirstConsumer 必须要设置为 WaitForFirstConsumer

随机的绑定有可能导致，一个pod关联了不同主机的pv导致启动失败

  • 查看
bash
[root@kube-master minio]# kubectl get sc | grep minio
minio-local-storage                 kubernetes.io/no-provisioner                  Retain          WaitForFirstConsumer   false          2m10s
[root@kube-master minio]# kubectl get sc | grep minio
minio-local-storage                 kubernetes.io/no-provisioner                  Retain          WaitForFirstConsumer   false          2m10s

1.3 创建pv ​

cat 2.pv.yaml

yaml
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-01
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01           ##此部分通过 kubectl get node --show-labels |grep kubernetes.io/hostname 查看
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-02
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-03
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-04
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01
###################
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-01
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-02
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-03
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-04
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-01
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-02
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-03
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-04
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-01
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01           ##此部分通过 kubectl get node --show-labels |grep kubernetes.io/hostname 查看
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-02
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-03
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: kube-node01-04
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node01
###################
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-01
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-02
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-03
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-02-04
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node02
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-01
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-02
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-03
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: k8snode-03-04
spec:
   capacity:
      storage: 1Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - kube-node03

每个pv都是使用local类型，节点亲和的方式和具体的节点进行绑定

  • 查看
bash
[root@kube-master minio]# kubectl get pv,pvc |grep minio
persistentvolume/k8snode-02-02                              1Gi        RWO            Retain           Available            minio-local-storage                2m56s
persistentvolume/k8snode-03-03                              1Gi        RWO            Retain           Available            minio-local-storage                2m56s
persistentvolume/kube-node01-01                             1Gi        RWO            Retain           Available            minio-local-storage                2m56s
[root@kube-master minio]# kubectl get pv,pvc |grep minio
persistentvolume/k8snode-02-02                              1Gi        RWO            Retain           Available            minio-local-storage                2m56s
persistentvolume/k8snode-03-03                              1Gi        RWO            Retain           Available            minio-local-storage                2m56s
persistentvolume/kube-node01-01                             1Gi        RWO            Retain           Available            minio-local-storage                2m56s

1.4 创建命名空间 ​

bash
kubectl create ns minio-tenant-1
kubectl create ns minio-tenant-1

1.5 创建集群 ​

要求: with 3 servers must have at least 6 volumes

bash
kubectl minio tenant create minio-1 --namespace minio-tenant-1 --servers 3 --volumes 12 --capacity 12Gi --storage-class minio-local-storage --image registry.cn-zhangjiakou.aliyuncs.com/hsuing/minio:RELEASE.2023-04-20T17-56-55Z --disable-tls

Tenant 'minio-1' created in 'minio-tenant-1' Namespace

  Username: KIHKL495T13VF7IPBKQ2
  Password: ZT6TnoGdPK0AOwQ1Trq2jSGacXKZOi5CIsfS5hc4
  Note: Copy the credentials to a secure location. MinIO will not display these again.

APPLICATION	SERVICE NAME   	NAMESPACE     	SERVICE TYPE	SERVICE PORT
MinIO      	minio          	minio-tenant-1	ClusterIP   	80
Console    	minio-1-console	minio-tenant-1	ClusterIP   	9090
kubectl minio tenant create minio-1 --namespace minio-tenant-1 --servers 3 --volumes 12 --capacity 12Gi --storage-class minio-local-storage --image registry.cn-zhangjiakou.aliyuncs.com/hsuing/minio:RELEASE.2023-04-20T17-56-55Z --disable-tls

Tenant 'minio-1' created in 'minio-tenant-1' Namespace

  Username: KIHKL495T13VF7IPBKQ2
  Password: ZT6TnoGdPK0AOwQ1Trq2jSGacXKZOi5CIsfS5hc4
  Note: Copy the credentials to a secure location. MinIO will not display these again.

APPLICATION	SERVICE NAME   	NAMESPACE     	SERVICE TYPE	SERVICE PORT
MinIO      	minio          	minio-tenant-1	ClusterIP   	80
Console    	minio-1-console	minio-tenant-1	ClusterIP   	9090

参数解释:

参数含义
–disable-tls关闭tls
enable-audit-logs是否启用审计，默认开启。启用审计时，需要准备额外存储给审计用。 这里有个奇怪的bug，必须用--enable-audit-logs=false，不能用enable-audit-logs false
servers服务器总数
volumes硬盘总数（4台服务器，每台4块盘）
capacity容量总数（假设一块磁盘1T，共16块）
storage-class存储类，使用刚才创建的minio-local-storage
namespace租户命名空间

1.5.1 查看pod ​

bash
[root@kube-master minio]# kubectl get all -nminio-tenant-1
NAME                 READY   STATUS    RESTARTS   AGE
pod/minio-1-ss-0-0   2/2     Running   0          59s
pod/minio-1-ss-0-1   2/2     Running   0          58s
pod/minio-1-ss-0-2   2/2     Running   0          58s

NAME                      TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE
service/minio             ClusterIP   192.168.176.21    <none>        80/TCP     60s
service/minio-1-console   ClusterIP   192.168.228.180   <none>        9090/TCP   60s
service/minio-1-hl        ClusterIP   None              <none>        9000/TCP   60s

NAME                            READY   AGE
statefulset.apps/minio-1-ss-0   3/3     59s
[root@kube-master minio]# kubectl get all -nminio-tenant-1
NAME                 READY   STATUS    RESTARTS   AGE
pod/minio-1-ss-0-0   2/2     Running   0          59s
pod/minio-1-ss-0-1   2/2     Running   0          58s
pod/minio-1-ss-0-2   2/2     Running   0          58s

NAME                      TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE
service/minio             ClusterIP   192.168.176.21    <none>        80/TCP     60s
service/minio-1-console   ClusterIP   192.168.228.180   <none>        9090/TCP   60s
service/minio-1-hl        ClusterIP   None              <none>        9000/TCP   60s

NAME                            READY   AGE
statefulset.apps/minio-1-ss-0   3/3     59s

1.5.2 通过UI查看 ​

image-20240716142221590

当为health显示绿色的时候，说明集群已经初始化完成

image-20240716142329115

1.5.3 通过命令查看 ​

bash
[root@kube-master minio]# kubectl minio tenant info minio-1
Tenant 'minio-1', Namespace 'minio-tenant-1', Total capacity 12 GiB

Current status: Initialized
MinIO version: registry.cn-zhangjiakou.aliyuncs.com/hsuing/minio:RELEASE.2023-04-20T17-56-55Z
MinIO service: minio/ClusterIP (port 80)
Console service: minio-1-console/ClusterIP (port 9090)

POOL	SERVERS	VOLUMES(SERVER)	CAPACITY(VOLUME)
0   	3      	4              	1.0 GiB
[root@kube-master minio]# kubectl minio tenant info minio-1
Tenant 'minio-1', Namespace 'minio-tenant-1', Total capacity 12 GiB

Current status: Initialized
MinIO version: registry.cn-zhangjiakou.aliyuncs.com/hsuing/minio:RELEASE.2023-04-20T17-56-55Z
MinIO service: minio/ClusterIP (port 80)
Console service: minio-1-console/ClusterIP (port 9090)

POOL	SERVERS	VOLUMES(SERVER)	CAPACITY(VOLUME)
0   	3      	4              	1.0 GiB

1.6 创建ingress ​

yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-s3-ing
  namespace: minio-tenant-1
  annotations:
    nginx.org/proxy-connect-timeout: "180s"
    nginx.org/proxy-read-timeout: "180s"
    nginx.org/client-max-body-size: "1024m"
spec:
  ingressClassName: nginx
  rules:
  - host: s3.ikubernetes.net
    http:
      paths:
      - backend:
          service:
            name: minio
            port:
              number: 80
        path: /
        pathType: Prefix
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-s3-ing
  namespace: minio-tenant-1
  annotations:
    nginx.org/proxy-connect-timeout: "180s"
    nginx.org/proxy-read-timeout: "180s"
    nginx.org/client-max-body-size: "1024m"
spec:
  ingressClassName: nginx
  rules:
  - host: s3.ikubernetes.net
    http:
      paths:
      - backend:
          service:
            name: minio
            port:
              number: 80
        path: /
        pathType: Prefix

这个是用于访问buckname里面的资源