SRE文档

深浅模式

image-20240630214729095

官当

1. ​

工作原理 ​

1、Prometheus通过Consul APl查询Consul的KV存储中保存的配置信息，然后从中获取关于服务的元数据；

2、Prometheus使用这些信息来构造目标服务的URL，并将其添加到服务发现的目标列表中。

3、当服务被注销或不可用时，PrometheQs将自动从其目标列表中删除该服务。

2. 安装 ​

2.1 单节点 ​

bash
docker run -id -expose=[8300,8301,8302,8500,8600]  --restart always-p 18300:8300 -p 18301:8301 -p 18302:8302 -p 18500:8500 -p18600:8600 --name Server1 -e 'cONSUL_LOCAL_CONFIG={"skip_leave_on_interrupt": true}' registry.cn-zhangjiakou.aliyuncs.com/hsuing/consul:v1.15.4 agent -server -bootstrap-expect=l -node=serverl -bind=0.0.0.0 -client=0.0.0.0 -ui -datacenterdc1
docker run -id -expose=[8300,8301,8302,8500,8600]  --restart always-p 18300:8300 -p 18301:8301 -p 18302:8302 -p 18500:8500 -p18600:8600 --name Server1 -e 'cONSUL_LOCAL_CONFIG={"skip_leave_on_interrupt": true}' registry.cn-zhangjiakou.aliyuncs.com/hsuing/consul:v1.15.4 agent -server -bootstrap-expect=l -node=serverl -bind=0.0.0.0 -client=0.0.0.0 -ui -datacenterdc1

参数说明:

bash
-expose：暴露出出来的端口，即consul启动所需的端口：8300,8301,8302,8500,8600

--restart：always表示容器挂了就自动重启

-p：建立宿主机与容器的端口映射

--name：容器名称

-e：环境变量，这里用于对consul进行配置

agent：容器中执行的命令，各参数含义：
	-server：表示节点是server类型
	-bootstrap-expect：表示集群中有几个server节点后开始选举leader，既然是单节点集群，那自然就是1了
	-node：节点名称
	-bind：集群内部通信地址，默认是0.0.0.0
	-client：客户端地址，默认是127.0.0.1
	-ui：启用consul的web页面管理
	-datacenter：数据中心
-expose：暴露出出来的端口，即consul启动所需的端口：8300,8301,8302,8500,8600

--restart：always表示容器挂了就自动重启

-p：建立宿主机与容器的端口映射

--name：容器名称

-e：环境变量，这里用于对consul进行配置

agent：容器中执行的命令，各参数含义：
	-server：表示节点是server类型
	-bootstrap-expect：表示集群中有几个server节点后开始选举leader，既然是单节点集群，那自然就是1了
	-node：节点名称
	-bind：集群内部通信地址，默认是0.0.0.0
	-client：客户端地址，默认是127.0.0.1
	-ui：启用consul的web页面管理
	-datacenter：数据中心

2.2 基于k8s集群安装 ​

2.2.1 创建rabc ​

yaml
[root@kube-master consul]# cat 1.consul-rabc.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul-server
  labels:
    app: consul
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: consul
  labels:
    app: consul
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul
subjects:
  - kind: ServiceAccount
    name: consul-server
    namespace: monitor
[root@kube-master consul]# cat 1.consul-rabc.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul-server
  labels:
    app: consul
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: consul
  labels:
    app: consul
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul
subjects:
  - kind: ServiceAccount
    name: consul-server
    namespace: monitor

2.2.2 创建存储 ​

采用nfs动态存储

yaml
[root@kube-master consul]# cat 2.consul-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: consul-data-pvc
  namespace: monitor
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "nfs-provisioner-storage"
  resources:
    requests:
      storage: 5Gi
[root@kube-master consul]# cat 2.consul-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: consul-data-pvc
  namespace: monitor
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "nfs-provisioner-storage"
  resources:
    requests:
      storage: 5Gi

2.2.3 创建sts服务 ​

yaml
[root@kube-master consul]# cat 3.consul-sts.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
  namespace: monitor
spec:
  serviceName: consul
  replicas: 3
  selector:
    matchLabels:
      k8s-app: consul
  template:
    metadata:
      labels:
        k8s-app: consul
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: k8s-app
                    operator: In
                    values:
                      - consul
              topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 10
      #serviceAccountName: consul-server
      #hostNetwork: true
      securityContext:
        fsGroup: 1000
      containers:
      - name: consul
        image: registry.cn-zhangjiakou.aliyuncs.com/hsuing/consul:v1.15.4
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 8500
            name: ui-port
          - containerPort: 8400
            name: alt-port
          - containerPort: 53
            name: udp-port
          - containerPort: 8443
            name: https-port
          - containerPort: 8080
            name: http-port
          - containerPort: 8301
            name: serflan
          - containerPort: 8302
            name: serfwan
          - containerPort: 8600
            name: consuldns
          - containerPort: 8300
            name: server
        env:
          - name: PODIP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: PODNAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        args:
          - "agent"
          - "-server"
          - "-bootstrap-expect=3"
          - "-ui"
          - "-config-file=/consul/config"
          - "-data-dir=/consul/data"
          - "-log-file=/consul/log"
          - "-bind=0.0.0.0"
          - "-client=0.0.0.0"
          - "-advertise=$(PODIP)"
          - "-retry-join=consul-0.consul.$(NAMESPACE).svc.cluster.local"
          - "-retry-join=consul-1.consul.$(NAMESPACE).svc.cluster.local"
          - "-retry-join=consul-2.consul.$(NAMESPACE).svc.cluster.local"
          - "-domain=cluster.local"
          - "-disable-host-node-id"
        resources:
          limits:
            cpu: "200m"
            memory: "512Mi"
          requests:
            cpu: "100m"
            memory: "128Mi"
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - consul leave
        readinessProbe:
          exec:
            command:
              - consul
              - members
          failureThreshold: 2
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        volumeMounts:
          - name: consul-data
            mountPath: /consul/data
            subPathExpr: data/$(PODNAME)
          - name: consul-data
            mountPath: /consul/config
            subPathExpr: config/$(PODNAME)
      volumes:
        - name: consul-data
          persistentVolumeClaim:
            claimName: consul-data-pvc
[root@kube-master consul]# cat 3.consul-sts.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
  namespace: monitor
spec:
  serviceName: consul
  replicas: 3
  selector:
    matchLabels:
      k8s-app: consul
  template:
    metadata:
      labels:
        k8s-app: consul
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: k8s-app
                    operator: In
                    values:
                      - consul
              topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 10
      #serviceAccountName: consul-server
      #hostNetwork: true
      securityContext:
        fsGroup: 1000
      containers:
      - name: consul
        image: registry.cn-zhangjiakou.aliyuncs.com/hsuing/consul:v1.15.4
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 8500
            name: ui-port
          - containerPort: 8400
            name: alt-port
          - containerPort: 53
            name: udp-port
          - containerPort: 8443
            name: https-port
          - containerPort: 8080
            name: http-port
          - containerPort: 8301
            name: serflan
          - containerPort: 8302
            name: serfwan
          - containerPort: 8600
            name: consuldns
          - containerPort: 8300
            name: server
        env:
          - name: PODIP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: PODNAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        args:
          - "agent"
          - "-server"
          - "-bootstrap-expect=3"
          - "-ui"
          - "-config-file=/consul/config"
          - "-data-dir=/consul/data"
          - "-log-file=/consul/log"
          - "-bind=0.0.0.0"
          - "-client=0.0.0.0"
          - "-advertise=$(PODIP)"
          - "-retry-join=consul-0.consul.$(NAMESPACE).svc.cluster.local"
          - "-retry-join=consul-1.consul.$(NAMESPACE).svc.cluster.local"
          - "-retry-join=consul-2.consul.$(NAMESPACE).svc.cluster.local"
          - "-domain=cluster.local"
          - "-disable-host-node-id"
        resources:
          limits:
            cpu: "200m"
            memory: "512Mi"
          requests:
            cpu: "100m"
            memory: "128Mi"
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - consul leave
        readinessProbe:
          exec:
            command:
              - consul
              - members
          failureThreshold: 2
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        volumeMounts:
          - name: consul-data
            mountPath: /consul/data
            subPathExpr: data/$(PODNAME)
          - name: consul-data
            mountPath: /consul/config
            subPathExpr: config/$(PODNAME)
      volumes:
        - name: consul-data
          persistentVolumeClaim:
            claimName: consul-data-pvc

2.2.4 创建svc ​

yaml
[root@kube-master consul]# cat 4.consul-svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: consul
  namespace: monitor
  labels:
    app: consul
spec:
  ports:
    - name: http
      protocol: TCP
      port: 8500
      targetPort: 8500
    - name: https
      protocol: TCP
      port: 8443
      targetPort: 8443
    - name: rpc
      protocol: TCP
      port: 8400
      targetPort: 8400
    - name: serflan-tcp
      protocol: TCP
      port: 8301
      targetPort: 8301
    - name: serflan-udp
      protocol: UDP
      port: 8301
      targetPort: 8301
    - name: serfwan-tcp
      protocol: TCP
      port: 8302
      targetPort: 8302
    - name: serfwan-udp
      protocol: UDP
      port: 8302
      targetPort: 8302
    - name: server
      protocol: TCP
      port: 8300
      targetPort: 8300
    - name: consuldns
      protocol: TCP
      port: 8600
      targetPort: 8600
  selector:
    k8s-app: consul
  clusterIP: None

---
apiVersion: v1
kind: Service
metadata:
  name: consul-dns
  namespace: monitor
  labels:
    app: consul
spec:
  selector:
    k8s-app: consul
  ports:
    - name: dns-tcp
      protocol: TCP
      port: 53
      targetPort: dns-tcp
    - name: dns-udp
      protocol: UDP
      port: 53
      targetPort: dns-udp

---
apiVersion: v1
kind: Service
metadata:
  name: consul-ui
  namespace: monitor
  labels:
    app: consul
spec:
  selector:
    k8s-app: consul
  ports:
    - name: http
      port: 80
      targetPort: 8500
[root@kube-master consul]# cat 4.consul-svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: consul
  namespace: monitor
  labels:
    app: consul
spec:
  ports:
    - name: http
      protocol: TCP
      port: 8500
      targetPort: 8500
    - name: https
      protocol: TCP
      port: 8443
      targetPort: 8443
    - name: rpc
      protocol: TCP
      port: 8400
      targetPort: 8400
    - name: serflan-tcp
      protocol: TCP
      port: 8301
      targetPort: 8301
    - name: serflan-udp
      protocol: UDP
      port: 8301
      targetPort: 8301
    - name: serfwan-tcp
      protocol: TCP
      port: 8302
      targetPort: 8302
    - name: serfwan-udp
      protocol: UDP
      port: 8302
      targetPort: 8302
    - name: server
      protocol: TCP
      port: 8300
      targetPort: 8300
    - name: consuldns
      protocol: TCP
      port: 8600
      targetPort: 8600
  selector:
    k8s-app: consul
  clusterIP: None

---
apiVersion: v1
kind: Service
metadata:
  name: consul-dns
  namespace: monitor
  labels:
    app: consul
spec:
  selector:
    k8s-app: consul
  ports:
    - name: dns-tcp
      protocol: TCP
      port: 53
      targetPort: dns-tcp
    - name: dns-udp
      protocol: UDP
      port: 53
      targetPort: dns-udp

---
apiVersion: v1
kind: Service
metadata:
  name: consul-ui
  namespace: monitor
  labels:
    app: consul
spec:
  selector:
    k8s-app: consul
  ports:
    - name: http
      port: 80
      targetPort: 8500

2.2.4 创建ingress ​

yaml
[root@kube-master consul]# cat 5.consul-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: monitor
  name: consul-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: consul.ikubernetes.net
    http:
      paths:
        - pathType: Prefix
          backend:
            service:
              name: consul
              port:
                number: 8500
          path: /
[root@kube-master consul]# cat 5.consul-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: monitor
  name: consul-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: consul.ikubernetes.net
    http:
      paths:
        - pathType: Prefix
          backend:
            service:
              name: consul
              port:
                number: 8500
          path: /
  • 查看consul成员
bash
[root@kube-master prometheus]# kubectl exec -n monitor consul-0 -- consul members
Node      Address             Status  Type    Build   Protocol  DC   Partition  Segment
consul-0  172.30.0.179:8301   alive   server  1.15.4  2         dc1  default    <all>
consul-1  172.23.127.73:8301  alive   server  1.15.4  2         dc1  default    <all>
consul-2  172.17.74.93:8301   alive   server  1.15.4  2         dc1  default    <all>
[root@kube-master prometheus]# kubectl exec -n monitor consul-0 -- consul members
Node      Address             Status  Type    Build   Protocol  DC   Partition  Segment
consul-0  172.30.0.179:8301   alive   server  1.15.4  2         dc1  default    <all>
consul-1  172.23.127.73:8301  alive   server  1.15.4  2         dc1  default    <all>
consul-2  172.17.74.93:8301   alive   server  1.15.4  2         dc1  default    <all>
  • 验证
bash
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

[root@kube-master consul]# kubectl exec -ti busybox -- nslookup consul.monitor.svc.cluster.local
Server:    192.168.0.10
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name:      consul.monitor.svc.cluster.local
Address 1: 172.17.74.87 consul-2.consul.monitor.svc.cluster.local
Address 2: 172.30.0.147 172-30-0-147.consul-dns.monitor.svc.cluster.local
Address 3: 172.23.127.73 consul-1.consul.monitor.svc.cluster.local
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

[root@kube-master consul]# kubectl exec -ti busybox -- nslookup consul.monitor.svc.cluster.local
Server:    192.168.0.10
Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name:      consul.monitor.svc.cluster.local
Address 1: 172.17.74.87 consul-2.consul.monitor.svc.cluster.local
Address 2: 172.30.0.147 172-30-0-147.consul-dns.monitor.svc.cluster.local
Address 3: 172.23.127.73 consul-1.consul.monitor.svc.cluster.local
  • 效果

image-20240701113758054

2.2.5 配置采集器 ​

yaml
    - job_name: service_discovery_consul
      honor_labels: true
      metrics_path: /metrics
      scheme: http
      consul_sd_configs:    #基于consul服务发现的配置
        - server: consul.monitor.svc.cluster.local:8500    #consul的监听地址
          services: []                 #匹配consul中所有的service
      relabel_configs:             #relabel_configs下面都是重写标签相关配置
      - source_labels: ['__meta_consul_tags']    #将__meta_consul_tags标签的至赋值给product
        target_label: 'servername'
      - source_labels: ['__meta_consul_dc']   #将__meta_consul_dc的值赋值给idc
        target_label: 'idc'
      - source_labels: ['__meta_consul_service']
        regex: "consul"  #匹配为"consul"的service
        action: drop       #执行的动作为删除
    - job_name: service_discovery_consul
      honor_labels: true
      metrics_path: /metrics
      scheme: http
      consul_sd_configs:    #基于consul服务发现的配置
        - server: consul.monitor.svc.cluster.local:8500    #consul的监听地址
          services: []                 #匹配consul中所有的service
      relabel_configs:             #relabel_configs下面都是重写标签相关配置
      - source_labels: ['__meta_consul_tags']    #将__meta_consul_tags标签的至赋值给product
        target_label: 'servername'
      - source_labels: ['__meta_consul_dc']   #将__meta_consul_dc的值赋值给idc
        target_label: 'idc'
      - source_labels: ['__meta_consul_service']
        regex: "consul"  #匹配为"consul"的service
        action: drop       #执行的动作为删除
  • 热更新
bash
curl -XPOST http://prometheus.ikubernetes.net/-/reload
curl -XPOST http://prometheus.ikubernetes.net/-/reload

3. 注册节点 ​

由于consul是在k8s中安装,要想采集k8s之外的数据,的把consul的服务暴露出去

3.1 创建svc ​

yaml
[root@kube-master consul]# cat node_service.yaml
kind: Service
apiVersion: v1
metadata:
  name: consul-ui-ip
  namespace: monitor
  labels:
    app: consul
spec:
  selector:
    k8s-app: consul
  type: NodePort
  sessionAffinity: None
  ports:
    - name: consul-ui-ip
      protocol: TCP
      port: 8500
      targetPort: 8500
      nodePort: 30007
[root@kube-master consul]# cat node_service.yaml
kind: Service
apiVersion: v1
metadata:
  name: consul-ui-ip
  namespace: monitor
  labels:
    app: consul
spec:
  selector:
    k8s-app: consul
  type: NodePort
  sessionAffinity: None
  ports:
    - name: consul-ui-ip
      protocol: TCP
      port: 8500
      targetPort: 8500
      nodePort: 30007
  • apply
bash
kubectl apply -f node_service.yaml

#查看
[root@kube-master consul]# kubectl get svc consul-ui-ip -nmonitor
NAME           TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
consul-ui-ip   NodePort   192.168.131.12   <none>        8500:30007/TCP   19m

#验证,随便一个k8s中的节点即可
[root@kube-master consul]# curl 10.103.236.201:30007
<a href="/ui/">Moved Permanently</a>.
kubectl apply -f node_service.yaml

#查看
[root@kube-master consul]# kubectl get svc consul-ui-ip -nmonitor
NAME           TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
consul-ui-ip   NodePort   192.168.131.12   <none>        8500:30007/TCP   19m

#验证,随便一个k8s中的节点即可
[root@kube-master consul]# curl 10.103.236.201:30007
<a href="/ui/">Moved Permanently</a>.

3.2 创建node-export ​

node-export为例

  • 创建
bash
docker run -d -p 9100:9100 \
-v "/proc:/host/proc" \
-v "/sys:/host/sys" \
-v "/:/rootfs" \
-v "/etc/localtime:/etc/localtime" \
--net=host \
registry.cn-zhangjiakou.aliyuncs.com/hsuing/node-exporter:latest \
--path.procfs /host/proc \
--path.sysfs /host/sys \
--collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"
docker run -d -p 9100:9100 \
-v "/proc:/host/proc" \
-v "/sys:/host/sys" \
-v "/:/rootfs" \
-v "/etc/localtime:/etc/localtime" \
--net=host \
registry.cn-zhangjiakou.aliyuncs.com/hsuing/node-exporter:latest \
--path.procfs /host/proc \
--path.sysfs /host/sys \
--collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"
  • 注入consul
bash
curl -X PUT -d '{"id": "middler2","name": "nodeexporter","address": "10.103.236.199","port":9100,"tags":["middleware"],"checks": [{"http":"http://10.103.236.199:9100/metrics","interval": "3s"}]}' http://10.103.236.201:30007/v1/agent/service/register
curl -X PUT -d '{"id": "middler2","name": "nodeexporter","address": "10.103.236.199","port":9100,"tags":["middleware"],"checks": [{"http":"http://10.103.236.199:9100/metrics","interval": "3s"}]}' http://10.103.236.201:30007/v1/agent/service/register
yaml
#参数说明
id : 注册ID 在consul中为唯一标识
name ：Service名称
address：自动注册绑定ip
port：自动注册绑定端口
tags：注册标签，可多个
checks : 健康检查
http:   检查数据来源
interval: 检查时间间隔
http://10.103.236.201:30007/v1/agent/service/register  consul注册接口
#参数说明
id : 注册ID 在consul中为唯一标识
name ：Service名称
address：自动注册绑定ip
port：自动注册绑定端口
tags：注册标签，可多个
checks : 健康检查
http:   检查数据来源
interval: 检查时间间隔
http://10.103.236.201:30007/v1/agent/service/register  consul注册接口
  • 效果

image-20240701151805983

image-20240701151846523

3.3 删除注入 ​

bash
#如果要注销掉某个服务，可以通过如下API命令操作

#格式
curl -X PUT http://10.103.236.201:30007/v1/agent/service/deregister/id
#注释
   id -> 先前创建时指定的名字 

#如
curl -X PUT http://10.103.236.201:30007/v1/agent/service/deregister/middler2
#如果要注销掉某个服务，可以通过如下API命令操作

#格式
curl -X PUT http://10.103.236.201:30007/v1/agent/service/deregister/id
#注释
   id -> 先前创建时指定的名字 

#如
curl -X PUT http://10.103.236.201:30007/v1/agent/service/deregister/middler2

nginx 代理consul

https://blog.csdn.net/kaikai0720/article/details/128615517

https://www.yuandangsheng.top/?p=3148

https://www.cnblogs.com/k-free-bolg/p/13181957.html