!版权声明：本博客内容均均为原创,每篇博文作为知识积累,写博不易,转载请注明出处。
目录[-]
描述：
上次已经配置了 Prometheus Operator 监控 Kubernetes，但是并没有监控 ETCD 的信息，所以这里我们配置一下监控 Etcd 信息。
系统环境：
Prometheus Operator版本： 0.29 (Prometheus Operator更名为Kube-prometheus，且版本变为：0.1.0)
Kubernetes 版本： 1.14.0
一、查看 Etcd 信息
获取 Etcd Pod 名称
$ kubectl get pods -n kube-system | grep etcd
etcd-k8s-master-2-11 1/1 Running 9 55d
查看 Etcd 描述信息
$ kubectl describe pod etcd-k8s-master-2-11 -n kube-system
......
Containers:
Command:
etcd
--advertise-client-urls=https://192.168.2.11:2379
--cert-file=/etc/kubernetes/pki/etcd/server.crt
--client-cert-auth=true
--data-dir=/var/lib/etcd
--initial-advertise-peer-urls=https://192.168.2.11:2380
--initial-cluster=k8s-master-2-11=https://192.168.2.11:2380
--key-file=/etc/kubernetes/pki/etcd/server.key
--listen-client-urls=https://127.0.0.1:2379,https://192.168.2.11:2379
--listen-peer-urls=https://192.168.2.11:2380
--name=k8s-master-2-11
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
--peer-client-cert-auth=true
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--snapshot-count=10000
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
......
可以看到 ETCD 的证书文件在 Kubernetes Master 节点的 “/etc/kubernetes/pki/etcd/” 文件夹下。
二、将证书存入 Kubernetes
利用 kubectl 命令将三个证书文件存入 Kubernetes 的 Secret 资源下。
$ kubectl create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key --from-file=/etc/kubernetes/pki/etcd/ca.crt -n monitoring
查看刚刚创建的资源
$ kubectl get secret etcd-certs -n monitoring
NAME TYPE DATA AGE
etcd-certs Opaque 3 1m
三、将证书挂入 Prometheus
编译 prometheus 资源
$ kubectl edit prometheus k8s -n monitoring
将证书挂入
apiVersion: /v1
kind: Prometheus
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","namespace":"monitoring"},"spec":{"alerting":{"alertmanagers":[{"name":"alertmanager-main","namespace":"monitoring","port":"web"}]},"baseImage":"quay-/prometheus/prometheus","nodeSelector":{"beta.kubernetes.io/os":"linux"},"replicas":2,"resources":{"requests":{"memory":"400Mi"}},"ruleSelector":{"matchLabels":{"prometheus":"k8s","role":"alert-rules"}},"securityContext":{"fsGroup":2000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"prometheus-k8s","serviceMonitorNamespaceSelector":{},"serviceMonitorSelector":{},"storage":{"volumeClaimTemplate":{"spec":{"resources":{"requests":{"storage":"8Gi"}},"storageClassName":"fast"}}},"version":"v2.7.2"}}
creationTimestamp: "-06-07T22:15:37Z"
generation: 5
labels:
prometheus: k8s
name: k8s
namespace: monitoring
resourceVersion: "2128109"
selfLink: /apis//v1/namespaces/monitoring/prometheuses/k8s
uid: c6daa0a1-8971-11e9-bc01-000c29d98697
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
baseImage: quay-/prometheus/prometheus
nodeSelector:
beta.kubernetes.io/os: linux
replicas: 2
resources:
requests:
memory: 400Mi
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
secrets: #------新增证书配置，将etcd证书挂入
- etcd-certs
更新完成后就可以在 Prometheus Pod 中看到上面挂入的 etcd 证书，我们可以进入 Pod 中查看：
$ kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring
/prometheus $ ls /etc/prometheus/secrets/etcd-certs/
ca.crt healthcheck-client.crt healthcheck-client.key
四、创建 Etcd Service & Endpoints
因为 ETCD 是独立于集群之外的，所以我们需要创建一个 Endpoints 将其代理到 Kubernetes 集群，然后创建一个 Service 绑定 Endpoints，然后 Kubernetes 集群的应用就可以访问 ETCD 了。
etcd-service.yaml
apiVersion: v1
kind: Service
metadata:
name: etcd-k8s
namespace: kube-system
labels:
k8s-app: etcd
spec:
type: ClusterIP
clusterIP: None #设置为None，不分配Service IP
ports:
- name: port
port: 2379
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: etcd-k8s
namespace: kube-system
labels:
k8s-app: etcd
subsets:
- addresses:
- ip: 192.168.2.11 #Etcd 所在节点的IP
ports:
- port: 2379 #Etcd 端口号
创建 Service & Endpoints
$ kubectl apply -f etcd-service.yaml
五、创建 ServiceMonitor
创建 Prometheus 监控资源，配置用于监控 Etcd 参数。
etcd-monitor.yaml
apiVersion: /v1
kind: ServiceMonitor
metadata:
name: etcd-k8s
namespace: monitoring
labels:
k8s-app: etcd-k8s
spec:
jobLabel: k8s-app
endpoints:
- port: port
interval: 30s
scheme: https
tlsConfig:
caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
insecureSkipVerify: true
selector:
matchLabels:
k8s-app: etcd
namespaceSelector:
matchNames:
- kube-system
创建 Etcd ServiceMonitor
$ kubectl apply -f etcd-monitor.yaml
六、查看 Prometheus 规则
创建完成后查看 Prometheus UI，可以看到已经有对应监控数据
七、Grafana 引入 ETCD 仪表盘
完成 Prometheus 配置后，直接打开 Grafana 页面，引入Dashboard，输入编号 “3070” 的仪表盘
可以看到监控 ETCD 的各个看板
