写在前面
学习K8s监控涉及到网上的教程大都不全或者有些旧，所以整理分享给小伙伴。博文内容为K8s集群通过helm方式创建kube-prometheus-stack监控平台教程折腾了一晚上，搞定了，一开始一直用prometheus-operator这个chart来装，报错各种找问题，后来才发现我的集群版本太高了，1.22的版本，而且prometheus-operator之后的版本改变了名字kube-prometheus-stack，旧的版本可能不兼容。人生不尽美好，追求自我注定孤独，而这就是生命的意义 ——黑塞《彼得卡门青》
环境版本
我的K8s集群版本
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$kubectl get nodesNAMESTATUS ROLES AGE VERSIONvms81.liruilongs.github.io Ready control-plane,master 34d v1.22.2vms82.liruilongs.github.io Ready <none> 34d v1.22.2vms83.liruilongs.github.io Ready <none> 34d v1.22.2
hrlm版本
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm versionversion.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}
prometheus-operator(旧名字)安装出现的问题
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm search repo prometheus-operatorNAME CHART VERSION APP VERSIONDESCRIPTIONali/prometheus-operator 8.7.0 0.35.0Provides easy monitoring definitions for Kubern...azure/prometheus-operator 9.3.2 0.38.1DEPRECATED Provides easy monitoring definitions...┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm install liruilong ali/prometheus-operatorError: failed to install CRD crds/crd-alertmanager.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm pull ali/prometheus-operator┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$
解决办法:新版本安装
直接下载kube-prometheus-stack(新)的chart包，通过命令安装：
/prometheus-community/helm-charts/releases/download/kube-prometheus-stack-30.0.1/kube-prometheus-stack-30.0.1.tgz
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$lsindex.yaml kube-prometheus-stack-30.0.1.tgz liruilonghelm liruilonghelm-0.1.0.tgz mysql mysql-1.6.4.tgz┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm listNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
解压chart包kube-prometheus-stack-30.0.1.tgz
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$tar -zxf kube-prometheus-stack-30.0.1.tgz
创建新的命名空间
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$cd kube-prometheus-stack/┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl create ns monitoringnamespace/monitoring created┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl config set-context $(kubectl config current-context) --namespace=monitoringContext "kubernetes-admin@kubernetes" modified.
进入文件夹，直接通过helm install liruilong .安装
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$lsChart.lock charts Chart.yaml CONTRIBUTING.md crds README.md templates values.yaml┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$helm install liruilong .
kube-prometheus-admission-create对应Pod的相关镜像下载不下来问题
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$kubectl get podsNAMEREADY STATUS RESTARTS AGEliruilong-kube-prometheus-admission-create--1-bn7x2 0/1ImagePullBackOff 033s
查看pod详细信息，发现是谷歌的一个镜像国内无法下载
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$kubectl describe pod liruilong-kube-prometheus-admission-create--1-bn7x2Name: liruilong-kube-prometheus-admission-create--1-bn7x2Namespace: monitoringPriority:0Node: vms83.liruilongs.github.io/192.168.26.83Start Time: Sun, 16 Jan 02:43:07 +0800Labels: app=kube-prometheus-stack-admission-createapp.kubernetes.io/instance=liruilongapp.kubernetes.io/managed-by=Helmapp.kubernetes.io/part-of=kube-prometheus-stackapp.kubernetes.io/version=30.0.1chart=kube-prometheus-stack-30.0.1controller-uid=2ce48cd2-a118-4e23-a27f-0228ef6c45e7heritage=Helmjob-name=liruilong-kube-prometheus-admission-createrelease=liruilongAnnotations: /podIP: 10.244.70.8//podIPs: 10.244.70.8/32Status: PendingIP: 10.244.70.8IPs:IP: 10.244.70.8Controlled By: Job/liruilong-kube-prometheus-admission-createContainers:create:Container ID:Image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068Image ID:Port:<none>Host Port:。。。。。。。。。。。。。。。。。。。。。。。。。。。
在dokcer仓库里找了一个类似的，通过kubectl edit修改
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0 替换为 ： docker.io/liangjw/kube-webhook-certgen:v1.1.1
或者也可以修改配置文件从新install(记得要把sha注释掉)
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$lsindex.yaml kube-prometheus-stack kube-prometheus-stack-30.0.1.tgz liruilonghelm liruilonghelm-0.1.0.tgz mysql mysql-1.6.4.tgz┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$cd kube-prometheus-stack/┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$lsChart.lock charts Chart.yaml CONTRIBUTING.md crds README.md templates values.yaml┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$cat values.yaml | grep -A 3 -B 2 kube-webhook-certgenenabled: trueimage:repository: docker.io/liangjw/kube-webhook-certgentag: v1.1.1#sha: "f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068"pullPolicy: IfNotPresent┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$helm del liruilong;helm install liruilong .
之后其他的相关pod正常创建中
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl get podsNAMEREADY STATUS RESTARTS AGEliruilong-grafana-5955564c75-zpbjq0/3ContainerCreating 027sliruilong-kube-prometheus-operator-5cb699b469-fbkw5 0/1ContainerCreating 027sliruilong-kube-state-metrics-5dcf758c47-bbwt4 0/1ContainerCreating 027sliruilong-prometheus-node-exporter-rfsc5 0/1ContainerCreating 028sliruilong-prometheus-node-exporter-vm7s9 0/1ContainerCreating 028sliruilong-prometheus-node-exporter-z9j8b 0/1ContainerCreating 028s
kube-state-metrics这个pod的镜像也没有拉取下来。应该也是相同的原因
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl get podsNAME READY STATUS RESTARTS AGEalertmanager-liruilong-kube-prometheus-alertmanager-0 2/2Running 03m35sliruilong-grafana-5955564c75-zpbjq 3/3Running 04m46sliruilong-kube-prometheus-operator-5cb699b469-fbkw51/1Running 04m46sliruilong-kube-state-metrics-5dcf758c47-bbwt4 0/1ImagePullBackOff 04m46sliruilong-prometheus-node-exporter-rfsc51/1Running 04m47sliruilong-prometheus-node-exporter-vm7s91/1Running 04m47sliruilong-prometheus-node-exporter-z9j8b1/1Running 04m47sprometheus-liruilong-kube-prometheus-prometheus-0 2/2Running 03m34s
同样 k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0 这个镜像没办法拉取
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl describe pod liruilong-kube-state-metrics-5dcf758c47-bbwt4Name: liruilong-kube-state-metrics-5dcf758c47-bbwt4Namespace: monitoringPriority:0Node: vms82.liruilongs.github.io/192.168.26.82Start Time: Sun, 16 Jan 02:59:53 +0800Labels: app.kubernetes.io/component=metricsapp.kubernetes.io/instance=liruilongapp.kubernetes.io/managed-by=Helmapp.kubernetes.io/name=kube-state-metricsapp.kubernetes.io/part-of=kube-state-metricsapp.kubernetes.io/version=2.3.0helm.sh/chart=kube-state-metrics-4.3.0pod-template-hash=5dcf758c47release=liruilongAnnotations: /podIP: 10.244.171.153//podIPs: 10.244.171.153/32Status: PendingIP: 10.244.171.153IPs:IP: 10.244.171.153Controlled By: ReplicaSet/liruilong-kube-state-metrics-5dcf758c47Containers:kube-state-metrics:Container ID:Image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0Image ID:Port:8080/TCP。。。。。。。。。。。。。。。。。。。。。。
同样的，我们通过docker仓库找一下相同的,然后通过kubectl edit pod修改一下
k8s.gcr.io/kube-state-metrics/kube-state-metrics 替换为： docker.io/dyrnq/kube-state-metrics:v2.3.0
可以先在节点机上拉取一下
┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$ ansible node -m shell -a "docker pull dyrnq/kube-state-metrics:v2.3.0"192.168.26.82 | CHANGED | rc=0 >>v2.3.0: Pulling from dyrnq/kube-state-metricse8614d09b7be: Pulling fs layer53ccb90bafd7: Pulling fs layere8614d09b7be: Verifying Checksume8614d09b7be: Download completee8614d09b7be: Pull complete53ccb90bafd7: Verifying Checksum53ccb90bafd7: Download complete53ccb90bafd7: Pull completeDigest: sha256:c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030Status: Downloaded newer image for dyrnq/kube-state-metrics:v2.3.0docker.io/dyrnq/kube-state-metrics:v2.3.0192.168.26.83 | CHANGED | rc=0 >>v2.3.0: Pulling from dyrnq/kube-state-metricse8614d09b7be: Pulling fs layer53ccb90bafd7: Pulling fs layere8614d09b7be: Verifying Checksume8614d09b7be: Download completee8614d09b7be: Pull complete53ccb90bafd7: Verifying Checksum53ccb90bafd7: Download complete53ccb90bafd7: Pull completeDigest: sha256:c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030Status: Downloaded newer image for dyrnq/kube-state-metrics:v2.3.0docker.io/dyrnq/kube-state-metrics:v2.3.0
修改完之后，会发现所有的pod都创建成功
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl get podsNAME READY STATUS RESTARTSAGEalertmanager-liruilong-kube-prometheus-alertmanager-0 2/2Running 0 61mliruilong-grafana-5955564c75-zpbjq 3/3Running 0 62mliruilong-kube-prometheus-operator-5cb699b469-fbkw51/1Running 0 62mliruilong-kube-state-metrics-5dcf758c47-bbwt4 1/1Running 7 (32m ago) 62mliruilong-prometheus-node-exporter-rfsc51/1Running 0 62mliruilong-prometheus-node-exporter-vm7s91/1Running 0 62mliruilong-prometheus-node-exporter-z9j8b1/1Running 0 62mprometheus-liruilong-kube-prometheus-prometheus-0 2/2Running 0 61m┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$
然后我们需要修改liruilong-grafana SVC的类型为NodePort，这样，物理机就可以访问了
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEalertmanager-operatedClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 33mliruilong-grafana ClusterIP 10.99.220.121 <none> 80/TCP 34mliruilong-kube-prometheus-alertmanager ClusterIP 10.97.193.228 <none> 9093/TCP 34mliruilong-kube-prometheus-operator ClusterIP 10.101.106.93 <none> 443/TCP 34mliruilong-kube-prometheus-prometheusClusterIP 10.105.176.19 <none> 9090/TCP 34mliruilong-kube-state-metrics ClusterIP 10.98.94.55<none> 8080/TCP 34mliruilong-prometheus-node-exporter ClusterIP 10.110.216.215 <none> 9100/TCP 34mprometheus-operated ClusterIP None <none> 9090/TCP 33m┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl edit svc liruilong-grafanaservice/liruilong-grafana edited┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEalertmanager-operatedClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 35mliruilong-grafana NodePort 10.99.220.121 <none> 80:30443/TCP 36mliruilong-kube-prometheus-alertmanager ClusterIP 10.97.193.228 <none> 9093/TCP 36mliruilong-kube-prometheus-operator ClusterIP 10.101.106.93 <none> 443/TCP 36mliruilong-kube-prometheus-prometheusClusterIP 10.105.176.19 <none> 9090/TCP 36mliruilong-kube-state-metrics ClusterIP 10.98.94.55<none> 8080/TCP 36mliruilong-prometheus-node-exporter ClusterIP 10.110.216.215 <none> 9100/TCP 36mprometheus-operated ClusterIP None <none> 9090/TCP 35m
通过secrets解密获取用户名密码
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets | grep grafanaliruilong-grafanaOpaque 338mliruilong-grafana-test-token-q8z8j kubernetes.io/service-account-token 338mliruilong-grafana-token-j94p8 kubernetes.io/service-account-token 338m┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets liruilong-grafana -o yamlapiVersion: v1data:admin-password: cHJvbS1vcGVyYXRvcg==admin-user: YWRtaW4=ldap-toml: ""kind: Secretmetadata:annotations:meta.helm.sh/release-name: liruilongmeta.helm.sh/release-namespace: monitoringcreationTimestamp: "-01-15T18:59:40Z"labels:app.kubernetes.io/instance: liruilongapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: grafanaapp.kubernetes.io/version: 8.3.3helm.sh/chart: grafana-6.20.5name: liruilong-grafananamespace: monitoringresourceVersion: "1105663"uid: c03ff5f3-deb5-458c-8583-787f41034469type: Opaque┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets liruilong-grafana -o jsonpath='{.data.admin-user}}'| base64 -dadminbase64: 输入无效┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets liruilong-grafana -o jsonpath='{.data.admin-password}}'| base64 -dprom-operatorbase64: 输入无效
得到用户名密码：admin/prom-operator
生活加油 ^ _ ^
