prometheus-operator
What is Prometheus Operator?featuresPrometheus Operator vs. kube-prometheusprometheus-operator自定义资源Prometheus Operator的github链接：参考What is Prometheus Operator?
Prometheus 是一套开源的系统监控、报警、时间序列数据库的组合，而 Prometheus Operator 是 CoreOS 开源的一套用于管理在 Kubernetes 集群上的 Prometheus 控制器，它是为了简化在 Kubernetes 上部署、管理和运行 Prometheus 和 Alertmanager 集群。
features
创建/销毁：使用操作员轻松为您的Kubernetes命名空间，特定应用程序或团队轻松启动Prometheus实例。
简单配置：从本机Kubernetes资源配置Prometheus的基础知识，如版本，持久性，保留策略和副本。
通过标签进行目标服务：根据熟悉的Kubernetes标签查询自动生成监控目标配置; 无需学习Prometheus特定的配置语言。
Prometheus Operator vs. kube-prometheus
Prometheus操作员将Prometheus配置为Kubernetes原生，并管理和操作Prometheus和Alertmanager集群。 这是关于完整端到端监控的难题之一。
kube-prometheus将Prometheus Operator与一系列清单相结合，以帮助开始监控Kubernetes本身以及运行在其上的应用程序。
kube-prometheus没有版本，并且以与Prometheus Operator相同的速度发布。 发行说明仅描述对操作员的更改，发布存档仅包含操作员代码的匹配更改。 对于kube-prometheus的更改，请始终引用此存储库的主分支。
kube-prometheus是一个单独的项目，将来会有自己的存储库[1] [operator-vs-kube。
prometheus-operator自定义资源
Prometheus，定义了所需的Prometheus部署。 运营商始终确保正在运行与资源定义匹配的部署。
ServiceMonitor，以声明方式指定应如何监视服务组。 操作员根据定义自动生成Prometheus刮削配置。
PrometheusRule，定义所需的Prometheus规则文件，可由包含Prometheus警报和记录规则的Prometheus实例加载。
Alertmanager，定义了所需的Alertmanager部署。 运营商始终确保正在运行与资源定义匹配的部署。
Prometheus Operator的github链接：
/coreos/prometheus-operator.git
目前已经移到 coreos/kube-prometheus
/coreos/kube-prometheus.git
Prometheus Operator所有yaml文件所在路径：
/coreos/prometheus-operator/contrib/kube-prometheus/manifests
移到
/coreos/kube-prometheus/manifests
/coreos/
编辑prometheus-operator-0.23.2目录下的bundle.yaml
修改项namespace: monitoring
apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:name: prometheus-operatorroleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: prometheus-operatorsubjects:- kind: ServiceAccountname: prometheus-operatornamespace: monitoring---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:name: prometheus-operatorrules:- apiGroups:- apiextensions.k8s.ioresources:- customresourcedefinitionsverbs:- '*'- apiGroups:- resources:- alertmanagers- prometheuses- prometheuses/finalizers- alertmanagers/finalizers- servicemonitors- prometheusrulesverbs:- '*'- apiGroups:- appsresources:- statefulsetsverbs:- '*'- apiGroups:- ""resources:- configmaps- secretsverbs:- '*'- apiGroups:- ""resources:- podsverbs:- list- delete- apiGroups:- ""resources:- services- endpointsverbs:- get- create- update- apiGroups:- ""resources:- nodesverbs:- list- watch- apiGroups:- ""resources:- namespacesverbs:- list- watch---apiVersion: apps/v1beta2kind: Deploymentmetadata:labels:k8s-app: prometheus-operatorname: prometheus-operatornamespace: monitoringspec:replicas: 1selector:matchLabels:k8s-app: prometheus-operatortemplate:metadata:labels:k8s-app: prometheus-operatorspec:containers:- args:- --kubelet-service=kube-system/kubelet- --logtostderr=true- --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1- --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.23.2image: quay.io/coreos/prometheus-operator:v0.23.2name: prometheus-operatorports:- containerPort: 8080name: httpresources:limits:cpu: 200mmemory: 200Mirequests:cpu: 100mmemory: 100MisecurityContext:allowPrivilegeEscalation: falsereadOnlyRootFilesystem: truenodeSelector:beta.kubernetes.io/os: linuxsecurityContext:runAsNonRoot: truerunAsUser: 65534serviceAccountName: prometheus-operator---apiVersion: v1kind: ServiceAccountmetadata:name: prometheus-operatornamespace: monitoring
执行创建
kubectl create -f bundle.yaml
部署kube-prometheus
kubectl create -f prometheus-operator/contrib/kube-prometheus/manifests
根据命名空间查询
kubectl get all -n monitoring [root@saas98 usr]$ kubectl get all -n monitoring NAME READY STATUS RESTARTS AGEpod/alertmanager-main-02/2Running 03h53mpod/alertmanager-main-12/2Running 03h52mpod/alertmanager-main-22/2Running 03h52mpod/grafana-5c54dbc48b-jvhcd1/1Running 03h54mpod/kube-state-metrics-fd9b964d5-srwkp4/4Running 03h49mpod/node-exporter-5ndbs2/2Running 03h54mpod/node-exporter-nts452/2Running 03h54mpod/node-exporter-pxtw52/2Running 03h54mpod/node-exporter-tvntn1/2CrashLoopBackOff 47 3h54mpod/node-exporter-wb7sx2/2Running 03h54mpod/prometheus-k8s-0 3/3Running 13h53mpod/prometheus-k8s-1 3/3Running 13h49mpod/prometheus-operator-76599f4b8c-zm5wl 1/1Running 03h50mNAME TYPE CLUSTER-IPEXTERNAL-IP PORT(S) AGEservice/alertmanager-main NodePort 10.99.5.67<none> 9093:30662/TCP3h54mservice/alertmanager-operated ClusterIP None <none> 9093/TCP,6783/TCP 3h53mservice/grafana NodePort 10.106.129.28 <none> 3000:31844/TCP3h54mservice/kube-state-metricsClusterIP None <none> 8443/TCP,9443/TCP 3h54mservice/node-exporter ClusterIP None <none> 9100/TCP 3h54mservice/prometheus-k8sNodePort 10.99.129.143 <none> 9090:31144/TCP3h54mservice/prometheus-operatedClusterIP None <none> 9090/TCP 3h53mservice/prometheus-operatorClusterIP None <none> 8080/TCP 3h54mNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGEdaemonset.apps/node-exporter 5 5 4 5 4 beta.kubernetes.io/os=linux 3h54mNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/grafana1/11 1 3h54mdeployment.apps/kube-state-metrics 1/11 1 3h54mdeployment.apps/prometheus-operator 1/11 1 3h54mNAMEDESIRED CURRENT READY AGEreplicaset.apps/grafana-5c54dbc48b1 1 1 3h54mreplicaset.apps/kube-state-metrics-6f5c6d88d5 0 0 0 3h54mreplicaset.apps/kube-state-metrics-fd9b964d51 1 1 3h53mreplicaset.apps/prometheus-operator-76599f4b8c 1 1 1 3h54mreplicaset.apps/prometheus-operator-f9fcb78bd 0 0 0 3h50mNAME READY AGEstatefulset.apps/alertmanager-main 3/33h53mstatefulset.apps/prometheus-k8s2/23h53m
修改访问方式（集群外部访问）
把svc的访问方式改为NodePort模式。
使用kubectl edit svc [svcname] -n monitoring方式修改
需要修改的是alertmanager-main，grafana，prometheus-k8s
例子：kubectl edit svc alertmanager-main -n monitoring
apiVersion: v1kind: Servicemetadata:name: grafananamespace: monitoringspec:type: NodePort#添加内容ports:- name: httpport: 3000targetPort: httpnodePort: 30100 #添加内容selector:app: grafana
[root@saas98 usr]$ kubectl get svc -n monitoringNAMETYPE CLUSTER-IPEXTERNAL-IP PORT(S) AGEalertmanager-main NodePort 10.99.5.67<none> 9093:30662/TCP3h55malertmanager-operated ClusterIP None <none> 9093/TCP,6783/TCP 3h54mgrafana NodePort 10.106.129.28 <none> 3000:31844/TCP3h55mkube-state-metricsClusterIP None <none> 8443/TCP,9443/TCP 3h55mnode-exporter ClusterIP None <none> 9100/TCP 3h55mprometheus-k8sNodePort 10.99.129.143 <none> 9090:31144/TCP3h55mprometheus-operatedClusterIP None <none> 9090/TCP 3h54mprometheus-operatorClusterIP None <none> 8080/TCP 3h55m
访问prometheus 端口 31144 例子http://118.31.17.205:31144/graph
通过访问http://118.31.17.205:31144/target 可以看到prometheus已经成功连接上了k8s的apiserver
访问alertmanager-main 例子：http://118.31.17.205:30662 alertmanager-main 30662
查看service-discovery http://118.31.17.205:31144/service-discovery
访问grafana 例子：http://118.31.17.205:31844 grafana 31844
输入密码就可以了（初始化用户名密码admin）
添加数据源
grafana默认已经添加了Prometheus数据源，可以直接用，grafana支持多种时序数据源，每种数据源都有各自的查询编辑器。
导入dashboard:导入面板，可以直接输入模板编号315在线导入，或者下载好对应的json模板文件本地导入，
官方面板模板下载地址：
/dashboards/315
/dashboards/8919
导入面板之后就可以看到对应的监控数据了,点击HOME选择查看，其实Grafana已经预定义了一系列Dashboard：
查看集群监控信息
参考
/coreos/prometheus-operator
