安装
建议将Prometheus Operator部署在专门的命名空间中，一般为monitoring。
kubectl create namespace monitoring
建议将Prometheus Operator部署在专门的命名空间中，一般为monitoring。
kubectl create namespace monitoring
为了监控etcd，需要为证书创建secret。
Prometheus Operator定义了etcd的ServiceMonitor，但需要https才能访问metrics，如果不导入证书，将无法访问，导致etcd无法监控。
kubectl create secret generic etcd-certs -nmonitoring \--from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt \--from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key \--from-file=/etc/kubernetes/pki/etcd/ca.crt
Helm v3安装时，在crds/目录中的清单文件会自动提交给Kubernetes。
helm install prometheus stable/prometheus-operator \--namespace monitoring \--set prometheusOperator.createCustomResource=false \--set kubeEtcd.serviceMonitor.scheme=https \--set kubeEtcd.serviceMonitor.caFile=/etc/prometheus/secrets/etcd-certs/ca.crt \--set kubeEtcd.serviceMonitor.certFile=/etc/prometheus/secrets/etcd-certs/healthcheck-client.crt \--set kubeEtcd.serviceMonitor.keyFile=/etc/prometheus/secrets/etcd-certs/healthcheck-client.key \--set prometheus.prometheusSpec.secrets={etcd-certs}
查看Kubernetes资源。
kubectl --namespace monitoring get allNAME READY STATUS RESTARTS AGEpod/alertmanager-prometheus-prometheus-oper-alertmanager-0 2/2Running 04m20spod/prometheus-grafana-dc56bc899-vprqs 2/2Running 04m56spod/prometheus-kube-state-metrics-67b765f8b8-wblcd 1/1Running 04m56spod/prometheus-prometheus-node-exporter-fxl6j1/1Running 04m56spod/prometheus-prometheus-node-exporter-r8vhc1/1Running 04m56spod/prometheus-prometheus-node-exporter-xcgkj1/1Running 04m56spod/prometheus-prometheus-oper-operator-58566dd678-5c2zm2/2Running 04m56spod/prometheus-prometheus-prometheus-oper-prometheus-0 3/3Running 14m9sNAMETYPE CLUSTER-IPEXTERNAL-IP PORT(S) AGEservice/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 4m20sservice/prometheus-grafana ClusterIP 10.1.45.41<none> 80/TCP 4m56sservice/prometheus-kube-state-metrics ClusterIP 10.1.35.41<none> 8080/TCP 4m56sservice/prometheus-operated ClusterIP None <none> 9090/TCP 4m9sservice/prometheus-prometheus-node-exporter ClusterIP 10.1.206.118 <none> 9100/TCP 4m56sservice/prometheus-prometheus-oper-alertmanager ClusterIP 10.1.248.72 <none> 9093/TCP 4m56sservice/prometheus-prometheus-oper-operator ClusterIP 10.1.170.8<none> 8080/TCP,443/TCP 4m56sservice/prometheus-prometheus-oper-prometheusClusterIP 10.1.132.191 <none> 9090/TCP 4m56sNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGEdaemonset.apps/prometheus-prometheus-node-exporter 3 3 3 3 3 <none>4m56sNAMEREADY UP-TO-DATE AVAILABLE AGEdeployment.apps/prometheus-grafana1/11 1 4m56sdeployment.apps/prometheus-kube-state-metrics 1/11 1 4m56sdeployment.apps/prometheus-prometheus-oper-operator 1/11 1 4m56sNAME DESIRED CURRENT READY AGEreplicaset.apps/prometheus-grafana-dc56bc899 1 1 1 4m56sreplicaset.apps/prometheus-kube-state-metrics-67b765f8b8 1 1 1 4m56sreplicaset.apps/prometheus-prometheus-oper-operator-58566dd678 1 1 1 4m56sNAME READY AGEstatefulset.apps/alertmanager-prometheus-prometheus-oper-alertmanager 1/14m20sstatefulset.apps/prometheus-prometheus-prometheus-oper-prometheus 1/14m9s
查看创建的CRDs。
kubectl get crd | grep coreosalertmanagers.-04-01T01:42:58Zpodmonitors. -04-01T01:42:58Zprometheuses.-04-01T01:42:58Zprometheusrules. -04-01T01:42:58Zservicemonitors. -04-01T01:42:59Z
为了在集群外访问Prometheus、Grafana和Alertmanager，我们安装开源网关Ambassador，通过该网关访问。后面我们配置Prometheus监控该网关。
添加Helm仓库，选择datawire/ambassadorChart。
helm repo add datawire https://www.getambassador.iohelm search repo ambassadorNAME CHART VERSION APP VERSIONDESCRIPTIONaliyuncs/ambassador 4.4.7 0.85.0A Helm chart for Datawire Ambassadordatawire/ambassador 6.2.2 1.3.1 A Helm chart for Datawire Ambassadordatawire/ambassador-operator 0.1.0 1.0.0 A Helm chart for Kubernetesstable/ambassador5.3.0 0.86.1A Helm chart for Datawire Ambassador
安装Ambassador Edge Stack。
kubectl create namespace ambassadorhelm install ambassador datawire/ambassador \--namespace ambassador \--set authService.create=false \--set crds.create=false \--set licenseKey.licenseKey=false \--set rateLimit.create=false \--set service.type=NodePort
如果如Ambassador系列-11-Helm安装Ambassador Edge Stack 1.1.0一文中提到的方式申请了License，可以在Helm命令行注册该License，licenseKey.value就是邮件中收到的License Key。
helm install ambassador datawire/ambassador \--namespace ambassador \--set authService.create=false \--set crds.create=false \--set licenseKey.secretName=ambassador-edge-stack \--set licenseKey.value=eyJhbGciOiJQUzUxXXXXXXXXXXXXXXXXXXXX.eyJsaWNlbnNlX2tleV92ZXJzaW9uIjoidjIiLCJjdXN0b21lcl9pZCI6InR3aW5nYW9Ac2luYS5jbiIsImN1c3RvbWVyX2VtYWlsIjoidHdpbmdhb0BzaW5hLmNuIiwiZW5hYmxlZF9mZWF0dXJlcyI6WyIiLCJmaWx0ZXIiLCJyYXRlbGltaXQiLCJ0cmFmZmljIiwiZGV2cG9ydGFsIl0sImVuZm9yY2VkX2xpbWl0cyI6W3sibCI6ImRldnBvcnRhbC1zZXJ2aWNlcyIsInYiOjV9LHsibCI6InJhdGVsaW1pdC1zZXJ2aWNlIiwidiI6NX0seyJsIjoiYXV0aGZpbHRlci1zZXJ2aWNlIiwidiI6NX1dLCJtZXRhZGF0YSI6e30sImV4cCI6MTYxMjUzMDk4NCwiaWF0IjoxNTgwOTXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.TKBPqYGitumz5iSbFQq8EN9KN_BAqCJs9x03K6W3WBJxUx4fp3Qc6whjc9lNZgNG6KfUh61DJ9dru8G-90SyjCxvz05QDvAUtL__7PYfTS-17Jq0ZJygOAC8hGtrOz8iCw--oFkAhpZ14mvc0-CpZEn0DgKAHel0WQY7nYGQ6aEh2GYQG80rf3KBSxZwbp-sawBANArwvCvWw1W_5tSpBy3FBG33J0IIb2rS9lAuFr0ZvVdocJr5vIKb1KQAH3Ww9sxLKfFdFOLN_5fUIsFiAOYiPuo0hpQp1BbIllxCYrKAMig3xKRIlJI7Z6C-YySSxBXXXXXXXXXXXXXXXXXXXX \--set rateLimit.create=false \--set service.type=NodePort
查看Kubernetes资源。
kubectl get all -nambassadorNAMEREADY STATUS RESTARTS AGEpod/ambassador-75b5688649-7pnl2 0/1Running 183spod/ambassador-75b5688649-jpvpv 0/1Running 183spod/ambassador-75b5688649-llkn2 0/1Running 183spod/ambassador-redis-8556cbb4c6-ssbt6 1/1Running 083sNAME TYPE CLUSTER-IPEXTERNAL-IP PORT(S) AGEservice/ambassador NodePort 10.1.110.49 <none> 80:38024/TCP,443:32271/TCP 83sservice/ambassador-admin ClusterIP 10.1.166.205 <none> 8877/TCP 83sservice/ambassador-redis ClusterIP 10.1.193.90 <none> 6379/TCP 83sNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/ambassador 0/33 0 83sdeployment.apps/ambassador-redis 1/11 1 83sNAME DESIRED CURRENT READY AGEreplicaset.apps/ambassador-75b5688649 3 3 0 83sreplicaset.apps/ambassador-redis-8556cbb4c6 1 1 1 83s
通过管理Service端口访问metrics，其实就是envoy的metrics。
curl http://10.1.166.205:8877/metrics# TYPE envoy_cluster_upstream_cx_connect_timeout counterenvoy_cluster_upstream_cx_connect_timeout{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_flow_control_paused_reading_total counterenvoy_cluster_upstream_flow_control_paused_reading_total{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_cx_close_notify counterenvoy_cluster_upstream_cx_close_notify{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_lb_recalculate_zone_structures counterenvoy_cluster_lb_recalculate_zone_structures{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_flow_control_resumed_reading_total counterenvoy_cluster_upstream_flow_control_resumed_reading_total{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_rq_timeout counterenvoy_cluster_upstream_rq_timeout{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_cx_connect_fail counterenvoy_cluster_upstream_cx_connect_fail{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_rq_cancelled counterenvoy_cluster_upstream_rq_cancelled{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_cx_rx_bytes_total counterenvoy_cluster_upstream_cx_rx_bytes_total{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_cx_overflow counterenvoy_cluster_upstream_cx_overflow{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_cx_destroy_remote counterenvoy_cluster_upstream_cx_destroy_remote{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0# TYPE envoy_cluster_upstream_cx_http2_total counterenvoy_cluster_upstream_cx_http2_total{envoy_cluster_name="cluster_127_0_0_1_8877_ambassador"} 0......
为Prometheus、Grafana和Alertmanager配置Mapping。没有找到Ambassador如何配置重定向后的URL重写，只能按照域名配置路由，但又发现浏览器在host请求头中竟然带了端口号，只能在mapping的host配置中加上端口号，如:32271。只能将就，留待后续优化，或者有没有人告诉我如何处理？
vi prometheus-mapping.yaml---apiVersion: getambassador.io/v2kind: Mappingmetadata:name: prometheus-mappingnamespace: ambassadorspec:host: :32271prefix: /service: prometheus-prometheus-oper-prometheus.monitoring:9090---apiVersion: getambassador.io/v2kind: Mappingmetadata:name: grafana-mappingnamespace: ambassadorspec:host: :32271prefix: /service: prometheus-grafana.monitoring:80---apiVersion: getambassador.io/v2kind: Mappingmetadata:name: alert-mappingnamespace: ambassadorspec:host: :32271prefix: /service: prometheus-prometheus-oper-alertmanager.monitoring:9093kubectl apply -f prometheus-mapping.yaml
在浏览器主机增加hosts，Windows系统在C:\Windows\System32\drivers\etc\hosts。
# Prometheus Start192.168.1.55 192.168.1.55 192.168.1.55 # Prometheus End
访问Prometheus，并切换到Targets页面，地址:32271/targets。
没有看出来monitoring/prometheus-prometheus-oper-kube-proxy出问题的原因，但可以看出使用node地址+Service端口访问，这是无法访问的。monitoring/prometheus-prometheus-oper-kubelet是通过https-metrics（10250）端口访问的。monitoring/prometheus-prometheus-oper-kube-controller-manager和monitoring/prometheus-prometheus-oper-kube-scheduler已经配置正确。
如果没有在helm命令行配置kubeEtcd.serviceMonitor参数，monitoring/prometheus-prometheus-oper-kube-etcd是无法监控的，如下图。
访问Grafana，缺省密码为prom-operator，获取方式：
helm show values stable/prometheus-operator | grep adminPasswordadminPassword: prom-operator
Grafana缺省内置了多个dashboard。
访问Alertmanager，地址：:32271/#/alerts。
监控Ambassador
我们从service/ambassador-admin抓取metrics，查看一下ports。
kubectl get service/ambassador-admin -oyaml -nambassadorapiVersion: v1kind: Servicemetadata:creationTimestamp: "-04-01T07:15:16Z"labels:app.kubernetes.io/instance: ambassadorapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: ambassadorapp.kubernetes.io/part-of: ambassadorhelm.sh/chart: ambassador-6.2.2product: aesservice: ambassador-adminname: ambassador-adminnamespace: ambassadorresourceVersion: "43258"selfLink: /api/v1/namespaces/ambassador/services/ambassador-adminuid: 955b23af-c023-4196-a8f7-224194bac419spec:clusterIP: 10.1.166.205ports:- name: ambassador-adminport: 8877protocol: TCPtargetPort: adminselector:app.kubernetes.io/instance: ambassadorapp.kubernetes.io/name: ambassadorsessionAffinity: Nonetype: ClusterIPstatus:loadBalancer: {}
查看Prometheus自定义资源，其中定义了serviceMonitorSelector.matchLabels=release: prometheus，Prometheus据此关联ServiceMonitor。
kubectl get prometheuses./prometheus-prometheus-oper-prometheus -nmonitoring -oyamlapiVersion: /v1kind: Prometheusmetadata:creationTimestamp: "-04-01T01:43:35Z"generation: 1labels:app: prometheus-operator-prometheuschart: prometheus-operator-8.5.0heritage: Helmrelease: prometheusname: prometheus-prometheus-oper-prometheusnamespace: monitoringresourceVersion: "14492"selfLink: /apis//v1/namespaces/monitoring/prometheuses/prometheus-prometheus-oper-prometheusuid: 2b97c71a-5ec6-41bf-a42a-565136821ae5spec:alerting:alertmanagers:- name: prometheus-prometheus-oper-alertmanagernamespace: monitoringpathPrefix: /port: webbaseImage: quay.io/prometheus/prometheusenableAdminAPI: falseexternalUrl: http://prometheus-prometheus-oper-prometheus.monitoring:9090listenLocal: falselogFormat: logfmtlogLevel: infopaused: falsepodMonitorNamespaceSelector: {}podMonitorSelector:matchLabels:release: prometheusportName: webreplicas: 1retention: 10droutePrefix: /ruleNamespaceSelector: {}ruleSelector:matchLabels:app: prometheus-operatorrelease: prometheussecurityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: prometheus-prometheus-oper-prometheusserviceMonitorNamespaceSelector: {}serviceMonitorSelector:matchLabels:release: prometheusversion: v2.13.1
创建ServiceMonitor。其中几个需要注意的关键点。
ServiceMonitor的name最终会反应到Prometheus的配置中，作为job_name。由于Prometheus自定义资源中定义了serviceMonitorSelector.matchLabels=release: prometheus，表示ServiceMonitor需要定义一个标签release: prometheus，Prometheus据此可以关联ServiceMonitor。ServiceMonitor的命名空间必须和Prometheus所在的命名空间相同，此处为monitoring。endpoints.port需要和Service中的拉取metrics的ports.name对应，此处和上面对应为ambassador-admin。namespaceSelector.matchNames需要和被监控的Service所在的命名空间相同，此处为ambassador。selector.matchLabels的标签必须和被监控的Service中能唯一标明身份的标签对应。
创建ambassador-admin服务对应的ServiceMonitor。
vi prometheus-serviceMonitorAmbassador.yaml---apiVersion: /v1kind: ServiceMonitormetadata:name: ambassador-monitorlabels:release: prometheusnamespace: monitoringspec:endpoints:- port: ambassador-adminnamespaceSelector:matchNames:- ambassadorselector:matchLabels:service: ambassador-adminkubectl apply -f prometheus-serviceMonitorAmbassador.yaml
Prometheus的Targets。
Prometheus监控指标rate(envoy_http_rq_total[1m])。
