k8s实战之部署Prometheus+Grafana可视化监控告警平台

dirtysea 发表于 2021-8-12 09:25:03

k8s实战之部署Prometheus+Grafana可视化监控告警平台

<h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;">主要内容</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">1.Prometheus架构<br style="box-sizing: border-box; outline: 0px;">2.K8S监控指标及实现思路<br style="box-sizing: border-box; outline: 0px;">3.在K8S平台部署Prometheus<br style="box-sizing: border-box; outline: 0px;">4.基于K8S服务发现的配置解析<br style="box-sizing: border-box; outline: 0px;">5.在K8S平台部署Grafana<br style="box-sizing: border-box; outline: 0px;">6.监控K8S集群中Pod、Node、资源对象<br style="box-sizing: border-box; outline: 0px;">7.使用Grafana可视化展示Prometheus监控数据<br style="box-sizing: border-box; outline: 0px;">8.告警规则与告警通知</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t2" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t2" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="1_Prometheus_12" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>1 Prometheus架构</h2><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t3" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t3" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="Prometheus__13" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>Prometheus 是什么</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">Prometheus（普罗米修斯）是一个最初在SoundCloud上构建的监控系统。自2012年成为社区开源项目，拥有非常活跃的开发人员和用户社区。为强调开源及独立维护，Prometheus于2016年加入云原生云计算基金会（CNCF），成为继Kubernetes之后的第二个托管项目。<br style="box-sizing: border-box; outline: 0px;">官网地址:<br style="box-sizing: border-box; outline: 0px;">https://prometheus.io<br style="box-sizing: border-box; outline: 0px;">https://github.com/prometheus</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t4" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t4" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="Prometheus__18" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>Prometheus 组成及架构</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><img src="https://img-blog.csdnimg.cn/20191130204511369.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"></p><ul style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; list-style: none; font-size: 16px; overflow: auto hidden; color: rgba(0, 0, 0, 0.75); font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Prometheus Server：收集指标和存储时间序列数据，并提供查询接口</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">ClientLibrary：客户端库</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Push Gateway：短期存储指标数据。主要用于临时性的任务</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Exporters：采集已有的第三方服务监控指标并暴露metrics</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Alertmanager：告警</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Web UI：简单的Web控制</li></ul><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t5" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t5" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="_27" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>数据模型</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">Prometheus将所有数据存储为时间序列；具有相同度量名称以及标签属于同一个指标。<br style="box-sizing: border-box; outline: 0px;">每个时间序列都由度量标准名称和一组键值对（也成为标签）唯一标识。<br style="box-sizing: border-box; outline: 0px;">时间序列格式：<br style="box-sizing: border-box; outline: 0px;">{=, …}<br style="box-sizing: border-box; outline: 0px;">示例：api_http_requests_total{method=”POST”, handler=”/messages”}</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t6" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t6" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="_33" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>作业和实例</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">实例：可以抓取的目标称为实例（Instances）<br style="box-sizing: border-box; outline: 0px;">作业：具有相同目标的实例集合称为作业</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;">（Job）
scrape_configs:
-job_name: ‘prometheus’
static_configs:
-targets: [‘localhost:9090’]
-job_name: ‘node’
static_configs:
-targets: [‘192.168.1.10:9090’]
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li></ul></pre><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t7" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t7" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="2_K8S_46" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>2 K8S监控指标及实现思路</h2><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t8" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t8" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="k8S_47" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>k8S监控指标</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><span style="box-sizing: border-box; outline: 0px; font-weight: 700;">Kubernetes本身监控</span></p><ul style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; list-style: none; font-size: 16px; overflow: auto hidden; color: rgba(0, 0, 0, 0.75); font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Node资源利用率</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Node数量</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Pods数量（Node）</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">资源对象状态</li><li style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-left: 32px; list-style-position: initial; list-style-image: initial;">Pod监控</li></ul><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">Pod数量（项目）<br style="box-sizing: border-box; outline: 0px;">容器资源利用率<br style="box-sizing: border-box; outline: 0px;">应用程序</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t9" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t9" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="PrometheusK8S_60" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>Prometheus监控K8S实现的架构</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><img src="https://img-blog.csdnimg.cn/20191130204758543.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/2019113020482297.png" alt="监控指标具体实现举例Pod性能 cAdvisor 容器CPUNode性能 node-exporter 节点CPU 内存利用率K8S资源对象 kube-state-metrics Pod/Deployment/Service" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">服务发现：<br style="box-sizing: border-box; outline: 0px;">https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t10" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t10" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="3_K8SPrometheus_69" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3 在K8S平台部署Prometheus</h2><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t11" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t11" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="31_70" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3.1集群环境</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><img src="https://img-blog.csdnimg.cn/20191130204905746.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"></p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t12" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t12" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="32__72" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3.2 项目地址</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">可以直接选择git clone项目代码，也可以后面自己编写，后面的操作也会有实现代码</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># git clone https://github.com/zhangdongdong7/k8s-prometheus.git
Cloning into 'k8s-prometheus'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
# cd k8s-prometheus/
# ls
alertmanager-configmap.yaml       kube-state-metrics-rbac.yaml          prometheus-rbac.yaml
alertmanager-deployment.yaml    kube-state-metrics-service.yaml       prometheus-rules.yaml
alertmanager-pvc.yaml             node_exporter-0.17.0.linux-amd64.tar.gzprometheus-service.yaml
alertmanager-service.yaml       node_exporter.sh                      prometheus-statefulset-static-pv.yaml
grafana.yaml                      OWNERS                               prometheus-statefulset.yaml
kube-state-metrics-deployment.yamlprometheus-configmap.yaml             README.md
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li></ul></pre><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t13" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t13" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="33_RBAC_90" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3.3 使用RBAC进行授权</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">RBAC（Role-Based Access Control，基于角色的访问控制）：负责完成授权（Authorization）工作。<br style="box-sizing: border-box; outline: 0px;">编写授权yaml</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim prometheus-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
   - ""
resources:
   - nodes
   - nodes/metrics
   - services
   - endpoints
   - pods
verbs:
   - get
   - list
   - watch
- apiGroups:
   - ""
resources:
   - configmaps
verbs:
   - get
- nonResourceURLs:
   - "/metrics"
verbs:
   - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: kube-system
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">30</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">31</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">32</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">33</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">34</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">35</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">36</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">37</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">38</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">39</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">40</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">41</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">42</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">43</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">44</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">45</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">46</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">47</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">48</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">49</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">50</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">51</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">52</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">53</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">54</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">55</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">56</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">57</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">创建</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl apply -f prometheus-rbac.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t14" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t14" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="34__156" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3.4 配置管理</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">使用Configmap保存不需要加密配置信息<br style="box-sizing: border-box; outline: 0px;">其中需要把nodes中ip地址根据自己的地址进行修改，其他不需要改动</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim prometheus-configmap.yaml
# Prometheus configuration format https://prometheus.io/docs/prometheus/latest/configuration/configuration/
apiVersion: v1
kind: ConfigMap #
metadata:
name: prometheus-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
prometheus.yml: |
rule_files:
- /etc/config/rules/*.rules

scrape_configs:
- job_name: prometheus
   static_configs:
   - targets:
   - localhost:9090

- job_name: kubernetes-nodes
   scrape_interval: 30s
   static_configs:
   - targets:
   - 192.168.73.135:9100 #ip地址根据自己的地址进行修改
   - 192.168.73.138:9100
   - 192.168.73.139:9100
   - 192.168.73.140:9100

- job_name: kubernetes-apiservers
   kubernetes_sd_configs:
   - role: endpoints
   relabel_configs:
   - action: keep
   regex: default;kubernetes;https
   source_labels:
   - __meta_kubernetes_namespace
   - __meta_kubernetes_service_name
   - __meta_kubernetes_endpoint_port_name
   scheme: https
   tls_config:
   ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
   insecure_skip_verify: true
   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

- job_name: kubernetes-nodes-kubelet
   kubernetes_sd_configs:
   - role: node
   relabel_configs:
   - action: labelmap
   regex: __meta_kubernetes_node_label_(.+)
   scheme: https
   tls_config:
   ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
   insecure_skip_verify: true
   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

- job_name: kubernetes-nodes-cadvisor
   kubernetes_sd_configs:
   - role: node
   relabel_configs:
   - action: labelmap
   regex: __meta_kubernetes_node_label_(.+)
   - target_label: __metrics_path__
   replacement: /metrics/cadvisor
   scheme: https
   tls_config:
   ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
   insecure_skip_verify: true
   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

- job_name: kubernetes-service-endpoints
   kubernetes_sd_configs:
   - role: endpoints
   relabel_configs:
   - action: keep
   regex: true
   source_labels:
   - __meta_kubernetes_service_annotation_prometheus_io_scrape
   - action: replace
   regex: (https?)
   source_labels:
   - __meta_kubernetes_service_annotation_prometheus_io_scheme
   target_label: __scheme__
   - action: replace
   regex: (.+)
   source_labels:
   - __meta_kubernetes_service_annotation_prometheus_io_path
   target_label: __metrics_path__
   - action: replace
   regex: ([^:]+)(?::\d+)?;(\d+)
   replacement: $1:$2
   source_labels:
   - __address__
   - __meta_kubernetes_service_annotation_prometheus_io_port
   target_label: __address__
   - action: labelmap
   regex: __meta_kubernetes_service_label_(.+)
   - action: replace
   source_labels:
   - __meta_kubernetes_namespace
   target_label: kubernetes_namespace
   - action: replace
   source_labels:
   - __meta_kubernetes_service_name
   target_label: kubernetes_name

- job_name: kubernetes-services
   kubernetes_sd_configs:
   - role: service
   metrics_path: /probe
   params:
   module:
   - http_2xx
   relabel_configs:
   - action: keep
   regex: true
   source_labels:
   - __meta_kubernetes_service_annotation_prometheus_io_probe
   - source_labels:
   - __address__
   target_label: __param_target
   - replacement: blackbox
   target_label: __address__
   - source_labels:
   - __param_target
   target_label: instance
   - action: labelmap
   regex: __meta_kubernetes_service_label_(.+)
   - source_labels:
   - __meta_kubernetes_namespace
   target_label: kubernetes_namespace
   - source_labels:
   - __meta_kubernetes_service_name
   target_label: kubernetes_name

- job_name: kubernetes-pods
   kubernetes_sd_configs:
   - role: pod
   relabel_configs:
   - action: keep
   regex: true
   source_labels:
   - __meta_kubernetes_pod_annotation_prometheus_io_scrape
   - action: replace
   regex: (.+)
   source_labels:
   - __meta_kubernetes_pod_annotation_prometheus_io_path
   target_label: __metrics_path__
   - action: replace
   regex: ([^:]+)(?::\d+)?;(\d+)
   replacement: $1:$2
   source_labels:
   - __address__
   - __meta_kubernetes_pod_annotation_prometheus_io_port
   target_label: __address__
   - action: labelmap
   regex: __meta_kubernetes_pod_label_(.+)
   - action: replace
   source_labels:
   - __meta_kubernetes_namespace
   target_label: kubernetes_namespace
   - action: replace
   source_labels:
   - __meta_kubernetes_pod_name
   target_label: kubernetes_pod_name
alerting:
   alertmanagers:
   - static_configs:
      - targets: ["alertmanager:80"]
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">30</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">31</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">32</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">33</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">34</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">35</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">36</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">37</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">38</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">39</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">40</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">41</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">42</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">43</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">44</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">45</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">46</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">47</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">48</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">49</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">50</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">51</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">52</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">53</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">54</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">55</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">56</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">57</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">58</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">59</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">60</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">61</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">62</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">63</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">64</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">65</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">66</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">67</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">68</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">69</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">70</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">71</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">72</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">73</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">74</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">75</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">76</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">77</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">78</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">79</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">80</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">81</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">82</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">83</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">84</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">85</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">86</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">87</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">88</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">89</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">90</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">91</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">92</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">93</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">94</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">95</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">96</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">97</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">98</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">99</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">100</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">101</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">102</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">103</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">104</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">105</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">106</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">107</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">108</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">109</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">110</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">111</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">112</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">113</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">114</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">115</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">116</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">117</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">118</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">119</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">120</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">121</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">122</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">123</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">124</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">125</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">126</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">127</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">128</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">129</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">130</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">131</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">132</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">133</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">134</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">135</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">136</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">137</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">138</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">139</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">140</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">141</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">142</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">143</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">144</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">145</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">146</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">147</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">148</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">149</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">150</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">151</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">152</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">153</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">154</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">155</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">156</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">157</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">158</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">159</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">160</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">161</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">162</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">163</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">164</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">165</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">166</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">167</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">168</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">169</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">170</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">171</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">创建</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl apply -f prometheus-configmap.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t15" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t15" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="35_prometheus_336" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3.5 有状态部署prometheus</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">这里使用storageclass进行动态供给，给prometheus的数据进行持久化，具体实现办法，可以查看之前的文章《k8s中的NFS动态存储供给》，除此之外可以使用静态供给的prometheus-statefulset-static-pv.yaml进行持久化</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim prometheus-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: prometheus
namespace: kube-system
labels:
k8s-app: prometheus
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v2.2.1
spec:
serviceName: "prometheus"
replicas: 1
podManagementPolicy: "Parallel"
updateStrategy:
type: "RollingUpdate"
selector:
matchLabels:
   k8s-app: prometheus
template:
metadata:
   labels:
   k8s-app: prometheus
   annotations:
   scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
   priorityClassName: system-cluster-critical
   serviceAccountName: prometheus
   initContainers:
   - name: "init-chown-data"
   image: "busybox:latest"
   imagePullPolicy: "IfNotPresent"
   command: ["chown", "-R", "65534:65534", "/data"]
   volumeMounts:
   - name: prometheus-data
      mountPath: /data
      subPath: ""
   containers:
   - name: prometheus-server-configmap-reload
      image: "jimmidyson/configmap-reload:v0.1"
      imagePullPolicy: "IfNotPresent"
      args:
         - --volume-dir=/etc/config
         - --webhook-url=http://localhost:9090/-/reload
      volumeMounts:
         - name: config-volume
         mountPath: /etc/config
         readOnly: true
      resources:
         limits:
         cpu: 10m
         memory: 10Mi
         requests:
         cpu: 10m
         memory: 10Mi

   - name: prometheus-server
      image: "prom/prometheus:v2.2.1"
      imagePullPolicy: "IfNotPresent"
      args:
         - --config.file=/etc/config/prometheus.yml
         - --storage.tsdb.path=/data
         - --web.console.libraries=/etc/prometheus/console_libraries
         - --web.console.templates=/etc/prometheus/consoles
         - --web.enable-lifecycle
      ports:
         - containerPort: 9090
      readinessProbe:
         httpGet:
         path: /-/ready
         port: 9090
         initialDelaySeconds: 30
         timeoutSeconds: 30
      livenessProbe:
         httpGet:
         path: /-/healthy
         port: 9090
         initialDelaySeconds: 30
         timeoutSeconds: 30
      # based on 10 running nodes with 30 pods each
      resources:
         limits:
         cpu: 200m
         memory: 1000Mi
         requests:
         cpu: 200m
         memory: 1000Mi

      volumeMounts:
         - name: config-volume
         mountPath: /etc/config
         - name: prometheus-data
         mountPath: /data
         subPath: ""
         - name: prometheus-rules
         mountPath: /etc/config/rules

   terminationGracePeriodSeconds: 300
   volumes:
   - name: config-volume
      configMap:
         name: prometheus-config
   - name: prometheus-rules
      configMap:
         name: prometheus-rules

volumeClaimTemplates:
- metadata:
   name: prometheus-data
spec:
   storageClassName: managed-nfs-storage #存储类根据自己的存储类名字修改
   accessModes:
   - ReadWriteOnce
   resources:
   requests:
      storage: "16Gi"
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">30</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">31</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">32</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">33</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">34</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">35</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">36</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">37</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">38</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">39</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">40</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">41</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">42</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">43</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">44</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">45</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">46</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">47</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">48</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">49</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">50</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">51</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">52</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">53</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">54</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">55</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">56</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">57</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">58</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">59</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">60</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">61</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">62</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">63</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">64</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">65</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">66</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">67</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">68</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">69</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">70</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">71</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">72</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">73</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">74</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">75</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">76</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">77</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">78</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">79</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">80</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">81</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">82</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">83</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">84</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">85</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">86</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">87</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">88</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">89</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">90</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">91</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">92</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">93</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">94</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">95</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">96</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">97</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">98</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">99</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">100</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">101</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">102</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">103</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">104</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">105</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">106</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">107</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">108</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">109</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">110</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">111</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">112</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">113</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">114</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">115</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">116</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">117</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">创建</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl apply -f prometheus-statefulset.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">检查状态</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl get pod -n kube-system
NAME                                  READY STATUS RESTARTS AGE
coredns-5bd5f9dbd9-wv45t             1/1 Running 1       8d
kubernetes-dashboard-7d77666777-d5ng4 1/1 Running 5       14d
prometheus-0                         2/2 Running 6       14d
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">可以看到一个prometheus-0的pod，这就刚才使用statefulset控制器进行的有状态部署，两个容器的状态为Runing则是正常，如果不为Runing可以使用kubectl describe pod prometheus-0 -n kube-system查看报错详情</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t16" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t16" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="36_service_471" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3.6 创建service暴露访问端口</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">此处使用nodePort固定一个访问端口，不适用随机端口，便于访问</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim prometheus-service.yaml
kind: Service
apiVersion: v1
metadata:
name: prometheus
namespace: kube-system
labels:
kubernetes.io/name: "Prometheus"
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
type: NodePort
ports:
- name: http
   port: 9090
   protocol: TCP
   targetPort: 9090
   nodePort: 30090 #固定的对外访问的端口
selector:
k8s-app: prometheus
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">创建</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl apply -f prometheus-service.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">检查</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl get pod,svc -n kube-system
NAME                                     READY STATUS RESTARTS AGE
pod/coredns-5bd5f9dbd9-wv45t             1/1 Running 1       8dpod/kubernetes-dashboard-7d77666777-d5ng4 1/1 Running 5       14dpod/prometheus-0                         2/2 Running 6       14dNAME                         TYPE    CLUSTER-IP EXTERNAL-IP PORT(S)          AGE
service/kube-dns             ClusterIP 10.0.0.2 <none>    53/UDP,53/TCP    13dservice/kubernetes-dashboard NodePort 10.0.0.127 <none>    443:30001/TCP    16dservice/prometheus          NodePort 10.0.0.33 <none>    9090:30090/TCP    13d
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li></ul></pre><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t17" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t17" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="37_web_507" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>3.7 web访问</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">使用任意一个NodeIP加端口进行访问，访问地址：http://NodeIP:Port ,此例就是：<code style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; font-size: 14px; line-height: 22px; color: rgb(199, 37, 78); background-color: rgb(249, 242, 244); border-radius: 2px; padding: 2px 4px;">http://192.168.73.139:30090</code><br style="box-sizing: border-box; outline: 0px;">访问成功的界面如图所示：</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t18" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t18" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="4_K8SGrafana_510" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>4 在K8S平台部署Grafana</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">通过上面的web访问，可以看出prometheus自带的UI界面是没有多少功能的，可视化展示的功能不完善，不能满足日常的监控所需，因此常常我们需要再结合Prometheus+Grafana的方式来进行可视化的数据展示<br style="box-sizing: border-box; outline: 0px;">官网地址：<br style="box-sizing: border-box; outline: 0px;">https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus<br style="box-sizing: border-box; outline: 0px;">https://grafana.com/grafana/download<br style="box-sizing: border-box; outline: 0px;">刚才下载的项目中已经写好了Grafana的yaml，根据自己的环境进行修改</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t19" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t19" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="41_StatefulSetgrafana_516" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>4.1 使用StatefulSet部署grafana</h2><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim grafana.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: grafana
namespace: kube-system
spec:
serviceName: "grafana"
replicas: 1
selector:
matchLabels:
   app: grafana
template:
metadata:
   labels:
   app: grafana
spec:
   containers:
   - name: grafana
   image: grafana/grafana
   ports:
      - containerPort: 3000
         protocol: TCP
   resources:
      limits:
         cpu: 100m
         memory: 256Mi
      requests:
         cpu: 100m
         memory: 256Mi
   volumeMounts:
      - name: grafana-data
         mountPath: /var/lib/grafana
         subPath: grafana
   securityContext:
   fsGroup: 472
   runAsUser: 472
volumeClaimTemplates:
- metadata:
   name: grafana-data
spec:
   storageClassName: managed-nfs-storage #和prometheus使用同一个存储类
   accessModes:
   - ReadWriteOnce
   resources:
   requests:
      storage: "1Gi"

---

apiVersion: v1
kind: Service
metadata:
name: grafana
namespace: kube-system
spec:
type: NodePort
ports:
- port : 80
targetPort: 3000
nodePort: 30091
selector:
app: grafana
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">30</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">31</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">32</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">33</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">34</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">35</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">36</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">37</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">38</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">39</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">40</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">41</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">42</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">43</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">44</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">45</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">46</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">47</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">48</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">49</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">50</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">51</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">52</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">53</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">54</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">55</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">56</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">57</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">58</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">59</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">60</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">61</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">62</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">63</li></ul></pre><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t20" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t20" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="42_Grafanaweb_582" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>4.2 Grafana的web访问</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">使用任意一个NodeIP加端口进行访问，访问地址：http://NodeIP:Port ,此例就是：http://192.168.73.139:30091<br style="box-sizing: border-box; outline: 0px;">成功访问界面如下，会需要进行账号密码登陆，默认账号密码都为admin，登陆之后会让修改密码<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130205739396.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">登陆之后的界面如下<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130205750341.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">第一步需要进行数据源添加，点击create your first data source数据库图标，根据下图所示进行添加即可<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130205809377.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">第二步，添加完了之后点击底部的绿色的Save&Test，会成功提示Data sourse is working，则表示数据源添加成功</p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t21" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t21" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="43_K8SPodNode_591" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>4.3 监控K8S集群中Pod、Node、资源对象数据的方法</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">1）Pod<br style="box-sizing: border-box; outline: 0px;">kubelet的节点使用cAdvisor提供的metrics接口获取该节点所有Pod和容器相关的性能指标数据，安装kubelet默认就开启了<br style="box-sizing: border-box; outline: 0px;">暴露接口地址：<br style="box-sizing: border-box; outline: 0px;">https://NodeIP:10255/metrics/cadvisor<br style="box-sizing: border-box; outline: 0px;">https://NodeIP:10250/metrics/cadvisor</p><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">2）Node<br style="box-sizing: border-box; outline: 0px;">需要使用node_exporter收集器采集节点资源利用率。<br style="box-sizing: border-box; outline: 0px;">https://github.com/prometheus/node_exporter<br style="box-sizing: border-box; outline: 0px;">使用文档：https://prometheus.io/docs/guides/node-exporter/</p><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">使用node_exporter.sh脚本分别在所有服务器上部署node_exporter收集器，不需要修改可直接运行脚本</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># cat node_exporter.sh #!/bin/bashwget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz

tar zxf node_exporter-0.17.0.linux-amd64.tar.gz
mv node_exporter-0.17.0.linux-amd64 /usr/local/node_exporter

cat <<EOF >/usr/lib/systemd/system/node_exporter.service

Description=https://prometheus.io

Restart=on-failure
ExecStart=/usr/local/node_exporter/node_exporter --collector.systemd --collector.systemd.unit-whitelist=(docker|kubelet|kube-proxy|flanneld).service

WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable node_exporter
systemctl restart node_exporter
# ./node_exporter.sh
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">检测node_exporter的进程，是否生效</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># ps -ef|grep node_exporter
root    6227    10 Oct08 ?    00:06:43 /usr/local/node_exporter/node_exporter --collector.systemd --collector.systemd.unit-whitelist=(docker|kubelet|kube-proxy|flanneld).service
root 118269 1175840 23:27 pts/0 00:00:00 grep --color=auto node_exporter
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">3）资源对象<br style="box-sizing: border-box; outline: 0px;">kube-state-metrics采集了k8s中各种资源对象的状态信息，只需要在master节点部署就行</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;">https://github.com/kubernetes/kube-state-metrics
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">创建rbac的yaml对metrics进行授权</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim kube-state-metrics-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube-state-metrics
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
resources:
- configmaps
- secrets
- nodes
- pods
- services
- resourcequotas
- replicationcontrollers
- limitranges
- persistentvolumeclaims
- persistentvolumes
- namespaces
- endpoints
verbs: ["list", "watch"]
- apiGroups: ["extensions"]
resources:
- daemonsets
- deployments
- replicasets
verbs: ["list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
verbs: ["list", "watch"]
- apiGroups: ["batch"]
resources:
- cronjobs
- jobs
verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
resources:
- horizontalpodautoscalers
verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kube-state-metrics-resizer
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
resources:
- pods
verbs: ["get"]
- apiGroups: ["extensions"]
resources:
- deployments
resourceNames: ["kube-state-metrics"]
verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kube-state-metrics
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kube-state-metrics-resizer
subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: kube-system
# kubectl apply -f kube-state-metrics-rbac.yaml
编写Deployment和ConfigMap的yaml进行metrics pod部署，不需要进行修改
# cat kube-state-metrics-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
k8s-app: kube-state-metrics
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.3.0
spec:
selector:
matchLabels:
   k8s-app: kube-state-metrics
   version: v1.3.0
replicas: 1
template:
metadata:
   labels:
   k8s-app: kube-state-metrics
   version: v1.3.0
   annotations:
   scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
   priorityClassName: system-cluster-critical
   serviceAccountName: kube-state-metrics
   containers:
   - name: kube-state-metrics
   image: lizhenliang/kube-state-metrics:v1.3.0
   ports:
   - name: http-metrics
      containerPort: 8080
   - name: telemetry
      containerPort: 8081
   readinessProbe:
      httpGet:
         path: /healthz
         port: 8080
      initialDelaySeconds: 5
      timeoutSeconds: 5
   - name: addon-resizer
   image: lizhenliang/addon-resizer:1.8.3
   resources:
      limits:
         cpu: 100m
         memory: 30Mi
      requests:
         cpu: 100m
         memory: 30Mi
   env:
      - name: MY_POD_NAME
         valueFrom:
         fieldRef:
            fieldPath: metadata.name
      - name: MY_POD_NAMESPACE
         valueFrom:
         fieldRef:
            fieldPath: metadata.namespace
   volumeMounts:
      - name: config-volume
         mountPath: /etc/config
   command:
      - /pod_nanny
      - --config-dir=/etc/config
      - --container=kube-state-metrics
      - --cpu=100m
      - --extra-cpu=1m
      - --memory=100Mi
      - --extra-memory=2Mi
      - --threshold=5
      - --deployment=kube-state-metrics
   volumes:
   - name: config-volume
      configMap:
         name: kube-state-metrics-config
---
# Config map for resource configuration.
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-state-metrics-config
namespace: kube-system
labels:
k8s-app: kube-state-metrics
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
# kubectl apply -f kube-state-metrics-deployment.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">30</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">31</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">32</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">33</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">34</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">35</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">36</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">37</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">38</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">39</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">40</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">41</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">42</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">43</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">44</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">45</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">46</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">47</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">48</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">49</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">50</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">51</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">52</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">53</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">54</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">55</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">56</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">57</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">58</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">59</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">60</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">61</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">62</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">63</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">64</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">65</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">66</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">67</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">68</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">69</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">70</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">71</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">72</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">73</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">74</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">75</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">76</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">77</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">78</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">79</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">80</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">81</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">82</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">83</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">84</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">85</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">86</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">87</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">88</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">89</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">90</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">91</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">92</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">93</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">94</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">95</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">96</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">97</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">98</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">99</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">100</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">101</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">102</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">103</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">104</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">105</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">106</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">107</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">108</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">109</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">110</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">111</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">112</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">113</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">114</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">115</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">116</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">117</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">118</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">119</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">120</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">121</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">122</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">123</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">124</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">125</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">126</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">127</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">128</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">129</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">130</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">131</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">132</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">133</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">134</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">135</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">136</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">137</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">138</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">139</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">140</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">141</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">142</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">143</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">144</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">145</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">146</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">147</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">148</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">149</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">150</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">151</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">152</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">153</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">154</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">155</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">156</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">157</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">158</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">159</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">160</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">161</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">162</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">163</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">164</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">165</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">166</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">167</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">168</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">169</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">170</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">171</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">172</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">173</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">174</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">175</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">176</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">177</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">178</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">179</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">180</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">181</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">182</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">183</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">184</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">185</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">186</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">187</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">188</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">189</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">190</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">191</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">192</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">193</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">194</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">195</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">196</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">197</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">198</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;">2.编写Service的yaml对metrics进行端口暴露
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># cat kube-state-metrics-service.yaml
apiVersion: v1
kind: Service
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "kube-state-metrics"
annotations:
prometheus.io/scrape: 'true'
spec:
ports:
- name: http-metrics
port: 8080
targetPort: http-metrics
protocol: TCP
- name: telemetry
port: 8081
targetPort: telemetry
protocol: TCP
selector:
k8s-app: kube-state-metrics
# kubectl apply -f kube-state-metrics-service.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"> 3.检查pod和svc的状态，可以看到正常运行了pod/kube-state-metrics-7c76bdbf68-kqqgd 和对外暴露了8080和8081端口
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl get pod,svc -n kube-system
NAME                                     READY STATUS RESTARTS AGE
pod/alertmanager-5d75d5688f-fmlq6       2/2 Running 0       9dpod/coredns-5bd5f9dbd9-wv45t             1/1 Running 1       9dpod/grafana-0                            1/1 Running 2       15dpod/kube-state-metrics-7c76bdbf68-kqqgd 2/2 Running 6       14dpod/kubernetes-dashboard-7d77666777-d5ng4 1/1 Running 5       16dpod/prometheus-0                         2/2 Running 6       15dNAME                         TYPE    CLUSTER-IP EXTERNAL-IP PORT(S)          AGE
service/alertmanager       ClusterIP 10.0.0.207 <none>    80/TCP          13dservice/grafana             NodePort 10.0.0.74 <none>    80:30091/TCP    15dservice/kube-dns             ClusterIP 10.0.0.2 <none>    53/UDP,53/TCP    14dservice/kube-state-metrics ClusterIP 10.0.0.194 <none>    8080/TCP,8081/TCP 14dservice/kubernetes-dashboard NodePort 10.0.0.127 <none>    443:30001/TCP    17dservice/prometheus          NodePort 10.0.0.33 <none>    9090:30090/TCP    14d#
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li></ul></pre><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t22" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t22" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="5_GrafanaPrometheus_876" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>5 使用Grafana可视化展示Prometheus监控数据</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">通常在使用Prometheus采集数据的时候我们需要监控K8S集群中Pod、Node、资源对象，因此我们需要安装对应的插件和资源采集器来提供api进行数据获取，在4.3中我们已经配置好，我们也可以使用Prometheus的UI界面中的Staus菜单下的Target中的各个采集器的状态情况，如图所示：<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210038868.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">只有当我们各个Target的状态都是UP状态时，我们可以使用自带的的界面去获取到某一监控项的相关的数据，如图所示：<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210052715.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">从上面的图中可以看出Prometheus的界面可视化展示的功能较单一，不能满足需求，因此我们需要结合Grafana来进行可视化展示Prometheus监控数据，在上一章节，已经成功部署了Granfana，因此需要在使用的时候添加dashboard和Panel来设计展示相关的监控项，但是实际上在Granfana社区里面有很多成熟的模板，我们可以直接使用，然后根据自己的环境修改Panel中的查询语句来获取数据<br style="box-sizing: border-box; outline: 0px;">https://grafana.com/grafana/dashboards<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210108238.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">推荐模板：</p><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">集群资源监控的模板号：3119，如图所示进行添加<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210124896.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210139253.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210155405.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">当模板添加之后如果某一个Panel不显示数据，可以点击Panel上的编辑，查询PromQL语句，然后去Prometheus自己的界面上进行调试PromQL语句是否可以获取到值，最后调整之后的监控界面如图所示<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210211623.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210225463.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">资源状态监控：6417<br style="box-sizing: border-box; outline: 0px;">同理，添加资源状态的监控模板，然后经过调整之后的监控界面如图所示，可以获取到k8s中各种资源状态的监控展示<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210238162.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">Node监控：9276<br style="box-sizing: border-box; outline: 0px;">同理，添加资源状态的监控模板，然后经过调整之后的监控界面如图所示，可以获取到各个node上的基本情况<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210253712.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"></p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t23" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t23" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="6_K8SAlertmanager_899" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>6 在K8S中部署Alertmanager</h2><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t24" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t24" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="61_Alertmanager_901" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>6.1 部署Alertmanager的实现步骤</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><img src="https://img-blog.csdnimg.cn/20191130210319348.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"></p><h2 style="box-sizing: border-box; outline: 0px; margin-top: 8px; margin-bottom: 16px; font-family: "PingFang SC", "Microsoft YaHei", SimHei, Arial, SimSun; font-size: 22px; color: rgb(79, 79, 79); line-height: 30px; font-variant-ligatures: no-common-ligatures;"><a name="t25" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a name="t25" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a><a id="62__903" style="box-sizing: border-box; outline: none; font-weight: normal; cursor: pointer; background-color: transparent; color: rgb(78, 161, 219);"></a>6.2 部署告警</h2><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">我们以Email来进行实现告警信息的发送</p><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">首先需要准备一个发件邮箱，开启stmp发送功能<br style="box-sizing: border-box; outline: 0px;">使用configmap存储告警规则，编写报警规则的yaml文件，可根据自己的实际情况进行修改和添加报警的规则，prometheus比zabbix就麻烦在这里，所有的告警规则需要自己去定义</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim prometheus-rules.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-rules
namespace: kube-system
data:
general.rules: |
groups:
- name: general.rules
   rules:
   - alert: InstanceDown
   expr: up == 0
   for: 1m
   labels:
      severity: error
   annotations:
      summary: "Instance {{ $labels.instance }} 停止工作"
      description: "{{ $labels.instance }} job {{ $labels.job }} 已经停止5分钟以上."
node.rules: |
groups:
- name: node.rules
   rules:
   - alert: NodeFilesystemUsage
   expr: 100 - (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"} * 100) > 80
   for: 1m
   labels:
      severity: warning
   annotations:
      summary: "Instance {{ $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高"
      description: "{{ $labels.instance }}: {{ $labels.mountpoint }} 分区使用大于80% (当前值: {{ $value }})"

   - alert: NodeMemoryUsage
   expr: 100 - (node_memory_MemFree_bytes+node_memory_Cached_bytes+node_memory_Buffers_bytes) / node_memory_MemTotal_bytes * 10
0 > 80
   for: 1m
   labels:
      severity: warning
   annotations:
      summary: "Instance {{ $labels.instance }} 内存使用率过高"
      description: "{{ $labels.instance }}内存使用大于80% (当前值: {{ $value }})"

   - alert: NodeCPUUsage
   expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"})) by (instance) * 100) > 60
   for: 1m
   labels:
      severity: warning
   annotations:
      summary: "Instance {{ $labels.instance }} CPU使用率过高"
      description: "{{ $labels.instance }}CPU使用大于60% (当前值: {{ $value }})"
# kubectl apply -f prometheus-rules.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">30</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">31</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">32</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">33</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">34</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">35</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">36</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">37</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">38</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">39</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">40</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">41</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">42</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">43</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">44</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">45</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">46</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">47</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">48</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">49</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">50</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">51</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"> 3.编写告警configmap的yaml文件部署，增加alertmanager告警配置，进行配置邮箱发送地址
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim alertmanager-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: alertmanager-config
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
data:
alertmanager.yml: |
global:
   resolve_timeout: 5m
   smtp_smarthost: 'xxx.com.cn:25'#登陆邮件进行查看
   smtp_from: 'xxx@126.com.cn' #根据自己申请的发件邮箱进行配置
   smtp_auth_username: 'xxx@qq.com.cn'
   smtp_auth_password: 'xxxxx'

receivers:
- name: default-receiver
   email_configs:
   - to: "zhangdongdong27459@goldwind.com.cn"

route:
   group_interval: 1m
   group_wait: 10s
   receiver: default-receiver
   repeat_interval: 1m
# kubectl apply -f alertmanager-configmap.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"> 4.创建PVC进行数据持久化，我这个yaml文件使用的跟Prometheus安装时用的存储类来进行自动供给，需要根据自己的实际情况修改
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim alertmanager-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alertmanager
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
spec:
storageClassName: managed-nfs-storage
accessModes:
- ReadWriteOnce
resources:
requests:
   storage: "2Gi"
# kubectl apply -f alertmanager-pvc.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;">5.编写deployment的yaml来部署alertmanager的pod
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim alertmanager-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: alertmanager
namespace: kube-system
labels:
k8s-app: alertmanager
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v0.14.0
spec:
replicas: 1
selector:
matchLabels:
   k8s-app: alertmanager
   version: v0.14.0
template:
metadata:
   labels:
   k8s-app: alertmanager
   version: v0.14.0
   annotations:
   scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
   priorityClassName: system-cluster-critical
   containers:
   - name: prometheus-alertmanager
      image: "prom/alertmanager:v0.14.0"
      imagePullPolicy: "IfNotPresent"
      args:
         - --config.file=/etc/config/alertmanager.yml
         - --storage.path=/data
         - --web.external-url=/
      ports:
         - containerPort: 9093
      readinessProbe:
         httpGet:
         path: /#/status
         port: 9093
         initialDelaySeconds: 30
         timeoutSeconds: 30
      volumeMounts:
         - name: config-volume
         mountPath: /etc/config
         - name: storage-volume
         mountPath: "/data"
         subPath: ""
      resources:
         limits:
         cpu: 10m
         memory: 50Mi
         requests:
         cpu: 10m
         memory: 50Mi
   - name: prometheus-alertmanager-configmap-reload
      image: "jimmidyson/configmap-reload:v0.1"
      imagePullPolicy: "IfNotPresent"
      args:
         - --volume-dir=/etc/config
         - --webhook-url=http://localhost:9093/-/reload
      volumeMounts:
         - name: config-volume
         mountPath: /etc/config
         readOnly: true
      resources:
         limits:
         cpu: 10m
         memory: 10Mi
         requests:
         cpu: 10m
         memory: 10Mi
   volumes:
   - name: config-volume
      configMap:
         name: alertmanager-config
   - name: storage-volume
      persistentVolumeClaim:
         claimName: alertmanager
# kubectl apply -f alertmanager-deployment.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">21</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">22</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">23</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">24</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">25</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">26</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">27</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">28</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">29</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">30</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">31</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">32</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">33</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">34</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">35</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">36</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">37</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">38</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">39</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">40</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">41</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">42</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">43</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">44</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">45</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">46</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">47</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">48</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">49</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">50</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">51</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">52</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">53</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">54</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">55</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">56</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">57</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">58</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">59</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">60</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">61</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">62</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">63</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">64</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">65</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">66</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">67</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">68</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">69</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">70</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">71</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">72</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">73</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">74</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">75</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">76</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">77</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">78</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">79</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">80</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"> 6.创建 alertmanager的service对外暴露的端口
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># vim alertmanager-service.yaml
apiVersion: v1
kind: Service
metadata:
name: alertmanager
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "Alertmanager"
spec:
ports:
- name: http
   port: 80
   protocol: TCP
   targetPort: 9093
selector:
k8s-app: alertmanager
type: "ClusterIP"
# kubectl apply -f alertmanager-service.yaml
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">17</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">18</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">19</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">20</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"> 7.检测部署状态，可以发现pod/alertmanager-5d75d5688f-fmlq6和service/alertmanager正常运行
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li></ul></pre><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl get pod,svc -n kube-system -o wide
NAME                                     READY STATUS RESTARTS AGE IP          NODE          NOMINATED NODE READINESS GATES
pod/alertmanager-5d75d5688f-fmlq6       2/2 Running 4       10d 172.17.15.2 192.168.73.140 <none>       <none>
pod/coredns-5bd5f9dbd9-qxvmz             1/1 Running 0       42m 172.17.33.2 192.168.73.138 <none>       <none>
pod/grafana-0                            1/1 Running 3       16d 172.17.31.2 192.168.73.139 <none>       <none>
pod/kube-state-metrics-7c76bdbf68-hv56m 2/2 Running 0       23h 172.17.15.3 192.168.73.140 <none>       <none>
pod/kubernetes-dashboard-7d77666777-d5ng4 1/1 Running 6       17d 172.17.31.4 192.168.73.139 <none>       <none>
pod/prometheus-0                         2/2 Running 8       16d 172.17.83.2 192.168.73.135 <none>       <none>

NAME                         TYPE    CLUSTER-IP EXTERNAL-IP PORT(S)          AGE SELECTOR
service/alertmanager       ClusterIP 10.0.0.207 <none>    80/TCP          14d k8s-app=alertmanager
service/grafana             NodePort 10.0.0.74 <none>    80:30091/TCP    16d app=grafana
service/kube-dns             ClusterIP 10.0.0.2 <none>    53/UDP,53/TCP    42m k8s-app=kube-dns
service/kube-state-metrics ClusterIP 10.0.0.194 <none>    8080/TCP,8081/TCP 15d k8s-app=kube-state-metrics
service/kubernetes-dashboard NodePort 10.0.0.127 <none>    443:30001/TCP    18d k8s-app=kubernetes-dashboard
service/prometheus          NodePort 10.0.0.33 <none>    9090:30090/TCP    15d k8s-app=prometheus
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">14</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">15</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">16</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">6.3 测试告警发送</p><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">登录prometheus默认的web界面，选择Alert菜单，则可以看到刚才我们使用prometheus-rules.yaml定义的的四个告警规则<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210513626.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">因为告警规则中定义了一个InstanceDown 的实例，所以我们可以停掉138服务器上的kubelet，来测试是否可以收到报警邮件发</p><pre class="prettyprint" style="box-sizing: border-box; outline: 0px; margin-bottom: 24px; padding: 8px 16px 6px 56px; position: relative; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; overflow: auto hidden; border-top: none; border-right: none; border-bottom: 1px solid rgb(245, 246, 247); border-left: none; border-image: initial; line-height: 22px; color: rgb(0, 0, 0); background-color: rgb(40, 44, 52); z-index: 9; font-variant-ligatures: no-common-ligatures;"><code class="has-numbering" style="box-sizing: border-box; outline: 0px; font-family: "Source Code Pro", "DejaVu Sans Mono", "Ubuntu Mono", "Anonymous Pro", "Droid Sans Mono", Menlo, Monaco, Consolas, Inconsolata, Courier, monospace, "PingFang SC", "Microsoft YaHei", sans-serif; line-height: 22px; color: rgb(171, 178, 191); border-radius: 4px; padding: 0.5em; display: block; overflow-x: auto; overflow-wrap: normal; min-width: 94%; text-size-adjust: none; word-break: break-all; position: unset;"># kubectl get node
NAME          STATUS ROLES AGE VERSION
192.168.73.135 Ready <none> 18d v1.15.2
192.168.73.138 Ready <none> 17d v1.15.2
192.168.73.139 Ready <none> 19d v1.15.2
192.168.73.140 Ready <none> 19d v1.15.2
# systemctl stop kubelet
# kubectl get node
NAME          STATUS ROLES AGE VERSION
192.168.73.135 NotReady <none> 18d v1.15.2
192.168.73.138 Ready    <none> 17d v1.15.2
192.168.73.139 Ready    <none> 19d v1.15.2
192.168.73.140 Ready    <none> 19d v1.15.2
</code><ul class="pre-numbering" style="box-sizing: border-box; outline: 0px; padding-top: 8px; padding-bottom: 8px; list-style: none; font-size: 16px; overflow: auto hidden; overflow-wrap: normal; position: absolute; width: 48px; top: 0px; left: 0px; text-align: right; word-break: break-all;"><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">1</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">2</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">3</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">4</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">5</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">6</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">7</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">8</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">9</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">10</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">11</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">12</li><li style="box-sizing: border-box; outline: 0px; margin-left: 0px; padding-right: 8px; padding-left: 8px; list-style: none; font-size: inherit; overflow-wrap: normal; border-right: 1px solid rgb(197, 197, 197); color: rgb(153, 153, 153); word-break: break-all;">13</li></ul></pre><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">稍微等一会，我们再刷新刚才的web上的告警规则界面，可以发现InstanceDown 的实例颜色变成粉红色了，并且显示2 active<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210547925.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">根据规则等待五分钟之后，我们去刷新刚才配置的告警收件箱，收到了一个封InstanceDown 的邮件提醒，邮件的发送间隔时间可以在alertmanager-configmap.yaml配置文件中进行设置，恢复刚才停止的kubelet，将不会收到告警邮件提醒<br style="box-sizing: border-box; outline: 0px;"><img src="https://img-blog.csdnimg.cn/20191130210611453.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述" style="box-sizing: border-box; outline: none; border: 0px none; margin-top: 24px; margin-bottom: 24px; max-width: 100%; cursor: zoom-in;"><br style="box-sizing: border-box; outline: 0px;">如果各位大佬看到这里，相信已经对于如何在k8s集群中部署一套Prometheus监控平台来可视化进行数据的展示和告警有了一定了解，读万卷书,不如行万里路，还是希望大家能够自己多去实操</p><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;"><br></p><p style="box-sizing: border-box; outline: 0px; margin-bottom: 16px; font-size: 16px; color: rgb(77, 77, 77); line-height: 26px; overflow: auto hidden; font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif; font-variant-ligatures: no-common-ligatures;">来源:https://blog.csdn.net/qq_40907977/article/details/103328864</p><p></p>

dirtysea 发表于 2021-8-12 16:17:01

参考 :https://blog.51cto.com/u_14143894/2438026
有关于Grafana的详细调试

七、使用Grafana可视化展示Prometheus监控数据

Grafana模版的id

• 集群资源监控：3119
• 资源状态监控：6417
• Node监控：9276

页: [1]

运维之家's Archiver

k8s实战之部署Prometheus+Grafana可视化监控告警平台