前言

  • 在上一篇文章优雅的使用Prometheus Operator中我们搭建起来了监控的基础堆栈。下来我们来监控我们的业务服务。
  • 监控这个SpringCloud的java微服务的方式有很多，比如说基于eureka的服务型和基于kubernetes的endpoints的注解服务发现。

基础指标依赖

  • 为了能让promethus采集到相关的指标数据我们需要加入一下依赖。
1
2
3
4
5
6
7
8
9

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.0.6</version>
</dependency>

基于eureka服务发现

  • 由于promethus默认提供的服务发现是不支持eureka的，but他支持consul，所以有老铁写了个桥接eureka-consul-adapter，让eurek伪装成consul，这样就能被promethus服务发现支持了。
  • 所以这里我们需要把这个依赖加入到eureka中。
1
2
3
4
5

<dependency>
    <groupId>at.twinformatics</groupId>
    <artifactId>eureka-consul-adapter</artifactId>
    <version>1.3.0</version>
</dependency>
  • 这里我们已经部署好了eureka。我们看下eureka的地址。
1
2
3
4

$ kubectl get po -owide | grep eureka
eureka-0                                   1/1     Running   0          33m     172.224.3.222   node1               <none>           <none>
eureka-1                                   1/1     Running   0          33m     172.224.7.94    node6               <none>           <none>
eureka-2                                   1/1     Running   0          34m     172.224.4.127   node2               <none>           <none>
  • 任选一个节点，查看我们的适配器是否已经成功生效 $ curl -i http://172.224.3.222:8761/v1/catalog/services HTTP/1.1 200 OK Connection: keep-alive
1
2
3
4
5
6

{
	"CONFIG-SERVICE": [],
	"DEMO1-SERVICE": [],
	"DEMO2-SERVICE": [],
	"EUREKA": []
}
  • 查看服务的详细 $ curl -i http://172.224.3.222:8761/v1/catalog/service/DEMO2-SERVICE HTTP/1.1 200 OK Connection: keep-alive
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[
	{
		"Address": "172.224.3.223",
		"Node": "DEMO2-SERVICE",
		"ServiceAddress": "172.224.3.223",
		"ServiceName": "DEMO2-SERVICE",
		"ServiceID": "demo2-service:8101@5826031",
		"ServicePort": 8101,
		"NodeMeta": {},
		"ServiceMeta": {
			"management.port": "8101"
		},
		"ServiceTags": []
	}
]
  • 查看指标是否能成功获取 $ curl -i http://172.224.3.223:8101/actuator/prometheus HTTP/1.1 200 OK Connection: keep-alive …
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total 1.0
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="direct",} 1685563.0
jvm_buffer_memory_used_bytes{id="mapped",} 0.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 67.0
...
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="direct",} 154.0
jvm_buffer_count_buffers{id="mapped",} 0.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 10297.0

这样我们就把spring boot配置完成了，接下来需要配置promethus。

追加自定义配置到promethus-oprator

我们需要继续修改文件prometheus-prometheus.yaml，我们从官方提供的文档可知，他为我们追加自定义的配置预留了一个属性additionalScrapeConfigs，我们只需将自定义的配置，通过这个配置就能追到上去。

additionalScrapeConfigs的格式为如下，为了方便我们使用，这里追加配置的方式，它采用了kubernetes的secret机制，我们自己编写的配置需要提交成一个secret，而后对应的配置，name就是secret的名称，key就是文件名。

1
2
3
4

spec:
  additionalScrapeConfigs:
    name:
    key:

有了基本概念之后我们就来追加我们的服务发现配置。

  • prometheus-additional.yaml
1
2
3
4

- job_name: "eureka-ds"
  consul_sd_configs:
    - server: "eureka-0.eureka.default.svc.cluster.local:8761"
  metrics_path: /actuator/prometheus
  • 创建
1

kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
  • 查看
1
2
3
4
5
6
7
8
9

$ kubectl get secret additional-configs -n monitoring -o yaml
apiVersion: v1
data:
  prometheus-additional.yaml: I2t1YmVjdGwgY3JlYXRlIHNlY3JldCBnZW5lcmljIGFkZGl0aW9uYWwtY29uZmlncyAtLWZyb20tZmlsZT1wcm9tZXRoZXVzLWFkZGl0aW9uYWwueWFtbCAtbiBtb25pdG9yaW5nCiNrdWJlY3RsIGdldCBzZWNyZXQgYWRkaXRpb25hbC1jb25maWdzIC1uIG1vbml0b3JpbmcgLW8geWFtbAoja3ViZWN0bCBkZWxldGUgc2VjcmV0IGFkZGl0aW9uYWwtY29uZmlncyAtbm1vbml0b3JpbmcKLSBqb2JfbmFtZTogImV1cmVrYS1kcyIKICBjb25zdWxfc2RfY29uZmlnczoKICAgIC0gc2VydmVyOiAiZXVyZWthLTAuZXVyZWthLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODc2MSIKICBtZXRyaWNzX3BhdGg6IC9hY3R1YXRvci9wcm9tZXRoZXVzCiMtIGpvYl9uYW1lOiAna3ViZXJuZXRlcy1zZXJ2aWNlLWVuZHBvaW50cycKIyAga3ViZXJuZXRlc19zZF9jb25maWdzOgojICAgIC0gcm9sZTogZW5kcG9pbnRzCiMgIHJlbGFiZWxfY29uZmlnczoKIyAgICAtIHNvdXJjZV9sYWJlbHM6IFtfX21ldGFfa3ViZXJuZXRlc19zZXJ2aWNlX2Fubm90YXRpb25fcHJvbWV0aGV1c19pb19zcHJpbmdzY3JhcGVdCiMgICAgICBhY3Rpb246IGtlZXAKIyAgICAgIHJlZ2V4OiB0cnVlCiMgICAgLSBzb3VyY2VfbGFiZWxzOiBbX19tZXRhX2t1YmVybmV0ZXNfc2VydmljZV9hbm5vdGF0aW9uX3Byb21ldGhldXNfaW9fc2NoZW1lXQojICAgICAgYWN0aW9uOiByZXBsYWNlCiMgICAgICB0YXJnZXRfbGFiZWw6IF9fc2NoZW1lX18KIyAgICAgIHJlZ2V4OiAoaHR0cHM/KQojICAgIC0gc291cmNlX2xhYmVsczogW19fbWV0YV9rdWJlcm5ldGVzX3NlcnZpY2VfYW5ub3RhdGlvbl9wcm9tZXRoZXVzX2lvX3BhdGhdCiMgICAgICBhY3Rpb246IHJlcGxhY2UKIyAgICAgIHRhcmdldF9sYWJlbDogX19tZXRyaWNzX3BhdGhfXwojICAgICAgcmVnZXg6ICguKykKIyAgICAtIHNvdXJjZV9sYWJlbHM6IFtfX2FkZHJlc3NfXywgX19tZXRhX2t1YmVybmV0ZXNfc2VydmljZV9hbm5vdGF0aW9uX3Byb21ldGhldXNfaW9fcG9ydF0KIyAgICAgIGFjdGlvbjogcmVwbGFjZQojICAgICAgdGFyZ2V0X2xhYmVsOiBfX2FkZHJlc3NfXwojICAgICAgcmVnZXg6IChbXjpdKykoPzo6XGQrKT87KFxkKykKIyAgICAgIHJlcGxhY2VtZW50OiAkMTokMgojICAgIC0gYWN0aW9uOiBsYWJlbG1hcAojICAgICAgcmVnZXg6IF9fbWV0YV9rdWJlcm5ldGVzX3NlcnZpY2VfbGFiZWxfKC4rKQojICAgIC0gc291cmNlX2xhYmVsczogW19fbWV0YV9rdWJlcm5ldGVzX25hbWVzcGFjZV0KIyAgICAgIGFjdGlvbjogcmVwbGFjZQojICAgICAgdGFyZ2V0X2xhYmVsOiBrdWJlcm5ldGVzX25hbWVzcGFjZQojICAgIC0gc291cmNlX2xhYmVsczogW19fbWV0YV9rdWJlcm5ldGVzX3NlcnZpY2VfbmFtZV0KIyAgICAgIGFjdGlvbjogcmVwbGFjZQojICAgICAgdGFyZ2V0X2xhYmVsOiBrdWJlcm5ldGVzX25hbWUK
kind: Secret
metadata:
  name: additional-configs
  namespace: monitoring
type: Opaque
  • 删除
1

kubectl delete secret additional-configs -nmonitoring
  • 有了secret之后呢我们就可以配置promethus了
1
2
3
4

spec:
  additionalScrapeConfigs:
    name: additional-configs #对应secret名
    key: prometheus-additional.yaml # 对应文件名
  • 完整配置如下
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: freemanliu/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: local-storage-promethues
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 10Gi
  • 重新apply，使其生效
1

kubectl apply -f manifests/
  • 再次通过SVCIP查看我们promethus的targets，可以看到已经成功采集到数据了，有一个错误是因为我们的config-service没有加入指标依赖。加入即可解决。

spring boot 微服务仪表盘

我们这里需要导入dashboard，需要的导入的id为4701。导入完成之后你将看到如下展示。