prometheus整合jmx_exporter 使用jmx_exporter监控Kafka

宁睿  论坛元老 | 2025-4-14 14:19:41 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 1949|帖子 1949|积分 5857

 docker-compose摆设kafka集群；单节点单zk-CSDN博客  
  
  
  springboot整合kafka；docker摆设kafka-CSDN博客

 
  
  kafka使用SSL加密和认证--todo_ssl.truststore.location-CSDN博客
  
  1. version: '3.8'
  3. services:
  4.   zookeeper1:
  5.     image: zookeeper:3.9.1
  6.     container_name: zookeeper1-cluster
  7.     hostname: zookeeper-cluster
  8.     ports:
  9.       - "2185:2181"
  10.     environment:
  11.       ZOOKEEPER_CLIENT_PORT: 2181
  12.       ZOOKEEPER_TICK_TIME: 2000
  13.       ZOOKEEPER_INIT_LIMIT: 10
  14.       ZOOKEEPER_SYNC_LIMIT: 5
  15.     volumes:
  16.       - ./zookeeper-data:/data
  17.       - ./zookeeper-datalog:/datalog
  18.     networks:
  19.       - kafka-cluster-network
  21.   kafka1:
  22.     image: bitnami/kafka:3.8
  23.     container_name: kafka1-cluster
  24.     depends_on:
  25.       - zookeeper1
  26.     ports:
  27.       - "9099:9092" # Kafka 客户端连接端口
  28.       - "9101:9101" # Prometheus 抓取指标的端口
  29.     environment:
  30.       KAFKA_BROKER_ID: 1
  31.       KAFKA_ZOOKEEPER_CONNECT: zookeeper-cluster:2181
  32.       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.118.20:9099
  33.       KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
  34.       KAFKA_CFG_MESSAGE_MAX_BYTES: 2097152
  35.       KAFKA_CFG_REPLICA_FETCH_MAX_BYTES: 3145728
  36.       KAFKA_CFG_FETCH_MAX_BYTES: 5242880
  37.       KAFKA_OPTS: "-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=9101:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml"
  38.     volumes:
  39.       - ./kafka1-data:/bitnami/kafka
  40.       - ./jmx_prometheus_javaagent-0.20.0.jar:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar
  41.       - ./kafka-jmx-exporter-config.yaml:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml
  42.     networks:
  43.       - kafka-cluster-network
  45.   kafka2:
  46.     image: bitnami/kafka:3.8
  47.     container_name: kafka2-cluster
  48.     depends_on:
  49.       - zookeeper1
  50.     ports:
  51.       - "9096:9092" # Kafka 客户端连接端口
  52.       - "9102:9102" # Prometheus 抓取指标的端口
  53.     environment:
  54.       KAFKA_BROKER_ID: 2
  55.       KAFKA_ZOOKEEPER_CONNECT: zookeeper-cluster:2181
  56.       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.118.20:9096
  57.       KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
  58.       KAFKA_CFG_MESSAGE_MAX_BYTES: 2097152
  59.       KAFKA_CFG_REPLICA_FETCH_MAX_BYTES: 3145728
  60.       KAFKA_CFG_FETCH_MAX_BYTES: 5242880
  61.       KAFKA_OPTS: "-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=9102:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml"
  62.     volumes:
  63.       - ./kafka2-data:/bitnami/kafka
  64.       - ./jmx_prometheus_javaagent-0.20.0.jar:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar
  65.       - ./kafka-jmx-exporter-config.yaml:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml
  66.     networks:
  67.       - kafka-cluster-network
  69.   kafka3:
  70.     image: bitnami/kafka:3.8
  71.     container_name: kafka3-cluster
  72.     depends_on:
  73.       - zookeeper1
  74.     ports:
  75.       - "9097:9092" # Kafka 客户端连接端口
  76.       - "9103:9103" # Prometheus 抓取指标的端口
  77.     environment:
  78.       KAFKA_BROKER_ID: 3
  79.       KAFKA_ZOOKEEPER_CONNECT: zookeeper-cluster:2181
  80.       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.118.20:9097
  81.       KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
  82.       KAFKA_CFG_MESSAGE_MAX_BYTES: 2097152
  83.       KAFKA_CFG_REPLICA_FETCH_MAX_BYTES: 3145728
  84.       KAFKA_CFG_FETCH_MAX_BYTES: 5242880
  85.       KAFKA_OPTS: "-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=9103:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml"
  86.     volumes:
  87.       - ./kafka3-data:/bitnami/kafka
  88.       - ./jmx_prometheus_javaagent-0.20.0.jar:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar
  89.       - ./kafka-jmx-exporter-config.yaml:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml
  90.     networks:
  91.       - kafka-cluster-network
  93.   kafka-ui:
  94.     image: provectuslabs/kafka-ui:v0.7.2
  95.     container_name: kafka-ui-cluster
  96.     hostname: kafka-ui-cluster
  97.     ports:
  98.       - "9098:8080"
  99.     depends_on:
  100.       - kafka1
  101.       - kafka2
  102.       - kafka3
  103.     environment:
  104.       KAFKA_CLUSTERS_0_NAME: kafka-cluster
  105.       KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: 192.168.118.20:9096,192.168.118.20:9097,192.168.118.20:9099
  106.       KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper-cluster:2181
  107.     networks:
  108.       - kafka-cluster-network
  110. networks:
  111.   kafka-cluster-network:
  112.     driver: bridge
复制代码
  下载    
jmx_prometheus_javaagent-0.20.0.jar
Central Repository: io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/


 
  或者https://github.com/prometheus/jmx_exporter/releases/tag/0.20.0
  

    下载 https://github.com/prometheus/jmx_exporter/tree/main/exampleskafka-jmx-exporter-config.yaml
  
https://github.com/prometheus/jmx_exporter/tree/main/examples
    1、在启动kafka时配置代理
  2、同时文件也挂载进去
 
  1. KAFKA_OPTS: "-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=9102:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml"
复制代码
  1.     volumes:
  2.       - ./kafka2-data:/bitnami/kafka
  3.       - ./jmx_prometheus_javaagent-0.20.0.jar:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar
  4.       - ./kafka-jmx-exporter-config.yaml:/opt/jmx_exporter/kafka-jmx-exporter-config.yaml
复制代码
  1. - "9103:9103" # Prometheus 抓取指标的端口
复制代码
启动服务后  查抄一下  是否配置成功 
http://192.168.118.20:9101/metrics

  
 摆设prometheus 和 grafana
  
  1. version: '3.8'
  3. services:
  4.   prometheus:
  5.     image: bitnami/prometheus:3.0.0
  6.     container_name: prometheus
  7.     hostname: prometheus
  8.     ports:
  9.       - "9090:9090" # Prometheus Web UI 端口
  10.     volumes:
  11.       - ./prometheus.yml:/etc/prometheus/prometheus.yml
  12.       - ./prometheus-data:/prometheus
  13.     command:
  14.       - '--config.file=/etc/prometheus/prometheus.yml'
  15.       - '--web.enable-lifecycle'
  16.       - '--storage.tsdb.retention.time=90d'
  17.       - "--storage.tsdb.path=/prometheus"
  18.       - "--web.enable-admin-api"
  19.     restart: always
  21.   grafana:
  22.     image: grafana/grafana:11.3.3
  23.     container_name: grafana
  24.     hostname: grafana
  25.     ports:
  26.       - "3000:3000" # Grafana Web UI 端口
  27.     environment:
  28.       GF_SECURITY_ADMIN_PASSWORD: admin # 设置 Grafana 的管理员密码
  29.     volumes:
  30.       - ./grafana-storage:/var/lib/grafana
  31.     restart: always
复制代码
  1. prometheus.yml  如下配置
  3. global:
  4.   scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  5.   evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  6.   # scrape_timeout is set to the global default (10s).
  8. # Alertmanager configuration
  9. alerting:
  10.   alertmanagers:
  11.     - static_configs:
  12.         - targets:
  13.           # - alertmanager:9093
  16. # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
  17. rule_files:
  18.   # - "first_rules.yml"
  19.   # - "second_rules.yml"
  21. # A scrape configuration containing exactly one endpoint to scrape:
  22. # Here it's Prometheus itself.
  23. scrape_configs:
  24.   - job_name: 'kafka'
  25.     static_configs:
  26.       - targets:
  27.           - '192.168.118.20:9101' # 对应 kafka1 的 JMX Exporter 端口
  28.           - '192.168.118.20:9102' # 对应 kafka2 的 JMX Exporter 端口
  29.           - '192.168.118.20:9103' # 对应 kafka3 的 JMX Exporter 端口
  31.   - job_name: "node"
  32.     static_configs:
  33.       - targets: ["localhost:9100"]
复制代码
grafana:    用户名和密码  都是  admin admin
  http://192.168.xxx.xxx:3000/
  

  

    https://grafana.com/grafana/dashboards/11962-kafka-metrics/
  https://grafana.com/grafana/dashboards/11962-kafka-metrics/
  

  
  
   https://grafana.com/grafana/dashboards/10973-kafka-dashboard333/
  

  
   Kafka Dashboard | Grafana Labs 
该模版需要更新yaml

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

本帖子中包含更多资源

您需要 登录 才可以下载或查看，没有账号？立即注册

x
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

宁睿

论坛元老
这个人很懒什么都没写!

楼主热帖

标签云

AI 运维 CIO 存储 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表