qidao123.com技术社区-IT企服评测·应用市场

标题: Prometheus +VictoriaMetrics+Granafa安装部署 [打印本页]
作者: 自由的羽毛    时间: 2025-1-18 19:46
标题: Prometheus +VictoriaMetrics+Granafa安装部署
测试环境

prometheus-2.54.1.linux-amd64.tar.gz
下载地址：
https://www.prometheus.io/download/
https://github.com/prometheus/prometheus/releases/download/v2.54.1/prometheus-2.54.1.linux-amd64.tar.gz
node_exporter-1.8.2.linux-amd64.tar.gz
下载地址：
https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
https://prometheus.io/download/#node_exporter
pushgateway-1.9.0.linux-amd64.tar.gz
下载地址：
https://www.prometheus.io/download/#pushgateway
https://github.com/prometheus/pushgateway/releases/download/v1.9.0/pushgateway-1.9.0.linux-amd64.tar.gz
victoria-metrics-linux-amd64-v1.103.0.tar.gz
下载地址：
https://github.com/VictoriaMetrics/VictoriaMetrics/releases
https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.103.0
https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.103.0/victoria-metrics-linux-amd64-v1.103.0.tar.gz
grafana-7.5.6-1.x86_64.rpm
下载地址：https://dl.grafana.com/oss/release/grafana-7.5.6-1.x86_64.rpm
CentOS 7.9
留意：prometheus，victoria-metrics，grafana，pushgateway 都可以安装在不同机器上，本文仅涉及学习实践，以是，都安装在同一台机器上了。
实践过程

VictoriaMetrics安装
  1. # wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.103.0/victoria-metrics-linux-amd64-v1.103.0.tar.gz
  2. # tar -xvzf victoria-metrics-linux-amd64-v1.103.0.tar.gz -C /usr/local/bin # 解压后会生成一个名为victoria-metrics-prod的二进制文件
  3. # 创建一个用于存储VictoriaMetrics数据的文件夹
  4. # mkdir -p /usr/data/victoria-metrics
复制代码
  1. # 创建服务
  2. # vi /etc/systemd/system/victoriametrics.service
  3. [Unit]
  4. Description=Victoria metrics service
  5. After=network.target
  7. [Service]
  8. Type=simple
  9. Restart=always
  10. TimeoutStartSec=30
  11. Restart=on-failure
  12. RestartSec=5s
  13. ExecStart=/usr/local/bin/victoria-metrics-prod -storageDataPath=/usr/data/victoria-metrics -retentionPeriod=30d -selfScrapeInterval=10s
  14. ExecStop=/bin/kill $MAINPID
  15. ExecReload=/bin/kill -HUP $MAINPID
  17. PrivateTmp=yes
  19. [Install]
  20. WantedBy=multi-user.target
复制代码
说明：
-storageDataPath  设置数据目录路径(如果目录路径不存在，程序启动时会自动创建)。VictoriaMetrics会将全部数据存储在此目录中。默认存储在当前工作目录中的victoria-metrics-data目录
-retentionPeriod  设置存储数据的保留。自动删除旧的数据。默认保留时间为1个月(31天)。 最小值为24h大概1d,-retentionPeriod=3设置数据仅存储3个月，-retentionPeriod=1d设置数据仅保留1天。
一般情况下，只必要设置上述两个参数标识即可，其它参数已经有充足好的默认值，仅在确实必要修改它们时才进行设置，实验./victoria-metrics-prod --help 可以检察全部可获取的参数形貌和及其默认值全部可获取的参数形貌和默认值。
默认的，VictoriaMetrics通过端口8428监听Prometheus查询API哀求
建议为 VictoriaMetrics设置监控
前台启动检察
  1. # /usr/local/bin/victoria-metrics-prod -storageDataPath=/usr/data/victoria-metrics -retentionPeriod=30d -selfScrapeInterval=10s
  2. 2024-09-03T16:33:42.187Z        info        VictoriaMetrics/lib/logger/flag.go:12        build version: victoria-metrics-20240828-135248-tags-v1.103.0-0-g5aeb759df9
  3. 2024-09-03T16:33:42.187Z        info        VictoriaMetrics/lib/logger/flag.go:13        command-line flags
  4. 2024-09-03T16:33:42.187Z        info        VictoriaMetrics/lib/logger/flag.go:20          -retentionPeriod="30d"
  5. 2024-09-03T16:33:42.187Z        info        VictoriaMetrics/lib/logger/flag.go:20          -selfScrapeInterval="10s"
  6. 2024-09-03T16:33:42.187Z        info        VictoriaMetrics/lib/logger/flag.go:20          -storageDataPath="/usr/data/victoria-metrics"
  7. 2024-09-03T16:33:42.187Z        info        VictoriaMetrics/app/victoria-metrics/main.go:73        starting VictoriaMetrics at "[:8428]"...
  8. 2024-09-03T16:33:42.187Z        info        VictoriaMetrics/app/vmstorage/main.go:107        opening storage at "/usr/data/victoria-metrics" with -retentionPeriod=30d
  9. 2024-09-03T16:33:42.189Z        info        VictoriaMetrics/lib/memory/memory.go:42        limiting caches to 611758080 bytes, leaving 407838720 bytes to the OS according to -memory.allowedPercent=60
  10. 2024-09-03T16:33:42.205Z        info        VictoriaMetrics/app/vmstorage/main.go:121        successfully opened storage "/usr/data/victoria-metrics" in 0.018 seconds; partsCount: 0; blocksCount: 0; rowsCount: 0; sizeBytes: 0
  11. 2024-09-03T16:33:42.205Z        info        VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:127        loading rollupResult cache from "/usr/data/victoria-metrics/cache/rollupResult"...
  12. 2024-09-03T16:33:42.207Z        info        VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:156        loaded rollupResult cache from "/usr/data/victoria-metrics/cache/rollupResult" in 0.001 seconds; entriesCount: 0, sizeBytes: 0
  13. 2024-09-03T16:33:42.207Z        info        VictoriaMetrics/app/victoria-metrics/main.go:84        started VictoriaMetrics in 0.020 seconds
  14. 2024-09-03T16:33:42.207Z        info        VictoriaMetrics/lib/httpserver/httpserver.go:121        starting server at http://127.0.0.1:8428/
  15. 2024-09-03T16:33:42.207Z        info        VictoriaMetrics/lib/httpserver/httpserver.go:122        pprof handlers are exposed at http://127.0.0.1:8428/debug/pprof/
  16. 2024-09-03T16:33:42.208Z        info        VictoriaMetrics/app/victoria-metrics/self_scraper.go:46        started self-scraping `/metrics` page with interval 10.000 seconds
  17. 2024-09-03T16:33:52.293Z        info        VictoriaMetrics/lib/storage/partition.go:202        creating a partition "2024_09" with smallPartsPath="/usr/data/victoria-metrics/data/small/2024_09", bigPartsPath="/usr/data/victoria-metrics/data/big/2024_09"
  18. 2024-09-03T16:33:52.295Z        info        VictoriaMetrics/lib/storage/partition.go:211        partition "2024_09" has been created
复制代码
关闭前台启动的进程，启动服务并设置开机启动
  1. # systemctl daemon-reload && sudo systemctl enable --now victoriametrics.service
  2. # 查看服务是否启动成功
  3. # systemctl status victoriametrics.service
复制代码
环境变量

全部VictoriaMetrics组件都允许通过%{ENV_VAR}语法在yaml配置文件（如-promgraft.config）和命令行标志中引用环境变量。比方，如果VictoriaMetrics启动时存在METRICS_AUTH_KEY=top-secret环境变量，则-metricsAuthKey=%{METRICS_AUTH_KEY} 会被VictoriaMetrics自动扩展为-metricsAuthKey=top-secret 。
VictoriaMetrics在启动时递归扩展环境变量中的%{ENV_VAR}引用。比方，如果 BAR=a%{BAZ} 且BAZ=bc，则FOO=%{BAR}环境变量扩展为FOO=abc。
此外，全部VictoriaMetrics组件都允许根据以下规则通过环境变量设置参数标识值：
开放防火墙端口
  1. # firewall-cmd --permanent --zone=public --add-port=8248/tcp
  2. success
  3. # firewall-cmd --reload
  4. success
复制代码
除了以二进制方式启动，VictoriaMetrics也支持Docker安装，具体可参考链接 https://hub.docker.com/r/victoriametrics/victoria-metrics/
参考链接

https://docs.victoriametrics.com/quick-start/
Prometheus安装与配置
  1. # wget https://github.com/prometheus/prometheus/releases/download/v2.54.1/prometheus-2.54.1.linux-amd64.tar.gz
  2. # tar -C /usr/local/ -xvzf prometheus-2.54.1.linux-amd64.tar.gz
  3. # cd /usr/local/prometheus-2.54.1.linux-amd64
  4. # ls
  5. console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool
  6. # ln -s /usr/local/prometheus-2.26.0.linux-amd64/prometheus /usr/local/bin/prometheus
  7. # cp prometheus.yml prometheus.yml.bak
  8. # echo ''> prometheus.yml
  9. # vi prometheus.yml
复制代码
将prometheus.yml内容替换为以下内容
  1. global:
  2.   scrape_interval:     15s # 默认，每15秒采样一次目标
  4. # 一份采样配置仅包含一个 endpoint 来做采样
  5. # 下面是 Prometheus 本身的endpoint:
  6. scrape_configs:
  7.   # job_name 将被被当作一个标签 `job=<job_name>`添加到该配置的任意时序采样.
  8.   - job_name: 'prometheus'
  10.     # 覆盖全局默认值，从该job每5秒对目标采样一次
  11.     scrape_interval: 5s
  13.     static_configs:
  14.       - targets: ['localhost:9090']
  16. remote_write:
  17.   - url: http://192.168.88.132:8428/api/v1/write
  18.     queue_config:
  19.       max_samples_per_send: 10000
  20.       capacity: 20000
  21.       max_shards: 30
复制代码
说明：
为了能给VictoriaMetrics发送数据，在Prometheus配置文件中，增加remote_write 配置。 添加以下代码到 Prometheus 配置文件(一般是在prometheus.yml):
  1. remote_write:
  2.   - url: http://<victoriametrics-addr>:8428/api/v1/write
复制代码
留意：添加时，必要替换为VictoriaMetrics主机名称大概IP地址，形如以下
  1. remote_write:
  2.   - url: http://192.168.88.132.71.170:8428/api/v1/write
复制代码
启动Prometheus服务
  1. # ./prometheus
  2. ts=2024-09-04T15:50:32.906Z caller=main.go:601 level=info msg="No time or size retention was set so using the default time retention" duration=15d
  3. ts=2024-09-04T15:50:32.906Z caller=main.go:645 level=info msg="Starting Prometheus Server" mode=server version="(version=2.54.1, branch=HEAD, revision=e6cfa720fbe6280153fab13090a483dbd40bece3)"
  4. ts=2024-09-04T15:50:32.906Z caller=main.go:650 level=info build_context="(go=go1.22.6, platform=linux/amd64, user=root@812ffd741951, date=20240827-10:56:41, tags=netgo,builtinassets,stringlabels)"
  5. ts=2024-09-04T15:50:32.906Z caller=main.go:651 level=info host_details="(Linux 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 localhost.localdomain (none))"
  6. ts=2024-09-04T15:50:32.906Z caller=main.go:652 level=info fd_limits="(soft=4096, hard=4096)"
  7. ts=2024-09-04T15:50:32.906Z caller=main.go:653 level=info vm_limits="(soft=unlimited, hard=unlimited)"
  8. ts=2024-09-04T15:50:32.917Z caller=web.go:571 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
  9. ts=2024-09-04T15:50:32.925Z caller=main.go:1160 level=info msg="Starting TSDB ..."
  10. ts=2024-09-04T15:50:32.930Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9090
  11. ts=2024-09-04T15:50:32.930Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
  12. ts=2024-09-04T15:50:32.932Z caller=head.go:626 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
  13. ts=2024-09-04T15:50:32.932Z caller=head.go:713 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=5.601µs
  14. ts=2024-09-04T15:50:32.933Z caller=head.go:721 level=info component=tsdb msg="Replaying WAL, this may take a while"
  15. ts=2024-09-04T15:50:32.933Z caller=head.go:793 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
  16. ts=2024-09-04T15:50:32.933Z caller=head.go:830 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.237µs wal_replay_duration=560.14µs wbl_replay_duration=141ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=5.601µs total_replay_duration=605.091µs
  17. ts=2024-09-04T15:50:32.938Z caller=main.go:1181 level=info fs_type=XFS_SUPER_MAGIC
  18. ts=2024-09-04T15:50:32.938Z caller=main.go:1184 level=info msg="TSDB started"
  19. ts=2024-09-04T15:50:32.938Z caller=main.go:1367 level=info msg="Loading configuration file" filename=prometheus.yml
  20. ts=2024-09-04T15:50:32.940Z caller=dedupe.go:112 component=remote level=info remote_name=b93975 url=http://192.168.88.132:8428/api/v1/write msg="Starting WAL watcher" queue=b93975
  21. ts=2024-09-04T15:50:32.940Z caller=dedupe.go:112 component=remote level=info remote_name=b93975 url=http://192.168.88.132:8428/api/v1/write msg="Starting scraped metadata watcher"
  22. ts=2024-09-04T15:50:32.945Z caller=main.go:1404 level=info msg="updated GOGC" old=100 new=75
  23. ts=2024-09-04T15:50:32.945Z caller=main.go:1415 level=info msg="Completed loading of configuration file" filename=prometheus.yml totalDuration=6.619214ms db_storage=25.792µs remote_storage=1.190631ms web_handler=652ns query_engine=18.267µs scrape=3.897727ms scrape_sd=49.586µs notify=1.164µs notify_sd=954ns rules=60.122µs tracing=62.555µs
  24. ts=2024-09-04T15:50:32.945Z caller=main.go:1145 level=info msg="Server is ready to receive web requests."
  25. ts=2024-09-04T15:50:32.945Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..."
  26. ts=2024-09-04T15:50:32.945Z caller=dedupe.go:112 component=remote level=info remote_name=b93975 url=http://192.168.88.132:8428/api/v1/write msg="Replaying WAL" queue=b93975
  27. ts=2024-09-04T15:50:40.288Z caller=dedupe.go:112 component=remote level=info remote_name=b93975 url=http://192.168.88.132:8428/api/v1/write msg="Done replaying WAL" duration=7.342804783s
复制代码
说明：如果盼望利用非默认配置文件，可以在实验命令时指定具体的配置文件，类似如下：
  1. # ./prometheus --config.file=./custom_prometheus.yml
复制代码
备注：重启命令
  1. kill -HUP `pid_of_prometheus`
复制代码
当然也可直接Ctrl + c停止Prometheus进程，然后重新运行。
Prometheus把传入的数据写入到本地存储的同时将数据复制到远程存储。这意味着即使远程存储不可用，存储在本地，--storage.tsdb.retention.time 指定数据保留期内的数据依然可用。
如果必要从多个 Prometheus 实例往 VictoriaMetrics 发送数据，添加external_labels配置到Prometheus配置文件的global结点，形如以下：
  1. global:
  2.   external_labels:
  3.     datacenter: dc-123
复制代码
如上，以上配置告诉 Prometheus 添加 datacenter=dc-123 标签到发送给远程存储的每个时间序列。标签名称可以是任意的，比如 datacenter 。标签值必须在全部 Prometheus 实例中保持唯一，如许，可以通过该标签过滤大概分组时序。
对于高负载的Prometheus实例（每秒200k+个样本），可以应用以下调优：
  1. remote_write:
  2.   - url: http://<victoriametrics-addr>:8428/api/v1/write    queue_config:      max_samples_per_send: 10000      capacity: 20000      max_shards: 30
复制代码
利用远程写入增加 Prometheus 约25%的内存利用率，这取决于数据形态(原文：Using remote write increases memory usage for Prometheus up to ~25% and depends on the shape of data)。如果你正面对太高内存消耗的题目，尝试低落 max_samples_per_send 和 capacity 参数配置值(留意：这两个参数是紧密相连的）检察更多关于远程写入调优.
建议升级Prometheus到 v2.12.0 或更高，因为之前版本利用 remote_write存在题目。
也可以检察下 vmagent 和 vmalert,  也是用于减少Prometheus更快和更少的资源消耗的一种选择方案。
参考链接：https://docs.victoriametrics.com/#prometheus-setup
创建服务
  1. # vi /etc/systemd/system/rometheus.service
复制代码
  1. [Unit]
  2. Description=Prometheus service
  3. After=network.target
  5. [Service]
  6. Type=simple
  7. Restart=always
  8. TimeoutStartSec=30
  9. Restart=on-failure
  10. RestartSec=5s
  11. ExecStart=/usr/local/prometheus-2.54.1.linux-amd64/prometheus --config.file=/usr/local/prometheus-2.54.1.linux-amd64/prometheus.yml
  12. ExecStop=/bin/kill $MAINPID
  13. ExecReload=/bin/kill -HUP $MAINPID
  15. [Install]
  16. WantedBy=multi-user.target
复制代码
留意：配置文件必须利用绝对路径，否则运行服务报错，找不到文件/prometheus.yml
先手动停止上述前台运行的prometheus，然后实验以下命令
  1. # systemctl daemon-reload && systemctl enable --now prometheus
  2. # systemctl status prometheus
复制代码
grafana安装与配置
  1. # yum install grafana-7.5.6-1.x86_64.rpm
复制代码
说明：如果利用yum命令安装找不到软件包，可以思量下载安装，如下
  1. # wget https://dl.grafana.com/oss/release/grafana-7.5.6-1.x86_64.rpm
  2. # yum install -y fontconfig urw-fonts
  3. # rpm -ivh grafana-7.5.6-1.x86_64.rpm
  4. warning: grafana-7.5.6-1.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 24098cb6: NOKEY
  5. Preparing...                          ################################# [100%]
  6. Updating / installing...
  7.    1:grafana-7.5.6-1                  ################################# [100%]
  8. ### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
  9. sudo /bin/systemctl daemon-reload
  10. sudo /bin/systemctl enable grafana-server.service
  11. ### You can start grafana-server by executing
  12. sudo /bin/systemctl start grafana-server.service
  13. POSTTRANS: Running script
  14. # /bin/systemctl daemon-reload
  15. ~]# /bin/systemctl enable grafana-server.service
  16. Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
  17. # /bin/systemctl start grafana-server.service
复制代码
说明：如果不实验yum install -y fontconfig urw-fonts命令，安装grafana时可能报错，如下
  1. warning: grafana-7.5.6-1.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 24098cb6: NOKEY
  2. error: Failed dependencies:
  3.         fontconfig is needed by grafana-7.5.6-1.x86_64
  4.         urw-fonts is needed by grafana-7.5.6-1.x86_64
复制代码
修改grafana配置[可选]
  1. # vim /etc/grafana/grafana.ini
复制代码
浏览器输入网址：http://:3000访问看看结果：

备注：grafana默认登录账号密码为admin/admin
参考链接：https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1
开放防火墙端口
  1. # firewall-cmd --permanent --zone=public --add-port=3000/tcp
  2. success
  3. # firewall-cmd --reload
  4. success
复制代码
创建Prometheus数据源

利用以下URL创建 Prometheus数据源 (创建数据源时仅修改URL，其它保持默认) :
  1. http://<victoriametrics-addr>:8428
复制代码
如下，替换 为VictoriaMetrics主机名称大概IP地址，形如http://192.168.55.88.132:8428，然后利用PromQL 或MetricsQL用创建的数据源创建图表

Access模式简介
访问模式控制如何处理对数据源的哀求。如果没有其他说明，Server(default)应该是首选方式。
参考链接：https://docs.victoriametrics.com/#grafana-setup
安装 pushgateway

何时利用pushgateway

pushgateway是一种中心服务，允许从Prometheus无法抓取的job中推送指标。有关具体信息，请参阅推送指标.
建议仅在某些有限的情况下利用Pushgateway。盲目利用pushgateway而不是Prometheus通常的拉取模型来收集一般指标时，会有几个陷阱：
当一个job的多个实例通过instance标签或其它类似的在Pushgateway中区分它们的指标时，后一点尤其重要。即使原始实例被重命名或删除，实例的指标依然保留在pushgateway中。这是因为pushgateway作为指标缓存的生命周期与向其推送指标的进程的生命周期从根本上是分开的。与Prometheus通常的拉取式监控相比：当实例消散（故意或无意）时，其指标将自动消散。利用Pushgateway时，情况并非如此，必须自己手动删除任何过期的指标或自动化此生命周期同步。
通常，Pushgateway的唯一有用利用场景是捕捉服务级别批处理作业的结果。“服务级别”的批处理作业是指与特定机器或作业实例在语义上无关的作业（比方，删除整个服务的多个用户的批处理作业）。此类作业的指标不应包括机器或实例标签，以将特定机器或实例的生命周期与推送的指标解耦。这减轻了Pushgateway中管理过期指标的负担。另请参阅监控批处理作业的最佳实践.
参考链接：
https://prometheus.io/docs/practices/pushing/?#when-to-use-the-pushgateway
安装pushgateway
  1. # wget https://github.com/prometheus/pushgateway/releases/download/v1.9.0/pushgateway-1.9.0.linux-amd64.tar.gz
  2. # tar -C /usr/local/ -xvzf pushgateway-1.9.0.linux-amd64.tar.gz
  3. # ln -s /usr/local/pushgateway-1.9.0.linux-amd64/pushgateway /usr/local/bin/pushgateway
  4. # pushgateway
  5. ts=2024-09-04T17:38:16.325Z caller=main.go:87 level=info msg="starting pushgateway" version="(version=1.9.0, branch=HEAD, revision=d1ca1a6a426126a09a21f745e8ffbaba550f9643)"
  6. ts=2024-09-04T17:38:16.325Z caller=main.go:88 level=info build_context="(go=go1.22.4, platform=linux/amd64, user=root@2167597b1e9c, date=20240608-15:04:08, tags=unknown)"
  7. ts=2024-09-04T17:38:16.328Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9091
  8. ts=2024-09-04T17:38:16.328Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9091
复制代码
配置Prometheus实例

为了访问pushgateway指标，必要精确配置本地运行的Prometheus实例。以下prometheus.yml示例配置文件将告诉Prometheus实例通过localhost:9100从 Node Exporter中抓取数据，以及抓取频率：
  1. global:
  2.   scrape_interval: 15s
  4. scrape_configs:
  5. - job_name: node
  6.   static_configs:
  7.   - targets: ['localhost:9100']
复制代码
修改prometheus配置文件
  1. global:
  2.   scrape_interval:     15s # 默认，每15秒采样一次目标
  4. # 一份采样配置仅包含一个 endpoint 来做采样
  5. # 下面是 Prometheus 本身的endpoint:
  6. scrape_configs:
  7.   # job_name 将被被当作一个标签 `job=<job_name>`添加到该配置的任意时序采样.
  8.   - job_name: 'prometheus'
  10.     # 覆盖全局默认值，从该job每5秒对目标采样一次
  11.     scrape_interval: 5s
  13.     static_configs:
  14.       - targets: ['localhost:9090']
  15.       
  16.   - job_name: 'pushgateway'
  17.     static_configs:
  18.       # 注意：不能写成['http://192.168.88.132:9090']，这样运行会报错
  19.       - targets: ['192.168.88.132:9091']
  20.       
  21. remote_write:
  22.   - url: http://192.168.88.132:8428/api/v1/write
  23.     queue_config:
  24.       max_samples_per_send: 10000
  25.       capacity: 20000
  26.       max_shards: 30
复制代码
重启Prometheus
  1. # system restart prometheus
  2. # system status prometheus
复制代码
创建服务
  1. # vi /etc/systemd/system/pushgateway.service
复制代码
  1. [Unit]
  2. Description=Pushgateway service
  3. After=network.target
  5. [Service]
  6. Type=simple
  7. Restart=always
  8. TimeoutStartSec=30
  9. Restart=on-failure
  10. RestartSec=5s
  11. ExecStart=/usr/local/pushgateway-1.9.0.linux-amd64/pushgateway
  12. ExecStop=/bin/kill $MAINPID
  13. ExecReload=/bin/kill -HUP $MAINPID
  15. [Install]
  16. WantedBy=multi-user.target
复制代码
先手动停止上述前台运行的pushgateway，然后实验以下命令
  1. # systemctl daemon-reload && systemctl enable --now pushgateway
  2. # systemctl status pushgateway
复制代码
Node Exporter安装与配置

留意：只需在必要被监控的机器上安装(本例中为一台redis服务器，IP：192.168.88.131)
  1. # wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
  2. # tar -C /usr/local/ -xvzf node_exporter-1.8.2.linux-amd64.tar.gz
  3. # ln -s /usr/local/node_exporter-1.8.2.linux-amd64/node_exporter  /usr/local/bin/node_exporter
  4. # node_exporter
  5. ts=2024-09-04T16:01:28.241Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.2, branch=HEAD, revision=f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)"
  6. ts=2024-09-04T16:01:28.241Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.5, platform=linux/amd64, user=root@03d440803209, date=20240714-11:53:45, tags=unknown)"
  7. ts=2024-09-04T16:01:28.242Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
  8. ts=2024-09-04T16:01:28.242Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
  9. ts=2024-09-04T16:01:28.242Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
  10. ts=2024-09-04T16:01:28.242Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
  11. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
  12. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=arp
  13. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=bcache
  14. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=bonding
  15. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=btrfs
  16. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=conntrack
  17. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=cpu
  18. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=cpufreq
  19. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=diskstats
  20. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=dmi
  21. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=edac
  22. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=entropy
  23. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=fibrechannel
  24. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=filefd
  25. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=filesystem
  26. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=hwmon
  27. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=infiniband
  28. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=ipvs
  29. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=loadavg
  30. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=mdadm
  31. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=meminfo
  32. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=netclass
  33. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=netdev
  34. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=netstat
  35. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=nfs
  36. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=nfsd
  37. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=nvme
  38. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=os
  39. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=powersupplyclass
  40. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=pressure
  41. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=rapl
  42. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=schedstat
  43. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=selinux
  44. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=sockstat
  45. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=softnet
  46. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=stat
  47. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=tapestats
  48. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=textfile
  49. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=thermal_zone
  50. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=time
  51. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=timex
  52. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=udp_queues
  53. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=uname
  54. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=vmstat
  55. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=watchdog
  56. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=xfs
  57. ts=2024-09-04T16:01:28.243Z caller=node_exporter.go:118 level=info collector=zfs
  58. ts=2024-09-04T16:01:28.244Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
  59. ts=2024-09-04T16:01:28.244Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100
复制代码
通过输出可知，Node Exporter已在运行，而且在端口9100暴露指标
Node Exporter 指标

通过哀求/metrics端点来确认指标是否已成功暴露：
  1. # curl http://localhost:9100/metrics
复制代码
看到类似下输出则表明暴露成功。
  1. # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
  2. # TYPE go_gc_duration_seconds summary
  3. go_gc_duration_seconds{quantile="0"} 0
  4. go_gc_duration_seconds{quantile="0.25"} 0
  5. go_gc_duration_seconds{quantile="0.5"} 0
  6. go_gc_duration_seconds{quantile="0.75"} 0
  7. go_gc_duration_seconds{quantile="1"} 0
  8. go_gc_duration_seconds_sum 0
  9. go_gc_duration_seconds_count 0
  10. ...
复制代码
Node Exporter现在暴露了Prometheus可以抓取的指标，包括输出中更深层级的各种体系指标(以node_为前缀）。要检察这些指标可以实验以下命令：
  1. # curl http://localhost:9100/metrics | grep "node_"
复制代码
配置Prometheus实例

修改prometheus配置文件
  1. global:
  2.   scrape_interval:     15s # 默认，每15秒采样一次目标
  4. # 一份采样配置仅包含一个 endpoint 来做采样
  5. # 下面是 Prometheus 本身的endpoint:
  6. scrape_configs:
  7.   # job_name 将被被当作一个标签 `job=<job_name>`添加到该配置的任意时序采样.
  8.   - job_name: 'prometheus'
  10.     # 覆盖全局默认值，从该job每5秒对目标采样一次
  11.     scrape_interval: 5s
  13.     static_configs:
  14.       - targets: ['localhost:9090']
  15.       
  16.   - job_name: 'pushgateway'
  17.     static_configs:
  18.       # 注意：不能写成['http://192.168.88.131:9091']，这样运行会报错
  19.       - targets: ['192.168.88.131:9091']
  20.       
  21.   - job_name: 'redis_node'
  22.     static_configs:
  23.       - targets: ['192.168.88.131:9100']
  24.       
  25. remote_write:
  26.   - url: http://192.168.88.132:8428/api/v1/write
  27.     queue_config:
  28.       max_samples_per_send: 10000
  29.       capacity: 20000
  30.       max_shards: 30
复制代码
重启Prometheus
通过Prometheus expression browser检察Node Exporter 指标


特定于Node Exporter的指标前缀为Node_，包括Node_cpu_seconds_total和Node_Exporter_build_info等指标。
点击以下链接检察一些示例指标
MetricMeaningrate(node_cpu_seconds_total{mode="system"}[1m])最近一分钟，每秒在体系模式下消耗的CPU时间平均值（以秒为单位）node_filesystem_avail_bytes非root用户可用的文件体系空间（以字节为单位）rate(node_network_receive_bytes_total[1m])最近一分钟，每秒接收的平均网络流量（以字节为单位）参考链接：https://prometheus.io/docs/guides/node-exporter/#monitoring-linux-host-metrics-with-the-node-exporter
验证Grafana能否正常展示数据



免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。



欢迎光临 qidao123.com技术社区-IT企服评测·应用市场 (https://dis.qidao123.com/) Powered by Discuz! X3.4