ElasticStack从入门到醒目

什么是ElasticStack
ElasticStack早期名称为elk
elk代表了三个组件

• ElasticSearch
  负责数据存储和检索。
  
• Logstash
  负责数据的采集，将源数据采集到ElasticSearch进行存储。
  
• Kibana
  负责数据的展示。雷同Granfa
  

由于Logstash是一个重量级产物，安装包超过300MB+，很多同砚只是用于采集日记，于是利用其他采集工具代替，比如flume，fluentd等产物替代。
厥后elastic公司也发现了这个问题，于是开发了一堆beats产物，其中典型代表就是Filebeat，metricbeat，heartbeat等。
而后，对于安全而言，又推出了xpack等相关组件，以及云环境的组件。
后期名称命名为elk stack(elk 技术栈)，厥后公司为了宣传ElasticStack
ElasticStack 架构
ElasticStack版本
https://www.elastic.co/   elastic官网
最新版本8+，8版本默认启用了https协议，我们先安装7.17版本，然后手动启动https协议。
后面再练习安装8版本
选择elastic安装方式，我们再Ubuntu上摆设elastic
二进制包摆设单机es环境
摆设

1. 1.下载elk安装包
2. root@elk:~# cat install_elk.sh
3. #!/bin/bash
4. wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-linux-x86_64.tar.gz
5. wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-linux-x86_64.tar.gz.sha512
6. shasum -a 512 -c elasticsearch-7.17.28-linux-x86_64.tar.gz.sha512
7. tar -xzf elasticsearch-7.17.28-linux-x86_64.tar.gz -C /usr/local
8. cd elasticsearch-7.17.28/
11. 2.修改配置文件
12. root@elk:~# vim /usr/local/elasticsearch-7.17.28/config/elasticsearch.yml
13. root@elk:~# egrep -v "^#|^$" /usr/local/elasticsearch-7.17.28/config/elasticsearch.yml
14. cluster.name: xu-elasticstack
15. path.data: /var/lib/es7
16. path.logs: /var/log/es7
17. network.host: 0.0.0.0
18. discovery.type: single-node
19. 相关参数说明:
20.         port
21.                 默认端口是9200
22.                 # By default Elasticsearch listens for HTTP traffic on the first free port it
23.                 # finds starting at 9200. Set a specific HTTP port here:
24.                 #
25.                 #http.port: 9200
26.                
27.         cluster.name
28.                 集群的名称
29.                
30.         path.data
31.                 ES的数据存储路径。
32.                
33.         path.logs
34.                 ES的日志存储路径。
35.                
36.         network.host
37.                 # elasticStack默认只允许本机访问
38.                 # By default Elasticsearch is only accessible on localhost. Set a different
39.                 # address here to expose this node on the network:
40.                 #
41.                 #network.host: 192.168.0.1
43.                 ES服务监听的地址。
44.                
45.         discovery.type
46.                 # 如果部署的es集群就需要配置discovery.seed_hosts和cluster.initial_master_nodes参数
47.                 # Pass an initial list of hosts to perform discovery when this node is started:
48.                 # The default list of hosts is ["127.0.0.1", "[::1]"]
49.                 #
50.                 #discovery.seed_hosts: ["host1", "host2"]
51.                 #
52.                 # Bootstrap the cluster using an initial set of master-eligible nodes:
53.                 #
54.                 #cluster.initial_master_nodes: ["node-1", "node-2"]               
55.                 指的ES集群的部署类型，此处的"single-node"，表示的是一个单点环境。
56.                
57. 3.如果此时直接启动elastic会报错
58. 3.1测试报错，官方给出的启动命令
59. Elasticsearch can be started from the command line as follows:
60. ./bin/elasticsearch
61. root@elk:~# /usr/local/elasticsearch-7.17.28/bin/elasticsearch
62. # 这些是java类型报错
63. Mar 17, 2025 7:44:51 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
64. WARNING: COMPAT locale provider will be removed in a future release
65. [2025-03-17T07:44:53,125][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elk] uncaught exception in thread [main]
66. org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
67.         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.28.jar:7.17.28]
68.         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.28.jar:7.17.28]
69.         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.28.jar:7.17.28]
70.         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.28.jar:7.17.28]
71.         at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.28.jar:7.17.28]
72.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.28.jar:7.17.28]
73.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.28.jar:7.17.28]
74. Caused by: java.lang.RuntimeException: can not run elasticsearch as root
75.         at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:107) ~[elasticsearch-7.17.28.jar:7.17.28]
76.         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183) ~[elasticsearch-7.17.28.jar:7.17.28]
77.         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.28.jar:7.17.28]
78.         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.28.jar:7.17.28]
79.         ... 6 more
80. uncaught exception in thread [main]
81. java.lang.RuntimeException: can not run elasticsearch as root  # 不允许root直接启动
82.         at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:107)
83.         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183)
84.         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
85.         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
86.         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
87.         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
88.         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
89.         at org.elasticsearch.cli.Command.main(Command.java:77)
90.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
91.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
92. For complete error details, refer to the log at /var/log/es7/xu-elasticstack.log
93. 2025-03-17 07:44:53,713764 UTC [1860] INFO  Main.cc@111 Parent process died - ML controller exiting
95. 3.2创建启动用户
96. root@elk:~# useradd -m  elastic
97. root@elk:~# id elastic
98. uid=1001(elastic) gid=1001(elastic) groups=1001(elastic)
99. # 通过elastic用户启动，此时还有一个报错
100. root@elk:~# su - elastic  -c  "/usr/local/elasticsearch-7.17.28/bin/elasticsearch"
101. could not find java in bundled JDK at /usr/local/elasticsearch-7.17.28/jdk/bin/java
102. # 系统中是存在java包的，但是elastic用户找不到，切换到elastic查看下
103. root@elk:~# ll /usr/local/elasticsearch-7.17.28/jdk/bin/java
104. -rwxr-xr-x 1 root root 12328 Feb 20 09:09 /usr/local/elasticsearch-7.17.28/jdk/bin/java*
105. root@elk:~# su - elastic
106. $ pwd     
107. /home/elastic
108. $ ls /usr/local/elasticsearch-7.17.28/jdk/bin/java
109. # 报错的原因就是权限被拒绝，也就是elastic没有权限访问java包
110. ls: cannot access '/usr/local/elasticsearch-7.17.28/jdk/bin/java': Permission denied
111. # 一层一层地向外找，最后能找到/usr/local/elasticsearch-7.17.28/jdk/bin目录没有权限，导致报错
112. root@elk:~#  chown elastic:elastic  -R /usr/local/elasticsearch-7.17.28/
113. root@elk:~# ll -d /usr/local/elasticsearch-7.17.28/jdk/bin/
114. drwxr-x--- 2 elastic elastic 4096 Feb 20 09:09 /usr/local/elasticsearch-7.17.28/jdk/bin//
116. # 此时再进行启动测试，发现其他错误
117. # 我们指定地path.data和path.log不存在，我们需要手动创建
118. java.lang.IllegalStateException: Unable to access 'path.data' (/var/lib/es7)
119. org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/var/lib/es7)
120. root@elk:~# install -d /var/{log,lib}/es7 -o elastic -g elastic
121. root@elk:~# ll -d /var/{log,lib}/es7
122. drwxr-xr-x 2 elastic elastic 4096 Mar 17 08:01 /var/lib/es7/
123. drwxr-xr-x 2 elastic elastic 4096 Mar 17 07:44 /var/log/es7/
125. # 现在重新启动服务，可以启动成功，检测端口
126. root@elk:~# su - elastic  -c  "/usr/local/elasticsearch-7.17.28/bin/elasticsearch"
127. root@elk:~# netstat -tunlp | egrep "9[2|3]00"
128. tcp6       0      0 :::9200                 :::*                    LISTEN      2544/java           
129. tcp6       0      0 :::9300                 :::*                    LISTEN      2544/java   

复制代码

通过浏览器访问9200
同时elastic提供了一个api，可以查看当前的主机数

1. [root@zabbix ~]# curl 192.168.121.21:9200/_cat/nodes
2. 172.16.1.21 40 97 0 0.11 0.29 0.20 cdfhilmrstw * elk
3. # 在命令行访问，由于目前的单节点部署es，所以node只有一个
5. # 前面我们启动es是前台启动前台启动会存在两个问题
6. 1.占用终端
7. 2.如果想结束es比较困难，所以这里一般我们采用后台运行的方式启动
8. 官方给我们的后台运行方式
9. elasticsearch 的-d参数
10. To run Elasticsearch as a daemon, specify -d on the command line, and record the process ID in a file using the -p option:
11. ./bin/elasticsearch -d -p pid
12. root@elk:~# su - elastic -c '/usr/local/elasticsearch-7.17.28/bin/elasticsearch -d'
14. # 常见报错问题
15. Q1：最大虚拟内存映射太小
16. bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
17. ERROR: Elasticsearch did not exit normally - check the logs at /var/log/es7/AAA.log
18. root@elk:~# sysctl -q vm.max_map_count
19. vm.max_map_count = 65530
20. root@elk:~# echo "vm.max_map_count = 262144" >> /etc/sysctl.d/es.conf
21. root@elk:~# sysctl -w vm.max_map_count=262144
22. vm.max_map_count = 262144
23. root@elk:~# sysctl -q vm.max_map_count
24. vm.max_map_count = 262144
27. Q2：es配置文件写错
28. java.net.UnknownHostException: single-node
31. Q3：出现lock字样说明已经有ES实例启动。杀死现有进程后再重新执行启动命令
32. java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
34. Q5：ES集群部署的有问题，缺少master角色。
35. {"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

复制代码

卸载环境

1. 1.停止elasticsearch
2. root@elk:~# kill `ps -ef | grep java | grep -v grep |awk '{print $2}'`
3. root@elk:~# ps -ef | grep java
4. root        4437    1435  0 09:21 pts/2    00:00:00 grep --color=auto java
6. 2.删除数据目录、日志目录、安装包、用户
7. root@elk:~# rm -rf /usr/local/elasticsearch-7.17.28/ /var/{lib,log}/es7/
8. root@elk:~# userdel -r elastic

复制代码

基于deb包安装ES单点

1. 1.安装deb包
2. root@elk:~# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-amd64.deb
4. 2.安装es
5. root@elk:~# dpkg -i elasticsearch-7.17.28-amd64.deb
6. # 通过二进制包安装es可以使用systemctl管理
7. ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
8. sudo systemctl daemon-reload
9. sudo systemctl enable elasticsearch.service
10. ### You can start elasticsearch service by executing
11. sudo systemctl start elasticsearch.service
12. Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
15. 3.修改es配置文件
16. root@elk:~# vim /etc/elasticsearch/elasticsearch.yml
17. root@elk:~# egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
18. cluster.name: xu-es
19. path.data: /var/lib/elasticsearch
20. path.logs: /var/log/elasticsearch
21. network.host: 0.0.0.0
22. discovery.type: single-node
24. 4.启动es
25. systemctl enable elasticsearch --now
26. # 查看es的service文件，下面的参数都是在二进制安装的时候我们自己做的
27. User=elasticsearch
28. Group=elasticsearch
29. ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quie
31. cat /usr/share/elasticsearch/bin/systemd-entrypoint
32. #!/bin/sh
34. # This wrapper script allows SystemD to feed a file containing a passphrase into
35. # the main Elasticsearch startup script
37. if [ -n "$ES_KEYSTORE_PASSPHRASE_FILE" ] ; then
38.   exec /usr/share/elasticsearch/bin/elasticsearch "$@" < "$ES_KEYSTORE_PASSPHRASE_FILE"
39. else
40.   exec /usr/share/elasticsearch/bin/elasticsearch "$@"
41. fi

复制代码

es常见术语

1. 1.索引 Index
2.         用户进行数据读写的单元
3. 2.分片 Shared
4.         一个索引至少要有一个分片，如果一个索引仅有一个分片，意味着该索引的数据只能全量存储在某个节点上，且分片是不可拆分的，隶属于某个节点。
5.         换句话说，分片是ES集群最小的调度单元。
6.         一个索引数据也可以被分散的存储在不同的分片上，且这些分片可以放在不同的节点，从而实现数据的分布式存储。
7. 3.副本 replica
8.         副本是针对分片来说的，一个分片可以有0个或多个副本。
9.         当副本数量为0时，意味着只有主分片(priamry shard)，当主分片所在的节点宕机时，数据就无法访问了。
10.         当副本数量大于0时，意味着同时存在主分片和副本分片(replica shard):
11.                 主分片负责数据的读写(read write,rw)
12.                 副本分片负责数据的读的负载均衡(read only,ro)
13. 4.文档 document
14.         指的是用户存储的数据。其中包含元数据和源数据。
15.         元数据：
16.                 用于描述源数据的数据。
17.         源数据:  
18.                 用户实际存储的数据。
19. 5.分配： allocation
20.         指的是将索引的不同分片(包含主分片和副本分片)分配到整个集群的过程。

复制代码

查看集群状态

1. # es提供了api /_cat/health
2. root@elk:~# curl 127.1:9200/_cat/health
3. 1742210504 11:21:44 xu-es green 1 1 3 3 0 0 0 0 - 100.0%
4. root@elk:~# curl 127.1:9200/_cat/health?v
5. epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
6. 1742210512 11:21:52  xu-es   green           1         1      3   3    0    0        0             0                  -                100.0%

复制代码

es集群环境摆设

1. 1.安装es集群服务
2. root@elk1:~# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-amd64.deb
3. root@elk1:~# dpkg -i elasticsearch-7.17.28-amd64.deb
4. root@elk2:~# dpkg -i elasticsearch-7.17.28-amd64.deb
5. root@elk3:~# dpkg -i elasticsearch-7.17.28-amd64.deb
8. 2.配置es，三台机器一样的配置
9. # 不需要配置discovery.type了
10. [root@elk1 ~]# grep -E "^(cluster|path|network|discovery|http)" /etc/elasticsearch/elasticsearch.yml
11. cluster.name: es-cluster
12. path.data: /var/lib/elasticsearch
13. path.logs: /var/log/elasticsearch
14. network.host: 0.0.0.0
15. http.port: 9200
16. discovery.seed_hosts: ["192.168.121.91", "192.168.121.92", "192.168.121.93"]
19. 3.启动服务
20. systemctl enable elasticsearch --now
22. 4.测试，带有*的是master节点
23. root@elk:~# curl 127.1:9200/_cat/nodes
24. 172.16.1.23  6 97 25 0.63 0.57 0.25 cdfhilmrstw - elk3
25. 172.16.1.22  5 96 23 0.91 0.76 0.33 cdfhilmrstw - elk2
26. 172.16.1.21 19 90 39 1.22 0.87 0.35 cdfhilmrstw * elk
27. root@elk:~# curl 127.1:9200/_cat/nodes?v
28. ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
29. 172.16.1.23            9          83   2    0.12    0.21     0.18 cdfhilmrstw -      elk3
30. 172.16.1.22            8          96   3    0.16    0.28     0.24 cdfhilmrstw -      elk2
31. 172.16.1.21           22          97   3    0.09    0.30     0.25 cdfhilmrstw *      elk
33. # 集群部署故障 没有uuid  集群缺少master
35. [root@elk3 ~]# curl http://192.168.121.92:9200/_cat/nodes?v
36. {"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
37. [root@elk3 ~]# curl 192.168.121.91:9200
38. {
39.   "name" : "elk91",
40.   "cluster_name" : "es-cluster",
41.   "cluster_uuid" : "_na_",
42.   ...
43. }
44. [root@elk3 ~]#
45. [root@elk3 ~]# curl 10.0.0.92:9200
46. {
47.   "name" : "elk92",
48.   "cluster_name" : "es-cluster",
49.   "cluster_uuid" : "_na_",
50.   ...
51. }
52. [root@elk3 ~]#
53. [root@elk3 ~]#
54. [root@elk3 ~]# curl 10.0.0.93:9200
55. {
56.   "name" : "elk93",
57.   "cluster_name" : "es-cluster",
58.   "cluster_uuid" : "_na_",
59.   ...
60. }
61. [root@elk3 ~]#
62. [root@elk3 ~]# curl http://192.168.121.91:9200/_cat/nodes
63. {"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
66. # 解决方式
67. 1.停止集群的ES服务
68. [root@elk91 ~]# systemctl stop elasticsearch.service
69. [root@elk92 ~]# systemctl stop elasticsearch.service
70. [root@elk93 ~]# systemctl stop elasticsearch.service
73. 2.删除数据，日志，和临时数据
74. [root@elk91 ~]# rm -rf /var/{lib,log}/elasticsearch/* /tmp/*
75. [root@elk92 ~]# rm -rf /var/{lib,log}/elasticsearch/* /tmp/*
76. [root@elk93 ~]# rm -rf /var/{lib,log}/elasticsearch/* /tmp/*
78. 3.添加配置项
79. [root@elk1 ~]# grep -E "^(cluster|path|network|discovery|http)" /etc/elasticsearch/elasticsearch.yml
80. cluster.name: es-cluster
81. path.data: /var/lib/elasticsearch
82. path.logs: /var/log/elasticsearch
83. network.host: 0.0.0.0
84. http.port: 9200
85. discovery.seed_hosts: ["192.168.121.91", "192.168.121.92", "192.168.121.93"]
86. cluster.initial_master_nodes: ["192.168.121.91", "192.168.121.92", "192.168.121.93"]  ######
88. 4.重启服务
90. 5.测试
94. es集群master选举流程
95. 1.启动时会检查集群是否有master，如果有则不发起选举master;
96. 1.刚开始启动，所有节点均为人自己是master，并向集群的其他节点发送信息（包含ClusterStateVersion，ID等）
97. 2.基于类似gossip协议获取所有可以参与master选举的节点列表;
98. 3.先比较"ClusterStateVersion"，谁最大，谁优先级高，会被选举出master;
99. 4.如果比不出来，则比较ID，谁的ID小，就优先成为master;
100. 5.当集群半数以上节点参与选举完成后，则完成master选举，比如有N个节点，仅需要"(N/2)+1"节点就可以确认master;
101. 6.master选举完成后，会向集群列表通报最新的master节点，此时才意味着选举完成;

复制代码

DSL

• ES相比与MySQL
  
  
  
  • MySQL属于关系型数据库:
    增删改查： 基于SQL
    
  • ES属于文档型数据库，和MangoDB很相似
    增删改查： DSL语句，属于ES独有的有种查询语言。
    针对模糊查询，mysql无法充实利用索引，性能较低，而是用ES查询模糊数据，是非常高效的。
    
  
  

往es中添加单条请求
利用postman进行测试

1. # 本质上是使用了curl
2. curl --location 'http://192.168.121.21:9200/test_linux/doc' \
3. --header 'Content-Type: application/json' \
4. --data '{
5.     "name": "孙悟空",
6.     "hobby": [
7.         "蟠桃",
8.         "紫霞仙子"
9.     ]
10. }
14. curl --location '192.168.121.21:9200/_bulk' \
15. --header 'Content-Type: application/json' \
16. --data '{ "create" : { "_index" : "test_linux_ss", "_id" : "1001" } }
17. { "name" : "猪八戒","hobby": ["猴哥","高老庄"] }
20. {"create": {"_index":"test_linux_ss","_id":"1002"}}
21. {"name":"白龙马","hobby":["驮唐僧","吃草"]}
22. '

复制代码

查询数据

1. curl --location '192.168.121.22:9200/test_linux_ss/_doc/1001' \
2. --data ''
6. curl --location --request GET '192.168.121.22:9200/test_linux_ss/_search' \
7. --header 'Content-Type: application/json' \
8. --data '{
9.     "query":{
10.         "match":{
11.             "name":"猪八戒"
12.         }
13.     }
14. }'

复制代码

删除数据

1. curl --location --request DELETE '192.168.121.22:9200/test_linux_ss/_doc/1001'

复制代码

kibana
摆设kibana
kibana是针对ES做的一款可视化工具。未来的操作都可以在ES中完成。

1. 1.下载kibana
2. root@elk:~# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.28-amd64.deb
4. 2.安装kibana
5. root@elk:~# dpkg -i kibana-7.17.28-amd64.deb
7. 3.修改配置文件
8. root@elk:~# vim /etc/kibana/kibana.yml
9. root@elk:~# grep -E "^(elasticsearch.host|i18n|server)" /etc/kibana/kibana.yml
10. server.port: 5601
11. server.host: "0.0.0.0"
12. elasticsearch.hosts: ["http://192.168.121.21:9200","http://192.168.121.22:9200","http://192.168.121.23:9200"]
13. i18n.locale: "zh-CN"
15. 4.启动kibana
16. root@elk:~# systemctl enable kibana.service --now
17. Synchronizing state of kibana.service with SysV service script with /lib/systemd/systemd-sysv-install.
18. Executing: /lib/systemd/systemd-sysv-install enable kibana
19. Created symlink /etc/systemd/system/multi-user.target.wants/kibana.service → /etc/systemd/system/kibana.service.
20. root@elk:~# netstat  -tunlp | grep 5601
21. tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      19392/node   

复制代码

web端访问测试
基于KQL基本利用
过滤数据
Filebeat
摆设Filebeat

1. 1.下载Filebeat
2. root@elk2:~# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.28-amd64.deb
4. 2.安装Filebeat
5. root@elk2:~# dpkg -i filebeat-7.17.28-amd64.deb
7. 3.编写Filebeat配置文件
8. # Filebeat需要我们自己创建config目录，然后编写配置文件
9. mkdir /etc/filebeat/config
10. vim /etc/filebeat/config/01-log-to-console.yaml
11. # Filebeat配置文件包含两个部分，Input和Output，Input指明要从哪里采集数据；Output指明数据存储到哪里，根据官方文档进行配置
12. # 目前还没有服务的运行日志，所以Input来源指定为特殊文件，Output去向输出的终端console
14. root@elk2:~# cat /etc/filebeat/config/01-log-to-console.yaml
15. # 定义数据从哪里来
16. filebeat.inputs:
17.   # 指定数据源的类型是log，表示从文件读取数据
18. - type: log
19.   # 指定文件的路径
20.   paths:
21.     - /tmp/student.log
23. # 定义数据到终端
24. output.console:
25.   pretty: true
27. 4.运行filebeat实例
28. filebeat -e -c /etc/filebeat/config/01-log-to-console.yaml
30. 5.创建student.log文件，并写入数据
31. root@elk2:~# echo ABC > /tmp/student.log
33. // 输出提示
34. {
35.   "@timestamp": "2025-03-18T14:48:42.432Z",
36.   "@metadata": {
37.     "beat": "filebeat",
38.     "type": "_doc",
39.     "version": "7.17.28"
40.   },
41.   "message": "ABC",   // 检测到的内容变化
42.   "input": {
43.     "type": "log"
44.   },
45.   "ecs": {
46.     "version": "1.12.0"
47.   },
48.   "host": {
49.     "name": "elk2"
50.   },
51.   "agent": {
52.     "type": "filebeat",
53.     "version": "7.17.28",
54.     "hostname": "elk2",
55.     "ephemeral_id": "7f116862-382c-48f4-8797-c4b689e6e6fe",
56.     "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf",
57.     "name": "elk2"
58.   },
59.   "log": {
60.     "offset": 0,   // 偏移量offset=0 表示从0开始
61.     "file": {
62.       "path": "/tmp/student.log"
63.     }
64.   }
65. }
68. # 向文件student.log中追加写入数据
69. root@elk2:~# echo 123 >> /tmp/student.log
71. // 查看filebeat输出提示
72. {
73.   "@timestamp": "2025-03-18T14:51:17.449Z",
74.   "@metadata": {
75.     "beat": "filebeat",
76.     "type": "_doc",
77.     "version": "7.17.28"
78.   },
79.   "log": {
80.     "offset": 4,  // 偏移量从4开始的
81.     "file": {
82.       "path": "/tmp/student.log"
83.     }
84.   },
85.   "message": "123",   // 统计到了数据 123
86.   "input": {
87.     "type": "log"
88.   },
89.   "ecs": {
90.     "version": "1.12.0"
91.   },
92.   "host": {
93.     "name": "elk2"
94.   },
95.   "agent": {
96.     "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf",
97.     "name": "elk2",
98.     "type": "filebeat",
99.     "version": "7.17.28",
100.     "hostname": "elk2",
101.     "ephemeral_id": "7f116862-382c-48f4-8797-c4b689e6e6fe"
102.   }
103. }

复制代码

Filebeat特性

1. # 使用echo的静默输出进行追加 # 此时filebeat采集不到数据
2. root@elk2:~# echo -n 456 >> /tmp/student.log
3. root@elk2:~# cat /tmp/student.log
4. ABC
5. 123
6. 456root@elk2:~#
7. root@elk2:~# echo -n abc  >> /tmp/student.log
8. root@elk2:~# cat /tmp/student.log
9. ABC
10. 123
11. 456789abcroot@elk2:~#
13. # 使用非静默写入数据  此时filebeat就可以采集到数据了
14. root@elk2:~# echo haha >> /tmp/student.log
15. root@elk2:~# cat /tmp/student.log
16. ABC
17. 123
18. 456789abchaha
21. // 查看filebeat输出信息
22. {
23.   "@timestamp": "2025-03-18T14:55:37.476Z",
24.   "@metadata": {
25.     "beat": "filebeat",
26.     "type": "_doc",
27.     "version": "7.17.28"
28.   },
29.   "host": {
30.     "name": "elk2"
31.   },
32.   "agent": {
33.     "name": "elk2",
34.     "type": "filebeat",
35.     "version": "7.17.28",
36.     "hostname": "elk2",
37.     "ephemeral_id": "7f116862-382c-48f4-8797-c4b689e6e6fe",
38.     "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf"
39.   },
40.   "log": {
41.     "offset": 8,  // 偏移量=8
42.     "file": {
43.       "path": "/tmp/student.log"
44.     }
45.   },
46.   "message": "456789abchaha",  // 采集到的数据是静默输出+非静默输出 的所有数据
47.   "input": {
48.     "type": "log"
49.   },
50.   "ecs": {
51.     "version": "1.12.0"
52.   }
53. }

复制代码

由此可以得出Filebeat的第一条性质：
filebeat默认是按行采集数据;

1. # 现在将Filebeat停止，修改student.log，如果重新开始进行数据采集，那么Filebeat是采集文件中所有的内容，还是只采集Filebeat停止后新增的内容
2. root@elk2:~# echo xixi >> /tmp/student.log
4. # 重启Filebeat
6. // 查看Filebeat的输出信息
7. {
8.   "@timestamp": "2025-03-18T15:00:51.759Z",
9.   "@metadata": {
10.     "beat": "filebeat",
11.     "type": "_doc",
12.     "version": "7.17.28"
13.   },
14.   "ecs": {
15.     "version": "1.12.0"
16.   },
17.   "host": {
18.     "name": "elk2"
19.   },
20.   "agent": {
21.     "type": "filebeat",
22.     "version": "7.17.28",
23.     "hostname": "elk2",
24.     "ephemeral_id": "81db6575-7f98-4ca4-a86f-4d0127c1e2a4",
25.     "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf",
26.     "name": "elk2"
27.   },
28.   "log": {
29.     "offset": 22,  // 偏移量也不是从0开始
30.     "file": {
31.       "path": "/tmp/student.log"
32.     }
33.   },
34.   "message": "xixi",  // 只采集Filebeat停止后新增的内容
35.   "input": {
36.     "type": "log"
37.   }
38. }
41. // 在上面有一条提示，当再次启动filebeat时回导入/var/lib/filebeat/registry/filebeat目录下的json文件
42. 2025-03-18T15:00:51.756Z        INFO        memlog/store.go:124        Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=5
43. // 在我们第一次启动时也会从/var/lib/filebeat/registry/filebeat目录下导入，但是第一次启动时不会存在该目录，自然也没有信息
45. // 查看/var/lib/filebeat/registry/filebeat下的json文件，在这个json文件中记录了offset值，这就是filebeat能够在停止后，重启不会从头开始记录文件内容
46. {"op":"set","id":1}
47. {"k":"filebeat::logs::native::1441831-64768","v":{"id":"native::1441831-64768","prev_id":"","timestamp":[431172511,1742309322],"ttl":-1,"identifier_name":"native","source":"/tmp/student.log","offset":0,"type":"log","FileStateOS":{"inode":1441831,"device":64768}}}
48. {"op":"set","id":2}
49. {"k":"filebeat::logs::native::1441831-64768","v":{"prev_id":"","source":"/tmp/student.log","type":"log","FileStateOS":{"inode":1441831,"device":64768},"id":"native::1441831-64768","offset":4,"timestamp":[434614328,1742309323],"ttl":-1,"identifier_name":"native"}}
50. {"op":"set","id":3}
51. {"k":"filebeat::logs::native::1441831-64768","v":{"id":"native::1441831-64768","identifier_name":"native","ttl":-1,"type":"log","FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","source":"/tmp/student.log","offset":8,"timestamp":[450912955,1742309478]}}
52. {"op":"set","id":4}
53. {"k":"filebeat::logs::native::1441831-64768","v":{"type":"log","identifier_name":"native","offset":22,"timestamp":[478003874,1742309738],"source":"/tmp/student.log","ttl":-1,"FileStateOS":{"inode":1441831,"device":64768},"id":"native::1441831-64768","prev_id":""}}
54. {"op":"set","id":5}
55. {"k":"filebeat::logs::native::1441831-64768","v":{"id":"native::1441831-64768","ttl":-1,"FileStateOS":{"device":64768,"inode":1441831},"identifier_name":"native","prev_id":"","source":"/tmp/student.log","offset":22,"timestamp":[478003874,1742309738],"type":"log"}}
56. {"op":"set","id":6}
57. {"k":"filebeat::logs::native::1441831-64768","v":{"offset":22,"timestamp":[759162512,1742310051],"type":"log","FileStateOS":{"device":64768,"inode":1441831},"id":"native::1441831-64768","prev_id":"","identifier_name":"native","source":"/tmp/student.log","ttl":-1}}
58. {"op":"set","id":7}
59. {"k":"filebeat::logs::native::1441831-64768","v":{"offset":22,"timestamp":[759368397,1742310051],"type":"log","FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","source":"/tmp/student.log","ttl":-1,"identifier_name":"native","id":"native::1441831-64768"}}
60. {"op":"set","id":8}
61. {"k":"filebeat::logs::native::1441831-64768","v":{"ttl":-1,"identifier_name":"native","id":"native::1441831-64768","source":"/tmp/student.log","timestamp":[761513338,1742310052],"FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","offset":27,"type":"log"}}
62. {"op":"set","id":9}
63. {"k":"filebeat::logs::native::1441831-64768","v":{"source":"/tmp/student.log","timestamp":[795028411,1742310356],"FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","offset":27,"ttl":-1,"type":"log","identifier_name":"native","id":"native::1441831-64768"}}

复制代码

这是Filebeat的第二个特性
filebeat默认会在"/var/lib/filebeat"目录下记录已经采集的文件offset信息，以便于下一次采集接着该位置继续采集数据;
Filebeat写入es

1. 将Filebeat的Output写入到es中
2. 根据官方文档查看配置
3. The Elasticsearch output sends events directly to Elasticsearch using the Elasticsearch HTTP API.
5. Example configuration:
7. output.elasticsearch:  
8.   hosts: ["https://myEShost:9200"]
10. root@elk2:~# cat /etc/filebeat/config/02-log-to-es.yaml
11. # 定义数据从哪里来
12. filebeat.inputs:
13.   # 指定数据源的类型是log，表示从文件读取数据
14. - type: log
15.   # 指定文件的路径
16.   paths:
17.     - /tmp/student.log
19. # 定义数据到终端
20. output.elasticsearch:  
21.   hosts:
22.     - 192.168.121.21:9200
23.     - 192.168.121.22:9200
24.     - 192.168.121.23:9200
25. # 删除filebeat的json文件
26. root@elk2:~# rm -rf /var/lib/filebeat
28. # 启动filebeat实例
29. root@elk2:~# filebeat -e -c  /etc/filebeat/config/02-log-to-es.yaml

复制代码

在kibana中收集到数据了
查看收集到的数据
设置刷新频率
自定义索引

1. # 索引名称我们可以自己定义，官方给出了定义方式 index参数设置
2. output.elasticsearch:
3.   hosts: ["http://localhost:9200"]
4.   index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"
5.   
6. root@elk2:~# cat /etc/filebeat/config/03-log-to-constom.yaml
7. # 定义数据从哪里来
8. filebeat.inputs:
9.   # 指定数据源的类型是log，表示从文件读取数据
10. - type: log
11.   # 指定文件的路径
12.   paths:
13.     - /tmp/student.log
15. # 定义数据到终端
16. output.elasticsearch:  
17.   hosts:
18.     - 192.168.121.21:9200
19.     - 192.168.121.22:9200
20.     - 192.168.121.23:9200
21.   # 自定义索引名称
22.   index: "test_filebeat-%{+yyyy.MM.dd}"
24. # 启动filebeat，此时会报错
25. root@elk2:~# filebeat -e -c /etc/filebeat/config/03-log-to-constom.yaml
26. 2025-03-19T02:55:18.951Z        INFO        instance/beat.go:698        Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
27. 2025-03-19T02:55:18.958Z        INFO        instance/beat.go:706        Beat ID: a109c2d1-fbb6-4b82-9416-29f9488ccabc
28. # 你必须要设置setup.template.name and setup.template.patter这两个参数，也就是说如果我们想自定义index名称，必须设置setup.template.name and setup.template.patter
29. 2025-03-19T02:55:18.958Z        ERROR        instance/beat.go:1027        Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified
30. Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified
32. # setup.template.name and setup.template.patter在官网中给出了提示、
33. If you change this setting, you also need to configure the setup.template.name and setup.template.pattern options (see Elasticsearch index template).
35. # 官方给出的实例
36. setup.template.name
37.         The name of the template. The default is filebeat. The Filebeat version is always appended to the given name, so the final name is filebeat-%{[agent.version]}.
38.        
39. setup.template.pattern
40.         The template pattern to apply to the default index settings. The default pattern is filebeat. The Filebeat version is always included in the pattern, so the final pattern is filebeat-%{[agent.version]}.
42. Example:
44. setup.template.name: "filebeat"
45. setup.template.pattern: "filebeat"
47. # 还需要设置shared和replicas 官方给出的默认设置
48. setup.template.settings:
49.   index.number_of_shards: 1
50.   index.number_of_replicas: 1
52. # 配置我们自己的索引模板(就是创建索引的规则)
53. root@elk2:~# cat /etc/filebeat/config/03-log-to-constom.yaml
54. # 定义数据从哪里来
55. filebeat.inputs:
56.   # 指定数据源的类型是log，表示从文件读取数据
57. - type: log
58.   # 指定文件的路径
59.   paths:
60.     - /tmp/student.log
62. # 定义数据到终端
63. output.elasticsearch:  
64.   hosts:
65.     - 192.168.121.21:9200
66.     - 192.168.121.22:9200
67.     - 192.168.121.23:9200
68.   # 自定义索引名称
69.   index: "test_filebeat-%{+yyyy.MM.dd}"
71. # 定义索引模板(就是创建索引的规则)的名称
72. setup.template.name: "test_filebeat"
73. # 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
74. setup.template.pattern: "test_filebeat-*"
75. # # 定义索引模板的规则信息
76. setup.template.settings:
77.   # 分片数
78.   index.number_of_shards: 3
79.   # 每个分片有多少个副本
80.   index.number_of_replicas: 0
83. # 启动filebeat 此时可以正常启动filebeat，但是在kibana中发现没有建立我们设置index，查看启动信息
84. root@elk2:~# filebeat  -e -c /etc/filebeat/config/03-log-to-constom.yaml
85. # 这里的大概意思就是ILM设置为了auto  如果启用了此配置，则忽略自定义索引的所有信息  所以我们要将ILM设置为false
86. 2025-03-19T03:10:02.548Z        INFO        [index-management]        idxmgmt/std.go:260        Auto ILM enable success.
87. 2025-03-19T03:10:02.558Z        INFO        [index-management.ilm]        ilm/std.go:170        ILM policy filebeat exists already.
88. 2025-03-19T03:10:02.559Z        INFO        [index-management]        idxmgmt/std.go:396        Set setup.template.name to '{filebeat-7.17.28 {now/d}-000001}' as ILM is enabled.
90. # 查看官网对于index lifecycle management  ILM配置
91. When index lifecycle management (ILM) is enabled, the default index is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index_num}", for example, "filebeat-8.17.3-2025-03-17-000001". Custom index settings are ignored when ILM is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Index lifecycle management (ILM) to learn how to change the index name.
93. # ilm默认是auto模式，支持true、false、auto
94. Enables or disables index lifecycle management on any new indices created by Filebeat. Valid values are true, false, and auto. When auto (the default) is specified on version 7.0 and later
95. setup.ilm.enabled: auto
97. # 添加ilm配置到我们自己的配置文件中
99. # 启动filebeat
100. root@elk2:~# filebeat  -e -c /etc/filebeat/config/03-log-to-constom.yaml

复制代码

索引模板已被创建

1. # 此时我想修改我的shared和replicas
2. # 直接修改配置文件 改为5shared 0repicas

复制代码

此时还是3shared 0replicas

1. # 这是由于 setup.template.overwrite 参数 默认是false也就是不覆盖
2. setup.template.overwrite
3. A boolean that specifies whether to overwrite the existing template. The default is false. Do not enable this option if you start more than one instance of Filebeat at the same time. It can overload Elasticsearch by sending too many template update requests.
6. # 设置setup.template.overwrite 为 true
7. root@elk2:~# cat /etc/filebeat/config/03-log-to-constom.yaml
8. # 定义数据从哪里来
9. filebeat.inputs:
10.   # 指定数据源的类型是log，表示从文件读取数据
11. - type: log
12.   # 指定文件的路径
13.   paths:
14.     - /tmp/student.log
16. # 定义数据到终端
17. output.elasticsearch:  
18.   hosts:
19.     - 192.168.121.21:9200
20.     - 192.168.121.22:9200
21.     - 192.168.121.23:9200
22.   # 自定义索引名称
23.   index: "test_filebeat-%{+yyyy.MM.dd}"
25. # 禁用索引生命周期管理(index lifecycle management，ILM)
26. # 如果启用了此配置，则忽略自定义索引的所有信息
27. setup.ilm.enabled: false
28. # 如果索引模板存在，是否覆盖，默认值为false，如果明确需要，则可以将其设置为ture。
29. # 但是官方建议将其设置为false，原因是每次写数据时，都会建立tcp链接，消耗资源。
30. setup.template.overwrite: true
31. # 定义索引模板(就是创建索引的规则)的名称
32. setup.template.name: "test_filebeat"
33. # 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
34. setup.template.pattern: "test_filebeat-*"
35. # # 定义索引模板的规则信息
36. setup.template.settings:
37.   # 分片数
38.   index.number_of_shards: 5
39.   # 每个分片有多少个副本
40.   index.number_of_replicas: 0
42. # 启动filebeat
43. # 此时就修改完成了shared和replicas

复制代码

Filebeat采集nginx实战

1. 1.安装nginx
2. root@elk2:~# apt install -y nginx
4. 2.启动nginx
5. root@elk2:~# systemctl start nginx
6. root@elk2:~# netstat -tunlp | grep 80
7. tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      17956/nginx: master
8. tcp6       0      0 :::80                   :::*                    LISTEN      17956/nginx: master
10. 3.测试访问
11. root@elk2:~# curl 127.1
12. # 日志位置
13. root@elk2:~# ll /var/log/nginx/access.log
14. -rw-r----- 1 www-data adm 86 Mar 19 06:58 /var/log/nginx/access.log
15. root@elk2:~# cat /var/log/nginx/access.log
16. 127.0.0.1 - - [19/Mar/2025:06:58:31 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
19. 4.编写Filebeat实例
20. root@elk2:~# cat /etc/filebeat/config/04-log-to-nginx.yaml
21. # 定义数据从哪里来
22. filebeat.inputs:
23.   # 指定数据源的类型是log，表示从文件读取数据
24. - type: log
25.   # 指定文件的路径
26.   paths:
27.     - /var/log/nginx/access.log*
29. # 定义数据到终端
30. output.elasticsearch:  
31.   hosts:
32.     - 192.168.121.21:9200
33.     - 192.168.121.22:9200
34.     - 192.168.121.23:9200
35.   # 自定义索引名称
36.   index: "test_filebeat-%{+yyyy.MM.dd}"
38. # 禁用索引生命周期管理(index lifecycle management，ILM)
39. # 如果启用了此配置，则忽略自定义索引的所有信息
40. setup.ilm.enabled: false
41. # 如果索引模板存在，是否覆盖，默认值为false，如果明确需要，则可以将其设置为ture。
42. # 但是官方建议将其设置为false，原因是每次写数据时，都会建立tcp链接，消耗资源。
43. setup.template.overwrite: true
44. # 定义索引模板(就是创建索引的规则)的名称
45. setup.template.name: "test_filebeat"
46. # 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
47. setup.template.pattern: "test_filebeat-*"
48. # # 定义索引模板的规则信息
49. setup.template.settings:
50.   # 分片数
51.   index.number_of_shards: 5
52.   # 每个分片有多少个副本
53.   index.number_of_replicas: 0
56. 5.启动filebeat
57. root@elk2:~# filebeat  -e -c /etc/filebeat/config/04-log-to-nginx.yaml

复制代码

Filebeat分析nginx日记
filebeat modules

1. # filebeat support any modules
2. # 关于filebeat模块官方的解释：他能简化filebeat分析的日志格式
3. # Filebeat modules simplify the collection, parsing, and visualization of common log formats.
4. # 默认情况下这些模块都是disable  在需要时我们需要自己设置为enabled
5. root@elk2:~# ls -l /etc/filebeat/modules.d/
6. total 300
7. -rw-r--r-- 1 root root   484 Feb 13 16:58 activemq.yml.disabled
8. -rw-r--r-- 1 root root   476 Feb 13 16:58 apache.yml.disabled
9. -rw-r--r-- 1 root root   281 Feb 13 16:58 auditd.yml.disabled
10. -rw-r--r-- 1 root root  2112 Feb 13 16:58 awsfargate.yml.disabled
11. 。。。
12. root@elk2:~# ls -l /etc/filebeat/modules.d/ | wc -l
13. 72
15. # 查看哪些模块处于enabled和disabled
16. root@elk2:~# filebeat modules list
19. # 启动模块
20. root@elk2:~# filebeat modules enable apache nginx mysql redis
21. Enabled apache
22. Enabled nginx
23. Enabled mysql
24. Enabled redis
26. # 停止模块
27. root@elk2:~# filebeat modules disable  apache  mysql redis
28. Disabled apache
29. Disabled mysql
30. Disabled redis

复制代码

配置filebeat监控nginx

1. # 需要在filebeat的配置文件中配置模块功能  在/etc/filebeat/filebeat.yml文件中规定了配置方式
2. filebeat.config.modules:
3.   # Glob pattern for configuration loading
4.   path: ${path.config}/modules.d/*.yml
6.   # Set to true to enable config reloading
7.   reload.enabled: false
9.   # Period on which files under path should be checked for changes
10.   #reload.period: 10s
12. module_nginx# 写入filebeat实例
13. root@elk2:~# cat /etc/filebeat/config/07-module-nginx-to-es.yaml
14. # Config modules
15. filebeat.config.modules:
16.   # Glob pattern for configuration loading  指定在哪个路径下加载
17.   path: ${path.config}/modules.d/*.yml
19.   # Set to true to enable config reloading 自动加载/etc/filebeat/modules.d/下面的yml文件。
20.   reload.enabled: true
22.   # Period on which files under path should be checked for changes
23.   #reload.period: 10s
25. output.elasticsearch:  
26.   hosts:
27.     - 192.168.121.21:9200
28.     - 192.168.121.22:9200
29.     - 192.168.121.23:9200
30.   index: "module_nginx-%{+yyyy.MM.dd}"
32. setup.ilm.enabled: false
33. setup.template.overwrite: true
34. setup.template.name: "module_nginx"
35. setup.template.pattern: "module_nginx-*"
36. setup.template.settings:
37.   index.number_of_shards: 5
38.   index.number_of_replicas: 0
41. # 准备nginx日志测试用例
42. root@elk2:~# cat /var/log/nginx/access.log
43. 192.168.121.1 - - [19/Mar/2025:16:42:23 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 CrKey/1.54.248666"
44. 1.168.121.1 - - [19/Mar/2025:16:42:26 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 CrKey/1.54.248666"
45. 92.168.121.1 - - [19/Mar/2025:16:42:29 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
46. 192.168.11.1 - - [19/Mar/2025:16:42:31 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 13; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Mobile Safari/537.36"
47. 192.168.121.1 - - [19/Mar/2025:16:42:40 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
51. # 启动filebeat实例
52. root@elk2:~# filebeat -e -c /etc/filebeat/config/07-module-nginx-to-es.yaml
58. # 在收集到的结果中包含正确的日志和错误的日志，我们如果想只收集正确的，需要设置nginx模板的配置文件 /etc/filebeat/module.d/nginx.yml
59. - module: nginx
60.   # Access logs
61.   access:
62.     enabled: true
64.     # Set custom paths for the log files. If left empty,
65.     # Filebeat will choose the paths depending on your OS.
66.     var.paths: ["/var/log/nginx/access.log"]
68.   # Error logs
69.   error:
70.     enabled: false    # 从true改为false
72.     # Set custom paths for the log files. If left empty,
73.     # Filebeat will choose the paths depending on your OS.
74.     #var.paths:
76.   # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
77.   ingress_controller:
78.     enabled: false
80.     # Set custom paths for the log files. If left empty,
81.     # Filebeat will choose the paths depending on your OS.
82.     #var.paths:

复制代码

kibana分析PV

1. pv：  page view  页面访问量
2. 1个request 就是一个 pv

复制代码

kibana分析IP
kibana分析带宽
kibana制作Dashboard
kibana分析设备
kibana分析操作体系占比
kibana分析全球用户占比
filebeat采集tomcat日记
摆设tomcat

1. [root@elk2 ~]# wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.5/bin/apache-tomcat-11.0.5.tar.gz
2. [root@elk2 ~]# tar xf apache-tomcat-11.0.5.tar.gz  -C /usr/local
4. # 配置环境变量
5. # es本身有jdk环境，我们将es的jdk环境配置到环境变量中，让tomcat调用es的jdk环境
6. # es的jdk环境目录
7. [root@elk2 ~]# ll /usr/share/elasticsearch/jdk/
8. # 添加环境变量
9. [root@elk2 ~]# vim /etc/profile.d/tomcat.sh
10. [root@elk2 ~]# source /etc/profile.d/tomcat.sh
11. [root@elk2 ~]# cat /etc/profile.d/tomcat.sh
12. #!/bin/bash
13. export JAVA_HOME=/usr/share/elasticsearch/jdk
14. export TOMCAT_HOME=/usr/local/apache-tomcat-11.0.5
15. export PATH=$PATH:$JAVA_HOME/bin:$TOMCAT_HOME/bin
16. [root@elk3 ~]# java -version
17. openjdk version "22.0.2" 2024-07-16
18. OpenJDK Runtime Environment (build 22.0.2+9-70)
19. OpenJDK 64-Bit Server VM (build 22.0.2+9-70, mixed mode, sharing)
21. # 因为tomcat默认的日志格式显示的信息很少，所以这里需要修改tomcat配置文件，修改日志格式
22. [root@elk3 ~]# vim /usr/local/apache-tomcat-11.0.5/conf/server.xml
23. ...
24.           <Host name="tomcat.test.com"  appBase="webapps"
25.                 unpackWARs="true" autoDeploy="true">
27.                 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
28.             prefix="tomcat.test.com_access_log" suffix=".json"
29. pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","request":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","http_user_agent":"%{User-Agent}i"}"/>
31.           </Host>
35. # 启动tomcat
36. [root@elk2 ~]# catalina.sh  start
37. [root@elk2 ~]# netstat -tunlp | grep 8080
38. tcp6       0      0 :::8080                 :::*                    LISTEN      98628/java   
41. # 访问测试
42. [root@elk2 ~]# cat /etc/hosts
43. 127.0.0.1 localhost
45. # The following lines are desirable for IPv6 capable hosts
46. ::1     ip6-localhost ip6-loopback
47. fe00::0 ip6-localnet
48. ff00::0 ip6-mcastprefix
49. ff02::1 ip6-allnodes
50. ff02::2 ip6-allrouters
53. 192.168.121.92 tomcat.test.com
55. [root@elk2 ~]# cat /usr/local/apache-tomcat-11.0.5/logs/tomcat.test.com_access_log.2025-03-23.json
56. {"clientip":"192.168.121.92","ClientUser":"-","authenticated":"-","AccessTime":"[23/Mar/2025:20:55:41 +0800]","request":"GET / HTTP/1.1","status":"200","SendBytes":"11235","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}

复制代码

配置filebeat监控tomcat

1. # 启动tomcat模块
2. [root@elk3 ~]# filebeat modules enable tomcat
3. Enabled tomcat
4. [root@elk3 ~]# ll /etc/filebeat/modules.d/tomcat.yml
5. -rw-r--r-- 1 root root 623 Feb 14 00:58 /etc/filebeat/modules.d/tomcat.yml
9. # 配置tomcat模块
10. [root@elk3 ~]# cat /etc/filebeat/modules.d/tomcat.yml
11. # Module: tomcat
12. # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-tomcat.html
14. - module: tomcat
15.   log:
16.     enabled: true
18.     # Set which input to use between udp (default), tcp or file.
19.     # var.input: udp
20.     var.input: file
21.     # var.syslog_host: tomcat.test.com
22.     # var.syslog_port: 8080
24.     # Set paths for the log files when file input is used.
25.     # var.paths:
26.     #   - /var/log/tomcat/*.log
27.     var.paths:
28.       - /usr/local/apache-tomcat-11.0.5/logs/tomcat.test.com_access_log.2025-03-23.json
30.     # Toggle output of non-ECS fields (default true).
31.     # var.rsa_fields: true
33.     # Set custom timezone offset.
34.     # "local" (default) for system timezone.
35.     # "+02:00" for GMT+02:00
36.     # var.tz_offset: local
38. # 配置filebeat
39. [root@elk3 ~]# cat /etc/filebeat/config/02-tomcat-es.yaml
40. filebeat.config.modules:
41.   path: ${path.config}/modules.d/*.yml
42.   reload.enabled: true
43. output.elasticsearch:
44.   hosts:
45.   - 192.168.121.91:9200
46.   - 192.168.121.92:9200
47.   - 192.168.121.93:9200
48.   index: test-modules-tomcat-%{+yyyy.MM.dd}
49. setup.ilm.enabled: false
50. setup.template.name: "test-modules-tomcat"
51. setup.template.pattern: "test-modules-tomcat-*"
52. setup.template.overwrite: true
53. setup.template.settings:
54.   index.number_of_shards: 5
55.   index.number_of_replicas: 0
58. # 启动filebeat
59. [root@elk3 ~]# filebeat -e -c /etc/filebeat/config/02-tomcat-es.yaml

复制代码

filebeat processors
filebeat 处理器
https://www.elastic.co/guide/en/beats/filebeat/7.17/filtering-and-enhancing-data.html

1. # 在我们使用filebeat采集tomcat日志时，由于tomcat的日志在我们的设置下是json格式的，我们想要得出json格式中的具体信息，需要通过filebeat processors进行进一步配置
2. # decode JSON fields 参数可以实现解析json格式
3. # 官方配置
4. processors:
5.   - decode_json_fields:
6.       fields: ["field1", "field2", ...]
7.       process_array: false
8.       max_depth: 1
9.       target: ""
10.       overwrite_keys: false
11.       add_error_key: true
12. # fields：指定要对哪个字段进行json解析
13. # process_array：一个bool值，指定是否解析数字，默认是false，可选配置
14. # max_depth：最大解析深度，默认值为1 将解码中所示字段中的 JSON 对象fields，值为2还将解码嵌入在这些解析文档的字段中的对象，可选配置
15. # target：解码后的 JSON 将写入的字段。默认情况下，解码后的 JSON 对象将替换读取它的字符串字段。要将解码后的 JSON 字段合并到事件的根目录中，请指定 target一个空字符串 ( target: "")。请注意，null值 ( target:) 被视为未设置字段 可选配置
16. # overwrite_keys：布尔值，指定事件中的现有键是否被解码的 JSON 对象中的键覆盖。默认值为false。可选配置
17. # add_error_key：如果设置为 ，true并且在解码 JSON 密钥时发生错误，则该error字段将成为事件的一部分，并带有错误消息。如果设置为false，则事件的字段中不会有任何错误。默认值为false。可选配置
19. # 写入filebeat配置
20. [root@elk3 ~]# cat /etc/filebeat/config/02-tomcat-es.yaml,
21. cat: /etc/filebeat/config/02-tomcat-es.yaml,: No such file or directory
22. [root@elk3 ~]# cat /etc/filebeat/config/02-tomcat-es.yaml
23. filebeat.config.modules:
24.   path: ${path.config}/modules.d/*.yml
25.   reload.enabled: true
26. processors:
27.   - decode_json_fields:
28.       fields: ["event.original"]
29.       process_array: false
30.       max_depth: 1
31.       target: ""
32.       overwrite_keys: false
33.       add_error_key: true
34. output.elasticsearch:
35.   hosts:
36.   - 192.168.121.91:9200
37.   - 192.168.121.92:9200
38.   - 192.168.121.93:9200
39.   index: test-modules-tomcat-%{+yyyy.MM.dd}
40. setup.ilm.enabled: false
41. setup.template.name: "test-modules-tomcat"
42. setup.template.pattern: "test-modules-tomcat-*"
43. setup.template.overwrite: true
44. setup.template.settings:
45.   index.number_of_shards: 5
46.   index.number_of_replicas: 0
48. # 启动filebeat
49. [root@elk3 ~]# filebeat -e -c /etc/filebeat/config/02-tomcat-es.yaml
53. # 通过filebeat删除一个字段
54. processors:
55.   - drop_fields:
56.       when:
57.         condition
58.       fields: ["field1", "field2", ...]
59.       ignore_missing: false
60.       
61. The supported conditions are:
63. equals
64. contains
65. regexp
66. range
67. network
68. has_fields
69. or
70. and
71. not
73. # 删除status 是404的 event.module 字段值
74. [root@elk3 ~]# cat /etc/filebeat/config/02-tomcat-es.yaml
75. filebeat.config.modules:
76.   path: ${path.config}/modules.d/*.yml
77.   reload.enabled: true
78. processors:
79.   - decode_json_fields:
80.       fields: ["event.original"]
81.       process_array: false
82.       max_depth: 1
83.       target: ""
84.       overwrite_keys: false
85.       add_error_key: true
86.   - drop_fields:
87.       when:
88.         equals:
89.           status: "404"
90.       fields: ["event.module"]
91.       ignore_missing: false
92. output.elasticsearch:
93.   hosts:
94.   - 192.168.121.91:9200
95.   - 192.168.121.92:9200
96.   - 192.168.121.93:9200
97.   index: test-modules-tomcat-%{+yyyy.MM.dd}
98. setup.ilm.enabled: false
99. setup.template.name: "test-modules-tomcat"
100. setup.template.pattern: "test-modules-tomcat-*"
101. setup.template.overwrite: true
102. setup.template.settings:
103.   index.number_of_shards: 5
104.   index.number_of_replicas: 0
105. [root@elk3 ~]# filebeat -e -c /etc/filebeat/config/02-tomcat-es.yam

复制代码

filebeat采集es集群日记

1. # 启动模块
2. [root@elk1 ~]# filebeat modules enable elasticsearch
3. Enabled elasticsearch
5. [root@elk1 ~]# cat /etc/filebeat/config/02-es-log.yml
6. filebeat.config.modules:
7.   path: ${path.config}/modules.d/*.yml
8.   reload.enabled: true
10. output.elasticsearch:
11.   hosts:
12.   - 192.168.121.91:9200
13.   - 192.168.121.92:9200
14.   - 192.168.121.93:9200
15.   index: es-log-modules-eslog-%{+yyyy.MM.dd}
16. setup.ilm.enabled: false
17. setup.template.name: "es-log"
18. setup.template.pattern: "es-log-*"
19. setup.template.overwrite: true
20. setup.template.settings:
21.   index.number_of_shards: 5
22.   index.number_of_replicas: 0

复制代码

filebeat采集mysql日记

1. # 部署mysql
2. [root@elk1 ~]# wget https://dev.mysql.com/get/Downloads/MySQL-8.4/mysql-8.4.4-linux-glibc2.28-x86_64.tar.xz
3. [root@elk1 ~]# tar xf mysql-8.4.4-linux-glibc2.28-x86_64.tar.xz -C /usr/local/
6. # 准备启动脚本并授权
7. [root@elk1 ~]# cp /usr/local/mysql-8.4.4-linux-glibc2.28-x86_64/support-files/mysql.server /etc/init.d/
8. [root@elk1 ~]# vim /etc/init.d/mysql.server
9. [root@elk1 ~]# grep -E "^(basedir=|datadir=)" /etc/init.d/mysql.server
10. basedir=/usr/local/mysql-8.4.4-linux-glibc2.28-x86_64/
11. datadir=/var/lib/mysql
12. [root@elk1 ~]# useradd -m mysql
13. [root@elk1 ~]# install -d /var/lib/mysql -o mysql -g mysql
14. [root@elk1 ~]# ll -d /var/lib/mysql
15. drwxr-xr-x 2 mysql mysql 4096 Mar 25 17:05 /var/lib/mysql/
17. # 准备配置文件
18. [root@elk1 ~]# vim /etc/my.cnf
19. [root@elk1 ~]# cat /etc/my.cnf
20. [mysqld]
21. basedir=/usr/local/mysql-8.4.4-linux-glibc2.28-x86_64/
22. datadir=/var/lib/mysql
23. socket=/tmp/mysql80.sock
24. port=3306
26. [client]
27. socket=/tmp/mysql80.sock
29. # 启动数据库
30. [root@elk1 ~]# vim /etc/profile.d/mysql.sh
31. [root@elk1 ~]# cat /etc/profile.d/mysql.sh
32. #!/bin/bash
33. export MYSQL_HOME=/usr/local/mysql-8.4.4-linux-glibc2.28-x86_64/
34. export PATH=$PATH:$MYSQL_HOME/bin
35. [root@elk1 ~]# source  /etc/profile.d/mysql.sh
36. [root@elk1 ~]# mysqld --initialize-insecure  --user=mysql  --datadir=/var/lib/mysql  --basedir=/usr/local/mysql-8.4.4-linux-glibc2.28-x86_64
37. 2025-03-25T09:08:36.829914Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
38. 2025-03-25T09:08:36.842773Z 0 [System] [MY-013169] [Server] /usr/local/mysql-8.4.4-linux-glibc2.28-x86_64/bin/mysqld (mysqld 8.4.4) initializing of server in progress as process 7905
39. 2025-03-25T09:08:36.918780Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
40. 2025-03-25T09:08:37.818933Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
41. 2025-03-25T09:08:42.504501Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
42. 2025-03-25T09:08:46.909940Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.
43. [root@elk1 ~]# /etc/init.d/mysql.server start
44. Starting mysql.server (via systemctl): mysql.server.service.
45. [root@elk1 ~]# netstat -tunlp | grep 3306
46. tcp6       0      0 :::3306                 :::*                    LISTEN      8141/mysqld         
47. tcp6       0      0 :::33060                :::*                    LISTEN      8141/mysqld  
51. # 开启filebeat的模块
52. [root@elk1 ~]# filebeat modules enable mysql
53. Enabled mysql
55. # 配置filebeat
56. [root@elk1 ~]# cat /etc/filebeat/config/03-es-mysql-log.yaml
57. filebeat.config.modules:
58.   path: ${path.config}/modules.d/mysql.yml
59.   reload.enabled: true
61. output.elasticsearch:
62.   hosts:
63.   - 192.168.121.91:9200
64.   - 192.168.121.92:9200
65.   - 192.168.121.93:9200
66.   index: es-modules-mysql-%{+yyyy.MM.dd}
67. setup.ilm.enabled: false
68. setup.template.name: "es-modules-mysql"
69. setup.template.pattern: "es-modules-mysql-*"
70. setup.template.overwrite: true
71. setup.template.settings:
72.   index.number_of_shards: 5
73.   index.number_of_replicas: 0
74. # 配置mysql modules
75. [root@elk1 ~]# cat /etc/filebeat/modules.d/mysql.yml
76. # Module: mysql
77. # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-mysql.html
79. - module: mysql
80.   # Error logs
81.   error:
82.     enabled: true
84.     # Set custom paths for the log files. If left empty,
85.     # Filebeat will choose the paths depending on your OS.
86.     #var.paths:
87.     var.paths: ["/var/lib/mysql/elk1.err"]
89.   # Slow logs
90.   slowlog:
91.     enabled: true
93.     # Set custom paths for the log files. If left empty,
94.     # Filebeat will choose the paths depending on your OS.
95.     #var.paths:
97. # 启动filebeat实例
98. [root@elk1 ~]# filebeat -e -c /etc/filebeat/config/03-es-mysql-log.yaml

复制代码

filebeat采集redis

1. # 安装redis
2. [root@elk1 ~]# apt install -y redis
4. # redis日志文件位置
5. [root@elk1 ~]# cat /var/log/redis/redis-server.log
6. 8618:C 25 Mar 2025 17:18:37.442 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
7. 8618:C 25 Mar 2025 17:18:37.442 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8. 8618:C 25 Mar 2025 17:18:37.442 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=8618, just started
9. 8618:C 25 Mar 2025 17:18:37.442 # Configuration loaded
10.                 _._                                                  
11.            _.-``__ ''-._                                             
12.       _.-``    `.  `_.  ''-._           Redis 6.0.16 (00000000/0) 64 bit
13.   .-`` .-```.  ```\/    _.,_ ''-._                                   
14. (    '      ,       .-`  | `,    )     Running in standalone mode
15. |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
16. |    `-._   `._    /     _.-'    |     PID: 8618
17.   `-._    `-._  `-./  _.-'    _.-'                                   
18. |`-._`-._    `-.__.-'    _.-'_.-'|                                 
19. |    `-._`-._        _.-'_.-'    |           http://redis.io        
20.   `-._    `-._`-.__.-'_.-'    _.-'                                   
21. |`-._`-._    `-.__.-'    _.-'_.-'|                                 
22. |    `-._`-._        _.-'_.-'    |                                 
23.   `-._    `-._`-.__.-'_.-'    _.-'                                   
24.       `-._    `-.__.-'    _.-'                                       
25.           `-._        _.-'                                          
26.               `-.__.-'                                               
28. 8618:M 25 Mar 2025 17:18:37.446 # Server initialized
29. 8618:M 25 Mar 2025 17:18:37.446 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
30. 8618:M 25 Mar 2025 17:18:37.447 * Ready to accept connections
32. # 启动redis modules
33. [root@elk1 ~]# filebeat modules enable redis
34. Enabled redis
36. [root@elk1 ~]# cat /etc/filebeat/config/04-es-redis-log.yaml
37. filebeat.config.modules:
38.   path: ${path.config}/modules.d/redis.yml
39.   reload.enabled: true
41. output.elasticsearch:
42.   hosts:
43.   - 192.168.121.91:9200
44.   - 192.168.121.92:9200
45.   - 192.168.121.93:9200
46.   index: es-modules-redis-%{+yyyy.MM.dd}
47. setup.ilm.enabled: false
48. setup.template.name: "es-modules-redis"
49. setup.template.pattern: "es-modules-redis-*"
50. setup.template.overwrite: true
51. setup.template.settings:
52.   index.number_of_shards: 5
53.   index.number_of_replicas: 0
55. # 启动filebeat实例
56. [root@elk1 ~]# filebeat  -e -c /etc/filebeat/config/04-es-redis-log.yaml

复制代码

filebeat多行合并问题

1. # Manage multiline messages
2. #
3. parsers:
4. - multiline:
5.     type: pattern
6.     pattern: '^\['
7.     negate: true
8.     match: after
9.    
10. multiline.type
11.         Defines which aggregation method to use. The default is pattern. The other option is count which lets you aggregate constant number of lines.
12. multiline.pattern
13.         Specifies the regular expression pattern to match. Note that the regexp patterns supported by Filebeat differ somewhat from the patterns supported by Logstash. See Regular expression                 support for a list of supported regexp patterns. Depending on how you configure other multiline options, lines that match the specified regular expression are considered either                         continuations of a previous line or the start of a new multiline event. You can set the negate option to negate the pattern.
14. multiline.negate
15.         Defines whether the pattern is negated. The default is false.
16. multiline.match
17.         Specifies how Filebeat combines matching lines into an event. The settings are after or before. The behavior of these settings depends on what you specify for negate:

复制代码

manager multiline redis log message

1. # 通过manager multiline message优化redis集群日志采集规则
2. # type: filestream 是对 旧版本log的替代
3. [root@elk1 ~]# cat /etc/filebeat/config/04-es-redis-log.yaml
4. filebeat.inputs:
5. - type: filestream
6.   paths:
7.     - /var/log/redis/redis-server.log*
8.   # 配置解析器
9.   parsers:
10.     # 定义多行匹配
11.   - multiline:
12.       # 指定匹配的类型
13.       type: pattern
14.       # 定义匹配模式
15.       pattern: '^\d'
16.       # 参考官网: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
17.       negate: true
18.       match: after
19. output.elasticsearch:
20.   hosts:
21.   - 192.168.121.91:9200
22.   - 192.168.121.92:9200
23.   - 192.168.121.93:9200
24.   index: es-modules-redis-%{+yyyy.MM.dd}
25. setup.ilm.enabled: false
26. setup.template.name: "es-modules-redis"
27. setup.template.pattern: "es-modules-redis-*"
28. setup.template.overwrite: true
29. setup.template.settings:
30.   index.number_of_shards: 5
31.   index.number_of_replicas: 0
32. [root@elk1 ~]# filebeat -e -c /etc/filebeat/config/04-es-redis-log.yaml
33. # redis日志中的图案就筛选出来了

复制代码

manager multiline tomcat error log message

1. # tomcat error log path : /usr/local/apache-tomcat-11.0.5/logs/catalina.*
2. [root@elk2 ~]# cat /etc/filebeat/config/01-es-cluster-tomcat.yml
3. filebeat.inputs:
4. - type: filestream
5.   paths:
6.     - /usr/local/apache-tomcat-11.0.5/logs/catalina*
7.   parsers:
8.   - multiline:
9.       type: pattern
10.       pattern: '^\d'
11.       negate: true
12.       match: after
13. output.elasticsearch:
14.   hosts:
15.   - 192.168.121.91:9200
16.   - 192.168.121.92:9200
17.   - 192.168.121.93:9200
18.   index: test-modules-tomcat-elk2-%{+yyyy.MM.dd}
19. setup.ilm.enabled: false
20. setup.template.name: "test-modules-tomcat-elk2"
21. setup.template.pattern: "test-modules-tomcat-elk2*"
22. setup.template.overwrite: true
23. setup.template.settings:
24.   index.number_of_shards: 5
25.   index.number_of_replicas: 0

复制代码

filebeat多实例

1. 1.启动实例1
2. filebeat -e -c /etc/filebeat/config/01-log-to-console.yaml  --path.data /tmp/xixi
5. 2.启动实例2
6. filebeat -e -c /etc/filebeat/config/02-log-to-es.yaml  --path.data /tmp/haha
8. # 通过filebeat采集 /var/log/syslog  /var/log/auth.log
9. root@elk2:~# cat /etc/filebeat/config/05-log-to-syslog.yaml
10. # 定义数据从哪里来
11. filebeat.inputs:
12.   # 指定数据源的类型是log，表示从文件读取数据
13. - type: log
14.   # 指定文件的路径
15.   paths:
16.     - /var/log/syslog
18. # 定义数据到终端
19. output.elasticsearch:  
20.   hosts:
21.     - 192.168.121.21:9200
22.     - 192.168.121.22:9200
23.     - 192.168.121.23:9200
24.   # 自定义索引名称
25.   index: "test_syslog-%{+yyyy.MM.dd}"
27. # 禁用索引生命周期管理(index lifecycle management，ILM)
28. # 如果启用了此配置，则忽略自定义索引的所有信息
29. setup.ilm.enabled: false
30. # 如果索引模板存在，是否覆盖，默认值为false，如果明确需要，则可以将其设置为ture。
31. # 但是官方建议将其设置为false，原因是每次写数据时，都会建立tcp链接，消耗资源。
32. setup.template.overwrite: true
33. # 定义索引模板(就是创建索引的规则)的名称
34. setup.template.name: "test_syslog"
35. # 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
36. setup.template.pattern: "test_syslog-*"
37. # # 定义索引模板的规则信息
38. setup.template.settings:
39.   # 分片数
40.   index.number_of_shards: 5
41.   # 每个分片有多少个副本
42.   index.number_of_replicas: 0
43. root@elk2:~# cat /etc/filebeat/config/06-log-to-auth.yaml
44. # 定义数据从哪里来
45. filebeat.inputs:
46.   # 指定数据源的类型是log，表示从文件读取数据
47. - type: log
48.   # 指定文件的路径
49.   paths:
50.     - /var/log/auth.log
52. # 定义数据到终端
53. output.elasticsearch:  
54.   hosts:
55.     - 192.168.121.21:9200
56.     - 192.168.121.22:9200
57.     - 192.168.121.23:9200
58.   # 自定义索引名称
59.   index: "test_auth-%{+yyyy.MM.dd}"
61. # 禁用索引生命周期管理(index lifecycle management，ILM)
62. # 如果启用了此配置，则忽略自定义索引的所有信息
63. setup.ilm.enabled: false
64. # 如果索引模板存在，是否覆盖，默认值为false，如果明确需要，则可以将其设置为ture。
65. # 但是官方建议将其设置为false，原因是每次写数据时，都会建立tcp链接，消耗资源。
66. setup.template.overwrite: true
67. # 定义索引模板(就是创建索引的规则)的名称
68. setup.template.name: "test_auth"
69. # 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
70. setup.template.pattern: "test_auth-*"
71. # # 定义索引模板的规则信息
72. setup.template.settings:
73.   # 分片数
74.   index.number_of_shards: 5
75.   # 每个分片有多少个副本
76.   index.number_of_replicas: 0
79. # 通过多实例的方式启动filebeat
80. root@elk2:~# filebeat -e -c /etc/filebeat/config/05-log-to-syslog.yaml --path.data /tmp/xixi
81. root@elk2:~# filebeat -e -c /etc/filebeat/config/06-log-to-syslog.yaml --path.data /tmp/haha

复制代码

EFK分析web集群
摆设web集群

1. 1.部署tomcat服务器
2. # 192.168.121.92 192.168.121.93部署tomcat
3. 参照filebeat收集tomcat章节的部署tomcat方式
6. 2.部署nginx
7. # 192.168.121.91部署nginx
8. [root@elk1 ~]# apt install -y nginx
9. [root@elk1 ~]# vim /etc/nginx/nginx.conf
10. ...
11. upstream es-web{
12.     server 192.168.121.92:8080;
13.     server 192.168.121.93:8080;
14. }
15. server {
16.     server_name es.web.com;
17.     location / {
18.         proxy_pass http://es-web;
19.     }
20. }
21. ...
22. [root@elk1 ~]# nginx -t
23. [root@elk1 ~]# systemctl restart nginx
24. # 访问测试
25. [root@elk1 ~]# curl es.web.com

复制代码

采集web集群日记

1. # 91加载nginx模块
2. # 92 93 加载tomcat模块
3. [root@elk1 ~]# filebeat modules enable nginx
4. Enabled nginx
5. [root@elk2 ~]# filebeat modules enable tomcat
6. Enabled tomcat
7. [root@elk3 ~]# filebeat modules enable tomcat
8. Enabled tomcat
11. 1.配置nginx模块功能
12. [root@elk1 ~]# cat /etc/filebeat/modules.d/nginx.yml
13. # Module: nginx
14. # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-nginx.html
16. - module: nginx
17.   # Access logs
18.   access:
19.     enabled: true
21.     # Set custom paths for the log files. If left empty,
22.     # Filebeat will choose the paths depending on your OS.
23.     var.paths: /var/log/nginx/access.log
25.   # Error logs
26.   error:
27.     enabled: false
29.     # Set custom paths for the log files. If left empty,
30.     # Filebeat will choose the paths depending on your OS.
31.     #var.paths:
33.   # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
34.   ingress_controller:
35.     enabled: false
37.     # Set custom paths for the log files. If left empty,
38.     # Filebeat will choose the paths depending on your OS.
39.     #var.paths:
40. 2.配置tomcat模块功能
41. [root@elk2 ~]# cat /etc/filebeat/modules.d/tomcat.yml
42. # Module: tomcat
43. # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-tomcat.html
45. - module: tomcat
46.   log:
47.     enabled: true
49.     # Set which input to use between udp (default), tcp or file.
50.     var.input: file
51.     # var.syslog_host: localhost
52.     # var.syslog_port: 9501
54.     # Set paths for the log files when file input is used.
55.     var.paths:
56.       - /usr/local/apache-tomcat-11.0.5/logs/*.json
58.     # Toggle output of non-ECS fields (default true).
59.     # var.rsa_fields: true
61.     # Set custom timezone offset.
62.     # "local" (default) for system timezone.
63.     # "+02:00" for GMT+02:00
64.     # var.tz_offset: local
65. 3.配置91filebeat配置文件
66. [root@elk1 ~]# cat /etc/filebeat/config/01-es-web-nginx.yaml
67. filebeat.config.modules:
68.   # Glob pattern for configuration loading
69.   path: ${path.config}/modules.d/*.yml
71.   # Set to true to enable config reloading
72.   reload.enabled: true
73. output.elasticsearch:
74.   hosts:
75.   - 192.168.121.91:9200
76.   - 192.168.121.92:9200
77.   - 192.168.121.93:9200
78.   index: es-web-nginx-%{+yyyy.MM.dd}
79. setup.ilm.enabled: false
80. setup.template.name: "es-web-nginx"
81. setup.template.pattern: "es-web-nginx-*"
82. setup.template.overwrite: true
83. setup.template.settings:
84.   index.number_of_shards: 5
85.   index.number_of_replicas: 0
86. 4.92配置监控tomcat配置文件
87. [root@elk2 ~]# cat /etc/filebeat/config/01-es-web-tomcat.yaml
88. filebeat.config.modules:
89.   path: ${path.config}/modules.d/*.yml
90.   reload.enabled: true
91. processors:
92.   - decode_json_fields:
93.       fields: ["event.original"]
94.       process_array: false
95.       max_depth: 1
96.       target: ""
97.       overwrite_keys: false
98.       add_error_key: true
99.   - drop_fields:
100.       when:
101.         equals:
102.           status: "404"
103.       fields: ["event.module"]
104.       ignore_missing: false
105. output.elasticsearch:
106.   hosts:
107.   - 192.168.121.91:9200
108.   - 192.168.121.92:9200
109.   - 192.168.121.93:9200
110.   index: test-modules-tomcat91-%{+yyyy.MM.dd}
111. setup.ilm.enabled: false
112. setup.template.name: "test-modules-tomcat91"
113. setup.template.pattern: "test-modules-tomcat91-*"
114. setup.template.overwrite: true
115. setup.template.settings:
116.   index.number_of_shards: 5
117.   index.number_of_replicas: 0
118. 5.93配置监控tomcat配置文件
119. [root@elk3 ~]# cat /etc/filebeat/config/02-es-web-tomcat.yaml
120. filebeat.config.modules:
121.   path: ${path.config}/modules.d/*.yml
122.   reload.enabled: true
123. processors:
124.   - decode_json_fields:
125.       fields: ["event.original"]
126.       process_array: false
127.       max_depth: 1
128.       target: ""
129.       overwrite_keys: false
130.       add_error_key: true
131.   - drop_fields:
132.       when:
133.         equals:
134.           status: "404"
135.       fields: ["event.module"]
136.       ignore_missing: false
137. output.elasticsearch:
138.   hosts:
139.   - 192.168.121.91:9200
140.   - 192.168.121.92:9200
141.   - 192.168.121.93:9200
142.   index: test-modules-tomcat93-%{+yyyy.MM.dd}
143. setup.ilm.enabled: false
144. setup.template.name: "test-modules-tomcat93"
145. setup.template.pattern: "test-modules-tomcat93-*"
146. setup.template.overwrite: true
147. setup.template.settings:
148.   index.number_of_shards: 5
149.   index.number_of_replicas: 0
151. # 依次启动filebeat

复制代码

更改字段类型

1. 我们想统计带宽大小，但是此时是无法统计的
5. 这是一个字符类型的值
9. # 通过processors修改字段类型
10. # 官网配置
11. # 支持的类型
12. # The supported types include: integer, long, float, double, string, boolean, and ip.
14. processors:
15.   - convert:
16.       fields:
17.         - {from: "src_ip", to: "source.ip", type: "ip"}
18.         - {from: "src_port", to: "source.port", type: "integer"}
19.       ignore_missing: true
20.       fail_on_error: false
22. # 配置filebeat配置文件
23. filebeat.config.modules:
24.   path: ${path.config}/modules.d/*.yml
25.   reload.enabled: true
26. processors:
27.   - decode_json_fields:
28.       fields: ["event.original"]
29.       process_array: false
30.       max_depth: 1
31.       target: ""
32.       overwrite_keys: false
33.       add_error_key: true
34.   - convert:
35.       fields:
36.         - {from: "SendBytes", type: "long"}
37.   - drop_fields:
38.       when:
39.         equals:
40.           status: "404"
41.       fields: ["event.module"]
42.       ignore_missing: false
43. output.elasticsearch:
44.   hosts:
45.   - 192.168.121.91:9200
46.   - 192.168.121.92:9200
47.   - 192.168.121.93:9200
48.   index: test-modules-tomcat91-%{+yyyy.MM.dd}
49. setup.ilm.enabled: false
50. setup.template.name: "test-modules-tomcat91"
51. setup.template.pattern: "test-modules-tomcat91-*"
52. setup.template.overwrite: true
53. setup.template.settings:
54.   index.number_of_shards: 5
55.   index.number_of_replicas: 0

复制代码

Ansible摆设EFK集群

1. [root@ansible efk]# cat set_es.sh
2. #!/bin/bash
3. ansible-playbook 01-install-elaticsearch.yaml
4. ansible-playbook 02-install-kibana.yaml
5. ansible-playbook 03-install-filebeat.yaml
6. ansible-playbook 04-set-web.yaml
7. ansible-playbook 05-config-filebeat.yaml
9. [root@ansible efk]# bash set_es.sh
11. [root@ansible efk]# cat 01-install-elaticsearch.yaml
12. ---
13. - name: Install es cluster
14.   hosts: all
15.   tasks:
16.     - name: get es deb package
17.       get_url:
18.         url: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-amd64.deb
19.         dest: /root/
20.     - name: Install es
21.       shell:
22.         cmd: dpkg -i /root/elasticsearch-7.17.28-amd64.deb | cat
23.     - name: Configer es
24.       copy:
25.         src: conf/elasticsearch.yml
26.         dest: /etc/elasticsearch/elasticsearch.yml
27.     - name: start es
28.       systemd:
29.         name: elasticsearch
30.         state: started
31.         enabled: yes
34. [root@ansible efk]# cat 02-install-kibana.yaml
35. ---
36. - name: Install kibana
37.   hosts: elk1
38.   tasks:
39.     - name: Get kibana deb package
40.       get_url:
41.         url: https://artifacts.elastic.co/downloads/kibana/kibana-7.17.28-amd64.deb
42.         dest: /root
43.     - name: Install kibana
44.       shell:
45.         cmd: dpkg -i kibana-7.17.28-amd64.deb | cat
46.     - name: Config kibana
47.       copy:
48.         src: conf/kibana.yml
49.         dest: /etc/kibana/kibana.yml
50.     - name: Start kibana
51.       systemd:
52.         name: kibana
53.         state: started
54.         enabled: yes
55. [root@ansible efk]# cat 03-install-filebeat.yaml
56. ---
57. - name: Install filebeat
58.   hosts: elk
59.   tasks:
60.     - name: Get filebeat code
61.       get_url:
62.         url: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.28-amd64.deb
63.         dest: /root
64.     - name: Install filebeat
65.       shell:
66.         cmd: dpkg -i filebeat-7.17.28-amd64.deb | cat
67.     - name: Configer filebeat
68.       file:
69.         path: /etc/filebeat/config
70.         state: directory
72. [root@ansible efk]# cat 04-set-web.yaml
73. ---
74. - name: Set nginx
75.   hosts: elk1
76.   tasks:
77.     - name: Install nginx
78.       shell:
79.         cmd: apt install -y nginx | cat
80.     - name: config nginx
81.       copy:
82.         src: conf/nginx.conf
83.         dest: /etc/nginx/nginx.conf
84.     - name: start nginx
85.       systemd:
86.         name: nginx
87.         state: started
88.         enabled: yes
89.     - name: Configure hosts
90.       copy:
91.         content: 192.168.121.91 es.web.com
92.         dest: /etc/hosts
93. - name: Set tomcat
94.   hosts: elk2,elk3
95.   tasks:
96.     - name: Get tomcat code
97.       get_url:
98.         url: https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.5/bin/apache-tomcat-11.0.5.tar.gz
99.         dest: /root/
100.     - name: unarchive tomcat code
101.       unarchive:
102.         src: /root/apache-tomcat-11.0.5.tar.gz
103.         dest: /usr/local
104.         remote_src: yes
105.     - name: Configure jdk PATH
106.       copy:
107.         src: conf/tomcat.sh
108.         dest: /etc/profile.d/
109.     - name: reload profile
110.       shell:
111.         cmd: source /etc/profile.d/tomcat.sh | cat
112.     - name: Configure tomcat
113.       copy:
114.         src: conf/server.xml
115.         dest: /usr/local/apache-tomcat-11.0.5/conf/server.xml
116.     - name: start tomcat
117.       shell:
118.         cmd: catalina.sh  start |cat
119. [root@ansible efk]# cat 05-config-filebeat.yaml
120. ---
121. - name: configure filebeat
122.   hosts: elk1
123.   tasks:
124.     - name: enable nginx modules
125.       shell:
126.         cmd: filebeat modules enable nginx | cat
127.     - name: configure nginx modules
128.       copy:
129.         src: conf/nginx.yml
130.         dest: /etc/filebeat/modules.d/nginx.yml
131.     - name: configure filebeat
132.       copy:
133.         src: conf/01-es-cluster-nginx.yml
134.         dest: /etc/filebeat/config/01-es-cluster-nginx.yml
136. - name: configure filebeat
137.   hosts: elk2 elk3
138.   tasks:
139.     - name: enable tomcat modules
140.       shell:
141.         cmd: filebeat modules enable nginx | cat
142.     - name: configure tomcat modules
143.       copy:
144.         src: conf/tomcat.yml
145.         dest: /etc/filebeat/modules.d/tomcat.yml
146.     - name: configure filebeat
147.       template:
148.         src: conf/01-es-cluster-tomcat.yml.j2
149.         dest: /etc/filebeat/config/01-es-cluster-tomcat.yml

复制代码

logstash
安装配置logstash

1. 1.部署logstash
2. [root@elk3 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.28-amd64.deb
3. [root@elk3 ~]# dpkg -i logstash-7.17.28-amd64.deb
5. 2.创建符号链接，将Logstash命令添加到PATH环境变量
6. [root@elk3 ~]# ln -svf  /usr/share/logstash/bin/logstash /usr/local/bin/
7. '/usr/local/bin/logstash' -> '/usr/share/logstash/bin/logstash'
9. 3.基于命令行的方式启动实例，使用-e选项指定配置信息（不推荐）
10. [root@elk3 ~]# logstash -e "input { stdin { type => stdin } } output { stdout { codec => rubydebug } }"  --log.level warn
11. ...
12. The stdin plugin is now waiting for input:
13. 111111111111111111111111111
14. {
15.     "@timestamp" => 2025-03-13T06:51:32.821Z,
16.           "type" => "stdin",
17.        "message" => "111111111111111111111111111",
18.           "host" => "elk93",
19.       "@version" => "1"
20. }
22. 4.基于配置文件方式启动Logstash
23. [root@elk3 ~]# vim /etc/logstash/conf.d/01-stdin-to-stdout.conf
24. [root@elk3 ~]# cat /etc/logstash/conf.d/01-stdin-to-stdout.conf
25. input {
26.   stdin {
27.     type => stdin
28.   }
29. }
32. output {
33.   stdout {
34.     codec => rubydebug
35.   }
36. }
38. [root@elk3 ~]# logstash -f /etc/logstash/conf.d/01-stdin-to-stdout.conf
39. ...
40. 333333333333333333333333333333
41. {
42.           "type" => "stdin",
43.        "message" => "333333333333333333333333333333",
44.           "host" => "elk93",
45.     "@timestamp" => 2025-03-13T06:54:20.223Z,
46.       "@version" => "1"
47. }
49. # https://www.elastic.co/guide/en/logstash/7.17/plugins-inputs-file.html
50. [root@elk3 ~]# cat /etc/logstash/conf.d/02-file-to-stdout.conf
51. input {
52.   file {
53.     path => "/tmp/student.txt"
54.   }
55. }
58. output {
59.   stdout {
60.     codec => rubydebug
61.   }
62. }
63. [WARN ] 2025-03-26 09:40:52.788 [[main]<file] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
64. {
65.           "path" => "/tmp/student.txt",
66.       "@version" => "1",
67.     "@timestamp" => 2025-03-26T01:40:52.879Z,
68.        "message" => "aaaddd",
69.           "host" => "elk3"
70. }

复制代码

Logstash采集文本日记策略

1. Logstash采集策略和filebeat的采集策略类似
2.         1.以换行符为准，以行为单位进行采集
3.         2.也存在和filebeat类似的偏移量的概念
5. [root@elk3 ~]# ll /usr/share/logstash/data/plugins/inputs/file/.sincedb_782d533684abe27068ac85b78871b9fd
6. -rw-r--r-- 1 root root 53 Mar 26 09:57 /usr/share/logstash/data/plugins/inputs/file/.sincedb_782d533684abe27068ac85b78871b9fd
7. [root@elk3 ~]# cat /usr/share/logstash/data/plugins/inputs/file/.sincedb_782d533684abe27068ac85b78871b9fd
8. 408794 0 64768 12 1742955373.9715059 /tmp/student.txt  # 12就是偏移量
9. [root@elk3 ~]# cat /tmp/student.txt
10. ABC
11. 2025def
12. [root@elk3 ~]# ll -i /tmp/student.txt
13. 408794 -rw-r--r-- 1 root root 12 Mar 26 09:45 /tmp/student.txt
15. # 可以直接修改偏移量进行指定位置采集  我们修改偏移量到8  查看采集结果
16. {
17.     "@timestamp" => 2025-03-26T02:20:50.776Z,
18.           "host" => "elk3",
19.        "message" => "def",
20.           "path" => "/tmp/student.txt",
21.       "@version" => "1"
22. }

复制代码

start_position

1. 在filebeat中如果我们删除了filebeat的json文件，filebeat下一次采集从头开始，对于logstash来说，并不是这样
3. [root@elk3 ~]# rm -f /usr/share/logstash/data/plugins/inputs/file/.sincedb_782d533684abe27068ac85b78871b9fd
4. [root@elk3 ~]# logstash -f /etc/logstash/conf.d/02-file-to-stdout.conf
5. # 还是默认采集最后面的数据
6. [root@elk3 ~]# echo 123 >> /tmp/student.txt
7. [root@elk3 ~]# cat /tmp/student.txt
8. ABC
9. 2025def
10. 123
11. {
12.       "@version" => "1",
13.     "@timestamp" => 2025-03-26T02:26:17.008Z,
14.        "message" => "123",
15.           "host" => "elk3",
16.           "path" => "/tmp/student.txt"
17. }
19. // 这时就需要star_position参数
20. start_position
21. Value can be any of: beginning, end
22. Default value is "end"
24. [root@elk3 ~]# cat /etc/logstash/conf.d/02-file-to-stdout.conf
25. input {
26.   file {
27.     path => "/tmp/student.txt"
28.     start_position => "beginning"
29.   }
30. }
33. output {
34.   stdout {
35.     codec => rubydebug
36.   }
37. }
40. [root@elk3 ~]# rm -f /usr/share/logstash/data/plugins/inputs/file/.sincedb_782d533684abe27068ac85b78871b9fd
41. [root@elk3 ~]# logstash -f /etc/logstash/conf.d/02-file-to-stdout.conf
42. {
43.       "@version" => "1",
44.           "host" => "elk3",
45.        "message" => "2025def",
46.           "path" => "/tmp/student.txt",
47.     "@timestamp" => 2025-03-26T02:31:50.020Z
48. }
49. {
50.       "@version" => "1",
51.           "host" => "elk3",
52.        "message" => "ABC",
53.           "path" => "/tmp/student.txt",
54.     "@timestamp" => 2025-03-26T02:31:49.813Z
55. }
56. {
57.       "@version" => "1",
58.           "host" => "elk3",
59.        "message" => "123",
60.           "path" => "/tmp/student.txt",
61.     "@timestamp" => 2025-03-26T02:31:50.037Z
62. }

复制代码

filter plugins

1. # logstash的输出有很多的字段，如果有一些我不想要，可以使用filter plugins进行过滤
2. # 移除@version字段
3. [root@elk3 ~]# cat /etc/logstash/conf.d/02-file-to-stdout.conf
4. input {
5.   file {
6.     path => "/tmp/student.txt"
7.     start_position => "beginning"
8.   }
9. }
10. filter {
11.   mutate {
12.     remove_field => [ "@version" ]
13.   }
14. }
16. output {
17.   stdout {
18.     codec => rubydebug
19.   }
20. }
22. # -r模式启动logstash 可以实现重载
23. [root@elk3 ~]# logstash -r -f /etc/logstash/conf.d/02-file-to-stdout.conf
24. {
25.     "@timestamp" => 2025-03-26T03:01:02.078Z,
26.           "host" => "elk3",
27.        "message" => "111",
28.           "path" => "/tmp/student.txt"
29. }

复制代码

logstash架构
logstash多实例

1. 启动实例1：
2. [root@elk93 ~]# logstash -f /etc/logstash/conf.d/01-stdin-to-stdout.conf
5. 启动实例2：
6. [root@elk93 ~]# logstash -rf /etc/logstash/conf.d/02-file-to-stdout.conf  --path.data /tmp/logstash-multiple

复制代码

logstash与pipeline关系

1. - 一个Logstash实例可以有多个pipeline，若没有定义pipeline id，则默认为main pipeline。
2.        
3.        
4.         - 每个pipeline都有三个组件组成，其中filter插件是可选组件:
5.                 - input :
6.                         数据从哪里来 。
7.                        
8.                 - filter:
9.                         数据经过哪些插件处理，该组件是可选组件。
10.                        
11.                 - output:
12.                         数据到哪里去。

复制代码

logstash采集nginx日记

1. 1.安装nginx
2. [root@elk3 ~]# apt install -y nginx
5. 2.logstash采集nginx
6. [root@elk3 ~]# cat /etc/logstash/conf.d/03-nginx-grok.conf
7. input {
8.   file {
9.     path => "/var/log/nginx/access.log"
10.     start_position => "beginning"
11.   }
12. }
13. filter {
14.   mutate {
15.     remove_field => [ "@version" ]
16.   }
17. }
19. output {
20.   stdout {
21.     codec => rubydebug
22.   }
23. }
24. [root@elk3 ~]# logstash -r -f /etc/logstash/conf.d/03-nginx-grok.conf
25. {
26.           "host" => "elk3",
27.        "message" => "127.0.0.1 - - [26/Mar/2025:14:43:58 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"",
28.     "@timestamp" => 2025-03-26T06:45:24.375Z,
29.           "path" => "/var/log/nginx/access.log"
30. }
31. {
32.           "host" => "elk3",
33.        "message" => "127.0.0.1 - - [26/Mar/2025:14:43:57 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"",
34.     "@timestamp" => 2025-03-26T06:45:24.293Z,
35.           "path" => "/var/log/nginx/access.log"
36. }
37. {
38.           "host" => "elk3",
39.        "message" => "127.0.0.1 - - [26/Mar/2025:14:43:58 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"",
40.     "@timestamp" => 2025-03-26T06:45:24.373Z,
41.           "path" => "/var/log/nginx/access.log"
42. }

复制代码

grok plugins

1. # 基于正则提取
2. Logstash ships with about 120 patterns by default. You can find them here: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns.
4. [root@elk3 ~]# cat  /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.3.4/patterns/legacy/httpd
5. HTTPDUSER %{EMAILADDRESS}|%{USER}
6. HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
8. # Log formats
9. HTTPD_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" (?:-|%{NUMBER:response}) (?:-|%{NUMBER:bytes})
10. HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}
12. # Error logs
13. HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
14. HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[(?:%{WORD:module})?:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
15. HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
17. # Deprecated
18. COMMONAPACHELOG %{HTTPD_COMMONLOG}
19. COMBINEDAPACHELOG %{HTTPD_COMBINEDLOG}
22. # 配置logstash
23. [root@elk3 ~]# cat /etc/logstash/conf.d/03-nginx-grok.conf
24. input {
25.   file {
26.     path => "/var/log/nginx/access.log"
27.     start_position => "beginning"
28.   }
29. }
30. filter {
31.   mutate {
32.     remove_field => [ "@version" ]
33.   }
34.   # 基于正则提取任意文本，并将其封装为一个特定的字段。使用设置好的模板
35.   grok {
36.         match => { "message" => "%{HTTPD_COMMONLOG}" }
37.   }
38. }
40. output {
41.   stdout {
42.     codec => rubydebug
43.   }
44. }
46. [root@elk3 ~]# logstash -r -f /etc/logstash/conf.d/03-nginx-grok.conf
47. {
48.         "message" => "192.168.121.1 - - [26/Mar/2025:14:52:06 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Mobile Safari/537.36"",
49.            "path" => "/var/log/nginx/access.log",
50.         "request" => "/",
51.        "clientip" => "192.168.121.1",
52.            "host" => "elk3",
53.       "timestamp" => "26/Mar/2025:14:52:06 +0800",
54.            "auth" => "-",
55.            "verb" => "GET",
56.        "response" => "200",
57.           "ident" => "-",
58.     "httpversion" => "1.1",
59.      "@timestamp" => 2025-03-26T06:52:07.342Z,
60.           "bytes" => "396"
61. }

复制代码

useragent plugins

1. 用于提取用户的设备信息
3. [root@elk3 ~]# cat /etc/logstash/conf.d/03-nginx-grok.conf
4. input {
5.   file {
6.     path => "/var/log/nginx/access.log"
7.     start_position => "beginning"
8.   }
9. }
10. filter {
11.   mutate {
12.     remove_field => [ "@version" ]
13.   }
14.   grok {
15.         match => { "message" => "%{HTTPD_COMMONLOG}" }
16.   }
17.   useragent {
18.           # 指定从哪个字段解析用户设备信息
19.     source => 'message'
20.     # 将解析的结果存储到某个特定字段，若不指定，则默认放在顶级字段。
21.     target => "xu-ua"
22.   }      
24. }
26. [root@elk3 ~]# logstash -r -f /etc/logstash/conf.d/03-nginx-grok.conf
27. output {
28.   stdout {
29.     codec => rubydebug
30.   }
31. }
33. {
34.         "message" => "192.168.121.1 - - [26/Mar/2025:16:45:10 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 13; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Mobile Safari/537.36"",
35.        "clientip" => "192.168.121.1",
36.       "timestamp" => "26/Mar/2025:16:45:10 +0800",
37.         "request" => "/",
38.           "bytes" => "396",
39.            "verb" => "GET",
40.     "httpversion" => "1.1",
41.      "@timestamp" => 2025-03-26T08:45:11.587Z,
42.            "host" => "elk3",
43.            "auth" => "-",
44.           "xu-ua" => {
45.               "name" => "Chrome Mobile",
46.            "version" => "134.0.0.0",
47.                 "os" => "Android",
48.            "os_name" => "Android",
49.         "os_version" => "13",
50.             "device" => "Samsung SM-G981B",
51.            "os_full" => "Android 13",
52.              "minor" => "0",
53.           "os_major" => "13",
54.              "patch" => "0",
55.              "major" => "134"
56.     },
57.           "ident" => "-",
58.            "path" => "/var/log/nginx/access.log",
59.        "response" => "200"
60. }

复制代码

geoip plugins

1. 基于公网IP地址分析你的经纬度坐标点
3. [root@elk3 ~]# cat /etc/logstash/conf.d/03-nginx-grok.conf
4. input {
5.   file {
6.     path => "/var/log/nginx/access.log"
7.     start_position => "beginning"
8.   }
9. }
10. filter {
11.   mutate {
12.     remove_field => [ "@version" ]
13.   }
14.   grok {
15.         match => { "message" => "%{HTTPD_COMMONLOG}" }
16.   }
17.   useragent {
18.     source => 'message'
19.     target => "xu-ua"
20.   }      
21.   geoip {
22.         source => "clientip"
23.   }
25. }
26. [root@elk3 ~]# logstash -r -f /etc/logstash/conf.d/03-nginx-grok.conf
27. output {
28.   stdout {
29.     codec => rubydebug
30.   }
31. }
33. "geoip" => {
34.              "longitude" => -119.705,
35.          "country_code2" => "US",
36.            "region_name" => "Oregon",
37.               "timezone" => "America/Los_Angeles",
38.                     "ip" => "52.222.36.125",
39.         "continent_code" => "NA",
40.          "country_code3" => "US",
41.               "latitude" => 45.8401,
42.           "country_name" => "United States",
43.               "dma_code" => 810,
44.            "postal_code" => "97818",
45.            "region_code" => "OR",
46.               "location" => {
47.             "lat" => 45.8401,
48.             "lon" => -119.705
49.         },
50.              "city_name" => "Boardman"
51.     }

复制代码

date plugins

1. [root@elk3 ~]# cat /etc/logstash/conf.d/03-nginx-grok.conf
2. input {
3.   file {
4.     path => "/var/log/nginx/access.log"
5.     start_position => "beginning"
6.   }
7. }
8. filter {
9.   mutate {
10.     remove_field => [ "@version" ]
11.   }
12.   grok {
13.         match => { "message" => "%{HTTPD_COMMONLOG}" }
14.   }
15.   useragent {
16.     source => 'message'
17.     target => "xu-ua"
18.   }      
19.   geoip {
20.         source => "clientip"
21.   }
22.   date {
23.         # 匹配日期字段，将其转换为日期格式，将来存储到ES，基于官方的示例对号入座对应的格式即可。
24.     # https://www.elastic.co/guide/en/logstash/7.17/plugins-filters-date.html#plugins-filters-date-match
25.     match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
26.     # 将match匹配的日期修改后的值直接覆盖到指定字段，若不定义，则默认覆盖"@timestamp"。
27.     target => "xu-timestamp"
28.   }
30. }
32. output {
33.   stdout {
34.     codec => rubydebug
35.   }
36. }
38. [root@elk3 ~]# logstash -r -f /etc/logstash/conf.d/03-nginx-grok.conf
39. "xu-timestamp" => 2025-03-26T09:17:18.000Z,

复制代码

mutate plugins

1. 如果我们想统计带宽 我们会发现 "bytes" => "396"
2. 是字符串类型，不能累加，所以要使用mutate plugins转换类型
4. [root@elk3 ~]# cat /etc/logstash/conf.d/03-nginx-grok.conf
5. input {
6.   file {
7.     path => "/var/log/nginx/access.log"
8.     start_position => "beginning"
9.   }
10. }
11. filter {
12.   mutate {
13.     convert => {
14.       "bytes" => "integer"
15.     }
16.     remove_field => [ "@version" ]
17.   }
18.   grok {
19.         match => { "message" => "%{HTTPD_COMMONLOG}" }
20.   }
21.   useragent {
22.     source => 'message'
23.     target => "xu-ua"
24.   }      
25.   geoip {
26.         source => "clientip"
27.   }
28.   date {
29.     match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
30.     target => "xu-timestamp"
31.   }
33. }
35. output {
36.   stdout {
37.     codec => rubydebug
38.   }
39. }

复制代码

logstash 采集日记输出到es

1. [root@elk3 ~]# cat /etc/logstash/conf.d/08-nginx-to-es.conf
2. input {
3.   file {
4.     path => "/var/log/nginx/access.log"
5.     start_position => "beginning"
6.   }
7. }
10. filter {
11.   grok {
12.     match => { "message" => "%{HTTPD_COMMONLOG}" }
13.   }
15.   useragent {
16.     source => "message"
17.     target => "xu_user_agent"
18.   }
20.   geoip {
21.     source => "clientip"
22.   }
24.   date {
25.     match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
26.     target => "xu-timestamp"
27.   }
29.   # 对指定字段进行转换处理
30.   mutate {
31.     # 将指定字段转换成我们需要转换的类型
32.     convert => {
33.       "bytes" => "integer"
34.     }
35.    
36.     remove_field => [ "@version","host","message" ]
37.   }
38. }
40. output {
41.   stdout {
42.     codec => rubydebug
43.   }
45.   elasticsearch {
46.       # 对应的ES集群主机列表
47.       hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]
48.       # 对应的ES集群的索引名称
49.       index => "xu-elk-nginx"
50.   }
51. }
53. 存在的问题:
54.         Failed (timed out waiting for connection to open). Sleeping for 0.02
56. 问题描述:
57.         此问题在 ElasticStack 7.17.28版本中，可能会出现Logstash无法写入ES的情况。
58.        
59. TODO:
60.         需要调研官方是否做了改动，导致无法写入成功，需要额外的参数配置。
62. 临时解决方案:
63.         - 回退版本到7.17.23版本。
64.         - 注释掉geoip的配置

复制代码

解决在写入es时geoip plugins识别时间过长问题

1. 通过查看官网，我们可以看到geoip模块可以指定数据库， 我们通过指定数据库的方式来解决这个问题
3. 1.查看Logstash本地默认的geoip插件
4. [root@elk3 ~]# tree /usr/share/logstash/data/plugins/filters/geoip/1742980310/
5. /usr/share/logstash/data/plugins/filters/geoip/1742980310/
6. ├── COPYRIGHT.txt
7. ├── elastic-geoip-database-service-agreement-LICENSE.txt
8. ├── GeoLite2-ASN.mmdb
9. ├── GeoLite2-ASN.tgz
10. ├── GeoLite2-City.mmdb
11. ├── GeoLite2-City.tgz
12. ├── LICENSE.txt
13. └── README.txt
15. 0 directories, 8 files
17. 2.配置logstash
18. [root@elk3 ~]# cat /etc/logstash/conf.d/04-nginx-to-es.conf
19. input {
20.   file {
21.     path => "/var/log/nginx/access.log"
22.     start_position => "beginning"
23.   }
24. }
26. filter {
27.   mutate {
28.     convert => {
29.       "bytes" => "integer"
30.     }
31.     remove_field => [ "@version" ]
32.   }
33.   grok {
34.         match => { "message" => "%{HTTPD_COMMONLOG}" }
35.   }
36.   useragent {
37.     source => 'message'
38.     target => "xu-ua"
39.   }      
40.   geoip {
41.         source => "clientip"
42.         database => "/usr/share/logstash/data/plugins/filters/geoip/1742980310/GeoLite2-City.mmdb"
43.   }
44.   date {
45.     match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
46.     target => "xu-timestamp"
47.   }
49. }
51. output {
52.   stdout {
53.     codec => rubydebug
54.   }
55.   elasticsearch {
56.         index => "xu-logstash"
57.         hosts => ["http://192.168.121.91:9200","http://192.168.121.92:9200","http://192.168.121.93:9200"]
58.   }
59. }
60. [root@elk3 ~]#

复制代码

解决geoip.location数据类型不正确问题
这时的经纬度是float类型，是不能出图的
在kibana中创建索引模板
ELFK架构
json插件案例

1. graph LR
2. filebeat--->|发送|logstash
4. 让logstash接收filebeat收集到的数据，logstash优先于filebeat启动
5. 也就是logstash中的input plugins type 是beats
7. # 配置logstash
8. [root@elk3 ~]# grep -v "^#" /etc/logstash/conf.d/05-beat-es.conf
9. input {
10.   beats {
11.       port => 5044
12.   }
13. }
15. filter {
16.   mutate {
17.     remove_field => [ "@version","host","agent","ecs","tags","input","log" ]
18.   }
19.   json {
20.      source => "message"
21.    }
22. }
24. output {
25.   stdout {
26.     codec => rubydebug
27.   }
28. }
31. # 配置filebeat
32. [root@elk1 ~]# cat /etc/filebeat/config/05-json.yaml
33. filebeat.inputs:
34. - type: filestream
35.   paths:
36.     - /tmp/student.json
37. output.logstash:
38.   hosts: ["192.168.121.93:5044"]
40. # 先启动logstash，在启动filebeat
41. [root@elk3 conf.d]# logstash -rf 05-beat-es.conf
42. [root@elk3 ~]# netstat -tunlp | grep 5044
43. tcp6       0      0 :::5044                 :::*                    LISTEN      120181/java   
45. # 在启动filebeat
46. [root@elk1 ~]# filebeat  -e -c /etc/filebeat/config/05-json.yaml
49. // 准备测试数据
50. {
51.   "name":"aaa",
52.   "hobby":["写小说","唱歌"]
53. }
54. {
55.   "name":"bbb",
56.   "hobby":["健身","台球","打豆豆"]
57. }
58. {
59.   "name":"ccc",
60.   "hobby":["乒乓球","游泳","游戏"]
61. }
62. {
63.    "name": "ddd",
64.    "hobby": ["打游戏","打篮球"]
65. }
67. # 查看采集结果，由于filebeat的采集规则是按行采集，就导致我们准备的一条数据它采集出来了多条
68. "message" => "   "name": "ddd",",
69. "message" => "   "hobby": ["打游戏","打篮球"]",
70. ...
72. # 需要使用filebeat的多行合并进行处理
73. [root@elk1 ~]# cat /etc/filebeat/config/05-json.yaml
74. filebeat.inputs:
75. - type: filestream
76.   paths:
77.     - /tmp/student.json
78.   parsers:
79.     - multiline:
80.         type: count
81.         count_lines: 4
82. output.logstash:
83.   hosts: ["192.168.121.93:5044"]
85. # 查看数据采集情况
86. {
87.        "message" => "{\n  "name":"aaa",\n  "hobby":["写小说","唱歌"]\n}",
88.           "name" => "aaa",
89.          "hobby" => [
90.         [0] "写小说",
91.         [1] "唱歌"
92.     ],
93.     "@timestamp" => 2025-03-27T07:46:14.390Z
94. }

复制代码

写入es

1. [root@elk3 ~]# grep -v "^#" /etc/logstash/conf.d/05-beat-es.conf
2. input {
3.   beats {
4.       port => 5044
5.   }
6. }
8. filter {
9.   mutate {
10.     remove_field => [ "@version","host","agent","ecs","tags","input","log" ]
11.   }
12.   json {
13.      source => "message"
14.    }
15. }
17. output {
18.   stdout {
19.     codec => rubydebug
20.   }
21.   elasticsearch {
22.         hosts => ["http://192.168.121.91:9200"]
23.   }
24. }

复制代码

ELFK架构梳理之电商指标分享项目案例

1. 1.生成测试数据
2. [root@elk1 ~]# cat gen-log.py
3. #!/usr/bin/env python
4. # -*- coding: UTF-8 -*-
5. # @author : Jason Yin
7. import datetime
8. import random
9. import logging
10. import time
11. import sys
13. LOG_FORMAT = "%(levelname)s %(asctime)s [com.oldboyedu.%(module)s] - %(message)s "
14. DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
16. # 配置root的logging.Logger实例的基本配置
17. logging.basicConfig(level=logging.INFO, format=LOG_FORMAT, datefmt=DATE_FORMAT, filename=sys.argv[1]
18. , filemode='a',)
19. actions = ["浏览页面", "评论商品", "加入收藏", "加入购物车", "提交订单", "使用优惠券", "领取优惠券",
20. "搜索", "查看订单", "付款", "清空购物车"]
22. while True:
23.     time.sleep(random.randint(1, 5))
24.     user_id = random.randint(1, 10000)
25.     # 对生成的浮点数保留2位有效数字.
26.     price = round(random.uniform(15000, 30000),2)
27.     action = random.choice(actions)
28.     svip = random.choice([0,1,2])
29.     logging.info("DAU|{0}|{1}|{2}|{3}".format(user_id, action,svip,price))
30. [root@elk1 ~]#  python3 gen-log.py /tmp/apps.log
32. 2.查看数据内容
33. [root@elk1 ~]# tail -f /tmp/apps.log
34. ...
35. INFO 2025-03-27 17:03:10 [com.oldboyedu.gen-log] - DAU|7973|加入购物车|0|19300.65
36. INFO 2025-03-27 17:03:13 [com.oldboyedu.gen-log] - DAU|8617|加入购物车|2|19720.57
37. INFO 2025-03-27 17:03:14 [com.oldboyedu.gen-log] - DAU|6879|搜索|2|24774.85
38. INFO 2025-03-27 17:03:19 [com.oldboyedu.gen-log] - DAU|804|付款|2|21352.22
39. INFO 2025-03-27 17:03:22 [com.oldboyedu.gen-log] - DAU|3014|清空购物车|0|19908.62
40. ...
42. # 启动logstash实例
43. [root@elk3 conf.d]# cat 06-beats_apps-to-es.conf
44. input {
45.   beats {
46.       port => 9999
47.   }
48. }
49. filter {
50. mutate {
51.     split => { "message" => "|" }
53.     add_field => {
54.       "other" => "%{[message][0]}"
55.       "userId" => "%{[message][1]}"
56.       "action" => "%{[message][2]}"
57.       "svip" => "%{[message][3]}"
58.       "price" => "%{[message][4]}"
59.     }
60. }
61. mutate{
62.         split => { "other" => " " }
64.     add_field => {
65.        datetime => "%{[other][1]} %{[other][2]}"
66.     }
67.    
68.     convert => {
69.        "price" => "float"
70.      }
71.         remove_field => [ "@version","host","agent","ecs","tags","input","log","message","other"]
72.   }
73. }
74. output {
75. #  stdout {
76. #    codec => rubydebug
77. #  }
79.   elasticsearch {
80.      index => "linux96-logstash-elfk-apps"
81.      hosts => ["http://192.168.121.91:9200","http://192.168.121.92:9200","http://192.168.121.93:9200"]
82.   }
83. }
84. # 启动filebeat实例
85. [root@elk1 ~]# cat /etc/filebeat/config/06-filestream-to-logstash.yml
86. filebeat.inputs:
87. - type: filestream
88.   paths:
89.     - /tmp/apps.log
91. output.logstash:
92.   hosts: ["192.168.121.93:9999"]

复制代码

ELK架构
logstash if语句

1. 在logstash中支持if语句，假如有多个input，可以通过if来对不同的input做不同的过滤，以及不同的输出
3. # 配置logstash if
4. [root@elk3 ~]# cat /etc/logstash/conf.d/08-logstash-if.conf
5. input {
6.         beats {
7.                 port => 9999
8.                 type => "xu-filebeat"
9.         }
10.         file {
11.                 path => "/var/log/nginx/access.log"
12.                 start_position => "beginning"
13.             type => "xu-file"
14.         }
15.         tcp {
16.                 port => 8888
17.             type => "xu-tcp"
18.         }
19. }
21. fileter {
22.         if [type] == "xu-tcp" {
23.            mutate {
24.        add_field => {
25.           school => "school1"
26.           class => "one"
27.        }
28.        remove_field => [ "@version","port"]
29.      }
30.         } else if [type] == "xu-filebeat" {
31.                 mutate {
32.         split => { "message" => "|" }
34.         add_field => {
35.           "other" => "%{[message][0]}"
36.           "userId" => "%{[message][1]}"
37.           "action" => "%{[message][2]}"
38.           "svip" => "%{[message][3]}"
39.           "price" => "%{[message][4]}"
40.           "address" => "1.1.1.1"
41.         }
43.       }
45.       mutate {
47.         split => { "other" => " " }
49.         add_field => {
50.            datetime => "%{[other][1]} %{[other][2]}"
51.         }
52.         
53.         convert => {
54.            "price" => "float"
55.          }
57.         remove_field => [ "@version","host","agent","ecs","tags","input","log","message","other"]
58.       }
60.       date {
61.         match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ]
62.       }
63.         } else {
64.                 grok {
65.         match => { "message" => "%{HTTPD_COMMONLOG}" }
66.       }
67.    
68.       useragent {
69.         source => "message"
70.         target => "xu_user_agent"
71.       }
72.    
73.       geoip {
74.         source => "clientip"
75.         database => "/usr/share/logstash/data/plugins/filters/geoip/CC/GeoLite2-City.mmdb"
76.       }
77.    
78.       date {
79.         match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
80.         target => "xu-timestamp"
81.       }
82.      
83.       mutate {
84.         convert => {
85.           "bytes" => "integer"
86.         }
87.         
88.         add_field => {
89.            office => "https://studylinux.cn"
90.         }
92.         remove_field => [ "@version","host","message" ]
93.       }
94.         }
95. }
97. output {
98.         if [type] == "xu-filebeat" {
99.       elasticsearch {
100.          index => "xu-logstash-if-filebeat"
101.          hosts => ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
102.       }
103.   } else if [type] == "xu-tcp" {
104.       elasticsearch {
105.          index => "xu-logstash-if-tcp"
106.          hosts => ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
107.       }
109.   }else {
110.       elasticsearch {
111.          index => "xu-logstash-if-file"
112.          hosts => ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
113.       }
114.   }
115. }
116. [root@elk3 ~]#

复制代码

pipeline

1. # pipline配置文件位置
2. [root@elk3 ~]# ll /etc/logstash/pipelines.yml
3. -rw-r--r-- 1 root root 285 Feb 18 18:52 /etc/logstash/pipelines.yml
5. # 修改pipline配置文件
6. [root@elk3 ~]# tail -4 /etc/logstash/pipelines.yml
7. - pipeline.id: xixi
8.   path.config: "/etc/logstash/conf.d/01-file-to-stdout.conf"
9. - pipeline.id: haha
10.   path.config: "/etc/logstash/conf.d/03-nginx-grok.conf"
12. # 启动logstash，可以通过logstash -r直接启动不需要再指定配置文件了
13. [root@elk3 ~]# logstash  -r
14.         # 直接会有一个报错 ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
15.         # logstash默认会去 /usr/share/logstash/config/pipelines.yml找这个文件，此时我们做一个软链接即可
17. # 配置软链接
18. [root@elk3 ~]# mkdir /usr/share/logstash/config/
19. [root@elk3 ~]# ln -svf /etc/logstash/pipelines.yml  /usr/share/logstash/config/
20. '/usr/share/logstash/config/pipelines.yml' -> '/etc/logstash/pipelines.yml'
21. [root@elk3 ~]# logstash  -r
22. ...
23. [INFO ] 2025-03-29 10:16:50.372 [[xixi]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"xixi"}
24. [INFO ] 2025-03-29 10:16:54.380 [[haha]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"haha"}
25. ...

复制代码

ES集群安全
基于base_auth加密
es集群加密

1. # 在配置加密之前是可以正常访问的
2. [root@elk1 ~]# curl 127.1:9200/_cat/nodes
3. 192.168.121.92  6 94 0 0.05 0.03 0.00 cdfhilmrstw - elk2
4. 192.168.121.91 22 95 4 0.25 0.27 0.25 cdfhilmrstw * elk1
5. 192.168.121.93 29 94 6 0.02 0.25 0.48 cdfhilmrstw - elk3
8. 1 生成证书文件
9. [root@elk1 ~]# /usr/share/elasticsearch/bin/elasticsearch-certutil cert -out /etc/elasticsearch/elastic-certificates.p12 -pass ""
10. ...
11. Certificates written to /etc/elasticsearch/elastic-certificates.p12
13. This file should be properly secured as it contains the private key for
14. your instance.
16. This file is a self contained file and can be copied and used 'as is'
17. For each Elastic product that you wish to configure, you should copy
18. this '.p12' file to the relevant configuration directory
19. and then follow the SSL configuration instructions in the product guide.
22. 2.把证书文件拷贝到其他节点
23. [root@elk1 ~]# chmod 640 /etc/elasticsearch/elastic-certificates.p12
24. [root@elk1 ~]# scp -r /etc/elasticsearch/elastic-certificates.p12  192.168.121.92:/etc/elasticsearch/elastic-certificates.p12
25. [root@elk1 ~]# scp - /etc/elasticsearch/elastic-certificates.p12  192.168.121.93:/etc/elasticsearch/elastic-certificates.p12
27. 3.修改ES集群的配置文件，并同步到所有节点
28. [root@elk1 ~]# tail -5 /etc/elasticsearch/elasticsearch.yml
29. xpack.security.enabled: true
30. xpack.security.transport.ssl.enabled: true
31. xpack.security.transport.ssl.verification_mode: certificate
32. xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
33. xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
34. [root@elk1 ~]# scp /etc/elasticsearch/elasticsearch.yml  192.168.121.92:/etc/elasticsearch/elasticsearch.yml
35. [root@elk1 ~]# scp /etc/elasticsearch/elasticsearch.yml  192.168.121.93:/etc/elasticsearch/elasticsearch.yml
37. 4.重启es
38. [root@elk1 ~]# systemctl restart elasticsearch.service
40. # 此时就不能直接访问了
41. [root@elk1 ~]# curl 127.1:9200
42. {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
45. 5.生成随机密码
46. [root@elk1 ~]# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords  auto
48. Changed password for user apm_system
49. PASSWORD apm_system = aBsQ3WI9ydUVTx2hk2JT
51. Changed password for user kibana_system
52. PASSWORD kibana_system = xoMBWbFyYmadDyrYcwyI
54. Changed password for user kibana
55. PASSWORD kibana = xoMBWbFyYmadDyrYcwyI
57. Changed password for user logstash_system
58. PASSWORD logstash_system = fWx19jXFHinpcraglh8E
60. Changed password for user beats_system
61. PASSWORD beats_system = NgKipgH0LfnFGFAazun6
63. Changed password for user remote_monitoring_user
64. PASSWORD remote_monitoring_user = Af4hu6PrhPYvn2S5zcEj
66. Changed password for user elastic
67. PASSWORD elastic = 0Nj2dpMTSNYurPqQHInA
70. [root@elk1 ~]# curl -u elastic:MSfRhWKA3lRhufYpxF9u 127.1:9200/_cat/nodes
71. 192.168.121.91 40 96 22 0.62 0.74 0.53 cdfhilmrstw - elk1
72. 192.168.121.92 17 96 20 0.44 0.67 0.36 cdfhilmrstw * elk2
73. 192.168.121.93 23 96 32 0.54 1.00 0.73 cdfhilmrstw - elk3
76. 6.kibana连接es
77.         6.1修改kibana配置文件
78. [root@elk1 ~]# tail -2 /etc/kibana/kibana.yml
79. elasticsearch.username: "kibana_system"
80. elasticsearch.password: "47UD4ZOypuWO100QciH4"
81.         6.2重启kibana
82. [root@elk1 ~]# systemctl restart kibana.service
83.         6.3web访问kibana

复制代码

重置es密码

1. 在es集群中有类似root用户的superuser的角色，我们可以创建一个用户属于superuser的角色，通过这个用户去修改elastic的密码
3. 1.创建一个超级管理员角色
4. [root@elk1 ~]# /usr/share/elasticsearch/bin/elasticsearch-users useradd xu -p 123456 -r superuser
6. 2.基于管理员修改密码
7. [root@elk1 ~]# curl -s --user xu:123456 -XPUT "http://localhost:9200/_xpack/security/user/elastic/_password?pretty" -H 'Content-Type: application/json' -d'
8.      {
9.        "password" : "654321"
10.      }'
11.      
12. [root@elk1 ~]# curl -uelastic:654321 127.1:9200/_cat/nodes
13. 192.168.121.91 35 96 7 0.38 0.43 0.52 cdfhilmrstw - elk1
14. 192.168.121.92 20 96 2 0.20 0.20 0.25 cdfhilmrstw * elk2
15. 192.168.121.93 27 97 5 0.10 0.18 0.38 cdfhilmrstw - elk3

复制代码

filebeat对接es加密

1. [root@elk1 ~]# cat /etc/filebeat/config/07-tcp-to-es_tls.yaml
2. filebeat.inputs:
3. - type: tcp
4.   host: "0.0.0.0:9000"
7. output.elasticsearch:
8.   hosts:
9.   - 192.168.121.91:9200
10.   - 192.168.121.92:9200
11.   - 192.168.121.93:9200
12.   # 指定连接ES集群的用户名
13.   username: "elastic"
14.   # 指定连接ES集群的密码
15.   password: "654321"
16.   index: xu-es-tls-filebeat
18. setup.ilm.enabled: false
19. setup.template.name: "xu-es-tls-filebeat"
20. setup.template.pattern: "xu-es-tls-filebeat-*"
21. setup.template.overwrite: true
22. setup.template.settings:
23.   index.number_of_shards: 3
24.   index.number_of_replicas: 0

复制代码

logstash对接es加密

1. [root@elk3 ~]# cat /etc/logstash/conf.d/09-tcp-to-es_tls.conf
2. input {
3.   tcp {
4.     port => 8888
5.   }
6. }  
8. output {
10.   elasticsearch {
11.     hosts => ["192.168.121.91:9200","192.168.121.92:9200","192.168.121.93:9200"]
12.     index => "oldboyedu-logstash-tls-es"
13.     user => elastic
14.     password => "654321"
15.   }
16. }

复制代码

api-key

1. 为什么要启用api-key
2.         为了安全性，使用用户名和密码的方式进行认证会暴露用户信息。
3.         ElasticSearch也支持api-key的方式进行认证。这样就可以保证安全性。api-key是不能用于登录kibana，安全性得到保障。
4.         而且可以基于api-key实现权限控制。
6. elasticsearch默认是没有启动api的，需要通过配置文件设置并启动api功能

复制代码

启动es api功能

1. [root@elk1 ~]# tail /etc/elasticsearch/elasticsearch.yml
2. # 启用api_key功能
3. xpack.security.authc.api_key.enabled: true
4. # 指定API密钥加密算法
5. xpack.security.authc.api_key.hashing.algorithm: pbkdf2
6. # 缓存的API密钥时间
7. xpack.security.authc.api_key.cache.ttl: 1d
8. # API密钥保存数量的上限
9. xpack.security.authc.api_key.cache.max_keys: 10000
10. # 用于内存中缓存的API密钥凭据的哈希算法
11. xpack.security.authc.api_key.cache.hash_algo: ssha256
13. [root@elk1 ~]# !scp
14. scp /etc/elasticsearch/elasticsearch.yml  192.168.121.93:/etc/elasticsearch/elasticsearch.yml
15. root@192.168.121.93's password:
16. elasticsearch.yml                                                                                                                                          100% 4270   949.6KB/s   00:00   
17. [root@elk1 ~]# scp /etc/elasticsearch/elasticsearch.yml  192.168.121.92:/etc/elasticsearch/elasticsearch.yml
18. root@192.168.121.92's password:
19. elasticsearch.yml
21. [root@elk1 ~]# systemctl restart elasticsearch.service

复制代码

创建api

1. # 解析api
2. [root@elk1 ~]# echo "TzBCTzY1VUJiWUdnVHlBNjZRTXc6eE9JWW9wT3dTT09Sam1UNE5RYnRjUQ==" | base64 -d ;echo
3. O0BO65UBbYGgTyA66QMw:xOIYopOwSOORjmT4NQbtcQ
6. # 配置filebeat
7. [root@elk1 ~]# cat /etc/filebeat/config/07-tcp-to-es_tls.yaml
8. filebeat.inputs:
9. - type: tcp
10.   host: "0.0.0.0:9000"
13. output.elasticsearch:
14.   hosts:
15.   - 192.168.121.91:9200
16.   - 192.168.121.92:9200
17.   - 192.168.121.93:9200
18.   #username: "elastic"
19.   #password: "654321"
20.   api_key: zvWA4JUBqFmHNaf3P8bM:d-goeFONRPelMuRxSr2Bxg
21.   index: xu-es-tls-filebeat
23. setup.ilm.enabled: false
24. setup.template.name: "xu-es-tls-filebeat"
25. setup.template.pattern: "xu-es-tls-filebeat-*"
26. setup.template.overwrite: true
27. setup.template.settings:
28.   index.number_of_shards: 3
29.   index.number_of_replicas: 0
30. [root@elk1 ~]# filebeat -e -c /etc/filebeat/config/07-tcp-to-es_tls.yaml

复制代码

基于ES的api创建api-key并实现权限管理
参考链接:
https://www.elastic.co/guide/en/beats/filebeat/7.17/beats-api-keys.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-privileges.html#privileges-list-cluster
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-privileges.html#privileges-list-indices

1. 1.创建api-key
2. # 发送请求
3. POST /_security/api_key
4. {
5.   "name": "jasonyin2020",
6.   "role_descriptors": {
7.     "filebeat_monitoring": {
8.       "cluster": ["all"],
9.       "index": [
10.         {
11.           "names": ["xu-es-apikey*"],
12.           "privileges": ["create_index", "create"]
13.         }
14.       ]
15.     }
16.   }
17. }
18. # 返回数据
19. {
20.   "id" : "0vXs4ZUBqFmHNaf3s8Zn",
21.   "name" : "jasonyin2020",
22.   "api_key" : "y1Vi5fL6RfGy_B47YWBXcw",
23.   "encoded" : "MHZYczRaVUJxRm1ITmFmM3M4Wm46eTFWaTVmTDZSZkd5X0I0N1lXQlhjdw=="
24. }
27. # 解析
28. [root@elk1 ~]# echo MHZYczRaVUJxRm1ITmFmM3M4Wm46eTFWaTVmTDZSZkd5X0I0N1lXQlhjdw== | base64 -d  ;echo
29. 0vXs4ZUBqFmHNaf3s8Zn:y1Vi5fL6RfGy_B47YWBXcw

复制代码

https
es集群配置https

1. 1.自建CA证书
2. [root@elk1 ~]# /usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /etc/elasticsearch/elastic-stack-ca.p12 --pass ""
3. [root@elk1 ~]# ll /etc/elasticsearch/elastic-stack-ca.p12
4. -rw------- 1 root elasticsearch 2672 Mar 29 20:44 /etc/elasticsearch/elastic-stack-ca.p12
7. 2.基于CA证书创建ES证书
8. [root@elk1 ~]# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/elastic-stack-ca.p12 --out /etc/elasticsearch/elastic-certificates-https.p12 --pass "" --days 3650 --ca-pass ""
9. [root@elk1 ~]# ll /etc/elasticsearch/elastic-stack-ca.p12
10. -rw------- 1 root elasticsearch 2672 Mar 29 20:44 /etc/elasticsearch/elastic-stack-ca.p12
11. [root@elk1 ~]# ll /etc/elasticsearch/elastic-certificates-https.p12
12. -rw------- 1 root elasticsearch 3596 Mar 29 20:48 /etc/elasticsearch/elastic-certificates-https.p12
15. 3.修改配置文件
16. [root@elk1 ~]# tail -2 /etc/elasticsearch/elasticsearch.yml
17. xpack.security.http.ssl.enabled: true
18. xpack.security.http.ssl.keystore.path: elastic-certificates-https.p12
20. [root@elk1 ~]# chmod 640 /etc/elasticsearch/elastic-certificates-https.p12
22. [root@elk1 ~]# scp  -rp /etc/elasticsearch/elastic{-certificates-https.p12,search.yml} 192.168.121.92:/etc/elasticsearch/
23. root@192.168.121.92's password:
24. elastic-certificates-https.p12                                                                                                                             100% 3596     1.6MB/s   00:00   
25. elasticsearch.yml                                                                                                                                          100% 4378     6.0MB/s   00:00   
26. [root@elk1 ~]# scp  -rp /etc/elasticsearch/elastic{-certificates-https.p12,search.yml} 192.168.121.93:/etc/elasticsearch/
27. root@192.168.121.93's password:
28. elastic-certificates-https.p12                                                                                                                             100% 3596   894.2KB/s   00:00   
29. elasticsearch.yml  
31. 4.重启ES集群
32. [root@elk1 ~]# systemctl restart elasticsearch.service
33. [root@elk2 ~]# systemctl restart elasticsearch.service
34. [root@elk3 ~]# systemctl restart elasticsearch.service
36. [root@elk1 ~]# curl https://127.1:9200/_cat/nodes -u elastic:654321 -k
37. 192.168.121.92 16 94 63 1.88 0.92 0.35 cdfhilmrstw - elk2
38. 192.168.121.91 14 96 30 0.79 0.90 0.55 cdfhilmrstw * elk1
39. 192.168.121.93  8 97 53 1.22 0.71 0.33 cdfhilmrstw - elk3
41. 5.修改kibana的配置跳过自建证书校验
42. [root@elk1 ~]# vim /etc/kibana/kibana.yml
43. ...
44. # 指向ES集群的地址协议为https
45. elasticsearch.hosts: ["https://192.168.121.91:9200","https://192.168.121.92:9200","https://192.168.121.93:9200"]
46. # 跳过证书校验
47. elasticsearch.ssl.verificationMode: none
48. [root@elk1 ~]# systemctl restart kibana.service

复制代码

filebeat对接https加密

1. # 编写filebeat配置文件
2. [root@elk92 filebeat]# cat 17-tcp-to-es-tls.yaml
3. filebeat.inputs:
4. - type: tcp
5.   host: "0.0.0.0:9000"
8. output.elasticsearch:
9.   hosts:
10.   - https://192.168.121.91:9200
11.   - https://192.168.121.92:9200
12.   - https://192.168.121.93:9200
13.   api_key: "m1wPlJUBrDbi_DeiIc-1:RcEw7Mk2QQKH_CGhMBnfbg"
14.   index: xu-es-apikey-tls-2025
15.   # 配置es集群的tls，此处跳过证书校验。默认值为: full
16.   # 参考链接:
17.   #         https://www.elastic.co/guide/en/beats/filebeat/7.17/configuration-ssl.html#client-verification-mode
18.   ssl.verification_mode: none
20. setup.ilm.enabled: false
21. setup.template.name: "xu"
22. setup.template.pattern: "xu*"
23. setup.template.overwrite: true
24. setup.template.settings:
25.   index.number_of_shards: 3
26.   index.number_of_replicas: 0

复制代码

logstash对接https加密

1. [root@elk93 logstash]# cat 13-tcp-to-es_api-key.conf
2. input {
3.   tcp {
4.     port => 8888
5.   }
6. }  
8. output {
10.   elasticsearch {
11.     hosts => ["192.168.121.91:9200","192.168.121.92:9200","192.168.121.93:9200"]
12.     index => "xu-api-key"
13.     #user => elastic
14.     #password => "123456"xu
15.     # 指定api-key的方式认证
16.     api_key => "oFwZlJUBrDbi_DeiLc9O:HWBj0LC2RWiUNTudV-6CBw"
17.    
18.     # 使用api-key则必须启动ssl
19.     ssl => true
21.     # 跳过ssl证书验证
22.     ssl_certificate_verification => false
23.   }
24. }
25. [root@elk93 logstash]#
26. [root@elk93 logstash]# logstash -rf 13-tcp-to-es_api-key.conf

复制代码

基于kibana实现RBAC
参考链接:
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-privileges.html
创建角色
创建用户
ES8摆设
单点摆设ES8集群

1. 环境准备：
2.         192.168.121.191 elk191
3.         192.168.121.192 elk192
4.         192.168.121.193 elk193
6. 1.获取安装包，安装es8
7. [root@elk191 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.17.3-amd64.deb
8. [root@elk191 ~]# dpkg -i elasticsearch-8.17.3-amd64.deb
11. # es8默认就支持https
12. --------------------------- Security autoconfiguration information ------------------------------
14. Authentication and authorization are enabled.
15. TLS for the transport and HTTP layers is enabled and configured.
17. The generated password for the elastic built-in superuser is : P0-MRYuCOTFj*4*rGNZk   # 内置elastic超级用户的密码
19. If this node should join an existing cluster, you can reconfigure this with
20. '/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
21. after creating an enrollment token on your existing cluster.
23. You can complete the following actions at any time:
25. Reset the password of the elastic built-in superuser with
26. '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.
28. Generate an enrollment token for Kibana instances with
29. '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.
31. Generate an enrollment token for Elasticsearch nodes with
32. '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.
34. -------------------------------------------------------------------------------------------------
35. ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
36. sudo systemctl daemon-reload
37. sudo systemctl enable elasticsearch.service
38. ### You can start elasticsearch service by executing
39. sudo systemctl start elasticsearch.service
42. 2.启动es8
43. [root@elk191 ~]# systemctl enable elasticsearch.service --now
44. Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /lib/systemd/system/elasticsearch.service.
45. [root@elk191 ~]# netstat  -tunlp | grep -E "9[2|3]00"
46. tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      1669/java           
47. tcp6       0      0 ::1:9300                :::*                    LISTEN      1669/java           
48. tcp6       0      0 :::9200                 :::*                    LISTEN      1669/java
50. 3.测试访问
51. [root@elk191 ~]# curl -u elastic:NVPLcMy0_n8aGL=UGAGc https://127.1:9200 -k
52. {
53.   "name" : "elk191",
54.   "cluster_name" : "elasticsearch",
55.   "cluster_uuid" : "-cw1TGvZSau0J2x-ThOJsg",
56.   "version" : {
57.     "number" : "8.17.3",
58.     "build_flavor" : "default",
59.     "build_type" : "deb",
60.     "build_hash" : "a091390de485bd4b127884f7e565c0cad59b10d2",
61.     "build_date" : "2025-02-28T10:07:26.089129809Z",
62.     "build_snapshot" : false,
63.     "lucene_version" : "9.12.0",
64.     "minimum_wire_compatibility_version" : "7.17.0",
65.     "minimum_index_compatibility_version" : "7.0.0"
66.   },
67.   "tagline" : "You Know, for Search"
68. }
69. [root@elk191 ~]# curl -u elastic:NVPLcMy0_n8aGL=UGAGc https://127.1:9200/_cat/nodes -k
70. 127.0.0.1 9 97 13 0.35 0.59 0.31 cdfhilmrstw * elk191

复制代码

摆设kibana8

1. 1.获取安装包，安装kibana
2. [root@elk191 ~]# wget  https://artifacts.elastic.co/downloads/kibana/kibana-8.17.3-amd64.deb
3. [root@elk191 ~]# dpkg -i kibana-8.17.3-amd64.deb
5. 2.配置kibana
6. [root@elk191 ~]# grep -vE "^$|^#" /etc/kibana/kibana.yml
7. server.port: 5601
8. server.host: "0.0.0.0"
9. logging:
10.   appenders:
11.     file:
12.       type: file
13.       fileName: /var/log/kibana/kibana.log
14.       layout:
15.         type: json
16.   root:
17.     appenders:
18.       - default
19.       - file
20. pid.file: /run/kibana/kibana.pid
21. i18n.locale: "zh-CN"
23. 3.启动kibana
24. [root@elk191 ~]# systemctl enable --now kibana.service
25. [root@elk191 ~]# ss -ntl | grep 5601
26. LISTEN 0      511               0.0.0.0:5601      0.0.0.0:*     
28. 4.生成kibana专用token
29. [root@elk191 ~]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
30. eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xMjEuMTkxOjkyMDAiXSwiZmdyIjoiZmNjMWI3MzJlNzIwMzMzMjI0ZDc5Zjk1YTUyZjIzZmUyNjMzMzYwZDIxY2Q0NzY3YjQ2ZjExZDhiOGYxZTFlZiIsImtleSI6IjdjNTk3SlVCeEI5S3NHd1ZPWVQ5OmYtN0FRWkhEUTVtMnlCZXdiMnJLbXcifQ==
32. 5.kiban服务器获取校验码
33. [root@elk191 ~]# /usr/share/kibana/bin/kibana-verification-code
34. Your verification code is:  414 756

复制代码

es8集群摆设

1. 1.拷贝配置文件到其他节点
2. [root@elk191 ~]# scp elasticsearch-8.17.3-amd64.deb 10.0.0.192:~
3. [root@elk191 ~]# scp elasticsearch-8.17.3-amd64.deb 10.0.0.193:~
5. 2.其他节点安装ES8软件包
6. [root@elk192 ~]# dpkg -i elasticsearch-8.17.3-amd64.deb
7. [root@elk193 ~]# dpkg -i elasticsearch-8.17.3-amd64.deb
9. # 配置es8
10. [root@elk191 ~]# grep -Ev "^$|^#" /etc/elasticsearch/elasticsearch.yml
11. cluster.name: xu-application
12. path.data: /var/lib/elasticsearch
13. path.logs: /var/log/elasticsearch
14. network.host: 0.0.0.0
15. discovery.seed_hosts: ["192.168.121.191","192.168.121.192","192.168.121.193"]
16. cluster.initial_master_nodes: ["192.168.121.191","192.168.121.192","192.168.121.193"]
17. xpack.security.enabled: true
18. xpack.security.enrollment.enabled: true
19. xpack.security.http.ssl:
20.   enabled: true
21.   keystore.path: certs/http.p12
22. xpack.security.transport.ssl:
23.   enabled: true
24.   verification_mode: certificate
25.   keystore.path: certs/transport.p12
26.   truststore.path: certs/transport.p12
27. http.host: 0.0.0.0
29. 3.在现有集群任意节点生成token令牌文件
30. [root@elk191 ~]#  /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
32. 4.在待加入节点使用token重新配置新节点的配置文件
33. [root@elk192 ~]# /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token  eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xMjEuMTkxOjkyMDAiXSwiZmdyIjoiMzIwODY0YzMxNmEyMDQ4YmIwYzVjNDNhY2FlZjQ4MTg2OTM3MmVhNTg2NjdiYTAwMjBjN2Y2ZTczN2YzNWU0MCIsImtleSI6IkE3RTY4SlVCU1BhTWhMRFN0VWdlOmdaM0dIS0RNUndld3o3ZWM0Qk1ySEEifQ==
34. [root@elk193 ~]# /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token  eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xMjEuMTkxOjkyMDAiXSwiZmdyIjoiMzIwODY0YzMxNmEyMDQ4YmIwYzVjNDNhY2FlZjQ4MTg2OTM3MmVhNTg2NjdiYTAwMjBjN2Y2ZTczN2YzNWU0MCIsImtleSI6IkE3RTY4SlVCU1BhTWhMRFN0VWdlOmdaM0dIS0RNUndld3o3ZWM0Qk1ySEEifQ==
36. 5.同步配置文件
37. [root@elk191 ~]# scp /etc/elasticsearch/elasticsearch.yml  192.168.121.192:/etc/elasticsearch/
38. [root@elk191 ~]# scp /etc/elasticsearch/elasticsearch.yml  192.168.121.193:/etc/elasticsearch/
41. 6.启动客户端es
42. [root@elk192 ~]# systemctl enable elasticsearch.service --now
43. [root@elk193 ~]# systemctl enable elasticsearch.service --now
45. 7.访问测试
46. [root@elk193 ~]# curl -u elastic:123456 -k https://192.168.121.191:9200/_cat/nodes
47. 192.168.121.191 17 97 10 0.61 0.55 0.72 cdfhilmrstw * elk191
48. 192.168.121.193 15 97 55 1.72 1.05 0.49 cdfhilmrstw - elk193
49. 192.168.121.192 13 97  4 0.25 0.45 0.52 cdfhilmrstw - elk192

复制代码

常见错误

1. 1.常见的错误处理Q1:
2. ERROR: Aborting enrolling to cluster. Unable to remove existing secure settings. Error was: Aborting enrolling to cluster. Unable to remove existing security configuration, elasticsearch.keystore did not contain expected setting [autoconfiguration.password_hash]., with exit code 74
5. 问题分析:
6. 说明本地已经有安全配置的相关参数设置了。删除之前的配置"elasticsearch.keystore"。
9. 解决方案:
10. rm -f /etc/elasticsearch/elasticsearch.keystore
15. 常见的错误处理Q2:
16. ERROR: Skipping security auto configuration because this node is configured to bootstrap or to join a multi-node cluster, which is not supported., with exit code 80
19. 解决方案:
20. export IS_UPGRADE=false
23. 常见的错误处理Q3：
24. ERROR: Aborting enrolling to cluster. This node doesn't appear to be auto-configured for security. Expected configuration is missing from elasticsearch.yml., with exit code 64
27. 错误分析:
28. 检查配置文件，发现会缺少安全相关的配置，可能是同步'elasticsearch.yml'失败了。
30. 解决方案:
31. 修改"/etc/elasticsearch/elasticsearch.yml"，添加安全配置即可，可以手动将elk192节点的配置拷贝到该节点即可。
33. 如果还是解决不了，可以对比elk191节点的配置和elk192的配置文件不同之处，然后将对应的配置拷贝过去即可，我此处测试缺少certs目录。
36. [root@elk191 ~]# scp -rp /etc/elasticsearch/certs/ 10.0.0.192:/etc/elasticsearch/
37. [root@elk191 ~]# scp /etc/elasticsearch/elasticsearch.yml 10.0.0.192:/etc/elasticsearch/
38. [root@elk191 ~]# scp /etc/elasticsearch/elasticsearch.keystore  10.0.0.192:/etc/elasticsearch/
39. [root@elk191 ~]# scp -rp /etc/elasticsearch/elasticsearch.yml  10.0.0.192:/etc/elasticsearch/
42. ERROR: Aborting enrolling to cluster. Unable to remove existing secure settings. Error was: Aborting enrolling to cluster. Unable to remove existing security configuration, elasticsearch.keystore did not contain expected setting [xpack.security.transport.ssl.keystore.secure_password]., with exit code 74

复制代码

es8和es7区别

1. - ES8和ES7对比部署
2.         1.ES8默认启用了https，支持认证等功能;
3.         2.ES8新增'elasticsearch-reset-password'脚本，对于elastic用户重置密码更加简单;
4.         3.ES8新增'elasticsearch-create-enrollment-token'脚本，可以为组件创建token信息，比如kibana组件；
5.         4.ES8新增kibana新增'kibana-verification-code'用于生成校验码。
6.         5.kibana支持更多的语言：English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR"
7.         6.kibana的webUI更加丰富，支持AI助手，手动创建索引等功能;
8.         7.ES8集群部署时，需要借助'elasticsearch-reconfigure-node'脚本来加入已存在的集群，默认就是单master节点的配置;

复制代码

ES7 JVM调优

1. 1.ES默认会吃掉物理机的一半内存
2. [root@elk91 ~]# ps -ef | grep java | grep Xms
3. elastic+   10045       1  2 Mar14 ?        00:56:32 /usr/share/elasticsearch/jdk/bin/java ...  -Xms1937m -Xmx1937m ...
5. 2.关于ES集群的调优原则
6.                 - 1.ES集群的JVM大小应该为物理机的一半，但是不的超过32GB；
7.                 - 2.比如你的集群内存为32GB，则默认应该为16GB，但是如果你的物理机是128GB，默认也会吃掉一半，因此我们需要手动配置为32GB；
9. 3.将ES内存设置为256Mb
10. [root@elk1 ~]# vim /etc/elasticsearch/jvm.options
11. [root@elk1 ~]# egrep "^-Xm[s|x]" /etc/elasticsearch/jvm.options
12. -Xms256m
13. -Xmx256m
15. 4.拷贝配置文件并滚动重启ES7集群
16. [root@elk1 ~]# scp /etc/elasticsearch/jvm.options 192.168.121.92:/etc/elasticsearch/
17. jvm.options                                                                                                                                                100% 3474     2.7MB/s   00:00   
18. [root@elk1 ~]# scp /etc/elasticsearch/jvm.options 192.168.121.93:/etc/elasticsearch/
19. jvm.options
21. [root@elk1 ~]# systemctl restart elasticsearch.service
22. [root@elk2 ~]# systemctl restart elasticsearch.service
23. [root@elk3 ~]# systemctl restart elasticsearch.service
25. 5.测试验证
26. [root@elk1 ~]# free -h
27.                total        used        free      shared  buff/cache   available
28. Mem:           3.8Gi       1.1Gi       1.9Gi       1.0Mi       800Mi       2.4Gi
29. Swap:          3.8Gi        26Mi       3.8Gi
30. [root@elk1 ~]# ps -ef | grep java | grep Xms
31. -Xms256m -Xmx256m
34. curl -k -u elastic:123456 https://127.1:9200/_cat/nodes
35. 192.168.121.92 68 67 94 4.01 2.12 0.96 cdfhilmrstw * elk2
36. 192.168.121.91 59 56 42 1.72 0.87 0.43 cdfhilmrstw - elk1
37. 192.168.121.93 63 61 92 3.30 2.26 1.14 cdfhilmrstw - elk3

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。