ToB企服应用市场:ToB评测及商务社交产业平台

标题: 安全运维 -- splunk 操作手册 [打印本页]

作者: 嚴華 时间: 2024-8-28 08:32
标题: 安全运维 -- splunk 操作手册
0x00 背景

splunk 一样平常运维操作条记。按照场景编号。

0x01 agent安装

下载链接 Splunk Universal Forwarder Previous Releases | Splunk
linux：
tar -zxvf splunkforwarder-8.0.3-a6754d8441bf-Linux-x86_64.tgz -C /opt
cp -r config /opt/splunkforwarder/etc/apps
vi /opt/splunkforwarder/etc/apps/prefix_app_inputs/local/inputs.conf
[monitor:///home/var/log/*log] #修改监控路径

/opt/splunkforwarder/bin/splunk start --accept-license # 输入app账号密码
/opt/splunkforwarder/bin/splunk show deploy-poll # 检查配置文件是否配置好DS指向
/opt/splunkforwarder/bin/splunk enable boot-start # 开启自启动
  windows:
Splunk 静默安装
msiexec -i splunkforwarder-9.0.3-a6754d8441bf-x64-release.msi SPLUNKUSERNAME=admin SPLUNKPASSWORD=pass AGREETOLICENSE=Yes /quiet

下面命令需要管理员权限：
echo f |xcopy /s /y /f deploymentclient.conf "%programfiles%\SplunkUniversalForwarder\etc\apps\yourcompany_all_deploymentclient\local\deploymentclient.conf"
  (yourcompany_all_deploymentclient 文件夹名字随便取）
deploymentclient.conf
[deployment-client]
  #phoneHomeIntervalInSecs = 600
  clientName = your_indexname
  [target-broker:deploymentServer]
  targetUri = siemds.yourcompany.com:8089
  yourcompany_all_deploymentclient 目录布局（app.conf 无关紧要）

splunk agent 从头重新读取日志，清空文件夹下缓存即可：
C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket
  0x02 splunk接入log（读取日志文件方式）

1.登录DS 后台
vim /opt/splunk/etc/development-apps/prefix_all_indexes/local/indexes.conf
新增索引 [prefix_mailcasph]
[prefix_mailcasph]
  homePath = volume:hotwarmdb/prefix_mailcasph/db
  coldPath = volume:colddb/prefix_mailcasph/colddb
  thawedPath = $SPLUNK_DB/prefix_mailcasph/thaweddb
  小tips: 如果是在web控制台操作，建议用命令4yy方式复制创建上面4行，表示从当前行开始复制4行，p粘贴。
inputs 负责数据采集，props负责数据解析

目录布局参考：

配置prefix_mailcasph_inputs文件夹：
vim prefix_mailcasph_inputs/local/app.conf (这个一般参考default/app.conf 不做修改)
[install]
state = enabled

[package]
check_for_updates = false

[ui]
is_visible = false
is_manageable = false
  vim prefix_mailcasph_inputs/local/inputs.conf （inputs.conf - Splunk Documentation）
[monitor://D:\Exchange Server\Logging\HttpProxy\Eas\HttpProxy*.log]
index = prefix_mailcasph
sourcetype = ms:exchange:http_proxy
crcSalt = <SOURCE>
disabled = false
  metadata 文件夹
local.meta 也是默认配置：
[]
access = read : [ * ], write : [ admin ]
export = system
  配置prefix_mailcasph_props文件夹
vim prefix_mailcasph_props/local/app.conf
[install]
state = enabled

[package]
check_for_updates = false

[ui]
is_visible = false
is_manageable = false
  props.conf (props.conf - Splunk Documentation) (来自splunk论坛某篇文章 https://www.splunk.com/en_us/blog/security/detecting-microsoft-exchange-vulnerabilities-0-8-days-later.html)
[ms:exchange:http_proxy]
  CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
FIELD_DELIMITER=,
KV_MODE=none
SHOULD_LINEMERGE=false
disabled=false
TIMESTAMP_FIELDS=DateTime
TRANSFORMS-killheader1 = kh1
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)\d{4}\-\d{2}\-\d{2}T
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ
MAX_TIMESTAMP_LOOKAHEAD = 25
NO_BINARY_CHECK=true
REPORT-extractfields = extractfields
  transforms.conf (来自splunk论坛某篇文章 https://www.splunk.com/en_us/blog/security/detecting-microsoft-exchange-vulnerabilities-0-8-days-later.html)
[kh1]
REGEX = ^DateTime
DEST_KEY = queue
FORMAT = nullQueue
[extractfields]
DELIMS=","
FIELDS=DateTime,RequestId,MajorVersion,MinorVersion,BuildVersion,RevisionVersion,ClientRequestId,Protocol,UrlHo
st,UrlStem,ProtocolAction,AuthenticationType,IsAuthenticated,AuthenticatedUser,Organization,AnchorMailbox,UserA
gent,ClientIpAddress,ServerHostName,HttpStatus,BackEndStatus,ErrorCode,Method,ProxyAction,TargetServer,TargetSe
rverVersion,RoutingType,RoutingHint,BackEndCookie,ServerLocatorHost,ServerLocatorLatency,RequestBytes,ResponseB
ytes,TargetOutstandingRequests,AuthModulePerfContext,HttpPipelineLatency,CalculateTargetBackEndLatency,GlsLaten
cyBreakup,TotalGlsLatency,AccountForestLatencyBreakup,TotalAccountForestLatency,ResourceForestLatencyBreakup,To
talResourceForestLatency,ADLatency,SharedCacheLatencyBreakup,TotalSharedCacheLatency,ActivityContextLifeTime,Mo
duleToHandlerSwitchingLatency,ClientReqStreamLatency,BackendReqInitLatency,BackendReqStreamLatency,BackendProce
ssingLatency,BackendRespInitLatency,BackendRespStreamLatency,ClientRespStreamLatency,KerberosAuthHeaderLatency,
HandlerCompletionLatency,RequestHandlerLatency,HandlerToModuleSwitchingLatency,ProxyTime,CoreLatency,RoutingLat
ency,HttpProxyOverhead,TotalRequestTime,RouteRefresherLatency,UrlQuery,BackEndGenericInfo,GenericInfo,GenericEr
rors,EdgeTraceId,DatabaseGuid,UserADObjectGuid,PartitionEndpointLookupLatency,RoutingStatus
  metadata 文件夹
local.meta 也是默认配置：
[]
access = read : [ * ], write : [ admin ]
export = system
  2.下发配置
2.1 DS 下发配置到CM （需要admin密码）
sudo -u splunk /opt/splunk/bin/splunk reload deploy-server -class <DS配置的服务器类>
  <DS配置的服务器类> 例如 yourcompany_windows_cas_apps 或者如果已经把 yourcompany_windows_cas_apps 加入到 yourcompany_indexer_apps 可以直接刷新 yourcompany_indexer_apps
这一步如果不熟练可以登录CM检查：
eg: 检查新增的index推送情况：
cat /opt/splunk/etc/master-apps/yourcompany_all_indexes/local/indexes.conf
  登录CM 检察index status: checking the status of the applied bundle
sudo -u splunk /opt/splunk/bin/splunk show cluster-bundle-status
  2.2 CM下发配置到indexer（需要admin密码）
sudo -u splunk /opt/splunk/bin/splunk apply cluster-bundle
  这一步如果不熟练可以登录恣意一台indexer检查：
eg: 检查新增的index推送情况：
cat /opt/splunk/etc/slave-apps/yourcompany_all_indexes/local/indexes.conf
  (上面这个路径是8.x.x的目录，9.x.x已更新为 /opt/splunk/etc/slave-apps.old, 升级之后使用 /opt/splunk/etc/peer-apps 文件夹, 需要去peer-apps 文件夹下查找)
如果没题目，在SH web应该可以搜索到数据啦。

0x03 splunk接入log（读取syslog方式）

linux 配置rsyslog

优点：
配置便捷，例如统一接入linux本地FW block日志：
vim /etc/rsyslog.conf （将下面load_balance更换为你情况中的load_balance_ip或index ip）
if $msg contains 'STATE_INVALID_DROP' then @load_balance:514
if $msg contains 'FINAL_REJECT' then @load_balance:514
systemctl restart rsyslog
等价于下面命令：
echo "if \$msg contains 'STATE_INVALID_DROP' then @10.111.11.11:514" >> /etc/rsyslog.conf && echo "if \$msg contains 'FINAL_REJECT' then @10.111.11.11:514" >> /etc/rsyslog.conf
  test syslog配置是否成功:
logger -p local0.info "l01test01 kernel: FINAL_REJECT: IN=ens160"
  缺点：
1.每加一台服务器，在后台推一次配置，如果需要增长的服务器数量多，则后台服务器推配置过于频仍。
2.需要确认原来有没有配置rsyslog转发，因为配置完需要重启，可能会影响原来配置的syslog数据接收。

0x04 splunk 安装app

splunk应用商店：
https://splunkbase.splunk.com/apps
阅读应用介绍，看应用是装在SH 还是 index 和 SH 都装。
1.在DS安装应用
因为是集群模式，以是登录DS web，选择 “应用 -- 管理应用 -- 从文件安装应用“。点击浏览，选择需要安装的应用，上载安装。
2.DS 下发到Deployer 和 CM
DS 后台，
推送到DP
  sudo -u splunk /opt/splunk/bin/splunk reload deploy-server -class yourcompany_searchhead_apps
  推送到CM
sudo -u splunk /opt/splunk/bin/splunk reload deploy-server -class yourcompany_indexer_apps
  yourcompany_indexer_apps 这个app 是在 DS -- 转发器 -- 服务器管理里定义的。
3.DP和CM分别推送到SH和Indexer
推送到SH
  sudo -u splunk /opt/splunk/bin/splunk apply shcluster-bundle -target https://10.111.xx.xxx:8089 # 集群模式恣意选择一台SH
  推送到Indexer
  sudo -u splunk /opt/splunk/bin/splunk apply cluster-bundle
   安装好app后可以通过syslog/文件等接入方式配置app的sourcetype来解析日志。
0x05 splunk 安装app解析日志

以radius为例：
1.自建索引和自建app解析：
新建索引参考上文此处略过。
DS 后台 /opt/splunk/etc/deployment-apps 新建 2个文件夹。
yourcompany_windows_radius_inputs
yourcompany_windows_radius_outputs
目录布局如下：（文件内容可以先阅读0x02章节）
yourcompany_windows_radius_inputs/metadata/local.meta
yourcompany_windows_radius_inputs/local/app.conf
yourcompany_windows_radius_inputs/local/inputs.conf
[monitor://C:\Windows\System32\LogFiles\WLAN\*]
index = yourcompany_radius
crcSalt = <SOURCE>
alwaysOpenFile = 1
disabled = false
sourcetype = radius

  [monitor://C:\Windows\System32\LogFiles\NPS\*]
index = yourcompany_radius
crcSalt = <SOURCE>
alwaysOpenFile = 1
disabled = false
sourcetype = radius

[monitor://C:\Windows\System32\LogFiles\IN*.log]
index = yourcompany_radius
crcSalt = <SOURCE>
alwaysOpenFile = 1
disabled = false
sourcetype = radius
  yourcompany_windows_radius_outputs
yourcompany_windows_radius_outputs/local/props.conf
[radius]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\"\w+\",\"\w+\"\,\d{2}
CHARSET=GB2312
TIME_FORMAT=%m/%d/%Y,%H:%M:%S,%N
TIME_PREFIX=\"\w+\",\"\w+\"\,
   这种 prop.conf 只能实现最简单的日志解析。
2.摆设agent文本方式接入日志
客户端服务器新增文件夹：
apps/config/radius/yourcompany_all_deploymentclient/local/deploymentclient.conf
[deployment-client]
  #phoneHomeIntervalInSecs = 600
  clientName = yourcompany_radius

  [target-broker:deploymentServer]
  targetUri = siemds.yourcompany.com:8089
  这时候发现可以接收到日志啦：(日志被简单分割)
"W10NPS0888","NAPS",06/12/2024,12:12:12,3,,"W10NPS0888\b7bbcccd777d",,,,,,,,0,"172.10.10.10","UA-77C7--UA-07C0707C-777",,,,,,,1,,16,"311 1 10.111.10.10 05/20/2024 12:12:12 11981996",,,,,,,,,"UA-01xxxxxxxxxxxxxxxx157",,,,,,,,,,,,,,,,,,,,,,,,,"Wired_BOX_Macbypass",1,,,,
  3.去应用商店查找官方app
运气很好，找到一个：
https://splunkbase.splunk.com/app/6989

看介绍说只需要安装到SH即可。(一般情况下Add-on 需要放SH和Index，而App只需要放SH)
下载传到DP后台，在DP推送到SH
sudo -u splunk /opt/splunk/bin/splunk apply shcluster-bundle -target https://10.111.xx.xxx:8089 # 集群模式恣意选择一台SH
  Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
修改DS配置，sourcetype改成app定义的。
/opt/splunk/etc/deployment-apps/yourcompany_windows_radius_inputs/local/inputs.conf
[monitor://C:\Windows\System32\LogFiles\WLAN\*]
index = yourcompany_radius
crcSalt = <SOURCE>
alwaysOpenFile = 1
disabled = false
sourcetype = microsoft:nps:radius

  [monitor://C:\Windows\System32\LogFiles\NPS\*]
index = yourcompany_radius
crcSalt = <SOURCE>
alwaysOpenFile = 1
disabled = false
sourcetype = microsoft:nps:radius

[monitor://C:\Windows\System32\LogFiles\IN*.log]
index = yourcompany_radius
crcSalt = <SOURCE>
alwaysOpenFile = 1
disabled = false
sourcetype = microsoft:nps:radius
   DS 配置推送到app：
sudo -u splunk /opt/splunk/bin/splunk reload deploy-server -class yourcompany_windows_radius_apps # DS转发器 -- 服务器管理里定义的
  yourcompany_windows_radius_apps 服务器类下面含有2个应用：
yourcompany_windows_ad_outputs、yourcompany_windows_radius_inputs。

yourcompany_windows_ad_outputs 这个应用的作用定义输出：
yourcompany_windows_ad_outputs/local/outputs.conf
[tcpout]
defaultGroup = myIndexers
forceTimebasedAutoLB = true

  [tcpout:myIndexers]
server=10.111.1.101:9998,10.111.1.102:9998,10.111.1.103:9998,10.111.1.104:9998 # 集群模式全部indexer
  tcpSendBufSz = 512000
  0x06 splunk 配置ldapsearch

例如，搜索OU !="ADreinforce" 且查找全部计算机并扫除全部禁用的计算机帐户，
可以用来检查US域控OU脱离：
| ldapsearch domain=US search="(&(objectCategory=computer)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))" attrs="distinguishedName,objectCategory,operatingSystem"|search distinguishedName="*OU=Domain*" AND NOT "OU=ADreinforce"
  参考来自 LDAP Queries in Privileged Identity
Troubleshoot the Splunk Supporting Add-on for Active Directory - Splunk Documentation
| ldapsearch domain=default search="(&(objectCategory=computer)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))"
  Centos 7 测试服务器 ldapsearch
yum install -y openldap openldap-clients nss-pam-ldapd
|ldapsearch -x -b "dc=pharmatechs,dc=com" search=(ou=ADreinforce)
| ldapsearch search="(&(ou=ADreinforce))"

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

欢迎光临 ToB企服应用市场:ToB评测及商务社交产业平台 (https://dis.qidao123.com/)