利用Kube-Bench对Kubernetes进行安全检测

卖不甜枣  金牌会员 | 2024-9-26 20:43:32 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 995|帖子 995|积分 2985

利用Kube-Bench对Kubernetes进行安全检测

1. 工具介绍

Kube-Bench是一个开源的Go语言工具，用于主动化查抄Kubernetes集群是否符合CIS Kubernetes基准。这些基准包括一系列关于Kubernetes配置和部署安全性的发起和最佳实践。
Kube-Bench实验了一系列针对Kubernetes组件（如kube-apiserver、etcd、kube-scheduler、kube-controller-manager等）的测试，来查抄它们是否按照CIS基准的推荐进行了配置。测试效果会被分类为PASS、FAIL或WARN，以清楚明白地显示哪些地方必要改进。
Kube-Bench是在运行时查抄Kubernetes环境的工具，它并不会修改系统。你可以定期运行kube-bench来审计你的Kubernetes环境，确保其保持在最佳的安全状态。
官方仓库：https://github.com/aquasecurity/kube-bench
2. CIS Kubernetes Benchmark支持

在Kubernetes环境中，CIS Kubernetes基准就是一套针对Kubernetes的安全配置最佳实践。例如，CIS Kubernetes基准会涵盖如何配置kubelet，如何限制API服务器上的权限等内容。
进入CIS（Center for Internet Security）官网可以下载Kubernetes Benchmark文件。
Kubernetes Benchmark、kube-bench config和Kubernetes配套关系如下：
SourceKubernetes Benchmarkkube-bench configKubernetes versionsCIS1.5.1cis-1.51.15CIS1.6.0cis-1.61.16-1.18CIS1.20cis-1.201.19-1.21CIS1.23cis-1.231.22-1.23CIS1.24cis-1.241.24CIS1.7cis-1.71.25CIS1.8cis-1.81.26CISGKE 1.0.0gke-1.0GKECISGKE 1.2.0gke-1.2.0GKECISEKS 1.0.1eks-1.0.1EKSCISEKS 1.1.0eks-1.1.0EKSCISEKS 1.2.0eks-1.2.0EKSCISACK 1.0.0ack-1.0ACKCISAKS 1.0.0aks-1.0AKSRHELRedHat OpenShift hardening guiderh-0.7OCP 3.10-3.11CISOCP4 1.1.0rh-1.0OCP 4.1-CIS1.6.0-k3scis-1.6-k3sk3s v1.16-v1.24DISAKubernetes Ver 1, Rel 6eks-stig-kubernetes-v1r6EKSCISTKGI 1.2.53tkgi-1.2.53vmwareCIS1.7.0-rkerke-cis-1.7rke v1.25-v1.27CIS1.7.0-rke2rke2-cis-1.6rke2 v1.25-v1.27CIS1.7.0-k3sk3s-cis-1.7k3s v1.25-v1.27 最新信息请访问 CIS Kubernetes Benchmark support 查看。
默认配置下，Kube-Bench将根据目标设备上运行的Kubernete版原来确定要运行的测试集。
3. 工具安装

3.1 二进制安装

下载地点：https://github.com/aquasecurity/kube-bench/releases
  1. [root@master1 ~]# wget https://github.com/aquasecurity/kube-bench/releases/download/v0.7.3/kube-bench_0.7.3_linux_amd64.tar.gz
  2. [root@master1 ~]# mkdir /opt/kube-bench
  3. # 将二进制文件解压到创建的目录
  4. [root@master1 ~]# tar xf kube-bench_0.7.3_linux_amd64.tar.gz -C /opt/kube-bench/
  6. # 查看文件内容
  7. [root@master1 ~]# cd /opt/kube-bench/
  8. [root@master1  kube-bench]# ls
  9. cfg  kube-bench
  10. [root@master1  kube-bench]# ls cfg/
  11. ack-1.0   cis-1.24           cis-1.6-k3s  eks-1.0.1                 gke-1.0       k3s-cis-1.7    rke2-cis-1.24  rke-cis-1.7
  12. aks-1.0   cis-1.24-microk8s  cis-1.7      eks-1.1.0                 gke-1.2.0     rh-0.7         rke2-cis-1.7   tkgi-1.2.53
  13. cis-1.20  cis-1.5            cis-1.8      eks-1.2.0                 k3s-cis-1.23  rh-1.0         rke-cis-1.23
  14. cis-1.23  cis-1.6            config.yaml  eks-stig-kubernetes-v1r6  k3s-cis-1.24  rke2-cis-1.23  rke-cis-1.24
  15. [root@master1 cfg]# cd cis-1.8/
  16. # 各个组件yaml文件记录了需要检测的详细信息
  17. [root@master1 cis-1.8]# ls
  18. config.yaml  controlplane.yaml  etcd.yaml  master.yaml  node.yaml  policies.yaml
复制代码
cfg/config.yaml包含了相关测试组件配置、配置文件路径、K8S版本和CIS标准映射等。如果k8s的某些配置文件自定义到了非默认的目次，修改config.yaml里的相应目次就行。
  1. [root@master1 kube-bench]# cat cfg/config.yaml
  2. ---
  3. ## Controls Files.
  4. # These are YAML files that hold all the details for running checks.
  5. #
  6. ## Uncomment to use different control file paths.
  7. # masterControls: ./cfg/master.yaml
  8. # nodeControls: ./cfg/node.yaml
  10. master:
  11.   components:
  12.     - apiserver
  13.     - scheduler
  14.     - controllermanager
  15.     - etcd
  16.     - flanneld
  17.     # kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark
  18.     - kubernetes
  19.     - kubelet
  21.   kubernetes:
  22.     defaultconf: /etc/kubernetes/config
  24.   apiserver:
  25.     bins:
  26.       - "kube-apiserver"
  27.       - "hyperkube apiserver"
  28.       - "hyperkube kube-apiserver"
  29.       - "apiserver"
  30.       - "openshift start master api"
  31.       - "hypershift openshift-kube-apiserver"
  32.     confs:
  33.       - /etc/kubernetes/manifests/kube-apiserver.yaml
  34.       - /etc/kubernetes/manifests/kube-apiserver.yml
  35.       - /etc/kubernetes/manifests/kube-apiserver.manifest
  36.       - /var/snap/kube-apiserver/current/args
  37.       - /var/snap/microk8s/current/args/kube-apiserver
  38.       - /etc/origin/master/master-config.yaml
  39.       - /etc/kubernetes/manifests/talos-kube-apiserver.yaml
  40.       - /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml
  41.     defaultconf: /etc/kubernetes/manifests/kube-apiserver.yaml
  43.   scheduler:
  44.     bins:
  45.       - "kube-scheduler"
  47. ...
  48. node:
  49.   components:
  50.     - kubelet
  51.     - proxy
  52.     # kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark
  53.     - kubernetes
  55.   kubernetes:
  56.     defaultconf: "/etc/kubernetes/config"
  58.   kubelet:
  59.     cafile:
  60.       - "/etc/kubernetes/pki/ca.crt"
  61.       - "/etc/kubernetes/certs/ca.crt"
  62.       - "/etc/kubernetes/cert/ca.pem"
  63.       - "/var/snap/microk8s/current/certs/ca.crt"
  64.       - "/var/lib/rancher/rke2/agent/server.crt"
  65.       - "/var/lib/rancher/rke2/agent/client-ca.crt"
  66.       - "/var/lib/rancher/k3s/agent/client-ca.crt"
  67. ...
  69. etcd:
  70.   components:
  71.     - etcd
  73.   etcd:
  74.     bins:
  75.       - "etcd"
  76.     datadirs:
  77.       - /var/lib/etcd/default.etcd
  78.       - /var/lib/etcd/data.etcd
  79.     confs:
  80.       - /etc/kubernetes/manifests/etcd.yaml
  81.       - /etc/kubernetes/manifests/etcd.yml
  82. ...
  84. controlplane:
  85.   components:
  86.     - apiserver
  88.   apiserver:
  89.     bins:
  90.       - "kube-apiserver"
  91.       - "hyperkube apiserver"
  92.       - "hyperkube kube-apiserver"
  93.       - "apiserver"
  95. policies:
  96.   components: []
  98. managedservices:
  99.   components: []
  101. version_mapping:
  102.   "1.15": "cis-1.5"
  103.   "1.16": "cis-1.6"
  104. ...
  106. target_mapping:
  107.   "cis-1.5":
  108.     - "master"
  109.     - "node"
  110.     - "controlplane"
  111.     - "etcd"
  112.     - "policies"
  113. ...
复制代码
3.2 源码安装

工具利用go语言开发，实验如下步骤进行源码安装：
  1. # 安装go
  2. yum install go
  3. # 配置go国内镜像源
  4. go env -w GO111MODULE=on
  5. go env -w GOPROXY=https://goproxy.cn,direct
  7. go install github.com/aquasecurity/kube-bench@latest
  8. #安装成功后位于当前用户的go目录下
  9. /root/go/bin
  11. ./kube-bench --help
  13. # Run all checks
  15. ./kube-bench
复制代码
4. 运行kube-bench检测

kube-bench支持etcd组件、master节点、node节点等检测。命令运行语法：
./kube-bench --config-dir <配置目次> --config <配置文件> run --targets=<必要检测的组件>
本文测试的K8S环境如下：
  1. # CPU架构
  2. [root@master1 ~]# lscpu | grep 架构
  3. 架构：                           x86_64
  4.   
  5. # K8S集群信息，master节点同时也承担worker节点的角色
  6. [root@master1 ~]# kubectl get nodes
  7. NAME        STATUS   ROLES                  AGE   VERSION
  8. master1   Ready    control-plane,master   44d   v1.27.6
  9. master2   Ready    control-plane,master   44d   v1.27.6
  10. master3   Ready    control-plane,master   44d   v1.27.6
  11. # K8S版本为1.27.6
  12. [root@master1 ~]# kubectl version
  13. WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
  14. Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:21:34Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
  15. Kustomize Version: v5.0.1
  16. Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:14:09Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
  18. # 操作系统为openEuler
  19. [root@master1 ~]# cat /etc/openEuler-latest
  20. openeulerversion=openEuler-22.03-LTS
  21. compiletime=2022-03-30-16-23-56
  22. gccversion=10.3.1-10.oe2203
  23. kernelversion=5.10.0-60.18.0.50.oe2203
  24. openjdkversion=1.8.0.312.b07-11.oe2203
复制代码
4.1 kube-bench检测etcd组件

  1. [root@master1 kube-bench]# ./kube-bench --config-dir ./cfg/ --config ./cfg/config.yaml run --targets=etcd
  2. [INFO] 2 Etcd Node Configuration
  3. [INFO] 2 Etcd Node Configuration
  4. [FAIL] 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated)
  5. [FAIL] 2.2 Ensure that the --client-cert-auth argument is set to true (Automated)
  6. [PASS] 2.3 Ensure that the --auto-tls argument is not set to true (Automated)
  7. [FAIL] 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated)
  8. [FAIL] 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated)
  9. [PASS] 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated)
  10. [WARN] 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual)
  12. == Remediations etcd ==
  13. 2.1 Follow the etcd service documentation and configure TLS encryption.
  14. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml
  15. on the master node and set the below parameters.
  16. --cert-file=</path/to/ca-file>
  17. --key-file=</path/to/key-file>
  19. 2.2 Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master
  20. node and set the below parameter.
  21. --client-cert-auth="true"
  23. 2.4 Follow the etcd service documentation and configure peer TLS encryption as appropriate
  24. for your etcd cluster.
  25. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the
  26. master node and set the below parameters.
  27. --peer-client-file=</path/to/peer-cert-file>
  28. --peer-key-file=</path/to/peer-key-file>
  30. 2.5 Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master
  31. node and set the below parameter.
  32. --peer-client-cert-auth=true
  34. 2.7 [Manual test]
  35. Follow the etcd documentation and create a dedicated certificate authority setup for the
  36. etcd service.
  37. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the
  38. master node and set the below parameter.
  39. --trusted-ca-file=</path/to/ca-file>
  42. == Summary etcd ==
  43. 2 checks PASS
  44. 4 checks FAIL
  45. 1 checks WARN
  46. 0 checks INFO
  48. == Summary total ==
  49. 2 checks PASS
  50. 4 checks FAIL
  51. 1 checks WARN
  52. 0 checks INFO
复制代码
根据汇总效果，检测出4个Fail，1个Warn，打印效果给出了相应的具体问题说明和解决方法。
4.2 kube-bench检测master节点

  1. [root@master1 kube-bench]# ./kube-bench --config-dir ./cfg/ --config ./cfg/config.yaml run --targets=master
  2. [INFO] 1 Control Plane Security Configuration
  3. [INFO] 1.1 Control Plane Node Configuration Files
  4. [PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated)
  5. [PASS] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated)
  6. [PASS] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated)
  7. [PASS] 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated)
  8. [PASS] 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated)
  9. [PASS] 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated)
  10. [FAIL] 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated)
  11. [FAIL] 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated)
  12. [WARN] 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual)
  13. [WARN] 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual)
  14. [PASS] 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated)
  15. [FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
  16. [FAIL] 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated)
  17. [PASS] 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated)
  18. [FAIL] 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated)
  19. [PASS] 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated)
  20. [FAIL] 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated)
  21. [PASS] 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated)
  22. [PASS] 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated)
  23. [WARN] 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual)
  24. [WARN] 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual)
  25. [INFO] 1.2 API Server
  26. [WARN] 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual)
  27. [PASS] 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated)
  28. [WARN] 1.2.3 Ensure that the --DenyServiceExternalIPs is set (Manual)
  29. [PASS] 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated)
  30. [FAIL] 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated)
  31. [PASS] 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
  32. [PASS] 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated)
  33. [PASS] 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated)
  34. [WARN] 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual)
  35. [PASS] 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated)
  36. [WARN] 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual)
  37. [WARN] 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual)
  38. [PASS] 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated)
  39. [PASS] 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated)
  40. [PASS] 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated)
  41. [FAIL] 1.2.16 Ensure that the --profiling argument is set to false (Automated)
  42. [FAIL] 1.2.17 Ensure that the --audit-log-path argument is set (Automated)
  43. [FAIL] 1.2.18 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)
  44. [FAIL] 1.2.19 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)
  45. [FAIL] 1.2.20 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)
  46. [WARN] 1.2.21 Ensure that the --request-timeout argument is set as appropriate (Manual)
  47. [PASS] 1.2.22 Ensure that the --service-account-lookup argument is set to true (Automated)
  48. [PASS] 1.2.23 Ensure that the --service-account-key-file argument is set as appropriate (Automated)
  49. [PASS] 1.2.24 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated)
  50. [PASS] 1.2.25 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated)
  51. [PASS] 1.2.26 Ensure that the --client-ca-file argument is set as appropriate (Automated)
  52. [PASS] 1.2.27 Ensure that the --etcd-cafile argument is set as appropriate (Automated)
  53. [WARN] 1.2.28 Ensure that the --encryption-provider-config argument is set as appropriate (Manual)
  54. [WARN] 1.2.29 Ensure that encryption providers are appropriately configured (Manual)
  55. [PASS] 1.2.30 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual)
  56. [INFO] 1.3 Controller Manager
  57. [WARN] 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual)
  58. [FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated)
  59. [PASS] 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated)
  60. [PASS] 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated)
  61. [PASS] 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated)
  62. [PASS] 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated)
  63. [PASS] 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated)
  64. [INFO] 1.4 Scheduler
  65. [FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)
  66. [PASS] 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated)
  68. == Remediations master ==
  69. 1.1.7 Run the below command (based on the file location on your system) on the control plane node.
  70. For example,
  71. chmod 600 /etc/kubernetes/manifests/etcd.yaml
  73. 1.1.8 Run the below command (based on the file location on your system) on the control plane node.
  74. For example,
  75. chown root:root /etc/kubernetes/manifests/etcd.yaml
  77. 1.1.9 Run the below command (based on the file location on your system) on the control plane node.
  78. For example, chmod 600 <path/to/cni/files>
  80. 1.1.10 Run the below command (based on the file location on your system) on the control plane node.
  81. For example,
  82. chown root:root <path/to/cni/files>
  84. 1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir,
  85. from the command 'ps -ef | grep etcd'.
  86. Run the below command (based on the etcd data directory found above).
  87. For example, chown etcd:etcd /var/lib/etcd
  89. 1.1.13 Run the below command (based on the file location on your system) on the control plane node.
  90. For example, chmod 600 /etc/kubernetes/admin.conf
  92. 1.1.15 Run the below command (based on the file location on your system) on the control plane node.
  93. For example,
  94. chmod 600 /etc/kubernetes/scheduler.conf
  96. 1.1.17 Run the below command (based on the file location on your system) on the control plane node.
  97. For example,
  98. chmod 600 /etc/kubernetes/controller-manager.conf
  100. 1.1.20 Run the below command (based on the file location on your system) on the control plane node.
  101. For example,
  102. chmod -R 600 /etc/kubernetes/pki/*.crt
  104. 1.1.21 Run the below command (based on the file location on your system) on the control plane node.
  105. For example,
  106. chmod -R 600 /etc/kubernetes/pki/*.key
  108. 1.2.1 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  109. on the control plane node and set the below parameter.
  110. --anonymous-auth=false
  112. 1.2.3 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  113. on the control plane node and remove the `DenyServiceExternalIPs`
  114. from enabled admission plugins.
  116. 1.2.5 Follow the Kubernetes documentation and setup the TLS connection between
  117. the apiserver and kubelets. Then, edit the API server pod specification file
  118. /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the
  119. --kubelet-certificate-authority parameter to the path to the cert file for the certificate authority.
  120. --kubelet-certificate-authority=<ca-string>
  122. 1.2.9 Follow the Kubernetes documentation and set the desired limits in a configuration file.
  123. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  124. and set the below parameters.
  125. --enable-admission-plugins=...,EventRateLimit,...
  126. --admission-control-config-file=<path/to/configuration/file>
  128. 1.2.11 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  129. on the control plane node and set the --enable-admission-plugins parameter to include
  130. AlwaysPullImages.
  131. --enable-admission-plugins=...,AlwaysPullImages,...
  133. 1.2.12 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  134. on the control plane node and set the --enable-admission-plugins parameter to include
  135. SecurityContextDeny, unless PodSecurityPolicy is already in place.
  136. --enable-admission-plugins=...,SecurityContextDeny,...
  138. 1.2.16 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  139. on the control plane node and set the below parameter.
  140. --profiling=false
  142. 1.2.17 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  143. on the control plane node and set the --audit-log-path parameter to a suitable path and
  144. file where you would like audit logs to be written, for example,
  145. --audit-log-path=/var/log/apiserver/audit.log
  147. 1.2.18 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  148. on the control plane node and set the --audit-log-maxage parameter to 30
  149. or as an appropriate number of days, for example,
  150. --audit-log-maxage=30
  152. 1.2.19 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  153. on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate
  154. value. For example,
  155. --audit-log-maxbackup=10
  157. 1.2.20 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  158. on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB.
  159. For example, to set it as 100 MB, --audit-log-maxsize=100
  161. 1.2.21 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  162. and set the below parameter as appropriate and if needed.
  163. For example, --request-timeout=300s
  165. 1.2.28 Follow the Kubernetes documentation and configure a EncryptionConfig file.
  166. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
  167. on the control plane node and set the --encryption-provider-config parameter to the path of that file.
  168. For example, --encryption-provider-config=</path/to/EncryptionConfig/File>
  170. 1.2.29 Follow the Kubernetes documentation and configure a EncryptionConfig file.
  171. In this file, choose aescbc, kms or secretbox as the encryption provider.
  173. 1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml
  174. on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold,
  175. for example, --terminated-pod-gc-threshold=10
  177. 1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml
  178. on the control plane node and set the below parameter.
  179. --profiling=false
  181. 1.4.1 Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file
  182. on the control plane node and set the below parameter.
  183. --profiling=false
  186. == Summary master ==
  187. 33 checks PASS
  188. 14 checks FAIL
  189. 13 checks WARN
  190. 0 checks INFO
  192. == Summary total ==
  193. 33 checks PASS
  194. 14 checks FAIL
  195. 13 checks WARN
  196. 0 checks INFO
复制代码
检测到14个Fail，可以根据给出的修补方案进行修改。
4.3 kube-bench检测worker节点

  1. [root@master1 kube-bench]# ./kube-bench --config-dir ./cfg/ --config ./cfg/config.yaml run --targets=node
  2. [INFO] 4 Worker Node Security Configuration
  3. [INFO] 4.1 Worker Node Configuration Files
  4. [FAIL] 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated)
  5. [PASS] 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated)
  6. [WARN] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual)
  7. [WARN] 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual)
  8. [PASS] 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated)
  9. [PASS] 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated)
  10. [WARN] 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual)
  11. [PASS] 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual)
  12. [FAIL] 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated)
  13. [PASS] 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated)
  14. [INFO] 4.2 Kubelet
  15. [PASS] 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
  16. [PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
  17. [PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated)
  18. [PASS] 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual)
  19. [PASS] 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual)
  20. [PASS] 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated)
  21. [PASS] 4.2.7 Ensure that the --hostname-override argument is not set (Manual)
  22. [PASS] 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual)
  23. [WARN] 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual)
  24. [PASS] 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated)
  25. [PASS] 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual)
  26. [WARN] 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual)
  27. [WARN] 4.2.13 Ensure that a limit is set on pod PIDs (Manual)
  29. == Remediations node ==
  30. 4.1.1 Run the below command (based on the file location on your system) on the each worker node.
  31. For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
  33. 4.1.3 Run the below command (based on the file location on your system) on the each worker node.
  34. For example,
  35. chmod 600 /etc/kubernetes/proxy.conf
  37. 4.1.4 Run the below command (based on the file location on your system) on the each worker node.
  38. For example, chown root:root /etc/kubernetes/proxy.conf
  40. 4.1.7 Run the following command to modify the file permissions of the
  41. --client-ca-file chmod 600 <filename>
  43. 4.1.9 Run the following command (using the config file location identified in the Audit step)
  44. chmod 600 /var/lib/kubelet/config.yaml
  46. 4.2.9 If using a Kubelet config file, edit the file to set `tlsCertFile` to the location
  47. of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile`
  48. to the location of the corresponding private key file.
  49. If using command line arguments, edit the kubelet service file
  50. /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and
  51. set the below parameters in KUBELET_CERTIFICATE_ARGS variable.
  52. --tls-cert-file=<path/to/tls-certificate-file>
  53. --tls-private-key-file=<path/to/tls-key-file>
  54. Based on your system, restart the kubelet service. For example,
  55. systemctl daemon-reload
  56. systemctl restart kubelet.service
  58. 4.2.12 If using a Kubelet config file, edit the file to set `TLSCipherSuites` to
  59. TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
  60. or to a subset of these values.
  61. If using executable arguments, edit the kubelet service file
  62. /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and
  63. set the --tls-cipher-suites parameter as follows, or to a subset of these values.
  64. --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
  65. Based on your system, restart the kubelet service. For example:
  66. systemctl daemon-reload
  67. systemctl restart kubelet.service
  69. 4.2.13 Decide on an appropriate level for this parameter and set it,
  70. either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting.
  73. == Summary node ==
  74. 15 checks PASS
  75. 2 checks FAIL
  76. 6 checks WARN
  77. 0 checks INFO
  79. == Summary total ==
  80. 15 checks PASS
  81. 2 checks FAIL
  82. 6 checks WARN
  83. 0 checks INFO
复制代码
检测到2个Fail，可以根据给出的修补方案进行修改。
4.4 实验全部检测

  1. # 不加targets执行全部检测
  2. [root@master1 kube-bench]# ./kube-bench --config-dir ./cfg/ --config ./cfg/config.yaml run
复制代码
4.5 手动设置不安全的参数进行检测

可以手动构造不安全的设置，利用工具进行检测：

  • 构造etcd不安全设置，将/etc/kubernetes/manifests/etcd.yaml文件里的client-cert-auth=true改为client-cert-auth=false。
  • 构造kube-apiserver不安全设置，将/etc/kubernetes/manifests/kube-apiserver.yaml文件里的authorization-mode授权模式改为AlwaysAllow。
修改后重启kubelet使配置见效，然后利用kube-bench检测是否能检测到，具体操作查看文末的参考资料。
5. 总结

kube-bench是一款强大的工具，它可以资助我们发现哪些Kubernetes配置没有遵循CIS的最佳实践，从而改正这些问题，加强我们的Kubernetes集群的安全性。虽然kube-bench不能捕获全部可能的安全问题，但它至少可以资助我们消除最常见的一些安全毛病。
6. 问题记录


  • Centos7.9 K8S 环境下面运行kube-bench遇到glibc配套问题，必要更新系统的glibc版本到2.32，由于更新glibc可能造成系统不稳固，改换了系统进行测试。
  1. [root@k8s-master kube-bench]# ./kube-bench --config-dir `pwd`/cfg --config `pwd`/cfg/config.yaml run --targets=etcd
  2. ./kube-bench: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./kube-bench)
  3. ./kube-bench: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./kube-bench)
复制代码
7. 参考资料

https://www.cnblogs.com/renshengdezheli/p/17640119.html

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

卖不甜枣

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

运维 CIO 存储 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表