详解K8s 镜像缓存管理kube-fledged

反转基因福娃  金牌会员 | 2024-5-17 10:07:15 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 560|帖子 560|积分 1680

本文分享自华为云社区《K8s 镜像缓存管理 kube-fledged 认知》，作者： 山河已无恙。
我们知道 k8s 上的容器调理需要在调理的节点行拉取当前容器的镜像，在一些特别场景中，

  • 需要快速启动和/或扩展的应用程序。例如，由于数据量激增，执行实时数据处理的应用程序需要快速扩展。
  • 镜像比较庞大，涉及多个版本，节点存储有限，需要动态清理不需要的镜像
  • 无服务器函数通常需要在几分之一秒内立即对传入事件和启动容器做出反应。
  • 在边沿设备上运行的 IoT 应用程序，需要容忍边沿设备和镜像镜像仓库之间的间歇性网络连接。
  • 如果需要从专用仓库中拉取镜像，而且无法授予每个人今后镜像仓库拉取镜像的访问权限，则可以在群集的节点上提供镜像。
  • 如果集群管理员或操作员需要对应用程序举行升级，并希望事先验证是否可以成功拉取新镜像。
kube-fledged 是一个 kubernetes operator，用于直接在 Kubernetes 集群的 worker 节点上创建和管理容器镜像缓存。它允许用户定义镜像列表以及这些镜像应缓存到哪些工作节点上（即拉取）。因此，应用程序 Pod 险些可以立即启动，由于不需要从镜像仓库中提取镜像。
kube-fledged 提供了 CRUD API 来管理镜像缓存的生命周期，并支持多个可设置的参数，可以根据本身的需要自定义功能。
Kubernetes 具有内置的镜像垃圾回收机制。节点中的 kubelet 会定期查抄磁盘使用率是否达到特定阈值（可通过标志举行设置）。一旦达到这个阈值，kubelet 会自动删除节点中所有未使用的镜像。
需要在发起的办理方案中实现自动和定期刷新机制。如果镜像缓存中的镜像被 kubelet 的 gc 删除，下一个刷新周期会将已删除的镜像拉入镜像缓存中。这可确保镜像缓存是最新的。
计划流程
https://github.com/senthilrch/kube-fledged/blob/master/docs/kubefledged-architecture.png
部署 kube-fledged

Helm 方式部署
  1. ──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$mkdir  kube-fledged
  3. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  4. └─$cd kube-fledged
  5. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  6. └─$export KUBEFLEDGED_NAMESPACE=kube-fledged
  7. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  8. └─$kubectl create namespace ${KUBEFLEDGED_NAMESPACE}
  9. namespace/kube-fledged created
  10. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  11. └─$helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/
  12. "kubefledged-charts" has been added to your repositories
  13. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  14. └─$helm repo update
  15. Hang tight while we grab the latest from your chart repositories...
  16. ...Successfully got an update from the "kubefledged-charts" chart repository
  17. ...Successfully got an update from the "kubescape" chart repository
  18. ...Successfully got an update from the "rancher-stable" chart repository
  19. ...Successfully got an update from the "skm" chart repository
  20. ...Successfully got an update from the "openkruise" chart repository
  21. ...Successfully got an update from the "awx-operator" chart repository
  22. ...Successfully got an update from the "botkube" chart repository
  23. Update Complete. ⎈Happy Helming!⎈
  24. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  25. └─$helm install --verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait
复制代码
实际部署中发现，由于网络问题，chart 无法下载，所以通过 make deploy-using-yaml 使用 yaml 方式部署
Yaml 文件部署
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  2. └─$git clone https://github.com/senthilrch/kube-fledged.git
  3. 正克隆到 'kube-fledged'...
  4. remote: Enumerating objects: 10613, done.
  5. remote: Counting objects: 100% (1501/1501), done.
  6. remote: Compressing objects: 100% (629/629), done.
  7. remote: Total 10613 (delta 845), reused 1357 (delta 766), pack-reused 9112
  8. 接收对象中: 100% (10613/10613), 34.58 MiB | 7.33 MiB/s, done.
  9. 处理 delta 中: 100% (4431/4431), done.
  10. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  11. └─$ls
  12. kube-fledged
  13. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  14. └─$cd kube-fledged/
  15. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
  16. └─$make deploy-using-yaml
  17. kubectl apply -f deploy/kubefledged-namespace.yaml
复制代码
第一次部署，发现镜像拉不下来
  1. ┌──[root@vms100.liruilongs.github.io]-[~]
  2. └─$kubectl get all -n kube-fledged
  3. NAME                                               READY   STATUS                  RESTARTS         AGE
  4. pod/kube-fledged-controller-df69f6565-drrqg        0/1     CrashLoopBackOff        35 (5h59m ago)   21h
  5. pod/kube-fledged-webhook-server-7bcd589bc4-b7kg2   0/1     Init:CrashLoopBackOff   35 (5h58m ago)   21h
  6. pod/kubefledged-controller-55f848cc67-7f4rl        1/1     Running                 0                21h
  7. pod/kubefledged-webhook-server-597dbf4ff5-l8fbh    0/1     Init:CrashLoopBackOff   34 (6h ago)      21h
  9. NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
  10. service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   21h
  11. service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   21h
  13. NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
  14. deployment.apps/kube-fledged-controller       0/1     1            0           21h
  15. deployment.apps/kube-fledged-webhook-server   0/1     1            0           21h
  16. deployment.apps/kubefledged-controller        0/1     1            0           21h
  17. deployment.apps/kubefledged-webhook-server    0/1     1            0           21h
  19. NAME                                                     DESIRED   CURRENT   READY   AGE
  20. replicaset.apps/kube-fledged-controller-df69f6565        1         1         0       21h
  21. replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         0       21h
  22. replicaset.apps/kubefledged-controller-55f848cc67        1         1         0       21h
  23. replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         0       21h
  24. ┌──[root@vms100.liruilongs.github.io]-[~]
  25. └─$
复制代码
这里我们找一下要拉取的镜像
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$cat *.yaml | grep image:
  3.       - image: senthilrch/kubefledged-controller:v0.10.0
  4.       - image: senthilrch/kubefledged-webhook-server:v0.10.0
  5.       - image: senthilrch/kubefledged-webhook-server:v0.10.0
复制代码
单独拉取一些，当前使用 ansible 在所有工作节点批量操作
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$ansible k8s_node -m shell -a "docker pull docker.io/senthilrch/kubefledged-cri-client:v0.10.0" -i host.yaml
复制代码
其他相关的镜像都拉取一下
操作完成之后容器状态全部正常
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$kubectl -n kube-fledged get all
  3. NAME                                               READY   STATUS    RESTARTS   AGE
  4. pod/kube-fledged-controller-df69f6565-wdb4g        1/1     Running   0          13h
  5. pod/kube-fledged-webhook-server-7bcd589bc4-j8xxp   1/1     Running   0          13h
  6. pod/kubefledged-controller-55f848cc67-klxlm        1/1     Running   0          13h
  7. pod/kubefledged-webhook-server-597dbf4ff5-ktbsh    1/1     Running   0          13h
  9. NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
  10. service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   36h
  11. service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   36h
  13. NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
  14. deployment.apps/kube-fledged-controller       1/1     1            1           36h
  15. deployment.apps/kube-fledged-webhook-server   1/1     1            1           36h
  16. deployment.apps/kubefledged-controller        1/1     1            1           36h
  17. deployment.apps/kubefledged-webhook-server    1/1     1            1           36h
  19. NAME                                                     DESIRED   CURRENT   READY   AGE
  20. replicaset.apps/kube-fledged-controller-df69f6565        1         1         1       36h
  21. replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         1       36h
  22. replicaset.apps/kubefledged-controller-55f848cc67        1         1         1       36h
  23. replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         1       36h
复制代码
验证是否安装成功
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
  2. └─$kubectl get pods -n kube-fledged -l app=kubefledged
  3. NAME                                          READY   STATUS    RESTARTS   AGE
  4. kubefledged-controller-55f848cc67-klxlm       1/1     Running   0          16h
  5. kubefledged-webhook-server-597dbf4ff5-ktbsh   1/1     Running   0          16h
  6. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
  7. └─$kubectl get imagecaches -n kube-fledged
  8. No resources found in kube-fledged namespace.
复制代码
使用 kubefledged

创建镜像缓存对象

根据 Demo 文件，创建镜像缓存对象
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
  2. └─$cd deploy/
  3. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  4. └─$cat kubefledged-imagecache.yaml
  5. ---
  6. apiVersion: kubefledged.io/v1alpha2
  7. kind: ImageCache
  8. metadata:
  9.   # Name of the image cache. A cluster can have multiple image cache objects
  10.   name: imagecache1
  11.   namespace: kube-fledged
  12.   # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  13.   labels:
  14.     app: kubefledged
  15.     kubefledged: imagecache
  16. spec:
  17.   # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  18.   cacheSpec:
  19.   # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  20.   - images:
  21.     - ghcr.io/jitesoft/nginx:1.23.1
  22.   # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  23.   - images:
  24.     - us.gcr.io/k8s-artifacts-prod/cassandra:v7
  25.     - us.gcr.io/k8s-artifacts-prod/etcd:3.5.4-0
  26.     nodeSelector:
  27.       tier: backend
  28.   # Specifies a list of image pull secrets to pull images from private repositories into the cache
  29.   imagePullSecrets:
  30.   - name: myregistrykey
复制代码
官方的 Demo 中对应的 镜像拉取不下来，所以换一下
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$docker pull us.gcr.io/k8s-artifacts-prod/cassandra:v7
  3. Error response from daemon: Get "https://us.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  4. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  5. └─$
复制代码
为了测试选择器标签的使用,我们找一个节点的标签单独做镜像缓存
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$kubectl get nodes  --show-labels
复制代码
同时我们直接从公有仓库拉取镜像，所以不需要 imagePullSecrets 对象
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$vim kubefledged-imagecache.yaml
复制代码
修改后的 yaml 文件

  • 添加了一个所有节点的 liruilong/my-busybox:latest 镜像缓存
  • 添加了一个 kubernetes.io/hostname: vms105.liruilongs.github.io 对应标签选择器的 liruilong/hikvision-sdk-config-ftp:latest 镜像缓存
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$cat kubefledged-imagecache.yaml
  3. ---
  4. apiVersion: kubefledged.io/v1alpha2
  5. kind: ImageCache
  6. metadata:
  7.   # Name of the image cache. A cluster can have multiple image cache objects
  8.   name: imagecache1
  9.   namespace: kube-fledged
  10.   # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  11.   labels:
  12.     app: kubefledged
  13.     kubefledged: imagecache
  14. spec:
  15.   # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  16.   cacheSpec:
  17.   # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  18.   - images:
  19.     - liruilong/my-busybox:latest
  20.   # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  21.   - images:
  22.     - liruilong/hikvision-sdk-config-ftp:latest
  23.     nodeSelector:
  24.       kubernetes.io/hostname: vms105.liruilongs.github.io
  25.   # Specifies a list of image pull secrets to pull images from private repositories into the cache
  26.   #imagePullSecrets:
  27.   #- name: myregistrykey
  28. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  29. └─$
复制代码
直接创建报错了
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$kubectl create -f kubefledged-imagecache.yaml
  3. Error from server (InternalError): error when creating "kubefledged-imagecache.yaml": Internal error occurred: failed calling webhook "validate-image-cache.kubefledged.io": failed to call webhook: Post "https://kubefledged-webhook-server.kube-fledged.svc:3443/validate-image-cache?timeout=1s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefledged.io")
  4. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  5. └─$kubectl get imagecaches -n kube-fledged
  6. No resources found in kube-fledged namespace.
  7. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  8. └─$
复制代码
办理办法，删除对应的对象，重新创建
我在当前项目的一个 issues 下面找到了办理办法 https://github.com/senthilrch/kube-fledged/issues/76
看起来这是由于 Webhook CA 是硬编码的，但是当 webhook 服务器启动时，会生成一个新的 CA 捆绑包并更新 webhook 设置。当发生另一个部署时，将重新应用原始 CA 捆绑包，而且 Webhook 哀求开始失败，直到再次重新启动 Webhook 组件以修补捆绑包init-server
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
  2. └─$make remove-kubefledged-and-operator
  3. # Remove kubefledged
  4. kubectl delete -f deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml
  5. error: resource mapping not found for name: "kube-fledged" namespace: "kube-fledged" from "deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml": no matches for kind "KubeFledged" in version "charts.helm.kubefledged.io/v1alpha2"
  6. ensure CRDs are installed first
  7. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
  8. └─$make deploy-using-yaml
  9. kubectl apply -f deploy/kubefledged-namespace.yaml
  10. namespace/kube-fledged created
  11. kubectl apply -f deploy/kubefledged-crd.yaml
  12. customresourcedefinition.apiextensions.k8s.io/imagecaches.kubefledged.io unchanged
  13. ....................
  14. kubectl rollout status deployment kubefledged-webhook-server -n kube-fledged --watch
  15. Waiting for deployment "kubefledged-webhook-server" rollout to finish: 0 of 1 updated replicas are available...
  16. deployment "kubefledged-webhook-server" successfully rolled out
  17. kubectl get pods -n kube-fledged
  18. NAME                                          READY   STATUS    RESTARTS   AGE
  19. kubefledged-controller-55f848cc67-76c4v       1/1     Running   0          112s
  20. kubefledged-webhook-server-597dbf4ff5-56h6z   1/1     Running   0          66s
复制代码
重新创建缓存对象，创建成功
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$kubectl create -f kubefledged-imagecache.yaml
  3. imagecache.kubefledged.io/imagecache1 created
  4. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  5. └─$kubectl get imagecaches -n kube-fledged
  6. NAME          AGE
  7. imagecache1   10s
  8. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  9. └─$
复制代码
查看当前被纳管的镜像缓存
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  2. └─$kubectl get imagecaches imagecache1 -n kube-fledged -o json
  3. {
  4.     "apiVersion": "kubefledged.io/v1alpha2",
  5.     "kind": "ImageCache",
  6.     "metadata": {
  7.         "creationTimestamp": "2024-03-01T15:08:42Z",
  8.         "generation": 83,
  9.         "labels": {
  10.             "app": "kubefledged",
  11.             "kubefledged": "imagecache"
  12.         },
  13.         "name": "imagecache1",
  14.         "namespace": "kube-fledged",
  15.         "resourceVersion": "20169836",
  16.         "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
  17.     },
  18.     "spec": {
  19.         "cacheSpec": [
  20.             {
  21.                 "images": [
  22.                     "liruilong/my-busybox:latest"
  23.                 ]
  24.             },
  25.             {
  26.                 "images": [
  27.                     "liruilong/hikvision-sdk-config-ftp:latest"
  28.                 ],
  29.                 "nodeSelector": {
  30.                     "kubernetes.io/hostname": "vms105.liruilongs.github.io"
  31.                 }
  32.             }
  33.         ]
  34.     },
  35.     "status": {
  36.         "completionTime": "2024-03-02T01:06:47Z",
  37.         "message": "All requested images pulled succesfully to respective nodes",
  38.         "reason": "ImageCacheRefresh",
  39.         "startTime": "2024-03-02T01:05:33Z",
  40.         "status": "Succeeded"
  41.     }
  42. }
  43. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
  44. └─$
复制代码
通过 ansible 来验证
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
  3. 192.168.26.102 | CHANGED | rc=0 >>
  4. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  5. 192.168.26.101 | CHANGED | rc=0 >>
  6. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  7. 192.168.26.103 | CHANGED | rc=0 >>
  8. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  9. 192.168.26.105 | CHANGED | rc=0 >>
  10. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  11. 192.168.26.100 | CHANGED | rc=0 >>
  12. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  13. 192.168.26.106 | CHANGED | rc=0 >>
  14. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  15. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  16. └─$
  17. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  18. └─$ansible all -m shell -a "docker images | grep liruilong/hikvision-sdk-config-ftp" -i host.yaml
  19. 192.168.26.102 | FAILED | rc=1 >>
  20. non-zero return code
  21. 192.168.26.100 | FAILED | rc=1 >>
  22. non-zero return code
  23. 192.168.26.103 | FAILED | rc=1 >>
  24. non-zero return code
  25. 192.168.26.105 | CHANGED | rc=0 >>
  26. liruilong/hikvision-sdk-config-ftp                                          latest            a02cd03b4342   4 months ago    830MB
  27. 192.168.26.101 | FAILED | rc=1 >>
  28. non-zero return code
  29. 192.168.26.106 | FAILED | rc=1 >>
  30. non-zero return code
  31. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  32. └─$
复制代码
开启自动刷新
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache=
  3. imagecache.kubefledged.io/imagecache1 annotated
  4. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  5. └─$
复制代码
添加镜像缓存

添加一个新的镜像缓存
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
  3. {
  4.     "apiVersion": "kubefledged.io/v1alpha2",
  5.     "kind": "ImageCache",
  6.     "metadata": {
  7.         "creationTimestamp": "2024-03-01T15:08:42Z",
  8.         "generation": 92,
  9.         "labels": {
  10.             "app": "kubefledged",
  11.             "kubefledged": "imagecache"
  12.         },
  13.         "name": "imagecache1",
  14.         "namespace": "kube-fledged",
  15.         "resourceVersion": "20175233",
  16.         "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
  17.     },
  18.     "spec": {
  19.         "cacheSpec": [
  20.             {
  21.                 "images": [
  22.                     "liruilong/my-busybox:latest",
  23.                     "liruilong/jdk1.8_191:latest"
  24.                 ]
  25.             },
  26.             {
  27.                 "images": [
  28.                     "liruilong/hikvision-sdk-config-ftp:latest"
  29.                 ],
  30.                 "nodeSelector": {
  31.                     "kubernetes.io/hostname": "vms105.liruilongs.github.io"
  32.                 }
  33.             }
  34.         ]
  35.     },
  36.     "status": {
  37.         "completionTime": "2024-03-02T01:43:32Z",
  38.         "message": "All requested images pulled succesfully to respective nodes",
  39.         "reason": "ImageCacheUpdate",
  40.         "startTime": "2024-03-02T01:40:34Z",
  41.         "status": "Succeeded"
  42.     }
  43. }
  44. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  45. └─$
复制代码
通过 ansible 确认
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
  3. 192.168.26.101 | FAILED | rc=1 >>
  4. non-zero return code
  5. 192.168.26.100 | FAILED | rc=1 >>
  6. non-zero return code
  7. 192.168.26.102 | FAILED | rc=1 >>
  8. non-zero return code
  9. 192.168.26.103 | FAILED | rc=1 >>
  10. non-zero return code
  11. 192.168.26.105 | FAILED | rc=1 >>
  12. non-zero return code
  13. 192.168.26.106 | FAILED | rc=1 >>
  14. non-zero return code
  15. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  16. └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
  17. 192.168.26.101 | CHANGED | rc=0 >>
  18. liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
  19. 192.168.26.102 | CHANGED | rc=0 >>
  20. liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
  21. 192.168.26.100 | CHANGED | rc=0 >>
  22. liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
  23. 192.168.26.103 | CHANGED | rc=0 >>
  24. liruilong/jdk1.8_191                                                        latest                                      17dbd4002a8c   5 years ago     170MB
  25. 192.168.26.105 | CHANGED | rc=0 >>
  26. liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB
  27. 192.168.26.106 | CHANGED | rc=0 >>
  28. liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB
  29. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  30. └─$
复制代码
删除镜像缓存
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$kubectl edit imagecaches imagecache1 -n kube-fledged
  3. imagecache.kubefledged.io/imagecache1 edited
  4. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  5. └─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
  6. {
  7.     "apiVersion": "kubefledged.io/v1alpha2",
  8.     "kind": "ImageCache",
  9.     "metadata": {
  10.         "creationTimestamp": "2024-03-01T15:08:42Z",
  11.         "generation": 94,
  12.         "labels": {
  13.             "app": "kubefledged",
  14.             "kubefledged": "imagecache"
  15.         },
  16.         "name": "imagecache1",
  17.         "namespace": "kube-fledged",
  18.         "resourceVersion": "20175766",
  19.         "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
  20.     },
  21.     "spec": {
  22.         "cacheSpec": [
  23.             {
  24.                 "images": [
  25.                     "liruilong/jdk1.8_191:latest"
  26.                 ]
  27.             },
  28.             {
  29.                 "images": [
  30.                     "liruilong/hikvision-sdk-config-ftp:latest"
  31.                 ],
  32.                 "nodeSelector": {
  33.                     "kubernetes.io/hostname": "vms105.liruilongs.github.io"
  34.                 }
  35.             }
  36.         ]
  37.     },
  38.     "status": {
  39.         "message": "Image cache is being updated. Please view the status after some time",
  40.         "reason": "ImageCacheUpdate",
  41.         "startTime": "2024-03-02T01:48:03Z",
  42.         "status": "Processing"
  43.     }
  44. }
复制代码
通过 Ansible 确认，可以看到无论是 mastere 上的节点照旧 work 的节点，对应的镜像缓存都被清理
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
  3. 192.168.26.102 | CHANGED | rc=0 >>
  4. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  5. 192.168.26.101 | CHANGED | rc=0 >>
  6. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  7. 192.168.26.105 | FAILED | rc=1 >>
  8. non-zero return code
  9. 192.168.26.100 | CHANGED | rc=0 >>
  10. liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
  11. 192.168.26.103 | FAILED | rc=1 >>
  12. non-zero return code
  13. 192.168.26.106 | FAILED | rc=1 >>
  14. non-zero return code
  15. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  16. └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
  17. 192.168.26.105 | FAILED | rc=1 >>
  18. non-zero return code
  19. 192.168.26.102 | FAILED | rc=1 >>
  20. non-zero return code
  21. 192.168.26.103 | FAILED | rc=1 >>
  22. non-zero return code
  23. 192.168.26.101 | FAILED | rc=1 >>
  24. non-zero return code
  25. 192.168.26.100 | FAILED | rc=1 >>
  26. non-zero return code
  27. 192.168.26.106 | FAILED | rc=1 >>
  28. non-zero return code
  29. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  30. └─$
复制代码
这里需要注意如果扫除所有的镜像缓存，那么需要把 images 下的数组 写成 "".
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  2. └─$kubectl edit imagecaches imagecache1 -n kube-fledged
  3. imagecache.kubefledged.io/imagecache1 edited
  4. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  5. └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
  6. 192.168.26.102 | FAILED | rc=1 >>
  7. non-zero return code
  8. 192.168.26.101 | FAILED | rc=1 >>
  9. non-zero return code
  10. 192.168.26.100 | FAILED | rc=1 >>
  11. non-zero return code
  12. 192.168.26.105 | FAILED | rc=1 >>
  13. non-zero return code
  14. 192.168.26.103 | FAILED | rc=1 >>
  15. non-zero return code
  16. 192.168.26.106 | FAILED | rc=1 >>
  17. non-zero return code
  18. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  19. └─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
  20. {
  21.     "apiVersion": "kubefledged.io/v1alpha2",
  22.     "kind": "ImageCache",
  23.     "metadata": {
  24.         "creationTimestamp": "2024-03-01T15:08:42Z",
  25.         "generation": 98,
  26.         "labels": {
  27.             "app": "kubefledged",
  28.             "kubefledged": "imagecache"
  29.         },
  30.         "name": "imagecache1",
  31.         "namespace": "kube-fledged",
  32.         "resourceVersion": "20176849",
  33.         "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
  34.     },
  35.     "spec": {
  36.         "cacheSpec": [
  37.             {
  38.                 "images": [
  39.                     ""
  40.                 ]
  41.             },
  42.             {
  43.                 "images": [
  44.                     "liruilong/hikvision-sdk-config-ftp:latest"
  45.                 ],
  46.                 "nodeSelector": {
  47.                     "kubernetes.io/hostname": "vms105.liruilongs.github.io"
  48.                 }
  49.             }
  50.         ]
  51.     },
  52.     "status": {
  53.         "completionTime": "2024-03-02T01:52:16Z",
  54.         "message": "All cached images succesfully deleted from respective nodes",
  55.         "reason": "ImageCacheUpdate",
  56.         "startTime": "2024-03-02T01:51:47Z",
  57.         "status": "Succeeded"
  58.     }
  59. }
  60. ┌──[root@vms100.liruilongs.github.io]-[~/ansible]
  61. └─$
复制代码
如果通过下面的方式删除，直接注释调对应的标签
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$cat kubefledged-imagecache.yaml
  3. ---
  4. apiVersion: kubefledged.io/v1alpha2
  5. kind: ImageCache
  6. metadata:
  7.   # Name of the image cache. A cluster can have multiple image cache objects
  8.   name: imagecache1
  9.   namespace: kube-fledged
  10.   # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  11.   labels:
  12.     app: kubefledged
  13.     kubefledged: imagecache
  14. spec:
  15.   # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  16.   cacheSpec:
  17.   # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  18.   #- images:
  19.     #- liruilong/my-busybox:latest
  20.   # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  21.   - images:
  22.     - liruilong/hikvision-sdk-config-ftp:latest
  23.     nodeSelector:
  24.       kubernetes.io/hostname: vms105.liruilongs.github.io
  25.   # Specifies a list of image pull secrets to pull images from private repositories into the cache
  26.   #imagePullSecrets:
  27.   #- name: myregistrykey
  28. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  29. └─$
复制代码
那么会报下面的错
  1. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
  2. └─$kubectl edit imagecaches imagecache1 -n kube-fledged
  3. error: imagecaches.kubefledged.io "imagecache1" could not be patched: admission webhook "validate-image-cache.kubefledged.io" denied the request: Mismatch in no. of image lists
  4. You can run `kubectl replace -f /tmp/kubectl-edit-4113815075.yaml` to try this update again.
复制代码
博文部门内容参考

© 文中涉及参考链接内容版权归原作者所有，如有侵权请告知,如果你认可它不要吝啬星星哦
https://github.com/senthilrch/kube-fledged
 
点击关注，第一时间了解华为云新鲜技能~
 

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。
回复

使用道具 举报

0 个回复

正序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

反转基因福娃

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

挺好的 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表