Overview
What is Kubernetes aggregation
Kubernetes apiserver aggregation AA 是Kubernetes提供的一种扩展API的方法，目前并没有GA
Difference between CRD and AA
众所周知，kubernetes扩展API的方法大概为三种：CRD、AA、手动扩展源码。根据CNCF分享中Min Kim说的AA更关注于实践，而用户无需了解底层的原理，这里使用过 kubebuilder， code-generator 的用户是很能体会到这点。官方也给出了CRD与AA的区别
API Access Control
Authentication
- CR: All strategies supported. Configured by root apiserver.
- AA: Supporting all root apiserver's authenticating strategies but it has to be done via authentication token review api except for authentication proxy which will cause an extra cost of network RTT.
Authorization
- CR: All strategies supported. Configured by root apiserver.
- AA: Delegating authorization requests to root apiserver via SubjectAccessReview api. Note that this approach will also cost a network RTT.
Admission Control
- CR: You could extend via dynamic admission control webhook (which is costing network RTT).
- AA: While You can develop and customize your own admission controller which is dedicated to your AA. While You can't reuse root-apiserver's built-in admission controllers nomore.
API Schema
Note: CR's integration with OpenAPI schema is being enhanced in the future releases and it will have a stronger integration with OpenAPI mechanism.
Validating
- CR: (landed in 1.12) Defined via OpenAPIv3 Schema grammar. more
- AA: You can customize any validating flow you want.
Conversion
- CR: (landed in 1.13) The CR conversioning (basically from storage version to requested version) could be done via conversioning webhook.
- AA: Develop any conversion you want.
SubResource
- CR: Currently only status and scale sub-resource supported.
- AA: You can customize any sub-resouce you want.
OpenAPI Schema
- CR: (landed in 1.13) The corresponding CRD's OpenAPI schema will be automatically synced to root-apiserver's openapi doc api.
- AA: OpenAPI doc has to be manually generated by code-generating tools.
Authentication
要想很好的使用AA，就需要对kubernetes与 AA 之间认证机制进行有一定的了解，这里涉及到一些概念
在下面的说明中，所有出现的APIServer都是指Kubernetes集群组件APIServer也可以为 root APIServer；所有的AA都是指 extension apiserver，就是自行开发的 AA。
客户端证书
客户端证书就是CA签名的证书，由客户端指定CA证书，在客户端连接时进行身份验证，在Kubernetes APIserver也使用了相同的机制。
默认情况下，APIServer在启动时指定参数 --client-ca-file ，这时APIServer会创建一个名为 extension-apiserver-authentication ，命名空间为 kube-system 下的 configMap。- $ kubectl get cm -A
- NAMESPACE NAME DATA AGE
- kube-system extension-apiserver-authentication 6 21h
- kubectl get cm extension-apiserver-authentication -n kube-system -o yaml
- I0622 14:24:00.509486 1 secure_serving.go:178] Serving securely on [::]:443
- I0622 14:24:00.509556 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
Token认证是指通过HTTP Header传入 Authorization: Bearer $TOKEN 的方式进行客户端认证，这也是Kubernetes集群内认证常用的方法。
在这种情况下，允许对APIServer进行认证也同样可以对AA进行认证。如果不想 AA 对同一集群进行身份验证，或AA在集群外部运行，可以将参数 --authentication-kubeconfig 以指定要使用的不同 Kubeconfig 认证。
下面实例是AA的启动参数- ./bin/apiserver -h|grep authentication-kubeconfig
- --authentication-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough righ
- ts to create tokenreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no cli
- ent CA is looked up in the cluster.
RequestHeader 认证是指，APIServer对来自AA代理连接进行的身份认证。
默认情况下，AA 从 extension-apiserver-authentication 中提到的 ConfigMap 中 提取 requestheader 客户端 CA 证书与 CN。如果主 Kubernetes APIServer 配置了选项 --requestheader-client-ca-file ，则它会填充此内容。
跳过客户端认证 --authentication-skip-lookup
授权
默认情况下，AA 服务器会通过自动注入到 Kubernetes 集群上运行的 pod 的连接信息和凭据，来连接到主 Kubernetes API 服务器。- E0622 11:20:12.375512 1 errors.go:77] Post "https://192.168.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews": write tcp 192.168.0.36:39324->192.168.0.1:443: write: connection reset by peer
默认情况下，Kubernetes 集群会启用RBAC，这就意味着AA 创建多个clusterrolebinding。
下面日志是 AA 对于集群中资源访问无权限的情况- E0622 09:01:26.750320 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.10/tools/cache/reflector.go:125: Failed to list *v1.MutatingWebhookConfiguration: mutatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:default:default" cannot list resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope
- E0622 09:01:29.357897 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.10/tools/cache/reflector.go:125: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:default" cannot list resource "namespaces" in API group "" at the cluster scope
- E0622 09:01:39.998496 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.10/tools/cache/reflector.go:125: Failed to list *v1.ValidatingWebhookConfiguration: validatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:default:default" cannot list resource "validatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope
apiserver-builder
apiserver-builder 项目就是创建AA的工具，可以参考 installing.md 来安装
初始化项目
初始化命令
- 这个是你的API资源的组，参考 k8s.io/api
- 如果组的名称是域名就设置为主域名，例如内置组
- /apis/authentication.k8s.io
- /apis/batch
- 生成的go mod 包名为你所在的目录的名称
- 例如，在firewalld目录下，go.mod 的名称为 firewalld
- apiserver-boot init repo --domain <your-domain>
- apiserver-boot init repo --domain <your-domain>fedoraproject.org
注：这里--domain设置为主域名就可以了，后面生成的group会按照格式 +
- apiserver-boot must be run from the directory containing the go package to bootstrap. This must
- be under $GOPATH/src/<package>.
创建一个GVK
- apiserver-boot create group version resource \
- --group firewalld \
- --version v1 \
- --kind PortRule
- type PortRule struct {
- metav1.TypeMeta `json:",inline"`
- metav1.ObjectMeta `json:"metadata,omitempty"`
- Spec PortRuleSpec `json:"spec,omitempty"`
- Status PortRuleStatus `json:"status,omitempty"`
- }
- // PortRuleSpec defines the desired state of PortRule
- type PortRuleSpec struct { // 这里内容都为空的，自己添加即可
- Name string `json:"name"`
- Host string `json:"host"`
- Port int `json:"port"`
- IsPremanent bool `json:"isPremanent,omitempty"`
- }
- // PortRuleStatus defines the observed state of PortRule
- type PortRuleStatus struct {
- }
apiserver-boot 没有专门用来生成代码的命令，可以执行任意生成命令即可，这里使用生成二进制执行文件命令，这个过程相当长。- apiserver-boot build executables
运行方式
运行方式无非三种，本地运行，集群内运行，集群外运行
running_locally
本地运行需要有一个etcd服务，不用配置ca证书，这里使用docker运行- docker run -d --name Etcd-server \
- --publish 2379:2379 \
- --publish 2380:2380 \
- --env ALLOW_NONE_AUTHENTICATION=yes \
- --env ETCD_ADVERTISE_CLIENT_URLS=http://etcd-server:2379 \
- bitnami/etcd:latest
- apiserver-boot build executables
- apiserver-boot run local
构建镜像
需要先构建容器镜像，apiserver-boot build container --image 这将生成代码，构建 apiserver 和controller二进制文件，然后构建容器映像。构建完成后还需要将对应的镜像push到仓库（可选）- apiserver-boot build config \
- --name <servicename> \
- --namespace <namespace to run in> \
- --image <image to run>
注，这个操作需要在支持Linux内核的环境下构建，wsl不具备内核功能故会报错，需要替换为wsl2，而工具是下载的，如果需要wsl1+Docker Desktop构建，需要自己修改
构建配置
- apiserver-boot build config \
- --name <servicename> \
- --namespace <namespace to run in> \
- --image <image to run>
<ul>在 |
