18.2 k8s-apiserver监控源码解读

雁过留声  金牌会员 | 2024-9-28 08:48:45 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 847|帖子 847|积分 2541

本节重点介绍 :



  • k8s代码库和模块地址

    • 下载 apiserver源码

  • apiserver中监控源码阅读
k8s源码地址分布

k8s代码库



  • 访问github上k8s堆栈，readme中给出了k8s 模块的代码地址
  • 举例图片

组件堆栈列表 地址

Repositories currently staged here:


  • k8s.io/api
  • k8s.io/apiextensions-apiserver
  • k8s.io/apimachinery
  • k8s.io/apiserver
  • k8s.io/cli-runtime
  • k8s.io/client-go
  • k8s.io/cloud-provider
  • k8s.io/cluster-bootstrap
  • k8s.io/code-generator
  • k8s.io/component-base
  • k8s.io/controller-manager
  • k8s.io/cri-api
  • k8s.io/csi-api
  • k8s.io/csi-translation-lib
  • k8s.io/kube-aggregator
  • k8s.io/kube-controller-manager
  • k8s.io/kube-proxy
  • k8s.io/kube-scheduler
  • k8s.io/kubectl
  • k8s.io/kubelet
  • k8s.io/legacy-cloud-providers
  • k8s.io/metrics
  • k8s.io/mount-utils
  • k8s.io/pod-security-admission
  • k8s.io/sample-apiserver
  • k8s.io/sample-cli-plugin
  • k8s.io/sample-controller
下载 apiserver 源码

  1. go get -d  k8s.io/apiserver
复制代码
分析apiserver 监控源码

以qps指标 apiserver_request_total为例

定位源码位置



  • 在源码目次全文搜索apiserver_request_total，选择.go文件
  • 举例图片

  • 发现位于 D:\go_path\pkg\mod\k8s.io\apiserver@v0.22.1\pkg\endpoints\metrics\metrics.go
  1.         requestCounter = compbasemetrics.NewCounterVec(
  2.                 &compbasemetrics.CounterOpts{
  3.                         Name:           "apiserver_request_total",
  4.                         Help:           "Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code.",
  5.                         StabilityLevel: compbasemetrics.STABLE,
  6.                 },
  7.                 []string{"verb", "dry_run", "group", "version", "resource", "subresource", "scope", "component", "code"},
  8.         )
复制代码


  • 分析，在这里定义了指标，并且指定了相关的标签，可以和我们在prometheus中查询到的效果匹配上
  2. apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="apiregistration.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="apiservices", scope="cluster", verb="WATCH", version="v1beta1"}
  3. 9014
  4. apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="apps", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="daemonsets", scope="cluster", verb="WATCH", version="v1"}
  5. 8863
  6. apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="apps", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="deployments", scope="cluster", verb="WATCH", version="v1"}
  7. 8830
  8. apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="bgpconfigurations", scope="cluster", verb="WATCH", version="v1"}
  9. 5575
  10. apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="bgppeers", scope="cluster", verb="WATCH", version="v1"}
  11. 2791
  12. apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="blockaffinities", scope="cluster", verb="WATCH", version="v1"}
  13. 2942
  14. apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="clusterinformations", scope="cluster", verb="WATCH", version="v1"}
  15. 2770
复制代码
处理哀求的metric 函数 MonitorRequest



  • 代码位置 D:\go_path\pkg\mod\k8s.io\apiserver@v0.22.1\pkg\endpoints\metrics\metrics.go
  • 源码如下
  1. // MonitorRequest handles standard transformations for client and the reported verb and then invokes Monitor to record
  2. // a request. verb must be uppercase to be backwards compatible with existing monitoring tooling.
  3. func MonitorRequest(req *http.Request, verb, group, version, resource, subresource, scope, component string, deprecated bool, removedRelease string, httpCode, respSize int, elapsed time.Duration) {
  4.         // We don't use verb from <requestInfo>, as this may be propagated from
  5.         // InstrumentRouteFunc which is registered in installer.go with predefined
  6.         // list of verbs (different than those translated to RequestInfo).
  7.         // However, we need to tweak it e.g. to differentiate GET from LIST.
  8.         reportedVerb := cleanVerb(CanonicalVerb(strings.ToUpper(req.Method), scope), verb, req)
  10.         dryRun := cleanDryRun(req.URL)
  11.         elapsedSeconds := elapsed.Seconds()
  12.         requestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component, codeToString(httpCode)).Inc()
  15.         // MonitorRequest happens after authentication, so we can trust the username given by the request
  16.         info, ok := request.UserFrom(req.Context())
  17.         if ok && info.GetName() == user.APIServerUser {
  18.                 apiSelfRequestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, resource, subresource).Inc()
  19.         }
  20.         if deprecated {
  21.                 deprecatedRequestGauge.WithContext(req.Context()).WithLabelValues(group, version, resource, subresource, removedRelease).Set(1)
  22.                 audit.AddAuditAnnotation(req.Context(), deprecatedAnnotationKey, "true")
  23.                 if len(removedRelease) > 0 {
  24.                         audit.AddAuditAnnotation(req.Context(), removedReleaseAnnotationKey, removedRelease)
  25.                 }
  26.         }
  27.         requestLatencies.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component).Observe(elapsedSeconds)
  30.         // We are only interested in response sizes of read requests.
  31.         if verb == "GET" || verb == "LIST" {
  32.                 responseSizes.WithContext(req.Context()).WithLabelValues(reportedVerb, group, version, resource, subresource, scope, component).Observe(float64(respSize))
  33.         }
  34. }
复制代码


  • 其中 WithLabelValues设置metric的值 ，Inc()代表counter +1
  1. requestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component, codeToString(httpCode)).Inc()
复制代码


  • 这个是k8s在prometheus sdk上做的封装，位置 D:\go_path\pkg\mod\k8s.io\component-base@v0.22.1\metrics\counter.go
  1. func (v *CounterVec) WithLabelValues(lvs ...string) CounterMetric {
  2.         if !v.IsCreated() {
  3.                 return noop // return no-op counter
  4.         }
  5.         if v.LabelValueAllowLists != nil {
  6.                 v.LabelValueAllowLists.ConstrainToAllowedList(v.originalLabels, lvs)
  7.         }
  8.         return v.CounterVec.WithLabelValues(lvs...)
  9. }
复制代码
自己哀求的计数



  • 如果userName是 system:apiserver ，那么把代表自身哀求的metric apiserver_selfrequest_total +1
  1.         info, ok := request.UserFrom(req.Context())
  2.         if ok && info.GetName() == user.APIServerUser {
  3.                 apiSelfRequestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, resource, subresource).Inc()
  4.         }
复制代码
要被废弃的api被哀求



  • metric丢弃的设置，apiserver_requested_deprecated_apis代表要被废弃的api被哀求了，打印信息
  1.         if deprecated {
  2.                 deprecatedRequestGauge.WithContext(req.Context()).WithLabelValues(group, version, resource, subresource, removedRelease).Set(1)
  3.                 audit.AddAuditAnnotation(req.Context(), deprecatedAnnotationKey, "true")
  4.                 if len(removedRelease) > 0 {
  5.                         audit.AddAuditAnnotation(req.Context(), removedReleaseAnnotationKey, removedRelease)
  6.                 }
  7.         }
复制代码


  • prometheus查询效果
  1. apiserver_requested_deprecated_apis{group="apiregistration.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="apiservices", version="v1beta1"}
  2. 1
  3. apiserver_requested_deprecated_apis{group="authorization.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="subjectaccessreviews", version="v1beta1"}
  4. 1
  5. apiserver_requested_deprecated_apis{group="certificates.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="certificatesigningrequests", version="v1beta1"}
  6. 1
  7. apiserver_requested_deprecated_apis{group="extensions", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="ingresses", version="v1beta1"}
  8. 1
  9. apiserver_requested_deprecated_apis{group="networking.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="ingressclasses", version="v1beta1"}
  10. 1
  11. apiserver_requested_deprecated_apis{group="networking.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="ingresses", version="v1beta1"}
  12. 1
  13. apiserver_requested_deprecated_apis{group="scheduling.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="priorityclasses", version="v1beta1"}
  14. 1
复制代码
设置哀求延迟值



  • 代码如下
  1. requestLatencies.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component).Observe(elapsedSeconds)
复制代码


  • requestLatencies定义了从0.05到60秒的bucket
  1.         requestLatencies = compbasemetrics.NewHistogramVec(
  2.                 &compbasemetrics.HistogramOpts{
  3.                         Name: "apiserver_request_duration_seconds",
  4.                         Help: "Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.",
  5.                         // This metric is used for verifying api call latencies SLO,
  6.                         // as well as tracking regressions in this aspects.
  7.                         // Thus we customize buckets significantly, to empower both usecases.
  8.                         Buckets: []float64{0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
  9.                                 1.25, 1.5, 1.75, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60},
  10.                         StabilityLevel: compbasemetrics.STABLE,
  11.                 },
  12.                 []string{"verb", "dry_run", "group", "version", "resource", "subresource", "scope", "component"},
  13.         )
复制代码


  • 分位值查询
  1. histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job="kubernetes-apiservers"}[5m])) by (verb, le))
复制代码


  • 效果图片

设置 http响应的巨细histogram



  • 设置代码
  1.         if verb == "GET" || verb == "LIST" {
  2.                 responseSizes.WithContext(req.Context()).WithLabelValues(reportedVerb, group, version, resource, subresource, scope, component).Observe(float64(respSize))
  3.         }
复制代码


  • responseSizes定义了 1KB 到1GB的巨细的bucket
  1.         responseSizes = compbasemetrics.NewHistogramVec(
  2.                 &compbasemetrics.HistogramOpts{
  3.                         Name: "apiserver_response_sizes",
  4.                         Help: "Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component.",
  5.                         // Use buckets ranging from 1000 bytes (1KB) to 10^9 bytes (1GB).
  6.                         Buckets:        compbasemetrics.ExponentialBuckets(1000, 10.0, 7),
  7.                         StabilityLevel: compbasemetrics.ALPHA,
  8.                 },
  9.                 []string{"verb", "group", "version", "resource", "subresource", "scope", "component"},
  10.         )
复制代码
追踪MonitorRequest调用链

InstrumentRouteFunc在prometheus的HandlerFunc底子上封装了k8s的go-restful

  1. // InstrumentRouteFunc works like Prometheus' InstrumentHandlerFunc but wraps
  2. // the go-restful RouteFunction instead of a HandlerFunc plus some Kubernetes endpoint specific information.
  3. func InstrumentRouteFunc(verb, group, version, resource, subresource, scope, component string, deprecated bool, removedRelease string, routeFunc restful.RouteFunction) restful.RouteFunction {
  4.         return restful.RouteFunction(func(req *restful.Request, response *restful.Response) {
  5.                 requestReceivedTimestamp, ok := request.ReceivedTimestampFrom(req.Request.Context())
  6.                 if !ok {
  7.                         requestReceivedTimestamp = time.Now()
  8.                 }
  10.                 delegate := &ResponseWriterDelegator{ResponseWriter: response.ResponseWriter}
  12.                 //lint:file-ignore SA1019 Keep supporting deprecated http.CloseNotifier
  13.                 _, cn := response.ResponseWriter.(http.CloseNotifier)
  14.                 _, fl := response.ResponseWriter.(http.Flusher)
  15.                 _, hj := response.ResponseWriter.(http.Hijacker)
  16.                 var rw http.ResponseWriter
  17.                 if cn && fl && hj {
  18.                         rw = &fancyResponseWriterDelegator{delegate}
  19.                 } else {
  20.                         rw = delegate
  21.                 }
  22.                 response.ResponseWriter = rw
  24.                 routeFunc(req, response)
  26.                 MonitorRequest(req.Request, verb, group, version, resource, subresource, scope, component, deprecated, removedRelease, delegate.Status(), delegate.ContentLength(), time.Since(requestReceivedTimestamp))
  27.         })
  28. }
复制代码
registerResourceHandlers 中根据不同的verb处理



  • 位置 D:\go_path\pkg\mod\k8s.io\apiserver@v0.22.1\pkg\endpoints\installer.go
  • LIST的verb代码如下
  1.                 case "LIST": // List all resources of a kind.
  2.                         doc := "list objects of kind " + kind
  3.                         if isSubresource {
  4.                                 doc = "list " + subresource + " of objects of kind " + kind
  5.                         }
  6.                         handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, deprecated, removedRelease, restfulListResource(lister, watcher, reqScope, false, a.minRequestTimeout))
  7.                         handler = utilwarning.AddWarningsHandler(handler, warnings)
  8.                         route := ws.GET(action.Path).To(handler).
  9.                                 Doc(doc).
  10.                                 Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
  11.                                 Operation("list"+namespaced+kind+strings.Title(subresource)+operationSuffix).
  12.                                 Produces(append(storageMeta.ProducesMIMETypes(action.Verb), allMediaTypes...)...).
  13.                                 Returns(http.StatusOK, "OK", versionedList).
  14.                                 Writes(versionedList)
复制代码


  • 上层被APIInstaller调用
  1. // Install handlers for API resources.
  2. func (a *APIInstaller) Install() ([]metav1.APIResource, []*storageversion.ResourceInfo, *restful.WebService, []error) {
  3.         var apiResources []metav1.APIResource
  4.         var resourceInfos []*storageversion.ResourceInfo
  5.         var errors []error
  6.         ws := a.newWebService()
  8.         // Register the paths in a deterministic (sorted) order to get a deterministic swagger spec.
  9.         paths := make([]string, len(a.group.Storage))
  10.         var i int = 0
  11.         for path := range a.group.Storage {
  12.                 paths[i] = path
  13.                 i++
  14.         }
  15.         sort.Strings(paths)
  16.         for _, path := range paths {
  17.                 apiResource, resourceInfo, err := a.registerResourceHandlers(path, a.group.Storage[path], ws)
  18.                 if err != nil {
  19.                         errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err))
  20.                 }
  21.                 if apiResource != nil {
  22.                         apiResources = append(apiResources, *apiResource)
  23.                 }
  24.                 if resourceInfo != nil {
  25.                         resourceInfos = append(resourceInfos, resourceInfo)
  26.                 }
  27.         }
  28.         return apiResources, resourceInfos, ws, errors
  29. }
复制代码
本节重点总结 :



  • k8s代码库和模块地址

    • 下载 apiserver源码

  • apiserver中监控源码阅读

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

本帖子中包含更多资源

您需要 登录 才可以下载或查看，没有账号？立即注册

x
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

雁过留声

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

存储 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表