From 9f08038f07f523a5a56343d4e9b10b4adc927e6f Mon Sep 17 00:00:00 2001 From: Solly Ross Date: Tue, 7 Aug 2018 15:11:47 -0400 Subject: [PATCH] Add a config walkthrough and update the readme A helful community member rightly pointed out that configuring the adapter was a bit confusing, and a step-by-step example would be useful. This adds such an example, and links to it from relevant places. --- README.md | 7 ++ docs/config-walkthrough.md | 230 +++++++++++++++++++++++++++++++++++++ docs/config.md | 7 ++ docs/walkthrough.md | 5 +- 4 files changed, 248 insertions(+), 1 deletion(-) create mode 100644 docs/config-walkthrough.md diff --git a/README.md b/README.md index 34da55c6..04207b3c 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,13 @@ metrics API suitable for use with the autoscaling/v2 Horizontal Pod Autoscaler in Kubernetes 1.6+. +Quick Links +----------- + +- [Config walkthrough](docs/config-walkthrough.md) and [config reference](docs/config.md). +- [End-to-end walkthrough](docs/walkthrough.md) +- [Deployment info and files](deploy/README.md) + Configuration ------------- diff --git a/docs/config-walkthrough.md b/docs/config-walkthrough.md new file mode 100644 index 00000000..e71ae1a5 --- /dev/null +++ b/docs/config-walkthrough.md @@ -0,0 +1,230 @@ +Configuration Walkthroughs +========================== + +*If you're looking for reference documentation on configuration, please +read the the [configuration reference](/docs/config.md)* + +Per-pod HTTP Requests +--------------------- + +### Background + +*The [full walkthrough](/docs/walkthrough.md) sets up a the background for +something like this* + +Suppose we have some frontend webserver, and we're trying to write an +configuration for the Promtheus adapter so that we can autoscale it based +on the HTTP requests per second that it receives. + +Before starting, we've gone and instrumented our frontend server with +a metric, `http_requests_total`. It is exposed with a single label, +`method`, breaking down the requests by HTTP verb. + +We've configured our Prometheus to collect the metric, and our promethues +adds the `kubernetes_namespace` and `kubernetes_pod_name` labels, +representing namespace and pod, respectively. + +If we query Prometheus, we see series that look like + +``` +http_requests_total{method="GET",kubernetes_namespace="production",kubernetes_pod_name="frontend-server-abcd-0123"} +``` + +### Configuring the adapter + +The adapter considers metrics in the following ways: + +1. First, It discovers the metrics available (*Discovery*) + +2. Then, it figures out which Kubernetes resources each metric is + associated with (*Association*) + +3. Then, it figures out how it should expose them to the custom metrics + API (*Naming*) + +4. Finally, it figures out how it should query Prometheus to get the + actual numbers (*Querying*) + +We need to inform the adapter how it should perform each of these steps +for our metric, `http_requests_total`, so we'll need to add a new +***rule***. Each rule in the adapter encodes these steps. Let's add a new +one to our configuration: + +```yaml +rules: +- {} +``` + +If we want to find all `http_requests_total` series ourselves in the +Prometheus dashboard, we'd write +`http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}` to +find all find all `http_requests_total` series that were associated with +a namespace and pod. + +We can add this to our rule in the `seriesQuery` field, to tell the +adapter how *discover* the right series itself: + +```yaml +rules: +- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' +``` + +Next, we'll need to tell the adapter how to figure out which Kubernetes +resources are associated with the metric. We've already said that +`kubernetes_namespace` represents the namespace name, and +`kubernetes_pod_name` represents the pod name. Since these names don't +quite follow a consistent pattern, we use the `overrides` section of the +`resources` field in our rule: + +```yaml +rules: +- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' + resources: + overrides: + kubernetes_namespace: {resource: "namespace"} + kubernetes_pod_name: {resource: "pod"} +``` + +This says that each label represents its corresponding resource. Since the +resources are in the "core" kubernetes API, we don't need to specify +a group. The adapter will automatically take care of pluralization, so we +can specify either `pod` or `pods`, just the same way as in `kubectl get`. +The resources can be any resource available in your kubernetes cluster, as +long as you've got a corresponding label. + +If our labels followed a consistent pattern, like `kubernetes_`, +we could specify `resources: {template: "kubernetes_<<.Resource>>"}` +instead of specifying an override for each resource. If you want to see +all resources currently available in your cluster, you can use the +`kubectl api-resources` command (but the list of available resources can +change as you add or remove CRDs or aggregated API servers). For more +information on resources, see [Kinds, Resources, and +Scopes](https://github.com/kubernetes-incubator/custom-metrics-apiserver/blob/master/docs/getting-started.md#kinds-resources-and-scopes) +in the custom-metrics-apiserver boilerplate guide. + +Now, cumulative metrics (like those that end in `_total`) aren't +particularly useful for autoscaling, so we want to convert them to rate +metrics in the API. We'll call the rate version of our metric +`http_requests_per_second`. We can use the the `name` field to tell the +adapter about that: + +```yaml +rules: +- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' + resources: + overrides: + kubernetes_namespace: {resource: "namespace"} + kubernetes_pod_name: {resource: "pod"} + name: + matches: "^(.*)_total" + as: "${1}_per_second" +``` + +Here, we've said that we should take the name matching +`_total`, and turning it into `_per_second`. + +Finally, we need to tell the adapter how to actually query Prometheus to +get some numbers. Since we want a rate, we might write: +`sum(rate(http_requests_total{kubernetes_namespace="production",kubernetes_pod_name=~"frontend-server-abcd-0123|fronted-server-abcd-4567"}) by (kubernetes_pod_name)`, +which would get us the total requests per second for each pod, summed across verbs. + +We can write something similar in the adapter, using the `metricsQuery` +field: + +```yaml +rules: +- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' + resources: + overrides: + kubernetes_namespace: {resource: "namespace"} + kubernetes_pod_name: {resource: "pod"} + name: + matches: "^(.*)_total" + as: "${1}_per_second" + metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)' +``` + +The adapter will automatically fill in the right series name, label +matchers, and group-by clause, depending on what we put into the API. +Since we're only working with a single metric anyway, we could replace +`<<.Series>>` with `http_requests_total`. + +Now, if we run an instance of the Prometheus adapter with this +configuration, we should see discovery information at +`$KUBERNETES/apis/custom.metrics.k8s.io/v1beta1/` of + +```json +{ + "kind": "APIResourceList", + "apiVersion": "v1", + "groupVersion": "custom.metrics.k8s.io/v1beta1", + "resources": [ + { + "name": "pods/http_requests_total", + "singularName": "", + "namespaced": true, + "kind": "MetricValueList", + "verbs": ["get"] + }, + { + "name": "namespaces/http_requests_total", + "singularName": "", + "namespaced": false, + "kind": "MetricValueList", + "verbs": ["get"] + } + ] +} +``` + +Notice that we get an entry for both "pods" and "namespaces" -- the +adapter exposes the metric on each resource that we've associated the +metric with (and all namespaced resources must be associated with +a namespace), and will fill in the `<<.GroupBy>>` section with the +appropriate label depending on which we ask for. + +We can now connect to +`$KUBERNETES/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second`, +and we should see + +```json +{ + "kind": "MetricValueList", + "apiVersion": "custom.metrics.k8s.io/v1beta1", + "metadata": { + "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second", + }, + "items": [ + { + "describedObject": { + "kind": "Pod", + "name": "frontend-server-abcd-0123", + "apiVersion": "/__internal", + }, + "metricName": "http_requests_per_second", + "timestamp": "2018-08-07T17:45:22Z", + "value": "16m" + }, + { + "describedObject": { + "kind": "Pod", + "name": "frontend-server-abcd-4567", + "apiVersion": "/__internal", + }, + "metricName": "http_requests_per_second", + "timestamp": "2018-08-07T17:45:22Z", + "value": "22m" + } + ] +} +``` + +This says that our server pods are receiving 16 and 22 milli-requests per +second (depending on the pod), which is 0.016 and 0.022 requests per +second, written out as a decimal. That's about what we'd expect with +little-to-no traffic except for the Prometheus scrape. + +If we added some traffic to our pods, we might see `1` or `20` instead of +`16m`, which would be `1` or `20` requests per second. We might also see +`20500m`, which would mean 20500 milli-requests per second, or 20.5 +requests per second in decimal form. diff --git a/docs/config.md b/docs/config.md index d5fccdf1..e4abb029 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,6 +1,10 @@ Metrics Discovery and Presentation Configuration ================================================ +*If you want a full walkthrough of configuring the adapter for a sample +metric, please read the [configuration +walkthrough](/docs/config-walkthrough.md)* + The adapter determines which metrics to expose, and how to expose them, through a set of "discovery" rules. Each rule is executed independently (so make sure that your rules are mutually exclusive), and specifies each @@ -123,6 +127,9 @@ resource: These two can be combined, so you can specify both a template and some individual overrides. +The resources mentioned can be any resource available in your kubernetes +cluster, as long as you've got a corresponding label. + Naming ------ diff --git a/docs/walkthrough.md b/docs/walkthrough.md index fb285463..19b6ae10 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -146,7 +146,10 @@ You may also need to modify the ConfigMap containing the metrics discovery configuration. If you're using the Prometheus configuration described above, it should work out of the box in common cases. Otherwise, read the [configuration documentation](/docs/config.md) to learn how to configure -the adapter for your particular metrics and labels. +the adapter for your particular metrics and labels. The [configuration +walkthrough](/docs/config-walkthrough.md) gives an end-to-end +configuration tutorial for configure the adapter for a scenario similar to +this one. ### The Registered API ###