From 5118c9ee1eaa669926f94d15b836154469f56175 Mon Sep 17 00:00:00 2001 From: Solly Ross Date: Tue, 7 Aug 2018 15:47:49 -0400 Subject: [PATCH] Add some FAQs to the README This adds some FAQs to the README containing information about certs, labelling, configuration, quantities, and multiple metrics. --- README.md | 90 +++++++++++++++++++++++++++++++++++++++++++++ docs/config.md | 2 +- docs/walkthrough.md | 2 + 3 files changed, 93 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 04207b3c..44dfa484 100644 --- a/README.md +++ b/README.md @@ -83,3 +83,93 @@ attention to: Operator](https://github.com/luxas/kubeadm-workshop#deploying-the-prometheus-operator-for-monitoring-services-in-the-cluster) - [Setting up the custom metrics adapter and sample app](https://github.com/luxas/kubeadm-workshop#deploying-a-custom-metrics-api-server-and-a-sample-app) + +FAQs +---- + +### Why do my metrics keep jumping between a normal value and a very large number? + +You're probably switching between whole numbers (e.g. `10`) and milli-quantities (e.g. `10500m`). +Worry not! This is just how Kubernetes represents fractional values. See the +[Quantity Values](/docs/walkthrough.md#quantity-values) section of the walkthrough for a bit more +information. + +### Why isn't my metric showing up? + +First, check your configuration. Does it select your metric? You can +find the [default configuration](/deploy/custom-metrics-config-map.yaml) +in the deploy directory, and more information about configuring the +adapter in the [docs](/docs/config.md). + +Next, check if the discovery information looks right. You should see the +metrics showing up as associated with the resources you expect at +`/apis/custom.metrics.k8s.io/v1beta1/` (you can use `kubectl get --raw +/apis/custom.metrics.k8s.io/v1beta1` to check, and can pipe to `jq` to +pretty-print the results, if you have it installed). If not, make sure +your series are labeled correctly. Consumers of the custom metrics API +(especially the HPA) don't do any special logic to associate a particular +resource to a particular series, so you have to make sure that the adapter +does it instead. + +For example, if you want a series `foo` to be associated with deployment +`bar` in namespace `somens`, make sure there's some label that represents +deployment name, and that the adapter is configured to use it. With the +default config, that means you'd need the query +`foo{namespace="somens",deployment="bar"}` to return some results in +Prometheus. + +Next, try using the `--v=6` flag on the adapter to see the exact queries +being made by the adapter. Try url-decoding the query and pasting it into +the Prometheus web console to see if the query looks wrong. + +### My query contains multiple metrics, how do I make that work? + +It's actually fairly straightforward, if a bit non-obvious. Simply choose one +metric to act as the "discovery" and "naming" metric, and use that to configure +the "discovery" and "naming" parts of the configuration. Then, you can write +whichever metrics you want in the `metricsQuery`! The series query can contain +whichever metrics you want, as long as they have the right set of labels. + +For example, if you have two metrics `foo_total` and `foo_count`, you might write + +```yaml +rules: +- seriesQuery: 'foo_total' + resources: {overrides: {system_name: {resource: "node"}}} + name: + matches: 'foo_total' + as: 'foo' + metricsQuery: 'sum(foo_total) by (<<.GroupBy>>) / sum(foo_count) by (<<.GroupBy>>)' +``` + +### I get errors about SubjectAccessReviews/system:anonymous/TLS/Certificates/RequestHeader! + +It's important to understand the role of TLS in the Kubernetes cluster. There's a high-level +overview here: https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md. + +All of the above errors generally boil down to misconfigured certificates. +Specifically, you'll need to make sure your cluster's aggregation layer is +properly configured, with requestheader certificates set up properly. + +Errors about SubjectAccessReviews failing for system:anonymous generally mean +that your cluster's given requestheader CA doesn't trust the proxy certificates +from the API server aggregator. + +On the other hand, if you get an error from the aggregator about invalid certificates, +it's probably because the CA specified in the `caBundle` field of your APIService +object doesn't trust the serving certificates for the adapter. + +If you're seeing SubjectAccessReviews failures for non-anonymous users, check your +RBAC rules -- you probably haven't given users permission to operate on resources in +the `custom.metrics.k8s.io` API group. + +### My metrics appear and disappear + +You probably have a Prometheus collection interval or computation interval +that's larger than your adapter's discovery interval. If the metrics +appear in discovery but occaisionally return not-found, those intervals +are probably larger than one of the rate windows used in one of your +queries. The adapter only considers metrics with datapoints in the window +`[now-discoveryInterval, now]` (in order to only capture metrics that are +still present), so make sure that your discovery interval is at least as +large as your collection interval. diff --git a/docs/config.md b/docs/config.md index e4abb029..64a3c6cd 100644 --- a/docs/config.md +++ b/docs/config.md @@ -188,7 +188,7 @@ Kubernetes resources. Then, if someone requested the metric `pods/http_request_per_second` for the pods `pod1` and `pod2` in the `somens` namespace, we'd have: -- `Series: "http_requests_total" +- `Series: "http_requests_total"` - `LabelMatchers: "pod=~\"pod1|pod2",namespace="somens"` - `GroupBy`: `pod` diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 42b8ad6e..e2da48d5 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -273,6 +273,8 @@ Try fetching the metrics again. You should see an increase in the rate after the collection interval specified in your Prometheus configuration has elapsed. If you leave it for a bit, the rate will go back down again. +### Quantity Values + Notice that the API uses Kubernetes-style quantities to describe metric values. These quantities use SI suffixes instead of decimal points. The most common to see in the metrics API is the `m` suffix, which means