diff --git a/README.md b/README.md index 4bfc29b5..c006b4cd 100644 --- a/README.md +++ b/README.md @@ -37,55 +37,33 @@ adapter talks to Prometheus and the main Kubernetes cluster: Presentation ------------ -The adapter gathers the names of available metrics from Prometheus at the -specified interval. Only metrics of the following forms are considered: +The adapter gathers the names of available metrics from Prometheus +a regular interval (see [Configuration](#configuration) above), and then +only exposes metrics that follow specific forms. -- "container" metrics (cAdvisor container metrics): series with a name - starting with `container_`, as well as non-empty `namespace` and - `pod_name` labels. +In general: -- "namespaced" metrics (metrics describing namespaced Kubernetes objects): - series with non-empty namespace labels (which don't start with - `container_`). +- Metrics must have the `namespace` label to be considered. -*Note*: Currently, metrics on non-namespaced objects (besides namespaces -themselves) are not supported. +- For each label on a metric, if that label name corresponds to + a Kubernetes resource (like `pod` or `service`), the metric will be + associated with that resource. -Metrics in Prometheus are converted in the custom-metrics-API metrics as -follows: +- Metrics ending in `_total` are assumed to be cumulative, and will be + exposed without the suffix as a rate metric. -1. The metric name and type are decided: - - For container metrics, the `container_` prefix is removed - - If the metric has the `_total` suffix, it is marked as a counter - metric, and the suffix is removed - - If the metric has the `_seconds_total` suffix, it is marked as - a seconds counter metric, and the suffix is removed. - - If the metric has none of the above suffixes, is is marked as a gauge - metric, and the metric name is used as-is - -2. Relevant resources are associated with the metric: - - container metrics are associated with pods only - - for non-container metrics, each label on the series is considered. If - that label represents a resource (without the group) available on the - server, the metric is associated with that resource. A metric may be - associated with multiple resources. - -When retrieving counter and seconds-counter metrics, the adapter requests -the metrics as a rate over the configured amount of time. For metrics -with multiple associated resources, the adapter requests the metric -aggregated over all non-requested metrics. - -The adapter does not consider resources consumed by the "POD" container, -which exists as part of all Kubernetes pods running in Docker simply -supports the existance of the pod's shared network namespace. +Detailed information can be found under [docs/format.md](docs/format.md). Example ------- -@luxas has an excellent example deployment of Prometheus, this adapter, -and a demo pod which serves a metric `http_requests_total`, which becomes -the custom metrics API metric `pods/http_requests`. It also autoscales on -that metric using the `autoscaling/v2alpha1` HorizontalPodAutoscaler. +A brief walkthrough exists in [docs/walkthrough.md](docs/walkthrough.md). + +Additionally, [@luxas](https://github.com/luxas) has an excellent example +deployment of Prometheus, this adapter, and a demo pod which serves +a metric `http_requests_total`, which becomes the custom metrics API +metric `pods/http_requests`. It also autoscales on that metric using the +`autoscaling/v2alpha1` HorizontalPodAutoscaler. It can be found at https://github.com/luxas/kubeadm-workshop. Pay special attention to: diff --git a/docs/format.md b/docs/format.md new file mode 100644 index 00000000..59ea6cef --- /dev/null +++ b/docs/format.md @@ -0,0 +1,44 @@ +Metrics Format and Presentation +=============================== + +The adapter gathers the names of available metrics from Prometheus at the +specified interval. Only metrics of the following forms are considered: + +- "container" metrics (cAdvisor container metrics): series with a name + starting with `container_`, as well as non-empty `namespace` and + `pod_name` labels. + +- "namespaced" metrics (metrics describing namespaced Kubernetes objects): + series with non-empty namespace labels (which don't start with + `container_`). + +*Note*: Currently, metrics on non-namespaced objects (besides namespaces +themselves) are not supported. + +Metrics in Prometheus are converted in the custom-metrics-API metrics as +follows: + +1. The metric name and type are decided: + - For container metrics, the `container_` prefix is removed + - If the metric has the `_total` suffix, it is marked as a counter + metric, and the suffix is removed + - If the metric has the `_seconds_total` suffix, it is marked as + a seconds counter metric, and the suffix is removed. + - If the metric has none of the above suffixes, is is marked as a gauge + metric, and the metric name is used as-is + +2. Relevant resources are associated with the metric: + - container metrics are associated with pods only + - for non-container metrics, each label on the series is considered. If + that label represents a resource (without the group) available on the + server, the metric is associated with that resource. A metric may be + associated with multiple resources. + +When retrieving counter and seconds-counter metrics, the adapter requests +the metrics as a rate over the configured amount of time. For metrics +with multiple associated resources, the adapter requests the metric +aggregated over all non-requested metrics. + +The adapter does not consider resources consumed by the "POD" container, +which exists as part of all Kubernetes pods running in Docker simply +supports the existance of the pod's shared network namespace. diff --git a/docs/walkthrough.md b/docs/walkthrough.md new file mode 100644 index 00000000..bfad6d2d --- /dev/null +++ b/docs/walkthrough.md @@ -0,0 +1,507 @@ +Walkthrough +=========== + +This walkthrough will go over the basics of setting up the Prometheus +adapter on your cluster and configuring an autoscaler to use application +metrics sourced from the adapter. + +Prerequisites +------------- + +### Cluster Configuration ### + +Before getting started, ensure that the main components of your +cluster are configured for autoscaling on custom metrics. As of +Kubernetes 1.7, this requires enabling the aggregation layer on the API +server and configuring the controller manager to use the metrics APIs via +their REST clients. + +Detailed instructions can be found in the Kubernetes documentation under +[Horizontal Pod +Autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics). + +Make sure that you've properly configured Heapster with the `--api-server` +flag, otherwise enabling custom metrics autoscaling support with disable +CPU autoscaling support. + +### Binaries and Images ### + +In order to follow this walkthrough, you'll need container images for +Prometheus and the custom metrics adapter. + +Both can be found on Dockerhub under `prom/prometheus` and +`directxman12/k8s-prometheus-adapter`, respectively. + +If you're feeling adventurous, you can build the latest version of the +custom metrics adapter by running `make docker-build`. + +Launching Prometheus and the Adapter +------------------------------------ + +In this walkthrough, it's assumed that you're deploying Prometheus into +its own namespace called `prom`. Most of the sample commands and files +are namespace-agnostic, but there are a few commands that rely on +namespace. If you're using a different namespace, simply substitute that +in for `prom` when it appears. + +### Prometheus Configuration ### + +If you've never deployed Prometheus before, you'll need an appropriate +Prometheus configuration. There's a extensive sample Prometheus +configuration in the Prometheus repository +[here](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml). +Be sure to also read the Prometheus documentation on [configuring +Prometheus](https://prometheus.io/docs/operating/configuration/). + +For the purposes of this walkthrough, you'll need the following +configuration options to be set: + +
+ +prom-cfg.yaml + +```yaml +# a short scrape interval means you can respond to changes in +# metrics more quickly +global: + scrape_interval: 15s + +# you need a scrape configuration for scraping from pods +scrape_configs: +- job_name: 'kubernetes-pods' + # if you want to use metrics on jobs, set the below field to + # true to prevent Prometheus from setting the `job` label + # automatically. + honor_labels: false + kubernetes_sd_configs: + - role: pod + # skip verification so you can do HTTPS to pods + tls_config: + insecure_skip_verify: true + # make sure your labels are in order + relabel_configs: + # these labels tell Prometheus to automatically attach source + # pod and namespace information to each collected sample, so + # that they'll be exposed in the custom metrics API automatically. + - source_labels: [__meta_kubernetes_namespace] + action: replace + target_label: namespace + - source_labels: [__meta_kubernetes_pod_name] + action: replace + target_label: pod + # these labels tell Prometheus to look for + # prometheus.io/{scrape,path,port} annotations to configure + # how to scrape + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + action: keep + regex: true + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] + action: replace + target_label: __metrics_path__ + regex: (.+) + - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] + action: replace + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + target_label: __address__ + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme] + action: replace + target_label: __scheme__ + regex: (.+) +``` + +
+ +Place your full configuration (it might just be the above) in a file (call +it `prom-cfg.yaml`, for example), and then create a ConfigMap containing +it: + +```shell +$ kubectl -n prom create configmap prometheus --from-file=prometheus.yml=prom-cfg.yaml +``` + +You'll be using this later when you launch Prometheus. + +### Launching the Deployment ### + +It's generally easiest to launch the adapter and Prometheus as two +containers in the same pod. You can use a deployment to manage your +adapter and Prometheus instance. + +First, we'll create a ServiceAccount for the deployment to run as: + +```yaml +$ kubectl -n prom create serviceaccount prom-cm-adapter +``` + +Start out with a fairly straightforward Prometheus deployment using the +ConfigMap from above, and proceed from there: + +
+ +prom-adapter.deployment.yaml [Prometheus only] + +```yaml +apiVersion: apps/v1beta1 +kind: Deployment +metadata: + name: prometheus +spec: + replicas: 1 + selector: + matchLabels: + app: prometheus + template: + metadata: + labels: + app: prometheus + spec: + serviceAccountName: prom-cm-adapter + containers: + - image: prom/prometheus:v1.6.1 + name: prometheus + args: + - -storage.local.retention=6h + - -storage.local.memory-chunks=500000 + # point prometheus at the configuration that you mount in below + - -config.file=/etc/prometheus/prometheus.yml + ports: + # this port exposes the dashboard and the HTTP API + - containerPort: 9090 + name: prom-web + protocol: TCP + volumeMounts: + # you'll mount your ConfigMap volume at /etc/prometheus + - mountPath: /etc/prometheus + name: prom-config + volumes: + # make your configmap available as a volume, so that you can + # mount in the Prometheus config from earlier + - name: config-volume + configMap: + name: prometheus +``` + +
+ +Save this file (for example, as `prom-adapter.deployment.yaml`). Now, +you'll need to modify the deployment to include the adapter. + +The adapter has several options, most of which it shares with any other +standard Kubernetes addon API server. This means that you'll need proper +API server certificates for it to function properly. To learn more about +which certificates are needed, and what they mean, see [Concepts: Auth and +Certificates](https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md). + +Once you've generated the necessary certificates, you'll need to place +them in a secret. This walkthrough assumes that you've set up your +cluster to automatically inject client certificates and CA certificates +(see the above concepts documentation). Make sure you've granted the +default service account for your namespace permission to fetch the +authentication CA ConfigMap: + +```shell +$ kubectl create rolebinding prom-ext-auth-reader --role="extension-apiserver-authentication-reader" --serviceaccount=prom:prom-cm-adapter +``` + +Then, store your serving certificates in a secret: + +```shell +$ kubectl -n prom create secret tls serving-cm-adapter --cert=/path/to/cm-adapter/serving.crt --key=/path/to/cm-adapter/serving.key +``` + +Finally, you'll need to make sure that the default service account for +your namespace has permission to list resources in the cluster: + +```shell +$ kubectl create clusterrole resource-lister --verb=list --resource="*" +$ kubectl create clusterrolebinding cm-adapter-resource-lister --clusterrole=resource-lister -- serviceaccount=prom:prom-cm-adapter +``` + +Next, amend the file above to run the adapter as well. You may need to +modify this part if you wish to inject the needed certificates a different +way. + +
+ +prom-adapter.deployment.yaml [Adapter & Prometheus] + +```yaml +... +spec: + containers: + ... + - image: directxman12/k8s-prometheus-adapter + name: cm-adapter + args: + - --secure-port=6443 + - --logtostderr=true + # use your serving certs + - --tls-cert-file=/var/run/serving-cert/tls.crt + - --tls-private-key-file=/var/run/serving-cert/tls.key + # Prometheus is running in the same pod, so you can say your Prometheus + # is at `localhost` + - --prometheus-url=http://localhost:9090 + # relist available Prometheus metrics every 1m + - --metrics-relist-interval=1m + # calculate rates for cumulative metrics over 30s periods. This should be *at least* + # as much as your collection interval for Prometheus. + - --rate-interval=30s + # re-discover new available resource names every 10m. You probably + # won't need to have this be particularly often, but if you add + # additional addons to your cluster, the adapter will discover there + # existance at this interval + - --discovery-interval=10m + - --v=4 + ports: + # this port exposes the custom metrics API + - containerPort: 6443 + name: https + protocol: TCP + volumeMounts: + - mountPath: /var/run/serving-certs + name: serving-certs + readOnly: true + volumes: + ... + - name: serving-certs + secret: + secretName: serving-cm-adapter +``` + +
+ +Next, create the deployment and expose it as a service, mapping port 443 +to your pod's port 443, on which you've exposed the custom metrics API: + +```shell +$ kubectl -n prom create -f prom-adapter.deployment.yaml +$ kubectl -n prom create service clusterip prometheus --tcp=443:6443 +``` + +### Registering the API ### + +Now that you have a running deployment of Prometheus and the adapter, +you'll need to register it as providing the +`custom-metrics.metrics.k8s.io/v1alpha` API. + +For more information on how this works, see [Concepts: +Aggregation](https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/aggregation.md). + +You'll need to create an API registration record for the +`custom-metrics.metrics.k8s.io/v1alpha1` API. In order to do this, you'll +need the base64 encoded version of the CA certificate used to sign the +serving certificates you created above. If the CA certificate is stored +in `/tmp/ca.crt`, you can get the base64-encoded form like this: + +```shell +$ base64 --w 0 < /tmp/ca.crt +``` + +Take the resulting value, and place it into the following file: + +
+ +cm-registration.yaml + +```yaml +apiVersion: apiregistration.k8s.io/v1beta1 +kind: APIService +metadata: + name: v1alpha1.custom-metrics.metrics.k8s.io +spec: + # this tells the aggregator how to verify that your API server is + # actually who it claims to be + caBundle: + # these specify which group and version you're registering the API + # server for + group: custom-metrics.metrics.k8s.io + version: v1alpha1 + # these control how the aggregator prioritizes your registration. + # it's not particularly relevant in this case. + groupPriorityMinimum: 1000 + versionPriority: 10 + # finally, this points the aggregator at the service for your + # API server that you created + service: + name: prometheus + namespace: prom +``` + +
+ +Register that registration object with the aggregator: + +```shell +$ kubectl -f cm-registration.yaml +``` + +### Double-Checking Your Work ### + +With that all set, your custom metrics API should show up in discovery. + +Try fetching the discovery information for it: + +```shell +$ kubectl get --raw /apis/custom-metrics.metrics.k8s.io/v1alpha1 +``` + +Since you don't have any metrics collected yet, you shouldn't see any +available resources, but the request should return successfully. Keep +this command in mind -- you'll want to use it later once you have a pod +producing custom metrics. + +Collecting Application Metrics +------------------------------ + +Now that you have a working pipeline for ingesting application metrics, +you'll need an application that produces some metrics. Any application +which produces Prometheus-formatted metrics will do. For the purposes of +this walkthrough, try out [@luxas](https://github.com/luxas)'s simple HTTP +counter in the `luxas/autoscale-demo` image on Dockerhub: + +
+ +sample-app.deploy.yaml + +```yaml +apiVersion: apps/v1beta1 +kind: Deployment +metadata: + name: sample-app +spec: + replicas: 1 + selector: + matchLabels: + app: sample-app + template: + metadata: + labels: + app: sample-app + annotations: + # based on your Prometheus config above, this tells prometheus + # to scrape this pod for metrics on port 8080 at "/metrics" + prometheus.io/scrape: true + prometheus.io/port: 8080 + prometheus.io/path: "/metrics" + spec: + containers: + - image: luxas/autoscale-demo + name: metrics-provider + ports: + - name: http + port: 8080 +``` + +
+ +Create this deployment, and expose it so that you can easily trigger +increases in metrics: + +```yaml +$ kubectl create -f sample-app.deploy.yaml +$ kubectl create service clusterip sample-app --tcp=80:8080 +``` + +This sample application provides some metrics on the number of HTTP +requests it receives. Consider the metric `http_requests_total`. First, +check that it appears in discovery using the command from [Double-Checking +Yor Work](#double-checking-your-work). The cumulative Prometheus metric +`http_requests_total` should have become the custom-metrics-API rate +metric `pods/http_requests`. Check out its value: + +```shell +$ kubectl get --raw "/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/pods/*/http_requests?selector=app%3Dsample-app" +``` + +It should be zero, since you're not currently accessing it. Now, create +a few requests with curl: + +```shell +$ curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics +``` + +Try fetching the metrics again. You should see an increase in the rate +after the collection interval specified in your Prometheus configuration +has elapsed. If you leave it for a bit, the rate will go back down again. + +Autoscaling +----------- + +Now that you have an application which produces custom metrics, you'll be +able to autoscale on it. As noted in the [HorizontalPodAutoscaler +walkthrough](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics), +there are three different types of metrics that the +HorizontalPodAutoscaler can handle. + +In this walkthrough, you've exposed some metrics that can be consumed +using the `Pods` metric type. + +Create a description for the HorizontalPodAutoscaler (HPA): + +
+ +sample-app-hpa.yaml + +```yaml +kind: HorizontalPodAutoscaler +apiVersion: autoscaling/v2alpha1 +metadata: + name: sample-app +spec: + scaleTargetRef: + # point the HPA at the sample application + # you created above + apiVersion: apps/v1beta1 + kind: Deployment + name: sample-app + # autoscale between 1 and 10 replicas + minReplicas: 1 + maxReplicas: 10 + metrics: + # use a "Pods" metric, which takes the average of the + # given metric across all pods controlled by the autoscaling target + - type: Pods + pods: + # use the metric that you used above: pods/http_requests + metricName: http_requests + # target 500 milli-requests per second, + # which is 1 request every two seconds + targetAverageValue: 500m +``` + +
+ +Create the HorizontalPodAutoscaler with + +``` +$ kubectl create -f sample-app-hpa.yaml +``` + +Then, like before, make some requests to the sample app's service. If you +describe the HPA, after the HPA sync interval has elapsed, you should see +the number of pods increase proportionally to the ratio between the actual +requests per second and your target of 1 request every 2 seconds. + +You can examine the HPA with + +```shell +$ kubectl describe hpa sample-app +``` + +You should see the HPA's last observed metric value, which should roughly +correspond to the rate of requests that you made. + +Next Steps +---------- + +For more information on how the HPA controller consumes different kinds of +metrics, take a look at the [HPA +walkthrough](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics). + +Also try exposing a non-cumulative metric from your own application, or +scaling on application on a metric provided by another application by +setting different labels or using the `Object` metric source type. + +For more information on how metrics are exposed by the Prometheus adapter, +see the [format documentation](./format.md).