diff --git a/docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md b/docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md index d9965aec..9ede03ec 100644 --- a/docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md +++ b/docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md @@ -15,7 +15,72 @@ This sample dashboard shows how to visualize the metrics with [Grafana](https:// 1. Make sure to have [Grafana](https://grafana.com/docs/grafana/latest/installation/) and [Prometheus](https://prometheus.io/docs/prometheus/latest/installation/) running in your cluster. 2. Make sure that Prometheus is properly scraping the metrics endpoints of the controller-manager and listeners. -3. Import the [dashboard](ARC-Autoscaling-Runner-Set-Monitoring.json) into Grafana. +3. Make sure that your scrape configuration maps the pod labels onto the scraped metrics, as described in [Scale set labels come from Prometheus, not from ARC](#scale-set-labels-come-from-prometheus-not-from-arc). +4. Import the [dashboard](ARC-Autoscaling-Runner-Set-Monitoring.json) into Grafana. + +## Scale set labels come from Prometheus, not from ARC + +The dashboard filters several panels on `actions_github_com_scale_set_name` and +`actions_github_com_scale_set_namespace`. **ARC does not emit these labels.** They are +produced by Prometheus from the labels that ARC sets on the listener pods: + +``` +Labels: + actions.github.com/scale-set-name=arc-runner-set + actions.github.com/scale-set-namespace=arc-runners + ... +``` + +During service discovery, Prometheus exposes those pod labels as metadata labels, with +the characters that are invalid in a label name replaced by `_`: + +``` +__meta_kubernetes_pod_label_actions_github_com_scale_set_name=arc-runner-set +__meta_kubernetes_pod_label_actions_github_com_scale_set_namespace=arc-runners +``` + +A common scrape configuration copies these onto the scraped metrics by stripping the +`__meta_kubernetes_pod_label_` prefix. For example, scoped to the `arc-systems` namespace: + +```yaml +scrape_configs: + - job_name: arc-metrics + honor_labels: true + kubernetes_sd_configs: + - role: pod + namespaces: + names: + - arc-systems + relabel_configs: + - action: labelmap + regex: __meta_kubernetes_pod_label_(.+) +``` + +If you use the Prometheus Operator, the equivalent is a `PodMonitor` (or `ServiceMonitor`) +with a `labelmap` entry in its `relabelings`. + +> [!IMPORTANT] +> Adding `actions_github_com_scale_set_name` or `actions_github_com_scale_set_namespace` to +> `listenerMetrics` in the scale set `values.yaml` does **not** populate them. The listener +> only knows about its own label set (`name`, `namespace`, `repository`, `organization`, +> `enterprise`, and the job labels), so any unknown label is exported with an empty value, +> for example `actions_github_com_scale_set_namespace=""`. + +### Troubleshooting empty panels + +If the `Startup Duration`, `Job Execution`, or `Running Jobs` panels are empty or report +`Cannot read properties of undefined (reading 'config')`, the scale set labels are most +likely missing from the scraped metrics. Query one of the metrics directly (for example +`gha_job_startup_duration_seconds_bucket`) in Prometheus and check the label values. If +`actions_github_com_scale_set_namespace` is empty or absent, you can either: + +- Update your scrape configuration to relabel the scraped pod labels, as shown above; or +- Replace `actions_github_com_scale_set_name` and `actions_github_com_scale_set_namespace` + in your copy of the dashboard with labels that your setup already produces, such as + `namespace`, or the raw + `__meta_kubernetes_pod_label_actions_github_com_scale_set_namespace` metadata label. + +Both options assume that your scrape configuration is already capturing the pod's labels. ## Required metrics @@ -50,6 +115,11 @@ The following metrics are required to be scraped by Prometheus in order to popul | workqueue_depth | name, namespace | ARC Controller | workqueue_queue_duration_seconds_sum | namespace | ARC Controller +> [!NOTE] +> The `actions_github_com_scale_set_name` and `actions_github_com_scale_set_namespace` labels +> are not emitted by ARC. They are added by Prometheus from the listener pod labels. See +> [Scale set labels come from Prometheus, not from ARC](#scale-set-labels-come-from-prometheus-not-from-arc). + ## Details This dashboard demonstrates some of the metrics provided by ARC and the underlying Kubernetes runtime. It provides a sample visualization of the behavior of the runner scale set, the ARC controllers, and the listeners. This should not be considered a comprehensive dashboard; it is a starting point that can be used with other metrics and logs to understand the health of the cluster. Review the [GitHub documentation detailing the Actions Runner Controller metrics and how to enable them](https://docs.github.com/en/enterprise-server@3.10/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#enabling-metrics).