public_git/actions-runner-controller
1
0
Fork 0
mirror of https://github.com/actions-runner-controller/actions-runner-controller.git synced 2026-09-30 15:46:18 +02:00
Code Issues Projects Releases Wiki Activity

docs: explain scale set metric labels come from Prometheus relabeling

Browse Source 
The Grafana dashboard sample filters on actions_github_com_scale_set_name
and actions_github_com_scale_set_namespace, which are not emitted by ARC.
They are produced by Prometheus relabeling of the listener pod labels.
Document the required scrape configuration, clarify that adding these
labels to listenerMetrics results in empty values, and add a
troubleshooting section for empty panels.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Nikola Jokic
2026-09-08 09:57:30 +02:00
co-authored by Copilot App
parent 54147cfa5e
commit 4b0eeaf1df
1 changed files with 71 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md
+71 -1
View File
@@ -15,7 +15,72 @@ This sample dashboard shows how to visualize the metrics with [Grafana](https://
1. Make sure to have [Grafana](https://grafana.com/docs/grafana/latest/installation/) and [Prometheus](https://prometheus.io/docs/prometheus/latest/installation/) running in your cluster.
2. Make sure that Prometheus is properly scraping the metrics endpoints of the controller-manager and listeners.
3. Import the [dashboard](ARC-Autoscaling-Runner-Set-Monitoring.json) into Grafana.
3. Make sure that your scrape configuration maps the pod labels onto the scraped metrics, as described in [Scale set labels come from Prometheus, not from ARC](#scale-set-labels-come-from-prometheus-not-from-arc).
4. Import the [dashboard](ARC-Autoscaling-Runner-Set-Monitoring.json) into Grafana.
## Scale set labels come from Prometheus, not from ARC
The dashboard filters several panels on `actions_github_com_scale_set_name` and
`actions_github_com_scale_set_namespace`. **ARC does not emit these labels.** They are
produced by Prometheus from the labels that ARC sets on the listener pods:
```
Labels:
actions.github.com/scale-set-name=arc-runner-set
actions.github.com/scale-set-namespace=arc-runners
...
```
During service discovery, Prometheus exposes those pod labels as metadata labels, with
the characters that are invalid in a label name replaced by `_`:
```
__meta_kubernetes_pod_label_actions_github_com_scale_set_name=arc-runner-set
__meta_kubernetes_pod_label_actions_github_com_scale_set_namespace=arc-runners
```
A common scrape configuration copies these onto the scraped metrics by stripping the
`__meta_kubernetes_pod_label_` prefix. For example, scoped to the `arc-systems` namespace:
```yaml
scrape_configs:
- job_name: arc-metrics
honor_labels: true
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- arc-systems
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
```
If you use the Prometheus Operator, the equivalent is a `PodMonitor` (or `ServiceMonitor`)
with a `labelmap` entry in its `relabelings`.
> [!IMPORTANT]
> Adding `actions_github_com_scale_set_name` or `actions_github_com_scale_set_namespace` to
> `listenerMetrics` in the scale set `values.yaml` does **not** populate them. The listener
> only knows about its own label set (`name`, `namespace`, `repository`, `organization`,
> `enterprise`, and the job labels), so any unknown label is exported with an empty value,
> for example `actions_github_com_scale_set_namespace=""`.
### Troubleshooting empty panels
If the `Startup Duration`, `Job Execution`, or `Running Jobs` panels are empty or report
`Cannot read properties of undefined (reading 'config')`, the scale set labels are most
likely missing from the scraped metrics. Query one of the metrics directly (for example
`gha_job_startup_duration_seconds_bucket`) in Prometheus and check the label values. If
`actions_github_com_scale_set_namespace` is empty or absent, you can either:
- Update your scrape configuration to relabel the scraped pod labels, as shown above; or
- Replace `actions_github_com_scale_set_name` and `actions_github_com_scale_set_namespace`
in your copy of the dashboard with labels that your setup already produces, such as
`namespace`, or the raw
`__meta_kubernetes_pod_label_actions_github_com_scale_set_namespace` metadata label.
Both options assume that your scrape configuration is already capturing the pod's labels.
## Required metrics
@@ -50,6 +115,11 @@ The following metrics are required to be scraped by Prometheus in order to popul
| workqueue_depth | name, namespace | ARC Controller
| workqueue_queue_duration_seconds_sum | namespace | ARC Controller
> [!NOTE]
> The `actions_github_com_scale_set_name` and `actions_github_com_scale_set_namespace` labels
> are not emitted by ARC. They are added by Prometheus from the listener pod labels. See
> [Scale set labels come from Prometheus, not from ARC](#scale-set-labels-come-from-prometheus-not-from-arc).
## Details
This dashboard demonstrates some of the metrics provided by ARC and the underlying Kubernetes runtime. It provides a sample visualization of the behavior of the runner scale set, the ARC controllers, and the listeners. This should not be considered a comprehensive dashboard; it is a starting point that can be used with other metrics and logs to understand the health of the cluster. Review the [GitHub documentation detailing the Actions Runner Controller metrics and how to enable them](https://docs.github.com/en/enterprise-server@3.10/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#enabling-metrics).