Scale Set Metrics ADR (#2568)
Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com>
This commit is contained in:
Nikola Jokic
GitHub
parent
c5ebe750dc
commit
91c8991835
|
|
@ -0,0 +1,213 @@
|
||||||
|
# Exposing metrics
|
||||||
|
|
||||||
|
Date: 2023-05-08
|
||||||
|
|
||||||
|
**Status**: Proposed
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Prometheus metrics are a common way to monitor the cluster. Providing metrics
|
||||||
|
can be a helpful way to monitor scale sets and the health of the ephemeral runners.
|
||||||
|
|
||||||
|
## Proposal
|
||||||
|
|
||||||
|
Two main components are driving the behavior of the scale set:
|
||||||
|
|
||||||
|
1. ARC controllers responsible for managing Kubernetes resources.
|
||||||
|
2. The `AutoscalingListener`, driver of the autoscaling solution responsible for
|
||||||
|
describing the desired state.
|
||||||
|
|
||||||
|
We can approach publishing those metrics in 3 different ways
|
||||||
|
|
||||||
|
### Option 1: Expose a metrics endpoint for the controller-manager and every instance of the listener
|
||||||
|
|
||||||
|
To expose metrics, we would need to create 3 additional resources:
|
||||||
|
|
||||||
|
1. `ServiceMonitor` - a resource used by Prometheus to match namespaces and
|
||||||
|
services from where it needs to gather metrics
|
||||||
|
2. `Service` for the `gha-runner-scale-set-controller` - service that will
|
||||||
|
target ARC controller `Deployment`
|
||||||
|
3. `Service` for each `gha-runner-scale-set` listener - service that will target
|
||||||
|
a single listener pod for each `AutoscalingRunnerSet`
|
||||||
|
|
||||||
|
#### Pros
|
||||||
|
|
||||||
|
- Easy to control which scale set exposes metrics and which does not.
|
||||||
|
- Easy to implement using helm charts in case they are enabled per chart
|
||||||
|
installation.
|
||||||
|
|
||||||
|
#### Cons
|
||||||
|
|
||||||
|
- With a cluster running many scale sets, we are going to create a lot of
|
||||||
|
resources.
|
||||||
|
- In case metrics are enabled on the controller manager level, and they should
|
||||||
|
be applied across all `AutoscalingRunnerSets`, it is difficult to inherit this
|
||||||
|
configuration by applying helm charts.
|
||||||
|
|
||||||
|
### Option 2: Create a single metrics aggregator service
|
||||||
|
|
||||||
|
To create an aggregator service, we can create a simple web application
|
||||||
|
responsible for publishing and gathering metrics. All listeners would be
|
||||||
|
responsible to communicate the metrics on each message, and controllers are
|
||||||
|
responsible to communicate the metrics on each reconciliation.
|
||||||
|
|
||||||
|
The application can be executed as a single pod, or as a side container next to
|
||||||
|
the manager.
|
||||||
|
|
||||||
|
#### Running the aggregator as a container in the controller-manager pod
|
||||||
|
|
||||||
|
**Pros:**
|
||||||
|
- It exists side by side and is following the life cycle of the controller
|
||||||
|
manager
|
||||||
|
- We don't need to introduce another controller managing the state of the pod
|
||||||
|
|
||||||
|
**Cons**
|
||||||
|
|
||||||
|
- Crashes of the aggregator can influence the controller manager execution
|
||||||
|
- The controller manager pod needs more resources to run
|
||||||
|
|
||||||
|
#### Running the aggregator in a separate pod
|
||||||
|
|
||||||
|
**Pros**
|
||||||
|
|
||||||
|
- Does not influence the controller manager pod
|
||||||
|
- The life cycle of the metric can be controlled by the controller manager (by
|
||||||
|
implementing another controller)
|
||||||
|
|
||||||
|
**Cons**
|
||||||
|
|
||||||
|
- We need to implement the controller that can spin up the aggregator in case of
|
||||||
|
the crash.
|
||||||
|
- If we choose not to implement the controller, the resource like `Deployment`
|
||||||
|
can be used to manage the aggregator, but we lose control over its life cycle.
|
||||||
|
|
||||||
|
#### Metrics webserver requirements
|
||||||
|
|
||||||
|
1. Create a web server with a single `/metrics` endpoint. The endpoint will have
|
||||||
|
`POST` and `GET` methods registered. The `GET` is used by Prometheus to
|
||||||
|
fetch the metrics, while the `POST` is going to be used by controllers and
|
||||||
|
listeners to publish their metrics.
|
||||||
|
2. `ServiceMonitor` - to target the metrics aggregator service
|
||||||
|
3. `Service` sitting in front of the web server.
|
||||||
|
|
||||||
|
**Pros**
|
||||||
|
|
||||||
|
- This implementation requires a few additional resources to be created
|
||||||
|
in a cluster.
|
||||||
|
- Web server is easy to implement and easy to document - all metrics are aggregated in a
|
||||||
|
single package, and the web server only needs to apply them to its state on
|
||||||
|
`POST`. The `GET` handler is simple.
|
||||||
|
- We can avoid Pushgateway from Prometheus.
|
||||||
|
|
||||||
|
**Cons**
|
||||||
|
|
||||||
|
- Another image that we need to publish on release.
|
||||||
|
- Change in metric configuration (on manager update) would require re-creation
|
||||||
|
of all listeners. This is not a big problem but is something to point out.
|
||||||
|
- Managing requests/limits can be tricky.
|
||||||
|
|
||||||
|
### Option 3: Use a Prometheus Pushgateway
|
||||||
|
|
||||||
|
#### Pros
|
||||||
|
|
||||||
|
- Using a supported way of pushing the metrics.
|
||||||
|
- Easy to implement using their library.
|
||||||
|
|
||||||
|
#### Cons
|
||||||
|
|
||||||
|
- In the Prometheus docs, they specify that: "Usually, the only valid use case
|
||||||
|
for Pushgateway is for capturing the outcome of a service-level batch job".
|
||||||
|
The listener does not really fit this criteria.
|
||||||
|
- Pushgateway is a single point of failure and potential bottleneck.
|
||||||
|
- You lose Prometheus's automatic instance health monitoring via the up metric (generated on every scrape).
|
||||||
|
- The Pushgateway never forgets series pushed to it and will expose them to Prometheus forever unless those series are manually deleted via the Pushgateway's API.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Since there are many ways in which you can collect metrics, we have decided not
|
||||||
|
to apply `prometheus-operator` resources nor `Service`.
|
||||||
|
|
||||||
|
The responsibility of the controller and the autoscaling listener is
|
||||||
|
only to expose metrics. It is up to the user to decide how to collect them.
|
||||||
|
|
||||||
|
When installing the ARC, the configuration for both the controller manager
|
||||||
|
and autoscaling listeners' metric servers is established.
|
||||||
|
|
||||||
|
### Controller metrics
|
||||||
|
|
||||||
|
By default, metrics server is listening on `0.0.0.0:8080`.
|
||||||
|
You can control the port of the metrics server using the `--metrics-addr` flag.
|
||||||
|
|
||||||
|
Metrics can be collected from `/metrics` endpoint
|
||||||
|
|
||||||
|
If the value of `--metrics-addr` is an empty string, metrics server won't be
|
||||||
|
started.
|
||||||
|
|
||||||
|
### Autoscaling listeners
|
||||||
|
|
||||||
|
By default, metrics server is listening on `0.0.0.0:8080`.
|
||||||
|
The endpoint used to expose metrics is `/metrics`.
|
||||||
|
|
||||||
|
You can control both the address and the endpoint using `--listener-metrics-addr` and `--listener-metrics-endpoint` flags.
|
||||||
|
|
||||||
|
If the value of `--listener-metrics-addr` is an empty string, metrics server won't be
|
||||||
|
started.
|
||||||
|
|
||||||
|
### Metrics exposed by the controller
|
||||||
|
|
||||||
|
To get a better understanding of health and workings of the cluster
|
||||||
|
resources, we need to expose the following metrics:
|
||||||
|
|
||||||
|
- `pending_ephemeral_runners` - Number of ephemeral runners in a pending state.
|
||||||
|
This information can show the latency between creating an `EphemeralRunner`
|
||||||
|
resource, and having an ephemeral runner pod started and ready to receive a
|
||||||
|
job.
|
||||||
|
- `running_ephemeral_runners` - Number of ephemeral runners currently running.
|
||||||
|
This information is helpful to see how many ephemeral runner pods are running
|
||||||
|
at any given time.
|
||||||
|
- `failed_ephemeral_runners` - Number of ephemeral runners in a `Failed` state.
|
||||||
|
This information is helpful to catch the faulty image, or some underlying
|
||||||
|
problem. When the ephemeral runner controller is not able to start the
|
||||||
|
ephemeral runner pod after multiple retries, it will set the state of the
|
||||||
|
`EphemeralRunner` to failed. Since the controller can not recover from this
|
||||||
|
state, it can be useful to set Prometheus alerts to catch this issue quickly.
|
||||||
|
|
||||||
|
### Metrics exposed by the `AutoscalingListener`
|
||||||
|
|
||||||
|
Since the listener is responsible for communicating the state with the actions
|
||||||
|
service, it can expose actions service related data through metrics. In
|
||||||
|
particular:
|
||||||
|
|
||||||
|
- `available_jobs` - Number of jobs with `runs-on` matching the runner scale set name. Jobs are not yet assigned but are acquired by the runner scale set.
|
||||||
|
- `acquired_jobs`- Number of jobs acquired by the scale set.
|
||||||
|
- `assigned_jobs` - Number of jobs assigned to this scale set.
|
||||||
|
- `running_jobs` - Number of jobs running (or about to be run).
|
||||||
|
- `registered_runners` - Number of registered runners.
|
||||||
|
- `busy_runners` - Number of registered runners running a job.
|
||||||
|
- `min_runners` - Number of runners desired by the scale set.
|
||||||
|
- `max_runners` - Number of runners desired by the scale set.
|
||||||
|
- `desired_runners` - Number of runners desired by the scale set.
|
||||||
|
- `idle_runners` - Number of registered runners not running a job.
|
||||||
|
- `available_jobs_total` - Total number of jobs available for the scale set (runs-on matches and scale set passes all the runner group permission checks).
|
||||||
|
- `acquired_jobs_total` - Total number of jobs acquired by the scale set.
|
||||||
|
- `assigned_jobs_total` - Total number of jobs assigned to the scale set.
|
||||||
|
- `started_jobs_total` - Total number of jobs started.
|
||||||
|
- `completed_jobs_total` - Total number of jobs completed.
|
||||||
|
- `job_queue_duration_seconds` - Time spent waiting for workflow jobs to get assigned to the scale set after queueing (in seconds).
|
||||||
|
- `job_startup_duration_seconds` - Time spent waiting for a workflow job to get started on the runner owned by the scale set (in seconds).
|
||||||
|
- `job_execution_duration_seconds` - Time spent executing workflow jobs by the scale set (in seconds).
|
||||||
|
|
||||||
|
### Metric names
|
||||||
|
|
||||||
|
Listener metrics belong to the `github_runner_scale_set` subsystem, so the names
|
||||||
|
are going to have the `github_runner_scale_set_` prefix.
|
||||||
|
|
||||||
|
Controller metrics belong to the `github_runner_scale_set_controller` subsystem,
|
||||||
|
so the names are going to have `github_runner_scale_set_controller` prefix.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Users can define alerts, monitor the behavior of both the actions-based metrics
|
||||||
|
(gathered from the listener) and the Kubernetes resource-based metrics
|
||||||
|
(gathered from the controller manager).
|
||||||
|
|
||||||
Loading…
Reference in New Issue