From 7e8d80689bd9cdc3dd55be9a31a5a1aaff450d61 Mon Sep 17 00:00:00 2001 From: toast-gear Date: Wed, 30 Mar 2022 09:53:59 +0100 Subject: [PATCH 1/8] docs: add limitations to runnersets + reorder --- README.md | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index a50250df..d08f17bf 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ ToC: - [Enterprise Runners](#enterprise-runners) - [RunnerDeployments](#runnerdeployments) - [RunnerSets](#runnersets) + - [Persistent Runners](#persistent-runners) - [Autoscaling](#autoscaling) - [Anti-Flapping Configuration](#anti-flapping-configuration) - [Pull Driven Scaling](#pull-driven-scaling) @@ -32,7 +33,7 @@ ToC: - [Runner Groups](#runner-groups) - [Runner Entrypoint Features](#runner-entrypoint-features) - [Using IRSA (IAM Roles for Service Accounts) in EKS](#using-irsa-iam-roles-for-service-accounts-in-eks) - - [Persistent Runners](#persistent-runners) + - [Software Installed in the Runner Image](#software-installed-in-the-runner-image) - [Using without cert-manager](#using-without-cert-manager) - [Troubleshooting](#troubleshooting) @@ -367,6 +368,8 @@ example-runnerdeploy2475ht2qbr mumoshu/actions-runner-controller-ci Running > This feature requires controller version => [v0.20.0](https://github.com/actions-runner-controller/actions-runner-controller/releases/tag/v0.20.0) +__Ensure you see the limitations before using this kind!!!!!__ + For scenarios where you require the advantages of a `StatefulSet`, for example persistent storage, ARC implements a runner based on Kubernete's StatefulSets, the RunnerSet. A basic `RunnerSet` would look like this: @@ -450,6 +453,20 @@ Under the hood, `RunnerSet` relies on Kubernetes's `StatefulSet` and Mutating We **Limitations** * For autoscaling the `RunnerSet` kind only supports pull driven scaling or the `workflow_job` event for webhook driven scaling. +* Whilst RunnerSets support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using RunnerSets in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default configuration, you will get a build up of PVCs and PVs if you deploy RunnerSets as ephemeral runners without some sort of custom solution for cleaning up the PVCs. +### Persistent Runners + +Every runner managed by ARC is "ephemeral" by default. The life of an ephemeral runner managed by ARC looks like this- ARC creates a runner pod for the runner. As it's an ephemeral runner, the `--ephemeral` flag is passed to the `actions/runner` agent that runs within the `runner` container of the runner pod. + +`--ephemeral` is an `actions/runner` feature that instructs the runner to stop and de-register itself after the first job run. + +Once the ephemeral runner has completed running a workflow job, it stops with a status code of 0, hence the runner pod is marked as completed, removed by ARC. + +As it's removed after a workflow job run, the runner pod is never reused across multiple GitHub Actions workflow jobs, providing you a clean environment per each workflow job. + +Although not generally recommended, it's possible to disable passing `--ephemeral` flag by explicitly setting `ephemeral: false` in the `RunnerDeployment` or `RunnerSet` spec. When disabled, your runner becomes "persistent". A persistent runner does not stop after workflow job ends, and in this mode `actions/runner` is known to clean only runner's work dir after each job. That means your runner's environment, including various actions cache, docker images stored in the `dind` and layer cache, is retained across multiple workflow job runs. + +Persistent runners are available as an option for some edge cases however they are not preferred as they can create challenges around providing a deterministic and secure environment. ### Autoscaling @@ -1311,21 +1328,6 @@ spec: securityContext: fsGroup: 1000 ``` - -### Persistent Runners - -Every runner managed by ARC is "ephemeral" by default. The life of an ephemeral runner managed by ARC looks like this- ARC creates a runner pod for the runner. As it's an ephemeral runner, the `--ephemeral` flag is passed to the `actions/runner` agent that runs within the `runner` container of the runner pod. - -`--ephemeral` is an `actions/runner` feature that instructs the runner to stop and de-register itself after the first job run. - -Once the ephemeral runner has completed running a workflow job, it stops with a status code of 0, hence the runner pod is marked as completed, removed by ARC. - -As it's removed after a workflow job run, the runner pod is never reused across multiple GitHub Actions workflow jobs, providing you a clean environment per each workflow job. - -Although not recommended, it's possible to disable passing `--ephemeral` flag by explicitly setting `ephemeral: false` in the `RunnerDeployment` or `RunnerSet` spec. When disabled, your runner becomes "persistent". A persistent runner does not stop after workflow job ends, and in this mode `actions/runner` is known to clean only runner's work dir after each job. That means your runner's environment, including various actions cache, docker images stored in the `dind` and layer cache, is retained across multiple workflow job runs. - -Persistent runners are available as an option for some edge cases however they are not preferred as they can create challenges around providing a deterministic and secure environment. - ### Software Installed in the Runner Image **Cloud Tooling**
From 98567dadc96d1d97b3e3504db0964bd3d24d3941 Mon Sep 17 00:00:00 2001 From: toast-gear Date: Wed, 30 Mar 2022 09:59:32 +0100 Subject: [PATCH 2/8] docs: fix index --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index d08f17bf..909c62dd 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,6 @@ ToC: - [Runner Groups](#runner-groups) - [Runner Entrypoint Features](#runner-entrypoint-features) - [Using IRSA (IAM Roles for Service Accounts) in EKS](#using-irsa-iam-roles-for-service-accounts-in-eks) - - [Software Installed in the Runner Image](#software-installed-in-the-runner-image) - [Using without cert-manager](#using-without-cert-manager) - [Troubleshooting](#troubleshooting) From 88d17c798882e0ca01212e4315839a24963ef2c0 Mon Sep 17 00:00:00 2001 From: toast-gear Date: Wed, 30 Mar 2022 10:00:34 +0100 Subject: [PATCH 3/8] docs: use the right font --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 909c62dd..82191a84 100644 --- a/README.md +++ b/README.md @@ -367,7 +367,7 @@ example-runnerdeploy2475ht2qbr mumoshu/actions-runner-controller-ci Running > This feature requires controller version => [v0.20.0](https://github.com/actions-runner-controller/actions-runner-controller/releases/tag/v0.20.0) -__Ensure you see the limitations before using this kind!!!!!__ +_Ensure you see the limitations before using this kind!!!!!_ For scenarios where you require the advantages of a `StatefulSet`, for example persistent storage, ARC implements a runner based on Kubernete's StatefulSets, the RunnerSet. From fd0092d13fe5122a79d614bcbe37788e0856be4d Mon Sep 17 00:00:00 2001 From: toast-gear Date: Wed, 30 Mar 2022 10:02:33 +0100 Subject: [PATCH 4/8] chore: new line for consistency --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 82191a84..20f5129b 100644 --- a/README.md +++ b/README.md @@ -453,6 +453,7 @@ Under the hood, `RunnerSet` relies on Kubernetes's `StatefulSet` and Mutating We * For autoscaling the `RunnerSet` kind only supports pull driven scaling or the `workflow_job` event for webhook driven scaling. * Whilst RunnerSets support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using RunnerSets in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default configuration, you will get a build up of PVCs and PVs if you deploy RunnerSets as ephemeral runners without some sort of custom solution for cleaning up the PVCs. + ### Persistent Runners Every runner managed by ARC is "ephemeral" by default. The life of an ephemeral runner managed by ARC looks like this- ARC creates a runner pod for the runner. As it's an ephemeral runner, the `--ephemeral` flag is passed to the `actions/runner` agent that runs within the `runner` container of the runner pod. From d26c8d6529b46281e72af46f24b6f9c34ee3d832 Mon Sep 17 00:00:00 2001 From: toast-gear Date: Wed, 30 Mar 2022 10:26:08 +0100 Subject: [PATCH 5/8] docs: add autoscaling also causes problems --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 20f5129b..c80dd811 100644 --- a/README.md +++ b/README.md @@ -452,7 +452,7 @@ Under the hood, `RunnerSet` relies on Kubernetes's `StatefulSet` and Mutating We **Limitations** * For autoscaling the `RunnerSet` kind only supports pull driven scaling or the `workflow_job` event for webhook driven scaling. -* Whilst RunnerSets support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using RunnerSets in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default configuration, you will get a build up of PVCs and PVs if you deploy RunnerSets as ephemeral runners without some sort of custom solution for cleaning up the PVCs. +* Whilst RunnerSets support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using RunnerSets in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default configuration or implementing autoscaling, you will get a build up of PVCs and PVs if you deploy RunnerSets as ephemeral runners without some sort of custom solution for cleaning up the PVCs. ### Persistent Runners From 7a750b928510c8a382389094947aedb5ff7fcdb9 Mon Sep 17 00:00:00 2001 From: toast-gear Date: Wed, 30 Mar 2022 10:35:32 +0100 Subject: [PATCH 6/8] docs: wording --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c80dd811..399735a6 100644 --- a/README.md +++ b/README.md @@ -452,7 +452,7 @@ Under the hood, `RunnerSet` relies on Kubernetes's `StatefulSet` and Mutating We **Limitations** * For autoscaling the `RunnerSet` kind only supports pull driven scaling or the `workflow_job` event for webhook driven scaling. -* Whilst RunnerSets support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using RunnerSets in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default configuration or implementing autoscaling, you will get a build up of PVCs and PVs if you deploy RunnerSets as ephemeral runners without some sort of custom solution for cleaning up the PVCs. +* Whilst `RunnerSets` support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using `RunnerSets` in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default ephemeral configuration or implementing autoscaling for your `RunnerSets`, you will get a build up of PVCs and PVs if you deploy RunnerSets as ephemeral runners without some sort of custom solution for cleaning up the PVCs. ### Persistent Runners From eb02f6f26e072f37f075fa58f34df5aa2c57f9ef Mon Sep 17 00:00:00 2001 From: toast-gear Date: Wed, 30 Mar 2022 10:36:34 +0100 Subject: [PATCH 7/8] docs: redundant words --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 399735a6..cc3f0afd 100644 --- a/README.md +++ b/README.md @@ -452,7 +452,7 @@ Under the hood, `RunnerSet` relies on Kubernetes's `StatefulSet` and Mutating We **Limitations** * For autoscaling the `RunnerSet` kind only supports pull driven scaling or the `workflow_job` event for webhook driven scaling. -* Whilst `RunnerSets` support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using `RunnerSets` in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default ephemeral configuration or implementing autoscaling for your `RunnerSets`, you will get a build up of PVCs and PVs if you deploy RunnerSets as ephemeral runners without some sort of custom solution for cleaning up the PVCs. +* Whilst `RunnerSets` support all runner modes as well as autoscaling, currently PVs are **NOT** automatically cleaned up as they are still bound to their respective PVCs when a runner is deleted by the controller. This has **major** implications when using `RunnerSets` in the standard runner mode, `ephemeral: true`, see [persistent runners](#persistent-runners) for more details. As a result of this, using the default ephemeral configuration or implementing autoscaling for your `RunnerSets`, you will get a build up of PVCs and PVs without some sort of custom solution for cleaning up. ### Persistent Runners From 14f9e7229ebe2061c35f5bbf60981a9ffe7a4ce9 Mon Sep 17 00:00:00 2001 From: Callum Tait <15716903+toast-gear@users.noreply.github.com> Date: Fri, 1 Apr 2022 15:49:15 +0100 Subject: [PATCH 8/8] docs: highlight why persistent are not ideal --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cc3f0afd..635499ee 100644 --- a/README.md +++ b/README.md @@ -464,7 +464,7 @@ Once the ephemeral runner has completed running a workflow job, it stops with a As it's removed after a workflow job run, the runner pod is never reused across multiple GitHub Actions workflow jobs, providing you a clean environment per each workflow job. -Although not generally recommended, it's possible to disable passing `--ephemeral` flag by explicitly setting `ephemeral: false` in the `RunnerDeployment` or `RunnerSet` spec. When disabled, your runner becomes "persistent". A persistent runner does not stop after workflow job ends, and in this mode `actions/runner` is known to clean only runner's work dir after each job. That means your runner's environment, including various actions cache, docker images stored in the `dind` and layer cache, is retained across multiple workflow job runs. +Although not generally recommended, it's possible to disable passing `--ephemeral` flag by explicitly setting `ephemeral: false` in the `RunnerDeployment` or `RunnerSet` spec. When disabled, your runner becomes "persistent". A persistent runner does not stop after workflow job ends, and in this mode `actions/runner` is known to clean only runner's work dir after each job. Whilst this can seem helpful it creates a non-deterministic environment which is not ideal for a CI/CD environment. Between runs your actions cache, docker images stored in the `dind` and layer cache, globally installed packages etc are retained across multiple workflow job runs which can cause issues which are hard to debug and inconsistent. Persistent runners are available as an option for some edge cases however they are not preferred as they can create challenges around providing a deterministic and secure environment.