Commit Graph
29 Commits
Author SHA1 Message Date
Nikola Jokic bf2b89a349 Merge remote-tracking branch 'origin/nikola-jokic-remove-integrity-hash-annotation' into nikola-jokic-revision-aware-outdated 2026-09-11 11:07:46 +02:00
Nikola JokicandCopilot Autofix powered by AI f862d6f9cb Fix formatting in ephemeralrunnerset_controller_test.go
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-09-11 09:04:34 +02:00
Nikola JokicandCopilot App bd0f75c3ec Wire APIReader into the hand-built EphemeralRunnerSet reconcilers in tests
Six specs construct an EphemeralRunnerSetReconciler literal and call
Reconcile directly instead of going through SetupWithManager, so the
APIReader backfill never runs and the field stays nil.

Nothing is broken today: those specs either publish patch ID 0 or return
via the stale-outdated path, so they never reach the scale-up branch
where the reader is consulted. But the next direct-Reconcile spec that
exercises scale-up with a non-zero patch ID would fail with "APIReader is
not configured" instead of the behaviour it meant to test, and the cause
would not be obvious from the failure.

Set the field the way SetupWithManager does in production, so the specs
exercise the same wiring the real controller has. The helper still errors
on a nil reader; falling back to the cached read is the race that fix
exists to prevent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-10 22:56:49 +02:00
Nikola JokicandCopilot App 60a6753d3b Make the Outdated phase revision-aware
An EphemeralRunnerSet goes Outdated when a child runner reports that the
Actions service rejected its runner spec, and the AutoscalingRunnerSet
then tears the listener down so the scale set stops taking jobs. Until
now nothing recorded which runner spec a given Outdated report was about,
so a report from a runner that predates a spec update kept the set
switched off even after the user had already fixed the spec.

Runners now carry the EphemeralRunnerSet's actionable revision as an
annotation, and Outdated runners are split into two groups:

  - stale outdated: created before the currently applied revision. Their
    verdict is about a spec that no longer exists, so they are deleted
    and replaced by runners built from the current spec.
  - outdated: created at or after the applied revision. Their verdict is
    about the current spec, so they hold the set in the Outdated phase.
    They are deliberately not delete-and-replaced: a fresh runner at the
    same revision would report Outdated again, forever.

patchAppliedActionableRevisionStatus now recomputes the phase in both
directions against the revision being applied, because it returns from
Reconcile without reaching updateStatus. The AutoscalingRunnerSet
teardown guard additionally requires the applied revision to have caught
up with the spec revision, so it does not act on an Outdated verdict for
a spec that has already been replaced.

terminated() now allocates a fresh slice instead of appending into the
finished slice, which could alias its backing array.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-10 22:56:49 +02:00
Nikola JokicandCopilot App e3656c69ff Defer scale up until the listener publishes a state that accounts for finished runners
Spec.Replicas is the count the listener asked for when it published
Spec.PatchID. When the EphemeralRunnerSet controller cleaned up finished
runners in the same reconcile, it then compared that count against a live
count the cleanup had just reduced, and created runners to replace jobs
that had already completed. Nobody asked for those runners.

Delete the finished runners, record the patch ID the cleanup belongs to in
Status.FinishedRunnerCleanupPatchID, and return, so the scaling decision is
made on the next reconcile against post-cleanup data. Scale up stays
suppressed while Spec.PatchID still equals that recorded patch ID: the gap
below Spec.Replicas is the one the cleanup opened, not new demand. The
listener marks itself dirty on every job completion and publishes a fresh
incrementing patch ID, so the suppression lifts as soon as it reports a
desired state that accounts for the completions.

Runners that are mid-deletion were not counted at all, which let the
controller over-create while deletions were still in flight. Count them
towards the scale-up total.

The cleanup helper is renamed to deleteTerminatedEphemeralRunners because
it is no longer specific to finished runners.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-10 22:56:48 +02:00
Nikola JokicandCopilot App ec793dfe9f Track runner spec updates with an actionable revision
When the AutoscalingRunnerSet's runner spec changes, the EphemeralRunnerSet
has to delete its idle and pending runners so they are rebuilt from the new
spec. That was detected by hashing Spec.EphemeralRunnerSpec into the
actions.github.com/integrity-hash annotation and comparing the annotation
against a freshly computed hash.

Replace it with a monotonic revision counter split across spec and status.
The AutoscalingRunnerSet controller bumps Spec.ActionableRevision when it
patches a new runner spec across; the EphemeralRunnerSet controller advances
Status.AppliedActionableRevision only after the cleanup has actually
succeeded. Because the applied marker lives in status and is written last, a
controller that crashes part-way through the cleanup comes back with the
applied revision still behind the spec revision and redoes the work, instead
of skipping runners that are still running the old spec.

The drift check uses apiequality.Semantic.DeepEqual rather than cmp.Equal or
reflect.DeepEqual. Most PodSpec collection fields carry omitempty, so a
template containing an explicitly empty value (`env: []`) is dropped when the
EphemeralRunnerSet is written and reads back as nil. A strict comparison would
report drift on every reconcile, bump the revision each time, and delete every
idle and pending runner forever. helpers_drift_test.go pins that behaviour with
a round-trip-through-JSON test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-10 22:56:48 +02:00
Nikola Jokic 54147cfa5e Introduce cache for lookups of desired resource state, to reduce the amount of allocations in the controller (#4568) 2026-09-07 13:49:28 +02:00
Nikola Jokic f6a3d738de Use metrics to display runner statuses instead of status field for EphemeralRunnerSet and AutoscalingRunnerSet (#4557) 2026-07-14 19:31:11 +02:00
Nikola Jokic 767e58e4b1 Upgrade resources in-place, causing 1-1 mapping between autoscaling runner set and ephemeral runner set (#4516) 2026-06-09 13:52:37 +02:00
Nikola JokicandCopilot Autofix powered by AI 9bc1c9e53e Shutdown the scaleset when runner is deprecated (#4404)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-19 13:30:20 +01:00
Nikola Jokic f99c6eda0b Moving to scaleset client for the controller (#4390) 2026-03-13 14:36:41 +01:00
Nikola Jokic 6d07b8d853 Add ephemeral runner finalizer during creation and check finalizer without requeue (#4320) 2025-11-20 23:06:27 +01:00
Nikola Jokic e46c929241 Azure Key Vault integration to resolve secrets (#4090) 2025-06-11 15:53:33 +02:00
Nikola Jokic cae7efa2c6 Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners (#4059) 2025-05-14 15:38:50 +02:00
Nikola Jokic b349ded2be Increase test timeouts to avoid CI test failures (#3554) 2024-06-21 13:45:48 +02:00
Nikola Jokic 9b51f25800 Rename imports in tests to remove double import and to improve readability (#3455) 2024-05-17 14:37:13 +02:00
Nikola Jokic 963ae48a3f Include self correction on empty batch and avoid removing pending runners when cluster is busy (#3426) 2024-04-16 12:55:25 +02:00
Nikola JokicandFrancesco Renzi 7a643a5107 Fix overscaling when the controller is much faster then the listener (#3371)
Co-authored-by: Francesco Renzi <rentziass@gmail.com>
2024-03-20 15:36:12 +01:00
Nikola Jokic 07bff8aa1e Extend the user agent and fix the build version for the listener app (#2892) 2023-09-14 20:10:49 +02:00
Tingluo Huang 261d4371b5 Update E2E test workflow. (#2395) 2023-03-14 09:00:07 -04:00
Nikola Jokic babbfc77d5 Surface EphemeralRunnerSet stats to AutoscalingRunnerSet (#2382) 2023-03-13 16:16:28 +01:00
c569304271 Add support for self-signed CA certificates (#2268)
Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com>
Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com>
Co-authored-by: Tingluo Huang <tingluohuang@github.com>
2023-03-09 17:23:32 +00:00
Francesco Renzi 40c905f25d Simplify the setup of controller tests (#2352) 2023-03-02 18:55:49 +00:00
Francesco Renzi 73e22a1756 Disable metrics serving in proxy tests (#2307) 2023-02-22 16:57:59 +00:00
6b4250ca90 Add support for proxy (#2286)
Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com>
Co-authored-by: Tingluo Huang <tingluohuang@github.com>
Co-authored-by: Ferenc Hammerl <fhammerl@github.com>
2023-02-21 17:33:48 +00:00
dependabot[bot]andYusuke Kuoka 219ba5b477 chore(deps): bump sigs.k8s.io/controller-runtime from 0.13.1 to 0.14.1 (#2132)
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
2023-01-27 09:23:28 +09:00
Nikola Jokic 882bfab569 Renaming autoScaling to autoscaling in tests matching the convention (#2201) 2023-01-23 17:03:01 +01:00
Tingluo Huang 4932412cd6 Fix L0 test to make it more reliable. (#2178) 2023-01-19 07:33:04 -05:00
622eaa34f8 Introduce new preview auto-scaling mode for ARC. (#2153)
Co-authored-by: Cory Miller <cory-miller@github.com>
Co-authored-by: Nikola Jokic <nikola-jokic@github.com>
Co-authored-by: Ava Stancu <AvaStancu@github.com>
Co-authored-by: Ferenc Hammerl <fhammerl@github.com>
Co-authored-by: Francesco Renzi <rentziass@github.com>
Co-authored-by: Bassem Dghaidi <Link-@github.com>
2023-01-17 12:06:20 -05:00