Commit Graph
127 Commits
Author SHA1 Message Date
Nikola JokicandCopilot App 0c17a55075 Guard Running phase transition with optimistic locking
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-12 23:37:31 +02:00
Nikola JokicandCopilot App 03a2cdbdec Let the listener own the EphemeralRunner Running phase transition
The controller derived Status.Phase directly from the pod phase, so a runner
became Running as soon as its pod started, whether or not it had picked up a
job. That made Running mean "the pod is up" instead of "the runner is busy",
and it left the EphemeralRunnerSet scale-down path unable to tell an idle
runner from one that is executing a job.

The listener already knows when a job is assigned to a specific runner, so
move the transition there. HandleJobStarted now reads the runner first and
only promotes it to Running when it is not terminal (Failed, Succeeded or
Outdated) and not being deleted, then patches the phase alongside the job
fields it already writes. The listener role gains "get" on ephemeralrunners
for that read.

On the controller side updateRunStatusFromPod keeps publishing the initial
Pending phase while the pod is starting, and no longer promotes to Running.
Runners waiting for work now stay Pending, so scale-down picks them before
runners that are actually executing a job.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-12 23:37:31 +02:00
Nikola JokicandCopilot App 6b62a10639 Take the reconcile snapshot lazily, right before the first mutation
Reconcilers deep copied the object they had just fetched on every single
reconcile, purely so a merge patch could be computed on the rare pass
that actually changes something. The copy is a full recursive walk and
allocation of the object, and the overwhelming majority of reconciles
throw it away untouched.

Introduce lazyCopy, which takes the snapshot on the first call to
Mutate and hands back the live object. Because Mutate is the only way
to reach the object, the snapshot cannot be taken after the mutation it
is supposed to be diffed against, which is the way this optimization is
usually gotten wrong.

Apply it to the four Reconcile entry points, and move the two
EphemeralRunnerSet status copies inside the branch that patches, so they
are only paid for when the status really changed.

While here, drop the short circuit in the EphemeralRunner finalizer
block: `addedFinalizers || AddFinalizer(...)` skipped adding the actions
finalizer whenever the first finalizer was added.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-12 23:28:18 +02:00
Nikola JokicandCopilot App 854c0517a7 Make the Outdated phase revision-aware
An EphemeralRunner that exits as Outdated marks its EphemeralRunnerSet
Outdated, and the AutoscalingRunnerSet then stops scaling it. Without a way
to tell which runner spec an Outdated report refers to, a report from a
runner built before the spec was updated keeps the set switched off after
the update that was supposed to fix it.

Stamp each runner with the actionable revision it was built from, and judge
Outdated reports against the revision the set has applied:

  - A runner whose revision is older than the applied one is reporting on a
    spec that has already been replaced. It is deleted and rebuilt from the
    current spec, and it does not hold the set Outdated.
  - A runner whose revision is current is reporting on the live spec, so the
    set stays Outdated. It is deliberately not delete-and-replaced: a fresh
    runner at the same revision would report Outdated again, forever.

Runners without the annotation parse to revision 0, which matches the zero
value of Status.AppliedActionableRevision, so existing runners keep their
current behaviour across an upgrade.

The phase is derived inside patchAppliedActionableRevisionStatus, from a
list read through the same authoritative reader as the set itself, because
the optimistic lock on that patch covers the EphemeralRunnerSet object only
and cannot vouch for a separately-read list. The runners are classified
against the live applied revision rather than the revision this call was
asked to apply: the caller reads the spec from the cache while this function
re-reads the status from the API server, so a lagging reconcile can arrive
with a target behind the live marker, and judging against it would flip a
set that has already moved on back to Outdated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-12 23:18:26 +02:00
Nikola JokicandCopilot App 03328aa14f Remove the integrity hash annotation (#4643)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-11 18:27:55 +02:00
484564e6d6 Defer scale up until the listener publishes a state that accounts for finished runners (#4642)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-09-11 15:29:02 +02:00
6f89d057c0 Track runner spec updates with an actionable revision (#4638)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-09-11 12:44:03 +00:00
Nikola JokicandCopilot App 5581cd0c54 Track AutoscalingRunnerSet updates with metadata.generation (#4636)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-11 14:38:30 +02:00
Nikola JokicandCopilot App 0cfedfbb2c Detect listener pod drift by comparing the pod, not a hash (#4635)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-09-10 22:56:43 +02:00
c475023e28 Fix EphemeralRunnerSet metadata drift check comparing against the wrong value (#4634)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-09-10 11:52:08 +02:00
KR Ravindra 5e540b6c71 Add default and per-controller max-concurrent-reconciles flags (#4626)
Signed-off-by: KR Ravindra <42912207+KR-Ravindra@users.noreply.github.com>
2026-09-09 09:50:13 +02:00
Nikola Jokic 7c68e1d318 Set listener qps and burst (#4558) 2026-09-08 15:49:58 +02:00
Nikola Jokic c3dfb396d3 Fix nil ResourceCache panic in stale scale set tests (#4628) 2026-09-08 12:34:45 +02:00
Nikola Jokic 54147cfa5e Introduce cache for lookups of desired resource state, to reduce the amount of allocations in the controller (#4568) 2026-09-07 13:49:28 +02:00
Diogo Torres 3cfdcf59fe Re-register the runner scale set when it is gone from the Actions service (#4571) 2026-09-03 08:16:06 +01:00
Nikola Jokic f6a3d738de Use metrics to display runner statuses instead of status field for EphemeralRunnerSet and AutoscalingRunnerSet (#4557) 2026-07-14 19:31:11 +02:00
Nikola Jokic 368e2f28b8 Use Patch instead of Update (#4533) 2026-07-10 12:34:40 +02:00
Nikola Jokic 767e58e4b1 Upgrade resources in-place, causing 1-1 mapping between autoscaling runner set and ephemeral runner set (#4516) 2026-06-09 13:52:37 +02:00
Nikola Jokic 30879de182 Fix patch on autoscaling runner set when creating a runner scale set (#4502) 2026-05-22 17:51:47 +02:00
Nikola Jokic 081b9ce1ee Fix secret reconciliation updates for the listener pod (#4492) 2026-05-11 15:04:07 +02:00
Junya Okabe 0f2a659878 Fix: Detect init container failure in EphemeralRunner controller (#4457) 2026-05-07 13:26:46 +02:00
Junya Okabe 8c84ab2f42 Fix empty GVK in OwnerReferences for modern controllers (#4475) 2026-04-29 19:29:09 +02:00
Junya Okabe a401686bd5 Add option to disable workqueue bucket rate limiter (#4451) 2026-04-22 23:26:39 +02:00
Nikola Jokic 802dc28d38 Add multi-label support to scalesets (#4408) 2026-03-19 15:29:40 +01:00
Nikola JokicandCopilot Autofix powered by AI 9bc1c9e53e Shutdown the scaleset when runner is deprecated (#4404)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-19 13:30:20 +01:00
Nikola Jokic dc7c858e68 Remove actions client (#4405) 2026-03-16 14:39:55 +01:00
Nikola Jokic f99c6eda0b Moving to scaleset client for the controller (#4390) 2026-03-13 14:36:41 +01:00
Nikola Jokic 1d9f626c53 Allow users to apply labels and annotations to internal resources (#4400) 2026-03-12 10:32:54 +01:00
gateixeiraandNikola Jokic 1f615c1a33 feat: add default linux nodeSelector to listener pod (#4377)
Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com>
2026-02-24 17:56:39 +01:00
Nikola Jokic 8b7fd9ffef Switch client to scaleset library for the listener and update mocks (#4383) 2026-02-24 14:17:31 +01:00
Nikola Jokic bfe78ccd5d Make restart pod more flexible to different failure scenarios (#4340) 2025-12-19 15:49:42 +01:00
Nikola Jokic 50038fba61 Re-schedule if the failed reason starts with OutOf (#4336) 2025-12-16 13:26:44 +01:00
Nikola Jokic 82d5579696 Restart the listener if pod is evicted (#4332) 2025-12-09 17:55:09 +01:00
Nikola Jokic 95d2107a6a Code style changes on the controller (#4324) 2025-11-21 14:20:44 +01:00
Nikola Jokic 6d07b8d853 Add ephemeral runner finalizer during creation and check finalizer without requeue (#4320) 2025-11-20 23:06:27 +01:00
Nikola JokicandCopilot 9f9409a4c1 Handle resource quota on status forbidden by retrying (#4305)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-10 13:58:25 +01:00
Nikola Jokic 3d73636407 Use combination of namespace, GitHub URL, and runner group when hashing the listener name (#4299) 2025-11-10 13:58:16 +01:00
Nikola Jokic 4d22089978 Delete listener resources without requeueing on each call (#4289) 2025-10-29 13:01:00 +01:00
Nikola Jokic 94a6f3cc3a Ensure ephemeral runner is deleted from the service on exit != 0 (#4260) 2025-10-06 11:38:56 +02:00
Nikola Jokic 088e2a3a90 Remove ephemeral runner when exit code != 0 and is patched with the job (#4239) 2025-09-17 21:40:37 +02:00
Nikola Jokic ddc2918a48 Requeue if create pod returns already exists error (#4201) 2025-08-14 17:00:48 +02:00
Nikola Jokic c27541140a Remove JIT config from ephemeral runner status field (#4191) 2025-08-04 12:35:04 +02:00
Nikola Jokic 9890c0592d Explicitly requeue during backoff ephemeral runner (#4152) 2025-06-27 12:05:43 +02:00
Nikola Jokic 3b5693eecb Remove check if runner exists after exit code 0 (#4142) 2025-06-27 11:11:39 +02:00
Nikola Jokic e46c929241 Azure Key Vault integration to resolve secrets (#4090) 2025-06-11 15:53:33 +02:00
Wim FournierandNikola Jokic d4af75d82e Delete config secret when listener pod gets deleted (#4033)
Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com>
2025-06-11 15:53:04 +02:00
Nikola Jokic 1dbb88cb9e Allow use of client id as an app id (#4057) 2025-05-16 16:21:06 +02:00
Nikola Jokic 43f1cd0dac Refactor resource naming removing unnecessary calculations (#4076) 2025-05-15 10:56:34 +02:00
Nikola JokicandCopilot 389d842a30 Relax version requirements to allow patch version mismatch (#4080)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-05-14 21:38:16 +02:00
Nikola Jokic cae7efa2c6 Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners (#4059) 2025-05-14 15:38:50 +02:00