Commit Graph
49 Commits
Author SHA1 Message Date
edi-oai 95b1269450 Defer in-place updates while VMs are restarting (#483) 2026-09-03 18:59:53 +01:00
edi-oai a838bb0299 Worker: introduce automatic "org.cirruslabs.orchard.worker-name" label (#481)
* Worker: introduce automatic "org.cirruslabs.orchard.worker-name" label

* TestLabelsCopy: use constants to fix linter error
2026-09-03 00:24:20 +01:00
edi-oai a3e08ffcbf Support running host processes alongside VMs (#482)
* Regenerate .pb with protoc-gen-go v1.36.11

* Support running host processes alongside VMs

* Replace existing host processes when starting a set
2026-09-03 00:24:09 +01:00
edi-oai 3fe0a284c5 Add endpoint support (#478)
* Add endpoint support

* Document endpoint network exposure
2026-09-02 16:08:39 +01:00
edi-oai 62acf3c265 Support dynamically updating Softnet policy (#477)
* Support dynamically updating Softnet policy

* Use cmpopts.EquateEmpty() when comparing two VMSpecs
2026-09-02 14:19:29 +01:00
Yibo Zhuang 9b8365edcf worker: wait for Tart and Vetu stop completion (#472)
Share one Stop completion per run and retain Stopping until both the stop command and VM cleanup finish. Expose transition conditions to reconciliation and return after requesting Stop so it cannot apply a generation or restart the VM early.

Cover both completion orders, repeated and concurrent calls, restart, and generation acknowledgement with bounded fake-command regressions. Allow process startup under parallel test load while preserving explicit shutdown ordering.
2026-09-01 21:59:50 +01:00
Yibo Zhuang 6cbee5352e Bound VM inventory initialization before worker registration (#467) 2026-08-19 09:58:56 -07:00
Yibo Zhuang 6cdb1b78d9 Recover controller sessions without stopping active VMs (#466) 2026-08-19 09:21:32 -07:00
Yibo Zhuang 2da158908c Wait for VM shutdown before freeing worker capacity (#465) 2026-08-18 07:38:36 -07:00
Nikolay Edigaryev 9092a9f172 Support Vetu virtualization on Linux in addition to Tart on macOS (#419)
* Support Vetu virtualization on Linux in addition to Tart on macOS

* api(portForward): ensure that rendezvousConn is closed

* Re-try SSH connections in integration tests

Because a VM might be still booting.
2026-03-16 11:12:28 +01:00
Nikolay Edigaryev f3b4eb42ca Worker: decouple updateWorker() and syncVMs() to use different triggers (#403)
* Worker: decouple updateWorker() and syncVMs() to use different triggers

* Actually return an error otherwise errgroup won't terminate
2026-02-06 18:16:40 +01:00
Nikolay Edigaryev 230a83c740 Worker: ignore HTTP 404 when updating VM (#397) 2026-02-05 13:01:05 +01:00
Nikolay Edigaryev 688238837a Implement server-side filtering for VMs by worker (#392)
* Implement server-side filtering for VMs by worker

* Parse more than one filter but error out when more than one is provided

* Fix off-by-one

* No need to use "\n" in Debugf()
2026-01-29 17:52:24 +01:00
Nikolay Edigaryev a234ec8995 orchard worker run: introduce "--synthetic" and "--workers" hidden flags (#390)
* orchard worker run: introduce "--synthetic" and "--workers" hidden flags

* Use slices.Clone() instead of directly assigning slices

* Use errgroup's context instead of command's context
2026-01-28 16:50:25 +01:00
Nikolay Edigaryev 7775515a73 Load testing: synthetic VMs, multiple worker support and Grafana k6 test (#389)
* Load testing: synthetic VMs, multiple worker support and Grafana k6 test

* echoserver: prevent fallthrough when Accept() fails

* Move default local-dev context logic to CreateDevController()

* Synthetic: add a random delay to startup script echoing
2026-01-28 10:54:55 +01:00
Nikolay Edigaryev ea89d01760 Worker: define VM interface and make tart.VM conform to it (#388)
* Worker: define VM interface and make tart.VM conform to it

* Hopefully produce better diff
2026-01-21 18:57:22 +01:00
Nikolay Edigaryev 5c162ce603 Power state fixes (#376)
* Do not call vm.Suspend() and vm.Stop() twice

* Do not attempt to Stop() or Suspend() the VM twice
2025-12-03 11:38:27 +01:00
Nikolay Edigaryev 76a552bade Ability to set VM's power state and retrieve backing Tart VM's name (#373)
* Ability to set VM's power state and retrieve backing Tart VM's name

* Validate user-provided "powerState" field

* Introduce TestSpecUpdatePowerStateSuspend

* Introduce TestSpecUpdatePowerStateStopped

* OpenAPI specification: add note about suspended VMs to "tartName" desc.

* Sometimes we need to wait more than 30 seconds
2025-12-02 16:43:17 -05:00
Nikolay Edigaryev 60303d11dd VM specification: allow suspendable VMs (#366) 2025-11-11 21:16:28 +04:00
Nikolay Edigaryev bafcf6fac2 Simplify state reconciliation and support changing Softnet settings (#364)
* Simplify state reconciliation and support changing Softnet settings

* Remove unused "updateFunc" parameter from syncOnDiskVMs()

* Don't take an address of a loop variable

* ensure → ensures

* updateVMState(): don't forget to update VMState

* Introduce TestSpecUpdateSoftnet integration test

* Update OpenAPI specification to include generation/observedGeneration
2025-11-06 20:56:31 +04:00
Nikolay Edigaryev 56260e7667 Worker: automatically scrape logical cores and memory size (#341) 2025-09-17 00:13:42 +04:00
Nikolay Edigaryev abcfee677d Work around Sequoia's "Local Network" permission with a helper process (#302)
* Work around Sequoia's "Local Network" permission with a helper process

* README.md: macOS 15 (Sequoia) warning

* Make "orchard dev" unix-specific too, otherwise Release fails

* Fix typo in "localNetworkHerlper"

* Slightly improve the macOS 15 (Sequoia) note

* orchard worker run: better documentation for --user

* Make sure privilege dropping is the first step we do in runWorker()
2025-04-10 18:01:19 +04:00
Nikolay Edigaryev c24db17aa5 Use VM status message to reflect pulling, cloning, configuring, etc. (#298) 2025-04-03 18:08:13 +04:00
Nikolay Edigaryev 2c8d36ef70 Always randomize MAC address (#251)
* Always randomize MAC address

* Worker: check DHCP lease time and print a warning if it's unconfigured

* Further improve the explanation

* Add two leases example to the explanation

* Add an example of the resulting /var/db/dhcpd_leases
2025-02-13 12:35:12 +00:00
Nikolay Edigaryev 26c8808506 Support scheduling by labels (#244) 2025-02-06 18:05:36 +04:00
Nikolay Edigaryev 581de320b9 Allow creating VMs with implicit CPU and memory (#243)
* Allow creating VMs with implicit CPU and memory

* Clarify why cpu/memory can be 0 a bit better

* Controller(API): don't forget to update DefaultCPU and DefaultMemory

* Add an integration test for implicit CPU and memory
2025-02-06 00:50:01 +04:00
Nikolay Edigaryev 88fba8004d Introduce WebSocket-based RPC v2 (#239)
* Introduce WebSocket-based RPC v2

* go test: add -ldflags="-B gobuildid"

* No need to change the "controller.workerNotifier.Notify()" error message

* No need to modify Protocol Buffers/gRPC generated code

* rpcWatch(): explain that connection shouldn't be normally be closed

* Avoid "port forwarding failed: " repetition in error messages

* Improve comments and avoid repetition in IP resolution errors
2025-01-30 17:33:32 +04:00
Nikolay Edigaryev 08769e00b4 Worker: do not consider on-disk VMs syncing error as fatal (#230) 2024-12-11 19:56:00 +04:00
Nikolay EdigaryevandFedor Korotkov ff0497b1d8 Produce OpenTelemetry metrics (#185)
* .golangci.yml: remove mentions of deprecated linters

* Fix "staticcheck" linter error by using grpc.NewClient

* Configure OpenTelemetry

Metrics only for now.

* Produce OpenTelemetry metrics

* Update DeploymentGuide.md

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>

* Update DeploymentGuide.md

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>

* Introduce "org.cirruslabs.orchard.controller.worker_status"

---------

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>
2024-06-24 18:19:51 +04:00
Nikolay Edigaryev 7fb0a85834 API(VM): new image FQN (fully-qualified name) field (#165) 2024-04-15 20:14:44 +04:00
Nikolay Edigaryev 510a25956f Asynchronous VM creation (#163) 2024-04-12 15:56:00 +04:00
Nikolay Edigaryev 063405672f Worker: better cleanup mechanisms (#139)
* Always Close() the Worker instance

* orchard list vms: show assigned worker for each of the VMs

* Stop the failed VMs before we schedule new VMs

To avoid violating resource constraints.

* syncOnDiskVMs: don't ignore running VMs

* Worker: show correct remote and local VM counts
2023-09-28 19:09:43 +00:00
Nikolay Edigaryev 3d0e375ede Don't stop and delete VMs that failed to clone (#125)
* NewVM() never returns an error

* Don't stop and delete VMs that failed to clone
2023-09-13 19:39:10 +04:00
Nikolay Edigaryev 3c3b8e8180 Do not treat controller registration error as fatal (#100) 2023-06-29 19:29:32 +04:00
Nikolay Edigaryev 60e564da88 Implement restart policy for VMs (#83)
* Implement restart policy for VMs

* Do not update VM.Resource, we only use it as a read-only specification

* Err()/setErr(): use atomic.Pointer instead of sync.Mutex
2023-04-24 19:30:08 +04:00
Nikolay Edigaryev 4eafec99a5 Fail VMs if the worker had crashed/is unhealthy (#70)
* Fail VMs if the worker had crashed/is unhealthy

* OnDiskName: properly handle cases when VM's name contains hyphens

* Worker: introduce Offline() method and check it before scheduling

* tart.List(): use Tart's JSON output

* OnDiskName: remove empty parts check

* Scheduler: move health-checking logic to a separate function

* Only fail "running" VMs

* Only fail orphaned VMs if they're in terminal state

* Integration tests

* Run healthCheckingLoopIteration() before schedulingLoopIteration()

* Worker: sync on-disk VMs only once at start
2023-04-03 16:47:49 +04:00
Fedor Korotkov f152043f19 Reactive Scheduling (#67)
Before we had two main loops: controller loop to assign VMs and worker loop to start VMs. Each of the loops was performed upon an interval every N seconds.

This change introduces a mechanism for reactively requesting loop execution:

 1. Controller loop will be executed upon VM creation to try to immediately schedule.
 2. A worker will be notified upon a VM assigment and worker loop will be requested to sync immediately.

 Fixes #31
2023-03-28 20:51:41 +04:00
Nikolay Edigaryev cb39836ee0 Resources support (#63)
* Resources support

* Ability to provide VM and worker resources via the CLI

* orchard dev: always listen on :6120

* orchard dev: support --resources

* REST API: provide resource defaults when creating VM

* OpenAPI: document "resources" field

* orchard dev: serve Swagger API documentation on /v1/

* Integration guide
2023-03-27 17:30:54 +04:00
Fedor KorotkovandNikolay Edigaryev 362ea85b4f Always require a client for running a worker (#52)
* Always require a client for running a worker

* Actually validate roles

* Delete worker

Fixes #46

* Update internal/worker/worker.go

Co-authored-by: Nikolay Edigaryev <edigaryev@gmail.com>

---------

Co-authored-by: Nikolay Edigaryev <edigaryev@gmail.com>
2023-03-24 17:44:20 +04:00
Nikolay Edigaryev af074f499d Remove UID for now and use machine ID to differentiate workers (#48)
* Remove UID for now and use machine ID to differentiate workers

* Rename MetadataWorkerKey back to MetadataWorkerNameKey
2023-03-23 23:38:54 +04:00
Nikolay Edigaryev 10f56bb5e3 Introduce "orchard ssh" and "orchard vnc" commands (#36)
* proxy.Connections(): handle "use of closed network connection" error

* Controller: less strict timeouts that work nicely for WebSockets

* Worker: only attempt connect to the gRPC once our UID is known

* Introduce "orchard ssh" and "orchard vnc" commands

* Worker: prevent context leak by moving logic into a separate function

* Fix linter errors

* Port forwarding integration test

* Check for "uname -mo" output
2023-03-21 14:58:24 -04:00
Fedor Korotkov fb3056d3ae Refactorings for simplify readability (#35) 2023-03-17 06:11:28 -04:00
Fedor Korotkov 3ecf98c039 Support startup/shutdown scripts (#33)
* Support `startup`/`shutdown` scripts

Fixes #26

* Fixed Go modules after rebase

* Fixes after rebase
2023-03-14 22:15:54 +04:00
Nikolay Edigaryev 47fef47d1c Port forwarding support (#30)
* Port forwarding support

* .golangci.yml: remove and replace deprecated and archived linters

* Client: pass credentials when calling WebSocket API methods

* API: require ServiceAccountRoleComputeWrite role for port forwarding

* Use Buf

* Rename Poll() RPC method to Watch()

* Split Rendezvous into two parts: Watcher and Proxy (#32)

* Split Rendezvous into two parts: Watcher and Proxy

* Implement Proxy cancellation

* Use Protocol Buffers structure directly in Watcher

* Fix TestWatcher after switching to Protocol Buffers structure

* portForwardVM(): ensure we also check for gin's context
2023-03-14 11:31:13 -04:00
Fedor KorotkovandNikolay Edigaryev 0582108ea6 Events Entity (#28)
* Generic Events

We can try to use these generic events for script execution and storing of the output logs in events with `log` kind.

* Lint issues

* Cleanup events upon VM deletion

* Basic integration test

* Run an actual VM in tests

* Apply suggestions from code review

Co-authored-by: Nikolay Edigaryev <edigaryev@gmail.com>

* Use POST

* Make newEventKey private

* Append events in batches

* Lint issues

* Private `scopePrefix`

---------

Co-authored-by: Nikolay Edigaryev <edigaryev@gmail.com>
2023-03-13 08:04:17 -04:00
Fedor Korotkov 165662bb0a Better state syncing and other improvements (#24) 2023-03-01 11:42:16 -05:00
Nikolay Edigaryev a7264370f5 Introduce "controller init" and generate self-signed X.509 certificate (#17) 2023-02-04 11:40:07 +04:00
Nikolay Edigaryev 6bcc02d815 Use golangci-lint (#15) 2023-01-31 22:22:28 +04:00
Nikolay EdigaryevandFedor Korotkov 92e8732d46 Initial version of the Orchard orchestration system (#3)
* Initial version of the Orchard orchestration system

* Update README.md

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>
2023-01-26 23:46:23 +04:00