diff --git a/.cirrus.yml b/.cirrus.yml index 23f40e5..8321c18 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -4,7 +4,7 @@ task: labels: name: dev-mini resources: - tart-vms: 1 + tart-vms: 2 pull_script: - tart pull ghcr.io/cirruslabs/macos-sonoma-base:latest test_script: diff --git a/Development.md b/DEVELOPMENT.md similarity index 100% rename from Development.md rename to DEVELOPMENT.md diff --git a/DeploymentGuide.md b/DeploymentGuide.md deleted file mode 100644 index 9988c90..0000000 --- a/DeploymentGuide.md +++ /dev/null @@ -1,168 +0,0 @@ -# Orchard Cluster Deployment Guide - -Orchard cluster consists of two components: Orchard Controller and a pool of Orchard Workers. Orchard Controller is -responsible for managing the cluster and scheduling of resources. Orchard Workers are responsible for executing the VMs. - -The following guide is split in two parts. First, we'll [deploy an Orchard Controller](#deploying-orchard-controller) and then we'll -[configure and register Orchard Workers](#configuring-orchard-workers) with Ansible. - -## Securing the communications with an Orchard Controller - -When an Orchard client or Worker connects to the Controller, they need to establish trust somehow and verify that they're talking to the right Controller and no [man-in-the-middle attack](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) is possible. - -Similarly to most web-browsers that rely on the PKI ([public key infrastructure](https://en.wikipedia.org/wiki/Public_key_infrastructure)), Orchard uses a hybrid approach by defaulting to automatic PKI verification (can be disabled by [`--no-pki`](#--no-pki-override)) and falling-back to a manual verification for self-signed certificates. - -In this section, we assume the following Controller types: - -* *Controller with a publicly valid certificate* - * can be configured by passing `--controller-cert` and `--controller-key` command-line arguments to `orchard controller run` -* *Controller with a self-signed certificate* - * configured automatically on first Controller start-up when no `--controller-cert` and `--controller-key` are presented - -Below we'll explain how Orchard client and Worker handle these two Controller types. - -### Client - -Client connects (or in other words, associates) with the Controller using a `orchard context create` command. - -The default mode allows for connecting to both Controller types (see above) by leveraging a fall-back mechanism: - -* Client first tries to connect to the Controller and validates its certificate using host's root CA set (can be disabled with [`--no-pki`](#--no-pki-override)) -* if the Client has encountered a *Controller with a publicly valid certificate*, that would be the last step and the association would succeed -* if the Client is dealing with *Controller with a self-signed certificate*, the Client will do another connection attempt to probe the Controller's certificate -* the probed Controller's certificate fingerprint is then presented to the user, and if the user agrees to trust it, the Client then considers that certificate to be trusted for a given Controller address -* Client finally connects to the Controller again with a trusted CA set containing only that certificate, execute the final API sanity checks and if everything is OK then the association succeeds - -Afterward, each interaction with the Controller (e.g. `orchard create vm` command) will stick to the chosen verification method and will re-verify the presented Controller's certificate against: - -* PKI association: host's root CA set -* non-PKI association: a trusted certificate stored in the Orchard's configuration file - -Another thing to note is that PKI and non-PKI associations will emit slightly different Boostrap Tokens for use in Worker: - -* PKI: Bootstrap Token won't include the Controller's certificate, thus forcing the Worker to validate the Controller's certificate against the Worker's host root CA set -* non-PKI: Bootstrap Token will include the Controller's certificate, thus forcing the Worker to validate the Controller's certificate only against that Controller's certificate - -### Worker - -Worker connects to the Controller using a `orchard worker run` command. - -The default mode allows for connecting to both Controller types (see above) by looking at the Bootstrap Token contents: - -* if the Bootstrap Token was generated on a Client that associated with a Controller with the help of the PKI - * the bootstrap token would contain no Controller certificate - * the Orchard Worker will try the PKI approach (can be disabled with [`--no-pki`](#--no-pki-override) to effectively prevent the Worker from connecting) and fail if certificate verification using PKI is possible -* if the Bootstrap Token was generated on a Client that associated with a Controller by utilizing a manual fingerprint verification - * the bootstrap token would contain a Controller certificate - * the Orchard Worker will try to connect to the Controller with a trusted CA set containing only that certificate - -#### `--no-pki` override - -If you're accessing a *Controller with a self-signed certificate* and want to additionally guard yourself against [CA compromises](https://en.wikipedia.org/wiki/Certificate_authority#CA_compromise) and other PKI-specific attacks, pass a `--no-pki` command-line argument to the following commands: - -* `orchard context create --no-pki` - * this will prevent the Client from using PKI and will let you interactively verify the Controller's certificate fingerprint before connecting -* `orchard worker run --no-pki` - * this will prevent the Worker from trying to use PKI when connecting to the Controller using a Bootstrap Token that has no certificate included in it - -Note that we've deliberately chosen not to use environment variables (e.g. `ORCHARD_NO_PKI`), because it's hard to be sure whether the variable was picked up by the command or not. - -Compared to environment variables, an invalid command-line argument will result in an error that wouldn't let you run the command. - -## Deploying Orchard Controller - -Orchard API is secured by default: all requests must be authenticated with credentials of a service account. -When you first run Orchard Controller, you can specify `ORCHARD_BOOTSTRAP_ADMIN_TOKEN` which will automatically -create a service account named `bootstrap-admin` with all privileges. Let's first generate `ORCHARD_BOOTSTRAP_ADMIN_TOKEN`: - -```bash -export ORCHARD_BOOTSTRAP_ADMIN_TOKEN=$(openssl rand -hex 32) -``` - -Now you can run Orchard Controller on a server of your choice. In the following sections you'll find several examples of -how to run Orchard Controller in various environments. Feel free to submit PRs with more examples. - -### Google Cloud Compute Engine - -An example below will deploy a single instance of Orchard Controller in Google Cloud Compute Engine in `us-central1` region. - -First, let's create a static IP address for our instance - -```bash -gcloud compute addresses create orchard-ip --region=us-central1 -export ORCHARD_IP=$(gcloud compute addresses describe orchard-ip --format='value(address)' --region=us-central1) -``` - -Once we have the IP address, we can create a new instance with Orchard Controller running inside a container: - -```bash -gcloud compute instances create-with-container orchard-controller \ - --machine-type=e2-micro \ - --zone=us-central1-a \ - --image-family cos-stable \ - --image-project cos-cloud \ - --tags=https-server \ - --address=$ORCHARD_IP \ - --container-image=ghcr.io/cirruslabs/orchard:latest \ - --container-env=PORT=443 \ - --container-env=ORCHARD_BOOTSTRAP_ADMIN_TOKEN=$ORCHARD_BOOTSTRAP_ADMIN_TOKEN \ - --container-mount-host-path=host-path=/home/orchard-data,mode=rw,mount-path=/data -``` - -Now you can create a new context for your local client: - -```bash -orchard context create --name production \ - --service-account-name bootstrap-admin \ - --service-account-token $ORCHARD_BOOTSTRAP_ADMIN_TOKEN \ - https://$ORCHARD_IP:443 -``` - -And select it as the default context: - -```bash -orchard context default production -``` - -## Configuring Orchard Workers - -First, create a service account limited with a minimal set of roles required for proper worker functioning: - -```bash -orchard create service-account worker-pool-m1 --roles "compute:read" --roles "compute:write" -``` - -Then, generate a bootstrap token: - -```shell -orchard get bootstrap-token worker-pool-m1 -``` - -Then, for each worker machine, start the worker as follows: - -```shell -orchard worker run --bootstrap-token -``` - -### Automation - -If you have a set of machines that you want to use as Orchard Workers, you can use Ansible to configure them. -Please refer a [separate repository](https://github.com/cirruslabs/ansible-orchard) where we prepared a basic -Ansible playbook for convenient setup. - -## Observability - -Both the controller and worker produce some useful OpenTelemetry metrics. Metrics are scoped with `org.cirruslabs.orchard` prefix and include information about resource utilization, statuses or workers, scheduling/pull time and many more. - -By default, the telemetry is sent to https://localhost:4317 using the gRPC protocol and to http://localhost:4318 using the HTTP protocol. - -You can override this by setting the [standard OpenTelemetry environment variable](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/) `OTEL_EXPORTER_OTLP_ENDPOINT`. - -Please refer to [OTEL Collector documentation](https://opentelemetry.io/docs/collector/) for instruction on how to setup a sidecar for the metrics collections or find out if your SaaS monitoring has an available OTEL endpoint (see [Honeycomb](https://docs.honeycomb.io/send-data/opentelemetry/) as an example). - -### Sending metrics to Google Cloud Platform - -There are two standard options of ingesting metrics procuded by Orchard controller and workers into the GCP: - -* [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) + [Google Cloud Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/googlecloudexporter/README.md) — open-source solution that can be later re-purposed to send metrics to any OTLP-compatible endpoint by swapping a single [exporter](https://opentelemetry.io/docs/collector/configuration/#exporters) -* [Ops Agent](https://cloud.google.com/monitoring/agent/ops-agent/otlp) — Google-backed solution with a syntax similar to OpenTelemetry Collector, but tied to GCP-only diff --git a/IntegrationGuide.md b/IntegrationGuide.md deleted file mode 100644 index 334e09a..0000000 --- a/IntegrationGuide.md +++ /dev/null @@ -1,189 +0,0 @@ -# Integrating with Orchard - -Orchard has a REST API that follows [OpenAPI specification](https://swagger.io/specification/) and is described in `api/openapi.yaml`. - -You can run `orchard dev` locally and navigate to http://127.0.0.1:6120/v1/ for interactive documentation. - -![](docs/orchard-api-documentation-browser.png) - -## Using the API - -Below you'll find examples of using Orchard API via vanilla Python's request library and Golang package that Orchard CLI build on top of. - -### Authentication - -When running in non-development mode, Orchard API expects a [basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) to be provided for each API call. - -Below you'll find two snippets that retrieve controller's information and output its version: - -**Python** - -```python -import requests -from requests.auth import HTTPBasicAuth - - -def main(): - # Authentication - basic_auth = HTTPBasicAuth("service account name", "service account token") - - response = requests.get("http://127.0.0.1:6120/v1/info", auth=basic_auth) - - print(response.json()["version"]) - - -if __name__ == '__main__': - main() -``` - -**Golang** - -```go -package main - -import ( - "context" - "fmt" - "github.com/cirruslabs/orchard/pkg/client" - "log" -) - -func main() { - client, err := client.New() - if err != nil { - log.Fatalf("failed to initialize Orchard API client: %v", err) - } - - controllerInfo, err := client.Controller().Info(context.Background()) - if err != nil { - log.Fatalf("failed to retrieve controller's information: %v", err) - } - - fmt.Println(controllerInfo.Version) -} -``` - -Note that we don't provide any credentials for Golang's version of the snippet: this is because Orchard's Golang API client (`github.com/cirruslabs/orchard/pkg/client`) has the ability to read the current's user Orchard context automatically. - -### Creating a VM - -A more intricate example would be spinning off a VM with a startup script that outputs date, reading its logs and removing it from the controller: - -**Python** - -```python -import time -import uuid - -import requests -from requests.auth import HTTPBasicAuth - - -def main(): - vm_name = str(uuid.uuid4()) - - basic_auth = HTTPBasicAuth("service account name", "service account token") - - # Create VM - response = requests.post("http://127.0.0.1:6120/v1/vms", auth=basic_auth, json={ - "name": vm_name, - "image": "ghcr.io/cirruslabs/macos-sonoma-base:latest", - "cpu": 4, - "memory": 4096, - "startup_script": { - "script_content": "date", - } - }) - response.raise_for_status() - - # Retrieve VM's logs - while True: - response = requests.get(f"http://127.0.0.1:6120/v1/vms/{vm_name}/events", auth=basic_auth) - response.raise_for_status() - - result = response.json() - - if isinstance(result, list) and len(result) != 0: - print(result[0]["payload"]) - break - - time.sleep(1) - - # Delete VM - response = requests.delete(f"http://127.0.0.1:6120/v1/vms/{vm_name}", auth=basic_auth) - response.raise_for_status() - - -if __name__ == '__main__': - main() -``` - -**Golang** - -```go -package main - -import ( - "context" - "fmt" - "github.com/cirruslabs/orchard/pkg/client" - v1 "github.com/cirruslabs/orchard/pkg/resource/v1" - "github.com/google/uuid" - "log" - "time" -) - -func main() { - vmName := uuid.New().String() - - client, err := client.New() - if err != nil { - log.Fatalf("failed to initialize Orchard API client: %v", err) - } - - // Create VM - err = client.VMs().Create(context.Background(), &v1.VM{ - Meta: v1.Meta{ - Name: vmName, - }, - Image: "ghcr.io/cirruslabs/macos-sonoma-base:latest", - CPU: 4, - Memory: 4096, - StartupScript: &v1.VMScript{ - ScriptContent: "date", - }, - }) - if err != nil { - log.Fatalf("failed to create VM: %v") - } - - // Retrieve VM's logs - for { - vmLogs, err := client.VMs().Logs(context.Background(), vmName) - if err != nil { - log.Fatalf("failed to retrieve VM logs") - } - - if len(vmLogs) != 0 { - fmt.Println(vmLogs[0]) - break - } - - time.Sleep(time.Second) - } - - // Delete VM - if err := client.VMs().Delete(context.Background(), vmName); err != nil { - log.Fatalf("failed to delete VM: %v", err) - } -} -``` - -## Resource management - -Some resources, such as `Worker` and `VM`, have a `resource` field which is a dictionary that maps between resource names and their amounts (amount requested or amount provided, depending on the resource) and is useful for scheduling. - -Well-known resources: - -* `org.cirruslabs.tart-vms` — number of Tart VM slots available on the machine or requested by the VM - * this number is `2` for workers and `1` for VMs by default diff --git a/README.md b/README.md index ad6d9f5..69eedcf 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ Orchard is an orchestration system for [Tart](https://github.com/cirruslabs/tart). Create a cluster of bare-metal Apple Silicon machines and manage dozens of VMs with ease! -## Quick start +## Usage -Start the Orchard in local development mode: +The fastest way to get started with Orchard is to use a local development mode: ```shell brew install cirruslabs/cli/orchard @@ -14,67 +14,7 @@ orchard dev ``` This will start Orchard Controller and a single Orchard Worker on your local machine. -For production deployments, please refer to the [Deployment Guide](./DeploymentGuide.md). -### Creating Virtual Machines +You can interact with the newly created cluster using the `orchard` CLI or programmatically, through the built-in REST API server. -Create a Virtual Machine resource: - -```shell -orchard create vm --image ghcr.io/cirruslabs/macos-sonoma-base:latest ventura-base -``` - -Check a list of VM resources to see if the Virtual Machine we've created above is already running: - -```shell -orchard list vms -``` - -### Accessing Virtual Machines - -#### SSH - -To SSH into a VM use the `orchard ssh` command: - -```shell -orchard ssh vm ventura-base -``` - -You can specify the `--username` and `--password` flags to specify the username/password pair to SSH. By default, `admin`/`admin` is used. - -You can also execute remote commands instead of spawning a login shell, similarly to the OpenSSH's `ssh` command: - -```shell -orchard ssh vm ventura-base "uname -a" -``` - -You can execute scripts remotely this way, by telling the remote command-line interpreter to read from the standard input and using the redirection operator as follows: - -```shell -orchard ssh vm ventura-base "bash -s" < script.sh -``` - -#### VNC - -Similar to `ssh` command, you can use `vnc` command to open Screen Sharing into a remote VM: - -```shell -orchard vnc vm --username=administrator --password=password101 ventura-base -``` - -From architecture perspective, Orchard has a lower level API for port forwarding that `ssh` and `vnc` commands are built on top of. -All port forwarding connections are done via the Orchard Controller instance which "proxies" a secure connection to the Orchard Workers. -Therefore, your workers can be located under a stricter firewall that only allows connections to the Orchard Controller instance. -Orchard Controller instance is secured by default and all API calls are authenticated and authorized. - -### Environment variables - -In addition to controlling the Orchard via the CLI arguments, there are environment variables that may be beneficial both when automating Orchard and in daily use: - -| Variable name | Description | -|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `ORCHARD_HOME` | Override Orchard's home directory. Useful when running multiple Orchard instances on the same host and when testing. | -| `ORCHARD_LICENSE_TIER` | The default license limit only allows connecting 4 Orchard Workers to the Orchard Controller. If you've purchased a [Gold Tier License](https://tart.run/licensing/), set this variable to `gold` to increase the limit to 20 Orchard Workers. And if you've purchased a [Platinum Tier License](https://tart.run/licensing/), set this variable to `platinum` to increase the limit to 200 Orchard Workers. | -| `ORCHARD_SERVICE_ACCOUNT_NAME` | Override service account name (used for controller API auth) on per-command basis | -| `ORCHARD_SERVICE_ACCOUNT_TOKEN` | Override service account token (used for controller API auth) on per-command basis | -| `ORCHARD_URL` | Override controller URL on per-command basis | +Please check out the [official documentation](https://tart.run/orchard/quick-start/) for more information and/or feel free to use [issues](https://github.com/cirruslabs/orchard/issues) for the remaining questions. diff --git a/docs/orchard-api-documentation-browser.png b/docs/orchard-api-documentation-browser.png deleted file mode 100644 index cbe1f6d..0000000 Binary files a/docs/orchard-api-documentation-browser.png and /dev/null differ