orchard/DeploymentGuide.md

Orchard Cluster Deployment Guide

Orchard cluster consists of two components: Orchard Controller and a pool of Orchard Workers. Orchard Controller is responsible for managing the cluster and scheduling of resources. Orchard Workers are responsible for executing the VMs.

The following guide is split in two parts. First, we'll deploy an Orchard Controller and then we'll configure and register Orchard Workers with Ansible.

Securing the communications with an Orchard Controller

When an Orchard client or Worker connects to the Controller, they need to establish trust somehow and verify that they're talking to the right Controller and no man-in-the-middle attack is possible.

Similarly to most web-browsers that rely on the PKI (public key infrastructure), Orchard uses a hybrid approach by defaulting to automatic PKI verification (can be disabled by --no-pki) and falling-back to a manual verification for self-signed certificates.

In this section, we assume the following Controller types:

• Controller with a publicly valid certificate
  • can be configured by passing --controller-cert and --controller-key command-line arguments to orchard controller run
• Controller with a self-signed certificate
  • configured automatically on first Controller start-up when no --controller-cert and --controller-key are presented

Below we'll explain how Orchard client and Worker handle these two Controller types.

Client

Client connects (or in other words, associates) with the Controller using a orchard context create command.

The default mode allows for connecting to both Controller types (see above) by leveraging a fall-back mechanism:

• Client first tries to connect to the Controller and validates its certificate using host's root CA set (can be disabled with --no-pki)
• if the Client has encountered a Controller with a publicly valid certificate, that would be the last step and the association would succeed
• if the Client is dealing with Controller with a self-signed certificate, the Client will do another connection attempt to probe the Controller's certificate
• the probed Controller's certificate fingerprint is then presented to the user, and if the user agrees to trust it, the Client then considers that certificate to be trusted for a given Controller address
• Client finally connects to the Controller again with a trusted CA set containing only that certificate, execute the final API sanity checks and if everything is OK then the association succeeds

Afterward, each interaction with the Controller (e.g. orchard create vm command) will stick to the chosen verification method and will re-verify the presented Controller's certificate against:

• PKI association: host's root CA set
• non-PKI association: a trusted certificate stored in the Orchard's configuration file

Another thing to note is that PKI and non-PKI associations will emit slightly different Boostrap Tokens for use in Worker:

• PKI: Bootstrap Token won't include the Controller's certificate, thus forcing the Worker to validate the Controller's certificate against the Worker's host root CA set
• non-PKI: Bootstrap Token will include the Controller's certificate, thus forcing the Worker to validate the Controller's certificate only against that Controller's certificate

Worker

Worker connects to the Controller using a orchard worker run command.

The default mode allows for connecting to both Controller types (see above) by looking at the Bootstrap Token contents:

• if the Bootstrap Token was generated on a Client that associated with a Controller with the help of the PKI
  • the bootstrap token would contain no Controller certificate
  • the Orchard Worker will try the PKI approach (can be disabled with --no-pki to effectively prevent the Worker from connecting) and fail if certificate verification using PKI is possible
• if the Bootstrap Token was generated on a Client that associated with a Controller by utilizing a manual fingerprint verification
  • the bootstrap token would contain a Controller certificate
  • the Orchard Worker will try to connect to the Controller with a trusted CA set containing only that certificate

--no-pki override

If you're accessing a Controller with a self-signed certificate and want to additionally guard yourself against CA compromises and other PKI-specific attacks, pass a --no-pki command-line argument to the following commands:

• orchard context create --no-pki
  • this will prevent the Client from using PKI and will let you interactively verify the Controller's certificate fingerprint before connecting
• orchard worker run --no-pki
  • this will prevent the Worker from trying to use PKI when connecting to the Controller using a Bootstrap Token that has no certificate included in it

Note that we've deliberately chosen not to use environment variables (e.g. ORCHARD_NO_PKI), because it's hard to be sure whether the variable was picked up by the command or not.

Compared to environment variables, an invalid command-line argument will result in an error that wouldn't let you run the command.

Deploying Orchard Controller

Orchard API is secured by default: all requests must be authenticated with credentials of a service account. When you first run Orchard Controller, you can specify ORCHARD_BOOTSTRAP_ADMIN_TOKEN which will automatically create a service account named bootstrap-admin with all privileges. Let's first generate ORCHARD_BOOTSTRAP_ADMIN_TOKEN:

export ORCHARD_BOOTSTRAP_ADMIN_TOKEN=$(openssl rand -hex 32)

Now you can run Orchard Controller on a server of your choice. In the following sections you'll find several examples of how to run Orchard Controller in various environments. Feel free to submit PRs with more examples.

Google Cloud Compute Engine

An example below will deploy a single instance of Orchard Controller in Google Cloud Compute Engine in us-central1 region.

First, let's create a static IP address for our instance

gcloud compute addresses create orchard-ip --region=us-central1
export ORCHARD_IP=$(gcloud compute addresses describe orchard-ip --format='value(address)' --region=us-central1)

Once we have the IP address, we can create a new instance with Orchard Controller running inside a container:

gcloud compute instances create-with-container orchard-controller \
  --machine-type=e2-micro \
  --zone=us-central1-a \
  --image-family cos-stable \
  --image-project cos-cloud \
  --tags=https-server \
  --address=$ORCHARD_IP \
  --container-image=ghcr.io/cirruslabs/orchard:latest \
  --container-env=PORT=443 \
  --container-env=ORCHARD_BOOTSTRAP_ADMIN_TOKEN=$ORCHARD_BOOTSTRAP_ADMIN_TOKEN \
  --container-mount-host-path=host-path=/home/orchard-data,mode=rw,mount-path=/data

Now you can create a new context for your local client:

orchard context create --name production \
  --service-account-name bootstrap-admin \
  --service-account-token $ORCHARD_BOOTSTRAP_ADMIN_TOKEN \
  https://$ORCHARD_IP:443

And select it as the default context:

orchard context default production

Configuring Orchard Workers

orchard create service-account worker-pool-m1 --roles "compute:read" --roles "compute:write"
orchard get bootstrap-token worker-pool-m1

Configuring Orchard Workers

If you have a set of machines that you want to use as Orchard Workers, you can use Ansible to configure them. Please refer a separate repository where we prepared a basic Ansible playbook for convenient setup.